Building an ARM cross compiler from source

There are many ways to get an arm cross compiler installed on your platform of choice, but sometimes using a pre-built binary toolchain is not an option. I had such a case, where I had to use the latest version of GCC because that was what the rest of the project was using. At the time, there were no pre-built toolchains available, which contained the latest version of GCC.

There are tools to make the job easier. One such a tool is crosstool-NG, but to be honest, I did not have much success with it. Anyway, I foolishly decided to do it the hard way because I also wanted to understand the process. What follows, is the recipe I followed to get everything built. I will try to point out some gotchas or tips along the way. Be warned, that this is a tedious process with a million different things that can go wrong. It literally took me weeks to get all the issues ironed out, researching issues as I went along. Anyway, enough waffling, lets get on with it!

Preparation
First, we need to obtain the source code packages for the different components that will eventually make up the toolchain. These are as follows:

Package Version
Binutils 2.24
Glibc 2.19
GMP 5.1.3
MPFR 3.1.2
Linux kernel 2.6.31.14
Boost 1.55.0

Technically, Boost is not part of the toolchain, but I am including it since we are building a C/C++ toolchain.

Download the source packages from their respective websites, or source control repositories. I don’t have links to these sites, Google is your friend.

I am using Ubuntu 12.10 as the host platform. Make sure that the build essentials package is installed:
sudo apt-get install build-essential

Also, if the compile fails somewhere along the line, make sure the libgmp-dev, libmpfr-dev and libmpc-dev are installed for your host platform.

The default shell on Ubuntu is Dash, which is incompatible with some of the shell scripts used in the building process. We need to make sure that /bin/sh points to bash, before we can begin:

sudo dpkg-reconfigure dash

A screen will popup, select ‘No’.

Environment
This process is loosely based on the description found in the book Building Embedded Linux Systems, but was adapted with the required changes for the latest versions being built. These step by step instructions are meant to be run from a single command line console, in the sequence described here. The whole process takes about an hour and a half, so take your time and concentrate on each step, since it is advisable to start over until the whole process goes through flawlessly.

Here are some steps to prepare the environment:

  1. Create directory from where we will work.
  2. Create two sub directories, one called ‘sources’ and the other one ‘build’.
  3. Extract the source packages that was downloaded, into the ‘sources’ directory. Each package will be in it’s own directory.
  4. Open a command console in the build directory and copy and paste the instructions into the console, as we go along.
  5. We will need write access to the destination path.

Building
The process makes use of environment variables, to keep track of stuff. Each section contains shell code to create a clean directory in the build directory, from where the actual build will take place. This makes it easy to start over if we need to, while keeping the source directory in it’s original state. Modify the following environment variable to reflect your environment and copy them to the console:

SRCDIR=/home/username/ARM/sources
BUILDMACH=i686-pc-linux-gnu
TARGETMACH=arm-unknown-linux-gnueabi
BUILDDIR=/home/username/ARM/build
INSTALLDIR=/opt/arm/$TARGETMACH
SYSROOTDIR=$INSTALLDIR/sysroot

Binutils
We start by building binutils. This contains the tools like the linker, which we will need to link the object files generated by GCC.

Do the configuring:

[ -d $BUILDDIR/binutils ] && rm -rf $BUILDDIR/binutils
mkdir $BUILDDIR/binutils
cd $BUILDDIR/binutils

BINUTILS_SRC=$SRCDIR/binutils-2.24

$BINUTILS_SRC/configure \
--disable-werror \
--build=$BUILDMACH \
--target=$TARGETMACH \
--prefix=$INSTALLDIR \
--with-sysroot=$SYSROOTDIR \
--disable-nls

The run the following commands individually:

make
make install

Kernel headers
The kernel headers will contain the header files and descriptions of things like integer size, of the target system. These commands can all be pasted in one go:

KERNEL_SRC=$SRCDIR/linux-2.6.31.14
TARGETARCH=arm

cd $KERNEL_SRC
make mrproper
make ARCH=$TARGETARCH integrator_defconfig

[ -d $SYSROOTDIR/usr ] && rm -rf $SYSROOTDIR/usr
mkdir $SYSROOTDIR/usr

make ARCH=$TARGETARCH headers_check
make ARCH=$TARGETARCH INSTALL_HDR_PATH=$SYSROOTDIR/usr headers_install

Bootstrap GCC
Now that the headers and binutils are in place, we are going to build the most basic of GCC compilers, called the bootstrap GCC. This compiler does not contain a C library yet, but it is perfectly suitable for writing bare metal code. This compiler will be used to build the C library later on.

Do the configuring:

GCC_SRC=$SRCDIR/gcc-4.8.2

[ -d $BUILDDIR/bootstrap-gcc ] && rm -rf $BUILDDIR/bootstrap-gcc
mkdir $BUILDDIR/bootstrap-gcc
cd $BUILDDIR/bootstrap-gcc

$GCC_SRC/configure \
--build=$BUILDMACH \
--host=$BUILDMACH \
--target=$TARGETMACH \
--prefix=$INSTALLDIR \
--without-headers \
--enable-bootstrap \
--enable-languages=c \
--disable-threads \
--enable-__cxa_atexit \
--disable-libmudflap \
--with-gnu-ld \
--with-gnu-as \
--disable-libssp \
--disable-libgomp \
--disable-nls \
--disable-shared

The following commands must be run one at a time:

make all-gcc install-gcc

make all-target-libgcc install-target-libgcc

ln -s $INSTALLDIR/lib/gcc/$TARGETMACH/4.8.2/libgcc.a $INSTALLDIR/lib/gcc/$TARGETMACH/4.8.2/libgcc_sh.a

GLibC headers
Building the GLibC library is a two step process. The reason for the two steps, is to get around the chicken and egg situation, that the GLibC library needs headers built by the pre processor, but the headers do not exist yet. To get around this, the developers introduced a process whereby only the headers are built. The output of this process will then be used by the actual compiling of the library.

Do the configuring:

LIBC_SRC=$SRCDIR/glibc-2.19

[ -d $BUILDDIR/libc ] && rm -rf $BUILDDIR/libc
mkdir -p $BUILDDIR/libc
cd $BUILDDIR/libc

echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache
export PATH=$INSTALLDIR/bin:$PATH

export CROSS=$TARGETMACH
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as

$LIBC_SRC/configure \
--build=$BUILDMACH \
--host=$TARGETMACH \
--prefix=$SYSROOTDIR/usr \
--with-headers=$SYSROOTDIR/usr/include \
--config-cache \
--enable-add-ons \
ports=yes \
--enable-kernel=2.6.31

Installing the headers:
make -k install-headers cross_compiling=yes install_root=$SYSROOTDIR

We need a symbolic link:
ln -s $INSTALLDIR/lib/gcc/$TARGETMACH/4.8.2/libgcc.a $INSTALLDIR/lib/gcc/$TARGETMACH/4.8.2/libgcc_eh.a

BuildingGLibC
Now that the headers are installed we build the actual C library.

Do the configuring:
[ -d $BUILDDIR/libc ] && rm -rf $BUILDDIR/libc
mkdir -p $BUILDDIR/libc
cd $BUILDDIR/libc

echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache

export CROSS=$TARGETMACH
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as

$LIBC_SRC/configure \
--build=$BUILDMACH \
--host=$TARGETMACH \
--prefix=/usr \
--with-headers=$SYSROOTDIR/usr/include \
--config-cache \
--enable-add-ons \
--enable-kernel=2.6.31 \
ports=yes \
CFLAGS=-O

Run the following commands one at a time:
make -k install-headers cross_compiling=yes install_root=$SYSROOTDIR

make

make install_root=${SYSROOTDIR} install

Building the next GCC
Now that we have a working C library, we are ready to build a version of GCC that makes use of it.

Do the configuring:
[ -d $BUILDDIR/final-gcc ] && rm -rf $BUILDDIR/final-gcc
mkdir -p $BUILDDIR/final-gcc
cd $BUILDDIR/final-gcc

echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache
export CC=gcc
export LD=ld
export AS=as

$GCC_SRC/configure \
--build=$BUILDMACH \
--target=$TARGETMACH \
--prefix=$INSTALLDIR \
--with-sysroot=${SYSROOTDIR} \
--enable-languages=c,c++ \
--with-gnu-as \
--with-gnu-ld \
--disable-multilib \
--disable-libmudflap \
--with-float=soft \
--disable-sjlj-exceptions \
--disable-nls \
--enable-threads=posix \
--enable-long-longx \
--enable-__cxa_atexit

Run the following commands one at a time:
make all-gcc
make install-gcc

Now that we have a version of GCC that contains a C library, we are ready to build the support libraries that will be needed by the final version of GCC.

GMP
GMP is a library with all sorts of arithmetic functions, as well as support for big numbers. This library will be needed by the final GCC.

Do the configuring:
LIBGMP_SRC=$SRCDIR/gmp-5.1.3

[ -d $BUILDDIR/gmp ] && rm -rf $BUILDDIR/gmp
mkdir -p $BUILDDIR/gmp
cd $BUILDDIR/gmp

export CROSS=$TARGETMACH
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as
export CFLAGS=-static

$LIBGMP_SRC/configure \
--build=$BUILDMACH \
--host=$TARGETMACH \
--prefix=$INSTALLDIR \
--disable-shared

Run the following two commands one at a time:
make
make install

MPFR
The MPFR library contains code that implements precision floating point calculations. This library will be needed by the final version of GCC.

Do the configuring:
LIBMPFR_SRC=$SRCDIR/mpfr-3.1.2

[ -d $BUILDDIR/mpfr ] && rm -rf $BUILDDIR/mpfr
mkdir -p $BUILDDIR/mpfr
cd $BUILDDIR/mpfr

export CROSS=$TARGETMACH
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as
export CFLAGS=-static

$LIBMPFR_SRC/configure \
--build=$BUILDMACH \
--host=$TARGETMACH \
--prefix=$INSTALLDIR \
--with-gmp=$INSTALLDIR

Run the following two commands one at a time:
make
make install

Building the final GCC
At last we have everything we need to build the final version of the GCC toolchain.

Do the configuring:
[ -d $BUILDDIR/final-gcc-2 ] && rm -rf $BUILDDIR/final-gcc-2
mkdir -p $BUILDDIR/final-gcc-2
cd $BUILDDIR/final-gcc-2

export CC=gcc
export LD=ld
export AS=as
export CFLAGS=

echo "libc_cv_forced_unwind=yes" > config.cache
echo "libc_cv_c_cleanup=yes" >> config.cache

$GCC_SRC/configure \
--prefix=$INSTALLDIR \
--target=$TARGETMACH \
--build=$BUILDMACH \
--with-sysroot=${SYSROOTDIR} \
--bindir=$INSTALLDIR/bin \
--disable-nls \
--enable-shared \
--disable-multilib \
--disable-libmudflap \
--disable-libgomp \
--enable-languages=c,c++ \
--enable-c99 \
--enable-long-long \
--with-float=soft \
--with-mode=arm \
--with-arch=armv5te \
--with-abi=aapcs-linux \
--with-mpfr=$INSTALLDIR \
--with-gmp=$INSTALLDIR

Run the following two commands one at a time:
make
make install

If everything went according plan, you should be able to run the following command to test your shiny new toolchain installation:
arm-unknown-linux-gnueabi-gcc -v

You should see something like this:
Using built-in specs.
COLLECT_GCC=arm-unknown-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/usr/lib/arm-unknown-linux-gnueabi/libexec/gcc/arm-unknown-linux-gnueabi/4.8.2/lto-wrapper
Target: arm-unknown-linux-gnueabi
Configured with: /home/username/ARM/sources/gcc-4.8.2/configure --prefix=/usr/lib/arm-unknown-linux-gnueabi --target=arm-unknown-linux-gnueabi --build=i686-pc-linux-gnu --with-sysroot=/usr/lib/sysroot/arm-unknown-linux-gnueabi --bindir=/usr/lib/arm-unknown-linux-gnueabi/bin --disable-nls --enable-shared --disable-multilib --disable-libmudflap --disable-libgomp --enable-languages=c,c++ --enable-c99 --enable-long-long --with-float=soft --with-mode=arm --with-arch=armv5te --with-abi=aapcs-linux --with-mpfr=/usr/lib/arm-unknown-linux-gnueabi --with-gmp=/usr/lib/arm-unknown-linux-gnueabi
Thread model: posix
gcc version 4.8.2 (GCC)

Building Boost libraries
As mentioned before, the boost libraries are technically not part of the toolchain. I am adding them, because some of my projects rely on them.

The Boost libraries make use of an annoying proprietary build system, that almost require one to study it completely just for the sake of compiling the source code. I am not going to try and explain the how’s or why’s, because I am not that familiar with it. Allot of Googling was required to figure this one out.

I would suggest, building Boost in a temporary build location. This makes it easier to clean up after the build, you simply delete the build directory. This is achieved by specifying the --build-dir= parameter in the build command.

Firstly, edit the file tools/build/v2/user-config.jam and add the following line under the ‘GCC configuration’ section:
using gcc : arm : arm-unknown-linux-gnueabi-g++ ;

Run the bootstrap shell script that sets everything up:
./bootstrap.sh

Now we can perform the actual build:
./b2 install toolset=gcc-arm --build-dir=$HOME/temp/boostbuild --prefix=$INSTALLDIR/usr

We’re almost done. Before we can use the libraries, we need to do some more housekeeping to get around boost peculiarities. A convention seems to exist to add ‘-mt’ to the end of library names to indicate a multi thread safe version of a library. As far as I can ascertain the multithreaded and normal libraries are exactly the same, so we can merely add symbolic links to the original library. Below are the libraries that I have come across, that needs ‘-mt’ versions. You might very well encounter more, just add the links and you should be OK. Just don’t blame me if this is not the case.

ln -s libboost_filesystem.a libboost_filesystem-mt.a
ln -s $INSTALLDIR/usr/lib/libboost_filesystem.so $INSTALLDIR/usr/lib/libboost_filesystem-mt.so
ln -s $INSTALLDIR/usr/lib/libboost_regex.a $INSTALLDIR/usr/lib/libboost_regex-mt.a
ln -s $INSTALLDIR/usr/lib/libboost_regex.so $INSTALLDIR/usr/lib/libboost_regex-mt.so
ln -s $INSTALLDIR/usr/lib/libboost_system.a $INSTALLDIR/usr/lib/libboost_system-mt.a
ln -s $INSTALLDIR/usr/lib/libboost_system.so $INSTALLDIR/usr/lib/libboost_system-mt.so

UPDATE:
In order to create libraries with the -mt suffix, we need to tell the build system at compile time, that we want the multithread safe libraries. The following parameters must be added to the build command above:
threading=multi --layout=tagged

That’s it, I hope you got this far without any major issues. If not, trace your previous steps CAREFULLY for any typos. It is advisable to start from scratch when an error occurred, I know it is tedious but there are no shortcuts in this complex process.