This page collects info on problems you might see if trying to bootstrap a Debian or Ubuntu cross-toolchain. There are a pile of things that tend to go wrong and as it's all horribly complicated it's useful to keep logs. Putting them here might be useful to someone.

Contents

  1. bootstrap status
  2. linux-libc-dev
  3. gcc
    1. Forgetting to uninstall the emdebian-style libraries
    2. Forgetting to install binutils-<target>
    3. forgetting to configure the control file before building
    4. stage1 compiler not installable due to missing gcc-<ver>-base
    5. compiler links missing
    6. not finding predefs.h header
    7. stage1 not using multiarch paths
    8. compiler include path setting
  4. eglibc
    1. Compiler not looking in system include paths
    2. stdc-predef.h not found
  5. Ubuntu Bootstrap
    1. Quantal

bootstrap status

This is the status for the Debian linux arches, doing a full toolchain bootstrap. As of 2014-03-22. Thanks to Helmut Grohne for determining all this.

alpha: eglibc stage1 libc6.1-dev:alpha conflicts with libc6-dev:amd64 arm64: gcc stage1 #742165 TODO:REBUILD armel: gcc stage1 compiles TODO:REBUILD armhf: gcc stage1 compiles TODO:REBUILD hppa: binutils stage1 compiles TODO:REBUILD i386: eglibc stage1 invokes cross-compiler with -m64 as host compiler ia64: eglibc stage1 does not find gcc's stage1 unwind.h m68k: gcc stage2 compiles TODO:REBUILD mips: gcc stage1 compiles TODO:REBUILD mipsel: gcc stage1 compiles TODO:REBUILD powerpc: gcc stage1 compiler broken, lacks headers TODO:REBUILD s390: gcc stage1 compiler broken, lacks headers TODO:REBUILD s390x: eglibc stage1 make install fails sparc: eglibc stage1 invokes cross-compiler with -m64 as host compiler x32: gcc stage1 builds broken compiler targets amd64 TODO:REBUILD

linux-libc-dev

DEB_BUILD_PROFILE=stage1 dpkg-buildpackage -aarm64 -B DEB_BUILD_PROFILE=stage1 dpkg-buildpackage -B

Note: 2nd build fails due to source changes. All these files are new:: arch/s390/include/asm/cmb.h.1 drivers/usb/gadget/f_{acm|ecm|obex|serial|subset}.c.1 lib/crc32.c.1

just manually delete them. Someone should fix the clean target.

gcc

Forgetting to uninstall the emdebian-style libraries

No lib....-cross libraries should be installed. The failure looks a lot like not installing binutils-xxx (see below). Update: I (Dima Kogan) observed these failures earlier, but can't reproduce it anymore.

Forgetting to install binutils-<target>

Does quite a lot of stuff, but then fails at libgcc configure: 

configure:3565: checking for suffix of object files
configure:3587: /home/wookey/linaro/armv8/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc -B/home/wookey/linaro/armv8/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/ -B/usr/aarch64-linux-gnu/bin/ -B/usr/aarch64-linux-gnu/lib/ -isystem /usr/aarch64-linux-gnu/include -isystem /usr/aarch64-linux-gnu/sys-include -isystem /home/wookey/linaro/armv8/toolchain/unstable/build/gcc-4.8-4.8.1/build/sys-include -c -g -O2  conftest.c
/home/wookey/linaro/armv8/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/as: 87: exec: -o: not found

or (building for armhf): 

Configuring in arm-linux-gnueabihf/libgcc
....
checking for arm-linux-gnueabihf-gcc...
/home/wookey/linaro/armv8/toolchain/unstablenew/build/armhfbuild/multiarched/gcc-4.7-4.7.2/build/./gcc/xgcc
-B/home/wookey/linaro/armv8/toolchain/unstablenew/build/armhfbuild/multiarched/gcc-4.7-4.7.2/build/./gcc/
-B/usr/arm-linux-gnueabihf/bin/ -B/usr/arm-linux-gnueabihf/lib/
-isystem /usr/arm-linux-gnueabihf/include -isystem
/usr/arm-linux-gnueabihf/sys-include
checking for suffix of object files... configure: error: in
/home/wookey/linaro/armv8/toolchain/unstablenew/build/armhfbuild/multiarched/gcc-4.7-4.7.2/build/arm-linux-gnueabihf/libgcc':
configure: error: cannot compute suffix of object files: cannot compile
See config.log' for more details.
make[3]: *** [configure-target-libgcc] Error 1
make[3]: Leaving directory
/home/wookey/linaro/armv8/toolchain/unstablenew/build/armhfbuild/multiarched/gcc-4.7-4.7.2/build'

config.log has: 

/home/wookey/linaro/armv8/toolchain/unstablenew/build/armhfbuild/multiarched/gcc-4.7-4.7.2/build/./gcc/as:
87: exec: -march=armv7-a: not found

This is because that as is a script: 

ORIGINAL_AS_FOR_TARGET="" 
...
 exec $original ${1+"$@"}

so if $original is null then it tries to run first paramater "-march=armv7-a" in this case.

Turns out this is code for 'you forgot to install binutils-arm-linux-gnueabihf' (i.e cross-binutils for target arch).

forgetting to configure the control file before building

Unless building for the same target arch/distro as gcc was already configured for you need to dpkg-buildpackage -d -T control before runing dpkg-buildpackage for the actual build. Otherwise substitubtion variables will be empty and stuff will break. (Or at least I think that's the problem)

dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/gcc-4.8-aarch64-linux-gnu/usr/lib/gcc-cross/aarch64-lin gnu/4.8/lto1 debian/gcc-4.8-aarch64-linux-gnu/usr/bin/aarch64-linux-gnu-gcov-4.8 debian/gcc-4.8-aarch64-linux-gnu/usr/bin/aarch64-l x-gnu-gcc-nm-4.8 debian/gcc-4.8-aarch64-linux-gnu/usr/bin/aarch64-linux-gnu-gcc-ar-4.8 debian/gcc-4.8-aarch64-linux-gnu/usr/lib/gcc oss/aarch64-linux-gnu/4.8/collect2 debian/gcc-4.8-aarch64-linux-gnu/usr/bin/aarch64-linux-gnu-gcc-4.8 debian/gcc-4.8-aarch64-linux- /usr/lib/gcc-cross/aarch64-linux-gnu/4.8/lto-wrapper debian/gcc-4.8-aarch64-linux-gnu/usr/bin/aarch64-linux-gnu-gcc-ranlib-4.8 were t linked against libm.so.6 (they use none of the library's symbols) dpkg-gencontrol: warning: can't parse dependency Recommends: libc6-dev-arm64-cross (>= 2.13-5) dpkg-gencontrol: error: error occurred while parsing Depends field: dnlcpp-4.8-aarch64-linux-gnu (= 4.8.1-10),

dh_gencontrol: dpkg-gencontrol -pgcc-4.8-aarch64-linux-gnu -ldebian/changelog -Tdebian/gcc-4.8-aarch64-linux-gnu.substvars -Pdebian c-4.8-aarch64-linux-gnu -v4.8.1-10 -Vlibgcc:Version=1:4.8.1-10 -Vgcc:Version=4.8.1-10 -Vgcc:EpochVersion=1:4.8.1-10 -Vgcc:SoftVersi 4.8 -Vgdc:Version= -Vgcj:Version=4.8.1-10 -Vgcj:SoftVersion=4.8 -Vgcj:BaseVersion=4.8 -Vgnat:Version=4.8.1-10 -Vbinutils:Version=2. 90.20131017 -Vdep:libgcc= -Vdep:libgccbiarch= -Vdep:libgccbiarchdev= -Vdep:libc=libc6 (>= 2.11) -Vdep:libcdev=libc6-dev-arm64-cross = 2.13-5) -Vdep:libcbiarch= -Vdep:libcbiarchdev= -Vdep:libunwinddev= -Vdep:libcxxbiarchdev= -Vdep:libcxxbiarchdbg= -Vdep:libgnat= - p:ecj= -Vdep:libcloog= -Vdep:gcj=gcc-4.8 (>= 4.8) -Vdep:gcjcross=gcj-4.8 (>= 4.8) -Vdep:gdccross=gdc-4.8 (>= 4.8) -Vconfl:lib32=lib i386 (<< 2.9-22) -Vmultiarch:breaks=gcc-4.1, gcc-4.3 (<< 4.3.6-1), gcc-4.4 (<< 4.4.6-4), gcc-4.5 (<< 4.5.3-2) -VBuilt-Using= -Vfort :mod-version=gfortran-mod-10 returned exit code 255 make[1]: *** [stamps/08-binary-stamp-gcc] Error 255

stage1 compiler not installable due to missing gcc-<ver>-base

This happens if you are building a versionof the compiler which is not set to produce the core packages (which are always produced by the latest version of the compiler only (e.g building gcc-4.7 when gcc-4.8 is in the archive too).

Having built a stage1 crosscompiler package try installing it: sudo dpkg -i ../gcc-4.8-aarch64-linux-gnu_4.8.1-6_amd64.deb ../cpp-4.8-aarch64-linux-gnu_4.8.1-6_amd64.deb ../libgcc-4.8-dev-arm64-cross_4.8.1-6_all.deb fails because: it needs gcc-4.8-base:amd64 in an exactly matching version.

Remove strict version? any gcc-4.8-base will do.

Can set with_gccxbase=yes to get dep on gcc-4.8-<triplet>-cross-base = strict version instead. this is set for stage2 but not stage1 in rules.conf. (any good reason?) setting this for stage1 as well lets us have gcc-4.8-aarch64-linux-gnu-base (= 4.8.1-6) missing instead (it didn't get built. rules.d/binary-base.mk gets included for with_gccbase or with_gccxbase...).

Can workaroud by using equivs to knock up a package: equivs-build --arch amd64 gcc-4.8-aarch64-linux-gnu-base with config: 

Section: misc
Priority: optional
Standards-Version: 3.6.2
Package: gcc-4.8-aarch64-linux-gnu-base
Version: 4.8.1-6
Architecture: all
Description: Fake gcc-4.8-aarch64-linux-gnu-base so stage1 gcc build is installable
 gcc-4.8-aarch64-linux-gnu-base didn't build - so faking it for now.

The bare compiler package installs versioned binaries. gcc-defaults normally sets up unversioned links. Do it manually for bootstrap:

sudo dpkg -i gcc-4.8-aarch64-linux-gnu-base_4.8.1-6_all.deb

sudo ln -s /usr/bin/aarch64-linux-gnu-gcc-4.8 /usr/bin/aarch64-linux-gnu-gcc sudo ln -s /usr/bin/aarch64-linux-gnu-cpp-4.8 /usr/bin/aarch64-linux-gnu-cpp sudo ln -s /usr/bin/aarch64-linux-gnu-gcov-4.8 /usr/bin/aarch64-linux-gnu-gcov

not finding predefs.h header

fails building libgcc (the first part of the build that uses cross-compiler): 

/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/xgcc -B/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/ -B/usr/aarch64-linux-gnu/bin/ -B/usr/aarch64-linux-gnu/lib/ -isystem /usr/aarch64-linux-gnu/include -isystem /usr/aarch64-linux-gnu/sys-include    -g -O2 -O2  -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE  -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include   -fPIC -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector -Dinhibit_libc  -fPIC -I. -I. -I../.././gcc -I../../../src/libgcc -I../../../src/libgcc/. -I../../../src/libgcc/../gcc -I../../../src/libgcc/../include  -DHAVE_CC_TLS  -o _fixunsdfsi.o -MT _fixunsdfsi.o -MD -MP -MF _fixunsdfsi.dep -DL_fixunsdfsi -c ../../../src/libgcc/libgcc2.c
In file included from /usr/include/features.h:341:0,
                 from /usr/include/limits.h:26,
                 from /home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include-fixed/limits.h:169,
                 from /home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include-fixed/syslimits.h:7,
                 from /home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include-fixed/limits.h:34,
                 from ../../../src/libgcc/libgcc2.c:1733:
/usr/include/stdc-predef.h:30:26: fatal error: bits/predefs.h: No such file or directory
compilation terminated.

/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/xgcc -B/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/ -B/usr/aarch64-linux-gnu/bin/ -B/usr/aarch64-linux-gnu/lib/ -isystem /usr/aarch64-linux-gnu/include -isystem /usr/aarch64-linux-gnu/sys-include    -g -O2 -O2  -g -O2 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE  -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition  -isystem ./include   -fPIC -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector -Dinhibit_libc  -fPIC -I. -I. -I../.././gcc -I../../../src/libgcc -I../../../src/libgcc/. -I../../../src/libgcc/../gcc -I../../../src/libgcc/../include  -DHAVE_CC_TLS  -o _fixsfdi.o -MT _fixsfdi.o -MD -MP -MF _fixsfdi.dep -DL_fixsfdi -c ../../../src/libgcc/libgcc2.c
make[5]: *** [_fixunsdfsi.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[5]: *** [_fixunssfsi.o] Error 1
make[5]: Leaving directory `/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/aarch64-linux-gnu/libgcc'

OK. so there is only one predefs.h on the system in 

/usr/include/x86_64-linux-gnu/bits/predefs.h

and it comes from libc6-dev:amd64

line 30 of /usr/include/stdc-predef.h is: 

#include <bits/predefs.h>

so compiler is apparently not looking in /usr/include/x86_64-linux-gnu (or is it really looking for the arm64 version of this file? that would be wrong - this is stage1 - no lib files needed. Why does it work on armhf/armel?)

Check paths: gets: 

ignoring nonexistent directory " and "
ignoring nonexistent directory "/usr/aarch64-linux-gnu/sys-include"
ignoring nonexistent directory "./include"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/aarch64-linux-gnu/4.7/include"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/aarch64-linux-gnu/4.7
/include-fixed"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/aarch64-linux-gnu/4.7/../../../../aarch64-linux-gnu/include"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/../../lib/gcc-cross/aarch64-linux-gnu/4.7/include"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/../../lib/gcc-cross/aarch64-linux-gnu/4.7/include-fixed"
ignoring nonexistent directory "/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/gcc/../lib/gcc-cross/../../lib/gcc-cross/aarch64-linux-gnu/4.7/../../../../aarch64-linux-gnu/include"
ignoring duplicate directory "."
ignoring duplicate directory "../../../src/libgcc/."
#include "..." search starts here:
#include <...> search starts here:
 .
 ../.././gcc
 ../../../src/libgcc
 ../../../src/libgcc/../gcc
 ../../../src/libgcc/../include
 /home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include
 /home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include-fixed
 /usr/include
End of search list.

so no /usr/include/x86_64-linux-gnu nor /usr/include/aarch64-linux-gnu
/usr/aarch64-linux-gnu/include and /usr/aarch64-linux-gnu/sys-include look wrong to me

I tried setting --with-sysroot=/ in rules.defs as that should enable multiarch. build still failed in the same place for the same reason.

I think the above is wrong, in that libgcc should not need any headers. and has been configured with --without-headers.

So the problem is really 

/home/wookey/linaro/armv8/toolchain/raringnew/arm64-cross-toolchain-base-1.92/gcc/build/./gcc/include-fixed/limits.h:34,

pulling in ../../../src/libgcc/libgcc2.c:1733:
or that pulling in /usr/include/stdc-predef.h:30:26:

which must be wrong?

hmm. this is fun.
fixed-includes has limits.h and syslimits.h
and there is #include_next limits.h in limits.h (protected by a define which is set in syslimits.h (still keeping up?). So now we fall through and find the 'real' (nearly the same) limits.h in /usr/include.

/usr/include/limits.h includes features.h, found in /usr/include

glibc 2.13 and 2.16 differ from here on: glibc2.16:
/usr/include/features.h includes stdc-predef.h (found in /usr/include)
/usr/include/stdc-predef.h includes /bits/predefs.h (and is designed to be implictly included first by the compiler)
bits/predefs.h is in /usr/include/x86_64-linux-gnu/bits/predefs.h

glib2.13:
/usr/include/features.h includes bits/predefs.h
bits/predefs.h is in /usr/include/x86_64-linux-gnu/bits/predefs.h
but, on a debian stable system there is also a /usr/include/bits/predefs.h (same size, date file, but that file is not owned by a package!

So if that extra version is often present then the build will work.
/usr/include/x86_64-linux-gnu/bits/ is owned by libc6-dev:amd64
/usr/include/bits/ is owned by libc6-dev-i386
even though the files inside it are now ownerless.

So the correct fix for this is for stage1 to use the multiarch include search paths.

There appears to be a second bug to do with libc6-dev-i386 leaving copies of the bits directory around.

adding --enable-multiarch to the configure setup doesn't fix it :-(

Discussion suggests that headers really shouldn't be needed here, and if they are they should be arm64 headers. And that fixincludes shouldn't happen, or at least shouldn't get used.

armhf build works fine... comparing bootstrap-protocol files may help.

OK. so the issue is actually that libgcc2 has three fns like this in it: 

#if defined(L_fixunsxfsi) && LIBGCC2_HAS_XF_MODE
/* Reenable the normal types, in case limits.h needs them.  */
#undef char
#undef short
#undef int
#undef long
#undef unsigned
#undef float
#undef double
#undef MIN
#undef MAX
#include <limits.h>

UWtype
__fixunsxfSI (XFtype a)
{
  if (a >= - (DFtype) Wtype_MIN)
    return (Wtype) (a + Wtype_MIN) - Wtype_MIN;
  return (Wtype) a;
}
#endif

which drag in limits.h even with --without-headers used.

This is because the fns used to use LONG_MIN from limits.h but now use Wtype_MIN so limits.h is no longer needed. This solves the problem of it not being available during 1st-stage build.

This is filed as gcc bug: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=55743 and is fixed in gcc 4.9 upstream, and in gcc-4.8 debian package

stage1 not using multiarch paths

gcc-4.8-4.8.1

DEB_STAGE=stage1 DEB_TARGET_ARCH=arm64 with_deps_on_target_arch_pkgs=yes dpkg-buildpackage -b -d fails with: 

checking for aarch64-linux-gnu-gcc...
/home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc -B/home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/
-B/usr/aarch64-linux-gnu/bin/ -B/usr/aarch64-linux-gnu/lib/
-isystem /usr/aarch64-linux-gnu/include -isystem /usr/aarch64-linux-gnu/sys-include
-isystem /home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/sys-include
checking for suffix of object files... configure: error: in /home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/aarch64-linux-gnu/libgcc':
configure: error: cannot compute suffix of object files: cannot compile

Log file has: 

configure:3578: /home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc -B/home/wookey/toolchain/unstable/build\
/gcc-4.8-4.8.1/build/./gcc/ -B/usr/aarch64-linux-gnu/bin/ -B/usr/aarch64-linux-gnu/lib/ -isystem /usr/aarch64-linux-gnu/incl\
ude -isystem /usr/aarch64-linux-gnu/sys-include -isystem /home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/sys-inclu\
de    -c -g -O2  conftest.c >&5
In file included from <command-line>:0:0:
/usr/include/stdc-predef.h:30:26: fatal error: bits/predefs.h: No such file or directory
 #include <bits/predefs.h>
                          ^
compilation terminated.
configure:3582: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "GNU C Runtime Library"
| #define PACKAGE_TARNAME "libgcc"
| #define PACKAGE_VERSION "1.0"
| #define PACKAGE_STRING "GNU C Runtime Library 1.0"
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL "http://www.gnu.org/software/libgcc/"
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }

This is _not_ the limits.h thing - we're past that point. the only predefs.h is in /usr/include/x86_64-linux-gnu/bits/predefs.h

not looking there due to lack of WITH_SYSROOT?

try again with: WITH_SYSROOT=/ DEB_BUILD_PROFILE=stage1 DEB_TARGET_ARCH=arm64 DEB_STAGE=stage1 dpkg-buildpackage -b -d same issue. 

/home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc -v
Using built-in specs.
COLLECT_GCC=/home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian/Linaro 4.8.1-10' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c --$
--disable-decimal-float --disable-libatomic --disable-libgomp --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libsanitizer --disable-thre$
--with-sysroot=/ --enable-multiarch --disable-shared --with-newlib --without-headers --with-system-zlib --enable-multiarch --enable-checking=release --build=x8$
--target=aarch64-linux-gnu --program-prefix=aarch64-linux-gnu-
Thread model: single

check search path with: 

/home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/cc1 -v
ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/4.8/include"
ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/4.8/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/../../aarch64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include

But the multiarch path is never printed there: /home/wookey/toolchain/unstable/build/gcc-4.8-4.8.1/build/./gcc/xgcc --print-multiarch 

aarch64-linux-gnu

looks OK. So why isn't it looking there?

compiler include path setting

default include paths are set in /src/gcc/cppdefault.c Controlled by: CROSS_DIRECTORY_STRUCTURE TARGET_SYSTEM_ROOT LOCAL_INCLUDE_DIR NATIVE_SYSTEM_HEADER_DIR CROSS_INCLUDE_DIR setting WITH_SYSROOT=/ and/or WITH_BUILD_SYSROOT=/ doesn't change the default paths.

existing arm-linux-gnueabi finished toolchain has: arm-linux-gnueabihf-gcc-4.7 -print-prog-name=cc1 -v 

#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/include-fixed
 /usr/lib/gcc-cross/arm-linux-gnueabihf/4.7/../../../../arm-linux-gnueabihf/include
 /usr/include

and: arm-linux-gnueabihf-gcc-4.7 --print-multiarch arm-linux-gnueabihf

Build a version of cross compiler against system libs: 

dpkg --add-architecture armel
apt-get update
apt-get install libc6-dev:armel libstdc++-4.8-dev:armel
DEB_TARGET_ARCH=armel DEB_CROSS_NO_BIARCH=yes with_deps_on_target_arch_pkgs=yes dpkg-buildpackage -b

sudo dpkg -i gcc-4.8-arm-linux-gnueabi_4.8.1-8_amd64.deb  cpp-4.8-arm-linux-gnueabi_4.8.1-8_amd64.deb g++-4.8-arm-linux-gnueabi_4.8.1-8_amd64.deb

finished compiler has good paths

arm-linux-gnueabi-gcc-4.8 -print-prog-name=cc1 -v 

ignoring nonexistent directory
"/usr/lib/gcc/arm-linux-gnueabi/4.8/../../../../arm-linux-gnueabi/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/arm-linux-gnueabi/4.8/include
 /usr/lib/gcc/arm-linux-gnueabi/4.8/include-fixed
 /usr/include

eglibc

Compiler not looking in system include paths

DEB_BUILD_PROFILE=bootstrap dpkg-buildpackage -b -uc -us -aarm64 -d

but it wants gcc-4.7 specifically. grr, so override gcc version:

DEB_GCC_VERSION=-4.8 DEB_BUILD_PROFILE=bootstrap dpkg-buildpackage -b -uc -us -aarm64 -d results in: "This configuration not matched in ../abi-tags" and a bit further back: 

In file included from posix/bits/posix1_lim.h:160:0,
                 from <stdin>:1:
                 ports/sysdeps/unix/sysv/linux/aarch64/nptl/bits/local_lim.h:39:26:
fatal error: linux/limits.h: No such file or directory

Check the search path: with `aarch64-linux-gnu-gcc-4.8 -print-prog-name=cc1` -v 

ignoring nonexistent directory "/usr/lib/gcc-cross/aarch64-linux-gnu/4.8/../../../../aarch64-linux-gnu/sys-include"
ignoring nonexistent directory "/usr/lib/gcc-cross/aarch64-linux-gnu/4.8/../../../../aarch64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc-cross/aarch64-linux-gnu/4.8/include
 /usr/lib/gcc-cross/aarch64-linux-gnu/4.8/include-fixed
End of search list.

So it is indeed not looking in /usr/include, or /usr/include/<triplet>. So the gcc stage1 build was duff.

stdc-predef.h not found

The build with DEB_GCC_VERSION=-4.8 may fail with this message: 

checking whether __attribute__((visibility())) is supported... no
configure: error: compiler support for visibility attribute is required

This is a red herring and what happens instead under the hood is: 

cc1: error: no include path in which to search for stdc-predef.h

The automatic inclusion of that file was introduced in gcc-4.8. The suggestion to pass -ffreestanding breaks other parts of the build. Adding -I/path/to/workaround via CC with that directory containing an empty stdc-predef.h works around this particular issue. It is not clear whether this is a gcc problem (of including a header that does not exist) or an eglibc problem (of not providing said header).

Ubuntu Bootstrap

This is info about the original quantal and raring arm64 cross-toolchain bootstraps in Ubuntu. Largely for historical interest, but it does cover a typical example of the sorts of issues one sees.

Quantal

Binutils was straightforward as the Quantal package already had aarch64 support and arm64 packaging. It just builds :-)

Linux-source required some work. linux-3.5.0 is current in quantal. Upstream work was on 3.6 and 3.7, although initially posted patches from July were for 3.5. Only headers are needed and base port is not invasive so version isn't too important. Taking the port from Linaro current armv8 tree http://git.linaro.org/gitweb?p=people/omarrmz/linux-aarch64.git;a=summary and adding Debian arm64 packaging in various places had only one bit of hackery needed (remove mention of drivers/irqchip from arm64/kconfig) to get a linux-source package built. The initial patch is here:

eglibc in Ubuntu is 2.15. supplied glibc patches were for trunk and then 2.16. eglibc2.15 does not build with gcc4.7, only 4.6. Initial gcc patches were for trunk and then 4.7. There was no public 4.6 port. eglibc2.16 was not available in Debian or Ubuntu, but there was a set of packaging in alioth, but no actual glibc source. eglibc Debian source layout is, for historical reasons not like upstream so a load of munging is needed to get the right layout. I did this hard way before dicovering the debian rules target to do it for me better. Then remove the gcc4.6 deps and rules-forcing. Add in glibc aarch64 patch and poke it till it applies. Fortunately glibc and eglibc are very similar here. Grok packaging layout and patch-naming scheme. Fix up submitted-fcntl-bits patch which moves AT_ GNU symbols and needs to put them back in again for each arch specifically. Turn off libselinux when building stage1 (on by default for linux in 2.16 packaging) Spend some time tracking down why builds always fail with EI_foo aliased to external symbol GI_foo errors 680439. Turns out to be gcc4.6 vs 4.7 issue, which boils down to pr33763, the patch for which breaks eglibc builds. So this actually needs to be fixed in gcc. Some hassling got this done quickly with a new pr33763 patch to put in gcc.

gcc packaging is complicated. Lots of patches applied for different arches, distros, options. Only the core C, C++ and fortran compliers are supported in initial port, so libssp, libmudflap, go, java, objectiveC, lidquadmath, ada, multilib etc all need to be disabled. pr33763 fix as above in order to even build eglibc stage1.

Next the gcc stage2 build fails with 'cannot find bits/predefs.h'. Some investigation finds that this is due to xgcc looking in the wrong place for lib headers. These are put in debian/tmp/ and a sysroot is set in order to find them, but the headers path given a) doesn't look in the sysroot and b) is for old /usr/<triplet>/include path instead of new multiarch /usr/include/<triplet>:/usr/include path. Work out where to poke this in multiple packaging layers.

Then libgcc.so ends up in non-multiarch dir. Turns out to be misapplication of part of Thibgs 'use multiarch paths everywhere', which I tried too late at night and forgot about.

The selinux still gets used for eglibc stage build. We still don't have one so patch changed to always skip selinux use if DEB_PROFILE=bootstrap (and set this in the arm64-cross-toolchain-base build for stage2 full build as well as stage1 - this needs doing properly).

Eventually get a toolchain - woohoo.

After this building extra c++ packages was straightforward.

And this can be installed in a chroot and used to build simple things like zlib (woohoo) but there is an issue. The c-library is provided in the old dpkg-cross location, not as libc6:arm64 . This means that anything non-trivial, i.e. that build-depends on another library, which itself has libc as an install dependency can't be built as the bottom of the multiarch library chain is missing. So we need to cross-build a real libc:arm64. (It could be faked up with equivs but this will cause problem with finding headers/libs later - in fact this whole toolchain mess is a bit of a bodge that needs sorting properly).

Building eglibc for arm64 is a simple matter of DEB_BUILD_PROFILE=bootstrap DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -d -aarm64 -kBEA7C52D -b (the -d is there because otherwise gettext gets in the way (it should be gettext:any in ubuntu))

This fails because the native part of the build (for getent and rpcgen) can't find errno.h. This is because libc6-dev:amd64 isn't installed (because it's not specified as an actual build-dep of eglibc, even though it's needed when crossing - we need to fix this). But this can't just be installed because an attempt to do so causes apt to remove a lot of essential packages (libnih, adduser, mount, rsyslog etc). This turns out to be because libnih1 has these dependencies: Pre-Depends: multiarch-support, libc6 (>> 2.15), libc6 (<< 2.16) which prevents libc 2.16 being installed. This is almost certainly a bug in eglibc's shlibdeps overrides. The easy fix is to build a version of libnih without this restriction, at which point libc6-dev:amd64 can be installed and eglibc can be cross-built for arm64.

Rebuilding eglibc:amd64 mean we now have matching libc6 for amd64 and arm64 and crossbuild-essential-arm64 should be installable, which opens the way for automated sbuild crossbuilds.

However, this wasn't quit enough as libgcc1:arm64 and linux-libc-dev:arm64 are needed to install libc6:arm64. We already have a libgcc1-arm64-cross with the right files in, but not the right layout, architecture, multiarchness or package name. The toolchain has support for a 'reverse-cross' to get the needed libgcc1 but it seemed easier to just mangle libgcc1-arm64-cross to make the right package. Linux-libc-dev is easiest to get by doing DEB_STAGE=stage1 build of the kernel. The kernel package version in quantal has moved on so the amd64 linux-libc-dev is newer. Bah - multi-arch fail. So build one of those too.

The toolchain bootstrap process needs to spit out real multiarch library packages to avoid this faffage. But finally libc6:arm64 is installable which means crossbuild-essential-arm64 is.