Tracking Issue for Aarch64 Images

[Alexhuszagh]

This is a tracking issue for building aarch64 images, meaning they run natively on linux/arm64 instead of linux/amd64, and what's confirmed to work, what needs to be fixed, and what's currently untested, so we can ensure the maintainers don't duplicate work, new contributors know where to start, and looking at what to patch for future builds. We can also likely add builds for PPC64LE, since almost all of the maintained packages that have install candidates for ARM64 also support PPC64LE. Currently, most of the linux GNU packages are/will be failing.

Note that the PPC64LE images are currently untested, but assumed to work.

Currently Succeeding

Builds of these images for aarch64 can be found here for debugging and testing purposes (these are not official images, not are they guaranteed to be stable).

• aarch64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
• s390x-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
• arm-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
• armv5te-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
• armv7-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
• armv7-unknown-linux-gnueabihf (x86_64, i686, arm64, pp64le)
• i586-unknown-linux-gnu (x86_64, arm64, pp64le)
• i686-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
• powerpc64le-unknown-linux-gnu (x86_64, i686, arm64)
• riscv64gc-unknown-linux-gnu (x86_64, i686, arm64, ppc64le)
• x86_64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
• aarch64-unknown-linux-musl (likely all architectures)
• arm-unknown-linux-musleabi (likely all architectures)
• arm-unknown-linux-musleabihf (likely all architectures)
• armv5te-unknown-linux-musleabi (likely all architectures)
• armv7-unknown-linux-musleabi (likely all architectures)
• armv7-unknown-linux-musleabihf (likely all architectures)
• i586-unknown-linux-musl (likely all architectures)
• i686-unknown-freebsd (likely all architectures)
• mips-unknown-linux-musl (likely all architectures)
• mips64-unknown-linux-muslabi64 (likely all architectures)
• mips64el-unknown-linux-muslabi64 (likely all architectures)
• mipsel-unknown-linux-musl (likely all architectures)
• sparcv9-sun-solaris (likely all architectures)
• thumbv6m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
• thumbv7em-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
• thumbv7em-none-eabihf (x86_64, armhf, arm64, pp64le, s390x)
• thumbv7m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
• thumbv7neon-unknown-linux-gnueabihf (x86_64, armhf, arm64, pp64le, s390x)
• x86_64-unknown-freebsd (likely all architectures)

Currently Failing

• arm-unknown-linux-gnueabihf (Unknown error in crosstool-ng build process)
• sparc64-unknown-linux-gnu (Unable to locate package g++-sparc64-linux-gnu, not available for ARM64, will need a full compile from source)
• mips-unknown-linux-gnu (Package for ARM64 and PPC64LE only in 22.04+, not in 20.04)
• mips64-unknown-linux-gnuabi64 (Unable to locate package g++-mips64-linux-gnuabi64, not available for ARM64, will need a full compile from source)
• mips64el-unknown-linux-gnuabi64 (Unable to locate package g++-mips64el-linux-gnuabi64, not available for ARM64, will need a full compile from source)
• mipsel-unknown-linux-gnu (Unable to locate package g++-mipsel-linux-gnu, not available for ARM64, will need a full compile from source)
• powerpc-unknown-linux-gnu (Unable to locate package g++-powerpc-linux-gnu, not available for ARM64, only x86_64 and PPC64LE, will need a full compile from source)
• powerpc64-unknown-linux-gnu (Unable to locate package g++-powerpc64-linux-gnu, not available for ARM64, will need a full compile from source)
• i686-pc-windows-gnu (unlikely to work without building WINE from source)
• x86_64-pc-windows-gnu (unlikely to work without building WINE from source)
• wasm32-unknown-emscripten (no linux/arm64 image for emsdk)
• x86_64-sun-solaris (fails while building GCC).

Failing, Cannot Fix

• aarch64-linux-android
• arm-linux-androideabi
• armv7-linux-androideabi
• i686-linux-android
• thumbv7neon-linux-androideabi
• x86_64-linux-android

The Android images have the following requirements, which means it will be impossible to build a cross-compiler for them without significant effort from a non-x86 host:

1. They require gcc-multilib and g++-multilib, which are not available on aarch64 and any host other than i686, x86_64, and s390x.
2. They use prebuilts for clang, gcc and go that assume an x86 host.
3. The NDKs assume an x86_64 host.

The only solution therefore would be to rebuild the entire NDK from source, get Android system, replace the prebuilts with native toolchains for a non-x86 host, update code so it uses the correct non-x86 prebuilts, etc. If this sounds like an issue, it is.

Currently Untested

• x86_64-unknown-dragonfly
• x86_64-unknown-illumos
• x86_64-unknown-linux-gnu:centos
• x86_64-unknown-linux-musl
• x86_64-unknown-netbsd

Contributing

If you'd like to contribute your own tests, successes, etc., feel free to do so with the following examples. The runners, std, C++, and dynamic library support can be found in the Github actions ci.yml.

# change this for specific targets
$ export TARGET=s390x-unknown-linux-gnu
$ cargo build-docker-image --platform linux/arm64/v8=aarch64-unknown-linux-gnu $TARGET
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu"
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE="ghcr.io/cross-rs/$TARGET:local"
$ export STD=1
$ export CPP=1
$ export DYLIB=1
$ export RUN=1
$ export RUNNERS="qemu-user qemu-system"
$ ci/test.sh

[Alexhuszagh]

If we install packages from Debian for some of these toolchains, we can support a lot more architectures. Potential solutions include:

• mips64-unknown-linux-gnuabi64: (amd64 arm64 i386 mips64el mipsel ppc64el)
• mips64el-unknown-linux-gnuabi64: (amd64 arm64 i386 mipsel ppc64el)
• mipsel-unknown-linux-gnu: (amd64 arm64 i386 mips64el ppc64el)

This only seems to affect MIPS however.

[DoumanAsh]

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

[Alexhuszagh]

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

That's unrelated (and an upstream bug): see #724. Try running cargo clean and then rebuilding. This can occur in any repository with a build script. It can happen fairly regularly if switching between toolchains. This is a cargo bug, but is exacerbated by how cross functions (swapping toolchains with the same target directory, which causes the bug to manifest).

[Emilgardis]

@CoreyCole this is the wrong issue for your problem, please create a new issue

as a quick mention, the table in the readme is only for the current :main images, you're using :latest in your example. Try

[target.aarch64-unknown-linux-gnu]
image = "ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main"

[abcd-ca]

I downloaded an aarch64 image to my host (my M1 mac). Would someone tell me how to use it? I've got a knowledge gap between having the native (M1, aarch64) image and this command:
CROSS_REMOTE=1 DOCKER_HOST=tcp://docker:2375/ cross build --release --target=armv7-unknown-linux-musleabihf.

I think for me the DOCKER_HOST part is wrong and when I try to run the image (start a container based on it) I see no port has been exposed.

Thank you

[Alexhuszagh]

@abcd-ca

Do you need CROSS_REMOTE? You should be able to run it natively on your Mac M1, and therefore don't need remote docker, unless you are connecting to the client remotely. You do have to provide the image name itself, however: these aren't named the same so you have to override the image name as follows:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

Does that help? All the image names and tags can be found here.

[abcd-ca]

@Alexhuszagh Thanks, that does look more like what I need however with that I get,

docker: Error response from daemon: image with reference ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64.

[Alexhuszagh]

Is your local machine the M1? Or are you connecting remotely to it? If so, then you do need CROSS_REMOTE along with the correct connection parameters (SSH connection? TCP?).

[abcd-ca]

Local is M1. No physically remote build computer yet. Hoping to avoid that