Rust support

[tonistiigi]

Allow building rust projects. Support installing rust via rustup (alpine/debian) and alpine packages (rust/rust-stdlib). Haven't tested official image. Debian package-based installs will probably not be supported as packages don't seem to be properly separated.

Currently have tested with these Dockerfile stages:

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git rust cargo
WORKDIR /work
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc rust-stdlib
RUN xx-clang --setup-target-triple
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
COPY --from=xx / /
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

The main complexity is that the binary releases use triple with unknown vendor instead of alpine and as C compiler always gets called with cc --target=wrong-triple it doesn't really matter that the CC is wrapped. I've added a way for cc-clang to work with any vendor by creating symlinks to the official ones. An alternative would be to try to rename the --target flag in the wrapper.

I think it makes sense to add xx-cargo script that would set the 3 env automatically. xx-cargo --setup-target could make the correct calls to rustup target add or xx-apk add rust-stdlib (and set up xx-clang for correct triple).

Risc-V doesn't currently work because the gnu triple is riscv64gc-* not riscv64. Might need some special override for that. In alpine:edge there is no riscv package for rust-stdlib yet, and no musl build in rustup either.

• add xx-cargo
• support riscv64gc-gnu triple
• CI testing

@crazy-max

[tonistiigi]

Added xx-cargo and riscv support. Works with both Alpine and Debian with either rustup or package based installs. RiscV only works on debian, musl support is blocked in rust-lang/libc#1994

Test stages with xx-cargo

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS blake
RUN apk add git
WORKDIR /src
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
RUN apk add clang lld musl-dev gcc git curl
COPY --from=xx / /
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN --mount=from=blake,src=/src/BLAKE3,rw cd b3sum && \
    xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
RUN apk add clang lld musl-dev gcc git rust cargo
COPY --from=xx / /
WORKDIR /work
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
COPY --from=xx / /
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    export XX_RUSTFLAGS="-C target-feature=+crt-static" && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && ls -lR /tmp/blake && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-apt
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl cargo
WORKDIR /work
COPY --from=xx / /
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN --mount=from=blake,src=/src/BLAKE3,rw git checkout 0.3.7 && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

[crazy-max]

musl support is blocked in rust-lang/libc#1994

@tonistiigi Looks like it has been merged: rust-lang/libc#2537

[tonistiigi]

@crazy-max So I think the last commit needs to be reverted to make CI work. Or do you have additional analysis?

[crazy-max]

@tonistiigi I looked across GitHub for XX_VENDOR or xx-info env but didn't find any use out there except here when dealing with xx-go, xx-apt, xx-apk and xx-verify.

I'm a bit worried about this case:

xx/base/xx-apt

Line 38 in 5596639

case "${XX_VENDOR}" in

I think we could just use case "$(xx-info vendor)" in.

[tonistiigi]

Do you understand what is causing the CI failure. Maybe there is a solution that would allow keeping that line. Having XX_VENDOR and xx-info vendor return different values doesn't make a lot of sense.

[crazy-max]

Do you understand what is causing the CI failure.

Yes it fails with xx-go because triple is wrong when XX_VENDOR is being exported in:

xx/base/xx-go

Line 6 in 5596639

export "${l?}"

[tonistiigi]

That file does not use XX_VENDOR afaics. So the issue needs to be already in xx-info not it xx-go