mudler · richiejp · Jul 28, 2026 · Jul 29, 2026
diff --git a/.agents/ci-caching.md b/.agents/ci-caching.md
@@ -231,6 +231,20 @@ This applies only to `Dockerfile.python` because:
 
 Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
 
+## The jetson wheels mirror (l4t builds)
+
+The `requirements-l4t12.txt` / `-l4t13.txt` files pull CUDA aarch64 torch wheels from `pypi.jetson-ai-lab.io` via `--extra-index-url`. That index has a history of multi-hour 502 outages, and a 502 on **any** project page aborts the whole uv resolution — uv consults every configured index for every requirement, so even PyPI-hosted packages die with it. To keep l4t builds green through outages, CI serves those wheels from a mirror it controls:
+
+- **Storage**: `ghcr.io/mudler/localai/jetson-wheels:{jp6-cu129,jp7-cu130}` — scratch OCI images holding the wheel subset, laid out like the upstream index (`/jp6/cu129/torch/<wheel>`).
+- **Sync**: `.github/workflows/jetson-wheels.yml` (Saturdays 03:00 UTC, ahead of the weekly `DEPS_REFRESH` re-resolve; also `workflow_dispatch` and master pushes touching its inputs) runs `scripts/jetson-wheels-sync.py` against the package list in `.github/jetson-wheels.json`. During an upstream outage the sync keeps the last-known-good wheels and exits green.
+- **Consumption**: `backend_build.yml` resolves the matching tag for `build-type: l4t` entries (cuda 12 → `jp6-cu129`, 13 → `jp7-cu130`) and passes it as the `JETSON_WHEELS_IMAGE` build-arg; `Dockerfile.python` bind-mounts it at `/jetson-wheels`; `installRequirements` in `backend/python/common/libbackend.sh` serves that directory on localhost as a PEP 503 index (`backend/python/common/pypi_mirror_server.py`) and rewrites the jetson index host in the requirements files to it. The local index 404s for anything it doesn't carry, which uv follows up on PyPI — only the jetson-built wheels resolve locally.
+- **Fallbacks**: if the mirror tag doesn't exist (bootstrap) `backend_build.yml` passes `scratch`, the mount is empty, and the build talks to the upstream index exactly as before. Builds outside CI (local, real Jetsons) never set `JETSON_WHEELS_IMAGE` and are unaffected.
+- **Cache interaction**: the bind mount's content is part of the `RUN ... make` layer's BuildKit hash, so a refreshed wheels image invalidates the install layer on the next build — no extra cache-buster needed.
+
+**Extending the package list**: a package a build needs from the jetson index but missing from `.github/jetson-wheels.json` resolves from PyPI instead — for compiled CUDA packages that silently means a CPU build. When adding an l4t backend with new compiled deps, add them to the list and dispatch `jetson-wheels.yml`.
+
+**Bootstrap** (one-time): `gh workflow run jetson-wheels.yml --ref master`, then make the `jetson-wheels` ghcr package public so anonymous pulls work (Settings → Packages).
+
 ## ccache for C++ backend builds
 
 `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:

diff --git a/.github/jetson-wheels.json b/.github/jetson-wheels.json
@@ -0,0 +1,25 @@
+{
+  "upstream": "https://pypi.jetson-ai-lab.io",
+  "indexes": {
+    "jp6/cu129": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ],
+    "jp7/cu130": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ]
+  }
+}
diff --git a/.github/workflows/backend_build.yml b/.github/workflows/backend_build.yml
@@ -181,6 +181,41 @@ jobs:
         id: deps_refresh
         run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
 
+      - name: Login to ghcr.io (jetson wheels mirror)
+        if: inputs.build-type == 'l4t'
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      # l4t builds pull their CUDA aarch64 torch wheels from
+      # pypi.jetson-ai-lab.io, which has a history of multi-hour 502 outages
+      # that fail every l4t job. jetson-wheels.yml mirrors those wheels into
+      # ghcr weekly; here we hand the mirror image to Dockerfile.python,
+      # which serves it as a local package index during pip install (see
+      # installRequirements in backend/python/common/libbackend.sh). Falls
+      # back to scratch — i.e. building straight against the upstream index —
+      # when the mirror tag doesn't exist yet, so the mirror can bootstrap
+      # without a chicken-and-egg failure.
+      - name: Resolve jetson wheels mirror image
+        id: jetson_wheels
+        if: inputs.build-type == 'l4t'
+        run: |
+          repo="ghcr.io/$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          case "${{ inputs.cuda-major-version }}" in
+            12) tag="jp6-cu129" ;;
+            13) tag="jp7-cu130" ;;
+            *) tag="" ;;
+          esac
+          img=""
+          if [ -n "$tag" ] && docker buildx imagetools inspect "$repo:$tag" >/dev/null 2>&1; then
+            img="$repo:$tag"
+          else
+            echo "jetson wheels image $repo:$tag not found; building against the upstream index"
+          fi
+          echo "image=$img" >> "$GITHUB_OUTPUT"
+
       - name: Build and push by digest
         id: build
         uses: docker/build-push-action@v7
@@ -201,6 +236,7 @@ jobs:
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
             BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
             BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
@@ -273,6 +309,7 @@ jobs:
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
             BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
             BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
+            JETSON_WHEELS_IMAGE=${{ steps.jetson_wheels.outputs.image || 'scratch' }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}

diff --git a/.github/workflows/jetson-wheels.yml b/.github/workflows/jetson-wheels.yml
@@ -0,0 +1,131 @@
+---
+name: 'sync jetson wheels mirror'
+
+# Mirrors the CUDA aarch64 wheels our l4t backends need from
+# pypi.jetson-ai-lab.io into scratch OCI images on ghcr
+# (ghcr.io/mudler/localai/jetson-wheels:<tag>, one tag per JetPack index).
+# backend_build.yml hands the matching tag to Dockerfile.python, which
+# bind-mounts it and serves it as a local package index during pip install
+# (see installRequirements in backend/python/common/libbackend.sh), so the
+# upstream index's recurring multi-hour 502 outages can no longer fail l4t
+# builds.
+#
+# The package subset lives in .github/jetson-wheels.json. A package that a
+# build needs from the jetson index but that is missing from that list will
+# resolve from PyPI instead — for compiled CUDA packages that silently means
+# a CPU build, so extend the list when adding an l4t backend with new
+# compiled deps.
+#
+# When upstream is unreachable the sync keeps the previously mirrored wheels
+# and exits green — the mirror serves last-known-good through outages. It
+# only fails when upstream is down and the tag has never been published
+# (bootstrap during an outage: nothing to serve yet).
+#
+# Triggers:
+#   - schedule (Saturdays 03:00 UTC) — refreshes ahead of base-images.yml
+#     (Saturdays 05:00 UTC) and the backend.yml weekly cron (Sundays), whose
+#     DEPS_REFRESH cache-bust re-resolves the python deps.
+#   - workflow_dispatch — manual one-off sync; also the bootstrap run:
+#       gh workflow run jetson-wheels.yml --ref master
+#   - push to master touching the config, the sync script, or this workflow.
+
+on:
+  schedule:
+    - cron: '0 3 * * 6'
+  workflow_dispatch:
+  push:
+    branches: [master]
+    paths:
+      - '.github/jetson-wheels.json'
+      - 'scripts/jetson-wheels-sync.py'
+      - '.github/workflows/jetson-wheels.yml'
+
+permissions:
+  contents: read
+  packages: write
+
+concurrency:
+  group: jetson-wheels-${{ github.repository }}
+  cancel-in-progress: false
+
+jobs:
+  sync:
+    if: github.repository == 'mudler/LocalAI'
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - index: 'jp6/cu129'
+            tag: 'jp6-cu129'
+          - index: 'jp7/cu130'
+            tag: 'jp7-cu130'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@master
+
+      - name: Login to ghcr.io
+        uses: docker/login-action@v4
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      - name: Compute image name
+        id: image
+        run: |
+          repo="ghcr.io/$(echo "${GITHUB_REPOSITORY}" | tr '[:upper:]' '[:lower:]')/jetson-wheels"
+          echo "ref=${repo}:${{ matrix.tag }}" >> "$GITHUB_OUTPUT"
+
+      # Seed the working dir with the current mirror contents so the sync is
+      # incremental and an upstream outage keeps last-known-good wheels.
+      - name: Pull current mirror contents
+        run: |
+          mkdir -p wheels
+          # The image is declared linux/arm64 (its only consumers are arm64
+          # l4t builds); pulling on this amd64 runner needs the explicit
+          # platform. The content is just wheel files — never executed here.
+          if docker pull --platform linux/arm64 "${{ steps.image.outputs.ref }}"; then
+            # scratch images have no command; docker create still needs one,
+            # but the container is never started so any path works.
+            cid="$(docker create "${{ steps.image.outputs.ref }}" /noop)"
+            docker export "${cid}" | tar -x -C wheels
+            docker rm "${cid}"
+            find wheels -name '*.whl' | sed 's/^/  existing: /'
+          else
+            echo "no existing mirror image (bootstrap run)"
+          fi
+
+      - name: Sync from upstream
+        id: sync
+        run: |
+          python3 scripts/jetson-wheels-sync.py \
+            --config .github/jetson-wheels.json \
+            --index '${{ matrix.index }}' \
+            --dest wheels \
+            --changed-file /tmp/jetson-wheels-changed
+          if [ -f /tmp/jetson-wheels-changed ]; then
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Push mirror image
+        if: steps.sync.outputs.changed == 'true'
+        run: |
+          cat > Dockerfile.jetson-wheels <<'EOF'
+          FROM scratch
+          COPY wheels/ /
+          EOF
+          # linux/arm64 because the consumers (l4t builds in
+          # backend_build.yml) build for arm64 and BuildKit refuses a
+          # platform-mismatched FROM; COPY-only, so no emulation is needed.
+          # provenance=false keeps the pushed ref a plain single manifest
+          # instead of an OCI index wrapping an attestation.
+          docker buildx build --push \
+            --platform linux/arm64 \
+            --provenance=false \
+            -f Dockerfile.jetson-wheels \
+            -t "${{ steps.image.outputs.ref }}" \
+            .
diff --git a/AGENTS.md b/AGENTS.md
@@ -19,7 +19,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 |------|-------------|
 | [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
 | [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
-| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
+| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, jetson wheels mirror for l4t builds (ghcr-hosted, survives pypi.jetson-ai-lab.io outages), prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
 | [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
 | [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
 | [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |

diff --git a/backend/Dockerfile.python b/backend/Dockerfile.python
@@ -1,6 +1,14 @@
 ARG BASE_IMAGE=ubuntu:24.04
 ARG APT_MIRROR=""
 ARG APT_PORTS_MIRROR=""
+# CI mirror of the CUDA aarch64 wheels from pypi.jetson-ai-lab.io, kept warm
+# by .github/workflows/jetson-wheels.yml so l4t builds survive the upstream
+# index's recurring multi-hour outages. The default (scratch) mounts an empty
+# directory, which makes installRequirements fall through to the upstream
+# index unchanged — local and Jetson-native builds are unaffected.
+ARG JETSON_WHEELS_IMAGE=scratch
+
+FROM ${JETSON_WHEELS_IMAGE} AS jetson-wheels
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -222,7 +230,8 @@ ENV FROM_SOURCE=${FROM_SOURCE}
 # and picks up newer wheels from PyPI / nightly indexes.
 ARG DEPS_REFRESH=initial
 
-RUN cd /${BACKEND} && PORTABLE_PYTHON=true make
+RUN --mount=type=bind,from=jetson-wheels,target=/jetson-wheels \
+    cd /${BACKEND} && PORTABLE_PYTHON=true JETSON_WHEELS_DIR=/jetson-wheels make
 
 # Package GPU libraries into the backend's lib directory.
 #

diff --git a/backend/python/common/libbackend.sh b/backend/python/common/libbackend.sh
@@ -475,6 +475,62 @@ function runProtogen() {
 }
 
 
+# When JETSON_WHEELS_DIR points at a directory of wheels mirrored from
+# pypi.jetson-ai-lab.io (CI bind-mounts the jetson-wheels OCI image there —
+# see backend/Dockerfile.python and .github/workflows/jetson-wheels.yml),
+# installRequirements serves it on localhost as a PEP 503 index and swaps the
+# jetson index host in the requirements files for the local one.
+#
+# The upstream index has a history of multi-hour 502 outages, and a 502 on
+# any project page aborts the whole uv resolution — uv consults every
+# configured index for every requirement, so even PyPI-hosted packages die
+# with it. The local index instead 404s for anything it doesn't carry, which
+# resolvers cleanly follow up on PyPI; only the jetson-built wheels (torch
+# and friends) resolve locally. When JETSON_WHEELS_DIR is unset — the
+# default, e.g. building on a real Jetson — nothing changes and the upstream
+# index is used as written in the requirements files.
+JETSON_PYPI_HOST="pypi.jetson-ai-lab.io"
+_JETSON_MIRROR_PID=""
+_JETSON_MIRROR_URL=""
+
+function _stopJetsonMirror() {
+    if [ -n "${_JETSON_MIRROR_PID}" ]; then
+        kill "${_JETSON_MIRROR_PID}" 2>/dev/null || true
+        _JETSON_MIRROR_PID=""
+        _JETSON_MIRROR_URL=""
+    fi
+}
+
+function _startJetsonMirror() {
+    local script_dir port_file port tries
+    # An empty dir is the JETSON_WHEELS_IMAGE=scratch default in
+    # Dockerfile.python: no mirror was provided, use upstream as-is.
+    if [ -z "$(find "${JETSON_WHEELS_DIR}" -name '*.whl' -print -quit 2>/dev/null)" ]; then
+        echo "jetson wheels dir ${JETSON_WHEELS_DIR} has no wheels, using upstream ${JETSON_PYPI_HOST}"
+        return 0
+    fi
+    script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+    port_file="$(mktemp)"
+    rm -f "${port_file}"
+    python3 "${script_dir}/pypi_mirror_server.py" --root "${JETSON_WHEELS_DIR}" --port-file "${port_file}" &
+    _JETSON_MIRROR_PID=$!
+    trap _stopJetsonMirror EXIT
+    tries=0
+    until [ -s "${port_file}" ]; do
+        tries=$((tries + 1))
+        if [ ${tries} -gt 50 ] || ! kill -0 "${_JETSON_MIRROR_PID}" 2>/dev/null; then
+            echo "WARNING: local jetson wheel mirror failed to start, using upstream ${JETSON_PYPI_HOST}"
+            _stopJetsonMirror
+            return 0
+        fi
+        sleep 0.2
+    done
+    port="$(cat "${port_file}")"
+    rm -f "${port_file}"
+    _JETSON_MIRROR_URL="http://127.0.0.1:${port}"
+    echo "serving jetson wheels from ${JETSON_WHEELS_DIR} at ${_JETSON_MIRROR_URL}"
+}
+
 # installRequirements looks for several requirements files and if they exist runs the install for them in order
 #
 #  - requirements-install.txt
@@ -520,18 +576,31 @@ function installRequirements() {
         export C_INCLUDE_PATH="${C_INCLUDE_PATH:-}:$(_portable_dir)/include/python${PYTHON_VERSION}"
     fi
 
+    if [ -n "${JETSON_WHEELS_DIR:-}" ] && [ -d "${JETSON_WHEELS_DIR}" ]; then
+        _startJetsonMirror
+    fi
+
+    local installFile
     for reqFile in ${requirementFiles[@]}; do
         if [ -f "${reqFile}" ]; then
+            installFile="${reqFile}"
+            if [ -n "${_JETSON_MIRROR_URL}" ] && grep -q "${JETSON_PYPI_HOST}" "${reqFile}"; then
+                installFile="$(mktemp)"
+                sed "s,https://${JETSON_PYPI_HOST},${_JETSON_MIRROR_URL},g" "${reqFile}" > "${installFile}"
+                echo "rewrote ${JETSON_PYPI_HOST} in ${reqFile} to the local wheel mirror (${installFile})"
+            fi
             echo "starting requirements install for ${reqFile}"
             if [ "x${USE_PIP}" == "xtrue" ]; then
-                pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${reqFile}"
+                pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${installFile}"
             else
-                uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${reqFile}"
+                uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement "${installFile}"
             fi
             echo "finished requirements install for ${reqFile}"
         fi
     done
 
+    _stopJetsonMirror
+
     runProtogen
 }