chore(tests): Avoid network, sleep and more during tests

.agents/building-and-testing.md

@@ -21,6 +21,9 @@ Let's say the user wants to build a particular backend for a given platform. For
 The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./tests/e2e`) are covered by a **strict, monotonic coverage ratchet**:
 
 - `make test-coverage` — runs the suites with `covermode=atomic` instrumentation and writes a merged profile to `coverage/coverage.out`. Uses the same prerequisites as `make test`.
+  - Prints per-root wall time and the slowest specs/hooks exceeding `COVERAGE_SLOW_SPEC_THRESHOLD` (default 3 seconds, capped by `COVERAGE_SLOW_SPEC_LIMIT`, default 25 per root); machine-readable root timings are written to `coverage/timings.tsv`.
+  - Verbose Ginkgo output is written to `coverage/logs/<root>.log`, with the prior run retained as `<root>.log.previous`. The terminal prints one status line per root and a short failure extract. If any suite fails, no merged profile is produced and the percentage ratchet is explicitly not run. A lock under `coverage/` rejects concurrent runs, which would otherwise corrupt their shared profiles and logs.
+  - Suites run in parallel by default and each recursive root invocation has a five-minute budget. Override auto-detected parallelism with `COVERAGE_PROCS`; tune diagnostics with `COVERAGE_SUITE_TIMEOUT` and `COVERAGE_PROGRESS_AFTER`. A timeout is a performance failure to investigate, not a reason to raise the committed default.
   - **`--coverpkg` (`COVERAGE_COVERPKG = core/...,pkg/...`):** coverage is attributed to the core+pkg packages, not just the package under test. This is what lets the in-process `tests/e2e` suite (which drives the real HTTP server over loopback via `application.New`) credit the `core/http/endpoints/...` handlers it exercises — folding it in roughly doubled endpoint coverage (e.g. `endpoints/openai` 13.6% → 52%). The denominator is therefore *all* of `core`+`pkg` (minus generated proto, dropped via `COVERAGE_EXCLUDE_RE`), so the number isn't comparable to a plain per-package figure.
   - **Integration suites (`COVERAGE_E2E_ROOTS = ./tests/e2e`)** run non-recursively (excludes `tests/e2e/distributed`, which needs containers) with `--label-filter=!real-models` (those need a downloaded model) against the mock backend built by `prepare-test`. `tests/integration` is deliberately excluded — it needs `make backends/local-store`, which the coverage CI job doesn't build.
   - **Flake note:** folding integration tests into a *strict* gate means a hard e2e failure (or a spec that silently stops running) can fail the coverage gate, not just the test. `--flake-attempts` absorbs transient retryable failures; covermode=atomic keeps line coverage deterministic otherwise.

.agents/ci-caching.md

@@ -1,5 +1,20 @@
 # CI Build Caching
 
+## Build network inventory and defensive proxy
+
+Backend and main-image builds use `cmd/build-proxy` as a strict HTTPS-intercepting
+proxy through the host network. Its short-lived CA is injected into the running
+BuildKit daemon and mounted over the conventional CA bundle during every
+Dockerfile `RUN`. Plain HTTP and opaque CONNECT traffic fail the job. Every
+destination is retained for 14 days as JSONL plus an aggregate host/method/byte
+summary. Responses are spooled and checked against `Content-Length`; GET/HEAD
+requests retry transient status codes or incomplete responses with exponential
+backoff capped at 500ms. Request headers, bodies, credentials and query strings
+are never recorded.
+
+The inventory is intended to size a future content-addressed cache and identify
+hosts worth adding to curated OCI mirrors, such as the Jetson wheels mirror.
+
 Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
 
 ## Workflow surfaces
@@ -231,6 +246,20 @@ This applies only to `Dockerfile.python` because:
 
 Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
 
+## The jetson wheels mirror (l4t builds)
+
+The `requirements-l4t12.txt` / `-l4t13.txt` files pull CUDA aarch64 torch wheels from `pypi.jetson-ai-lab.io` via `--extra-index-url`. That index has a history of multi-hour 502 outages, and a 502 on **any** project page aborts the whole uv resolution — uv consults every configured index for every requirement, so even PyPI-hosted packages die with it. To keep l4t builds green through outages, CI serves those wheels from a mirror it controls:
+
+- **Storage**: `ghcr.io/mudler/localai/jetson-wheels:{jp6-cu129,jp7-cu130}` — scratch OCI images holding the wheel subset, laid out like the upstream index (`/jp6/cu129/torch/<wheel>`).
+- **Sync**: `.github/workflows/jetson-wheels.yml` (Saturdays 03:00 UTC, ahead of the weekly `DEPS_REFRESH` re-resolve; also `workflow_dispatch` and master pushes touching its inputs) runs `scripts/jetson-wheels-sync.py` against the package list in `.github/jetson-wheels.json`. During an upstream outage the sync keeps the last-known-good wheels and exits green.
+- **Consumption**: `backend_build.yml` resolves the matching tag for `build-type: l4t` entries (cuda 12 → `jp6-cu129`, 13 → `jp7-cu130`) and passes it as the `JETSON_WHEELS_IMAGE` build-arg; `Dockerfile.python` bind-mounts it at `/jetson-wheels`; `installRequirements` in `backend/python/common/libbackend.sh` serves that directory on localhost as a PEP 503 index (`backend/python/common/pypi_mirror_server.py`) and rewrites the jetson index host in the requirements files to it. The local index 404s for anything it doesn't carry, which uv follows up on PyPI — only the jetson-built wheels resolve locally.
+- **Fallbacks**: if the mirror tag doesn't exist (bootstrap) `backend_build.yml` passes `scratch`, the mount is empty, and the build talks to the upstream index exactly as before. Builds outside CI (local, real Jetsons) never set `JETSON_WHEELS_IMAGE` and are unaffected.
+- **Cache interaction**: the bind mount's content is part of the `RUN ... make` layer's BuildKit hash, so a refreshed wheels image invalidates the install layer on the next build — no extra cache-buster needed.
+
+**Extending the package list**: a package a build needs from the jetson index but missing from `.github/jetson-wheels.json` resolves from PyPI instead — for compiled CUDA packages that silently means a CPU build. When adding an l4t backend with new compiled deps, add them to the list and dispatch `jetson-wheels.yml`.
+
+**Bootstrap** (one-time): `gh workflow run jetson-wheels.yml --ref master`, then make the `jetson-wheels` ghcr package public so anonymous pulls work (Settings → Packages).
+
 ## ccache for C++ backend builds
 
 `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:

.docker/apt-mirror.sh

@@ -7,8 +7,7 @@
 #
 # Inputs (env):
 #   APT_MIRROR        Replacement for archive.ubuntu.com and security.ubuntu.com
-#                     (e.g. "http://azure.archive.ubuntu.com" or
-#                      "https://mirrors.edge.kernel.org").
+#                     (e.g. "https://azure.archive.ubuntu.com").
 #                     Leave empty to keep upstream. The trailing "/ubuntu/..."
 #                     path is preserved by the rewrite.
 #   APT_PORTS_MIRROR  Replacement for ports.ubuntu.com (arm64/ppc64el/...).
@@ -18,8 +17,22 @@
 
 set -e
 
-if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
-    exit 0
+# BuildKit exposes the ephemeral interception CA at this dedicated path. Copy
+# it into the image trust bundle only when the proxy-enabled workflows supply
+# it; ordinary local and test builds retain their base-image trust unchanged.
+proxy_ca=/run/secrets/build_proxy_ca
+if [ -s "$proxy_ca" ]; then
+    # Keep the generated CA in the distribution-managed local certificate
+    # directory. Installing or upgrading ca-certificates later in this layer
+    # regenerates the bundle, so appending directly to it would be lost.
+    mkdir -p /usr/local/share/ca-certificates
+    cp "$proxy_ca" /usr/local/share/ca-certificates/localai-build-proxy.crt
+    if command -v update-ca-certificates >/dev/null 2>&1; then
+        update-ca-certificates
+    fi
+    cat > /etc/apt/apt.conf.d/99localai-build-proxy-ca <<EOF
+Acquire::https::CaInfo "$proxy_ca";
+EOF
 fi
 
 # Ubuntu 24.04 (noble) ships DEB822 sources at /etc/apt/sources.list.d/ubuntu.sources;
@@ -36,4 +49,11 @@ for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
     fi
 done
 
+# Build networking is HTTPS-only. Upgrade any untouched distribution defaults
+# as well (notably ports.ubuntu.com when a caller leaves its override empty).
+for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
+    [ -f "$f" ] || continue
+    sed -i -E 's,http://,https://,g' "$f"
+done
+
 echo "apt-mirror: rewrote sources (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.docker/bonsai-compile.sh

@@ -4,6 +4,8 @@
 
 set -euxo pipefail
 
+sh /LocalAI/.docker/install-build-proxy-ca.sh
+
 export CCACHE_DIR=/root/.ccache
 ccache --max-size=5G || true
 ccache -z || true

.docker/ik-llama-cpp-compile.sh

@@ -4,6 +4,8 @@
 
 set -euxo pipefail
 
+sh /LocalAI/.docker/install-build-proxy-ca.sh
+
 export CCACHE_DIR=/root/.ccache
 ccache --max-size=5G || true
 ccache -z || true

.docker/install-base-deps.sh

@@ -35,6 +35,22 @@
 
 set -eux
 
+# Some callers use this script without apt-mirror.sh. Ensure those first-stage
+# downloads also trust the proxy without masking the normal system CA bundle.
+proxy_ca=/run/secrets/build_proxy_ca
+if [ -s "$proxy_ca" ]; then
+    # Persist the generated CA as a local certificate so a subsequent
+    # ca-certificates package install cannot regenerate the bundle without it.
+    mkdir -p /usr/local/share/ca-certificates
+    cp "$proxy_ca" /usr/local/share/ca-certificates/localai-build-proxy.crt
+    if command -v update-ca-certificates >/dev/null 2>&1; then
+        update-ca-certificates
+    fi
+    cat > /etc/apt/apt.conf.d/99localai-build-proxy-ca <<EOF
+Acquire::https::CaInfo "$proxy_ca";
+EOF
+fi
+
 # --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
 if [ -x /usr/local/sbin/apt-mirror ]; then
     APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \

.docker/install-build-proxy-ca.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+# Install CI's optional HTTPS interception CA without replacing public roots.
+
+set -e
+
+proxy_ca=/run/secrets/build_proxy_ca
+if [ ! -s "$proxy_ca" ]; then
+    exit 0
+fi
+
+mkdir -p /usr/local/share/ca-certificates
+cp "$proxy_ca" /usr/local/share/ca-certificates/localai-build-proxy.crt
+update-ca-certificates

.docker/llama-cpp-compile.sh

@@ -4,6 +4,8 @@
 
 set -euxo pipefail
 
+sh /LocalAI/.docker/install-build-proxy-ca.sh
+
 export CCACHE_DIR=/root/.ccache
 ccache --max-size=5G || true
 ccache -z || true

.docker/turboquant-compile.sh

@@ -4,6 +4,8 @@
 
 set -euxo pipefail
 
+sh /LocalAI/.docker/install-build-proxy-ca.sh
+
 export CCACHE_DIR=/root/.ccache
 ccache --max-size=5G || true
 ccache -z || true

.github/actions/configure-apt-mirror/action.yml

@@ -20,28 +20,23 @@ inputs:
   github-hosted-mirror:
     description: 'archive/security mirror URL for github-hosted runners (empty = upstream)'
     required: false
-    default: 'http://azure.archive.ubuntu.com'
+    default: 'https://archive.ubuntu.com'
   github-hosted-ports-mirror:
     description: 'ports.ubuntu.com mirror URL for github-hosted runners (empty = upstream)'
     required: false
-    default: 'http://azure.ports.ubuntu.com'
+    default: 'https://ports.ubuntu.com'
   self-hosted-mirror:
     description: 'archive/security mirror URL for self-hosted runners (empty = upstream)'
     required: false
-    # HTTP, not HTTPS: the bare ubuntu:24.04 builder image doesn't ship
-    # ca-certificates, so the very first apt-get update over TLS would
-    # fail with "No system certificates available" before it can install
-    # anything. apt validates package integrity via GPG signatures, so
-    # plain HTTP is safe for the archive itself.
-    default: 'http://mirrors.edge.kernel.org'
+    default: 'https://mirrors.edge.kernel.org'
   self-hosted-ports-mirror:
     description: 'ports.ubuntu.com mirror URL for self-hosted runners (empty = upstream)'
     required: false
     # mirrors.edge.kernel.org does NOT carry /ubuntu-ports/ — only the
     # main /ubuntu/ archive — so arm64 builds 404 there. Leave ports
     # upstream by default. The original DDoS was on archive.ubuntu.com
     # so ports.ubuntu.com remains the path of least surprise.
-    default: ''
+    default: 'https://ports.ubuntu.com'
 
 outputs:
   effective-mirror:

.github/jetson-wheels.json

@@ -0,0 +1,25 @@
+{
+  "upstream": "https://pypi.jetson-ai-lab.io",
+  "indexes": {
+    "jp6/cu129": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ],
+    "jp7/cu130": [
+      "torch",
+      "torchvision",
+      "torchaudio",
+      "torchcodec",
+      "torchao",
+      "bitsandbytes",
+      "onnxruntime",
+      "ctranslate2"
+    ]
+  }
+}

.github/scripts/inject-build-proxy-ca.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ca="${LOCALAI_BUILD_PROXY_CA:?build proxy CA is unset}"
+builder="${BUILDER_NAME:?Buildx builder name is unset}"
+container="$(docker ps --filter "name=buildx_buildkit_${builder}0" --format '{{.ID}}' | head -n1)"
+if test -z "$container"; then
+  echo 'BuildKit container not found' >&2
+  exit 1
+fi
+
+docker exec "$container" mkdir -p /usr/local/share/ca-certificates /etc/ssl/certs
+docker cp "$ca" "$container:/usr/local/share/ca-certificates/localai-build-proxy.crt"
+docker exec "$container" sh -eu -c '
+  if command -v update-ca-certificates >/dev/null 2>&1; then
+    update-ca-certificates
+  else
+    cat /usr/local/share/ca-certificates/localai-build-proxy.crt >>/etc/ssl/certs/ca-certificates.crt
+  fi
+'
+docker restart "$container" >/dev/null

.github/scripts/start-build-proxy.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+set -euo pipefail
+output="${RUNNER_TEMP}/localai-build-proxy"
+mkdir -p "$output"
+CGO_ENABLED=0 GOCACHE="${RUNNER_TEMP}/go-build-cache" go build -o "$output/build-proxy" ./cmd/build-proxy
+nohup "$output/build-proxy" --listen 127.0.0.1:18080 --output "$output" >"$output/proxy.log" 2>&1 &
+echo "$!" >"$output/proxy.pid"
+for _ in $(seq 1 50); do
+  grep -q '^ca=' "$output/proxy.log" && break
+  sleep 0.1
+done
+grep '^proxy=' "$output/proxy.log"
+grep '^ca=' "$output/proxy.log"
+{
+  echo "LOCALAI_BUILD_PROXY=http://127.0.0.1:18080"
+  echo "LOCALAI_BUILD_PROXY_OUTPUT=$output"
+  echo "LOCALAI_BUILD_PROXY_CA=$output/ca/ca.crt"
+  echo "HTTP_PROXY=http://127.0.0.1:18080"
+  echo "HTTPS_PROXY=http://127.0.0.1:18080"
+  echo "http_proxy=http://127.0.0.1:18080"
+  echo "https_proxy=http://127.0.0.1:18080"
+  echo "SSL_CERT_FILE=$output/ca/ca.crt"
+  echo "CURL_CA_BUNDLE=$output/ca/ca.crt"
+  echo "REQUESTS_CA_BUNDLE=$output/ca/ca.crt"
+  echo "GIT_SSL_CAINFO=$output/ca/ca.crt"
+  echo "NODE_EXTRA_CA_CERTS=$output/ca/ca.crt"
+  echo "NO_PROXY=localhost,127.0.0.1"
+  echo "no_proxy=localhost,127.0.0.1"
+} >>"$GITHUB_ENV"

.github/scripts/stop-build-proxy.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+if test -z "${LOCALAI_BUILD_PROXY_OUTPUT:-}"; then
+  echo 'Build proxy did not start; skipping inventory finalization'
+  exit 0
+fi
+output="$LOCALAI_BUILD_PROXY_OUTPUT"
+if test -f "$output/proxy.pid"; then
+  kill -TERM "$(cat "$output/proxy.pid")" 2>/dev/null || true
+  for _ in $(seq 1 50); do
+    test -f "$output/summary.json" && break
+    sleep 0.1
+  done
+fi
+
+# Later artifact uploads and action post-hooks must not target a stopped proxy.
+{
+  echo 'HTTP_PROXY='
+  echo 'HTTPS_PROXY='
+  echo 'http_proxy='
+  echo 'https_proxy='
+  echo 'SSL_CERT_FILE='
+  echo 'CURL_CA_BUNDLE='
+  echo 'REQUESTS_CA_BUNDLE='
+  echo 'GIT_SSL_CAINFO='
+  echo 'NODE_EXTRA_CA_CERTS='
+} >>"$GITHUB_ENV"
+if test -f "$output/summary.json"; then
+  {
+    echo '### Build network inventory'
+    echo
+    echo 'HTTPS without the generated CA is reported as CONNECT because its HTTP method is encrypted.'
+    echo
+    echo '```json'
+    cat "$output/summary.json"
+    echo '```'
+  } >>"$GITHUB_STEP_SUMMARY"
+fi
+
+# A matrix cancellation can interrupt checkout or a BuildKit request at any
+# point. Preserve whatever inventory exists, but do not replace the canceled
+# conclusion with a misleading proxy-enforcement failure.
+if test "${LOCALAI_BUILD_JOB_STATUS:-}" = cancelled; then
+  echo 'Build was cancelled; skipping network inventory enforcement'
+  exit 0
+fi
+
+if ! test -s "$output/events.jsonl"; then
+  echo 'Build proxy produced no network inventory' >&2
+  exit 1
+fi
+if grep -qE '"method":"CONNECT"|"error":"plain HTTP is forbidden"' "$output/events.jsonl"; then
+  echo 'Build traffic bypassed HTTPS interception or attempted plain HTTP' >&2
+  exit 1
+fi