Skip to content

chore(tests): Avoid network, sleep and more during tests - #11050

Open
richiejp wants to merge 21 commits into
mudler:masterfrom
richiejp:perf/test-suite-10m
Open

chore(tests): Avoid network, sleep and more during tests#11050
richiejp wants to merge 21 commits into
mudler:masterfrom
richiejp:perf/test-suite-10m

Conversation

@richiejp

@richiejp richiejp commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR fixes #11048

Notes for Reviewers

  • feat(llama-cpp): route Score through the slot loop
  • feat(realtime): classifier wire types and pipeline config
  • feat(realtime): classifier response flow
  • fix(realtime): bound the VAD tick's scan window and buffer retention
  • fix(backend): let per-model threads override the global default
  • chore(gallery): single-thread the silero VAD
  • docs(realtime): classifier mode, VAD scan window, threads precedence
  • perf(llama-cpp): score all candidates in one batched decode
  • fix(realtime): gate scoring capacity by model usecase
  • fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step
  • feat(realtime): classifier argument slots via constrained completion
  • feat(realtime): splice filled slot values into classifier replies
  • fix(realtime): harden classifier slot completion
  • feat(realtime): prewarm the classifier scoring prompt on registration
  • perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix
  • fix(realtime): align classifier cache guidance
  • fix(llama-cpp): guard score task for fork backends
  • fix(dev): generate gRPC code before commit lint
  • test: make coverage failures observable
  • test: parallelize coverage without remote fixtures
  • test: add offline resource infrastructure
  • test: enforce offline resource replay
  • test: harden offline resource refresh
  • test: expose slow coverage waits
  • test: eliminate avoidable wall-clock waits
  • test: remove repeated fixture startup waits

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

@richiejp
richiejp force-pushed the perf/test-suite-10m branch from ec4ad92 to e9b8d2d Compare July 23, 2026 08:56
@richiejp
richiejp force-pushed the perf/test-suite-10m branch 2 times, most recently from 15a37b0 to 8a54235 Compare August 3, 2026 10:44
@richiejp

richiejp commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

@mudler ping

mudler
mudler reviewed Aug 4, 2026
View reviewed changes
Comment thread core/startup/model_preload.go Outdated

var status *galleryop.OpStatus
// wait for op to finish
poll := time.NewTicker(50 * time.Millisecond)

@mudler mudler Aug 4, 2026

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ops here can be quite long, and having a 50ms poll could be overkill

@richiejp richiejp Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, added exponential backoff up to a limit of 500ms

@richiejp richiejp mentioned this pull request Aug 6, 2026
Closed
2 tasks
@richiejp
richiejp force-pushed the perf/test-suite-10m branch 6 times, most recently from 0054143 to 4cc26d2 Compare August 7, 2026 15:00
richiejp and others added 17 commits August 7, 2026 21:13
@richiejp
test: make coverage failures observable
6ce5aeb 
Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: parallelize coverage without remote fixtures
add82e2 
Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: add offline resource infrastructure
54d8f9f 
Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: enforce offline resource replay
f68ddef 
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: harden offline resource refresh
fe9a2f2 
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: expose slow coverage waits
cbfabf9 
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: eliminate avoidable wall-clock waits
4334bdf 
Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: remove repeated fixture startup waits
3e49fc8 
Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: fix offline resource CI portability
d4aecc1 
Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
ci: cache Go modules before offline tests
7674687 
Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
test: drop the static network lint in favour of real isolation
01a70a9 
The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.

test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:

  +      "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",

Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.

Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.

Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@localai-org-maint-bot @richiejp
ci: keep hidden files in the offline test bundle artifact
8da8fe5 
Cherry-picked from 15a37b0 on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.

The other half of 15a37b0 moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
refactor: share bounded exponential backoff
c36cdcc 
Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
ci: mirror Jetson Python wheels
4449c5e 
Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
docs(agents): index the Jetson wheels mirror
449af4d 
Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
ci: add defensive build network proxy
62086f3 
Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
fix(kokoros): implement updated backend trait
815a53c 
Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
richiejp added 3 commits August 7, 2026 21:13
@richiejp
fix(ci): clear recovered proxy errors
d74adcc 
Do not mark a request failed when a later safe retry succeeds.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
ci: require HTTPS build interception
c3be397 
Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
fix(ci): preserve system trust in unproxied builds
b196ace 
Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.

Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.

Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
@richiejp
richiejp force-pushed the perf/test-suite-10m branch from 4cc26d2 to b196ace Compare August 7, 2026 21:10
@richiejp
fix(ci): persist build proxy trust
405ac69 
Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.

Assisted-by: Codex:gpt-5

Signed-off-by: Richard Palethorpe <io@richiejp.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mudler mudler mudler left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

LocalAI QA: Make tests go fast

3 participants

@richiejp @mudler @localai-org-maint-bot