fix(schema-load): extend cold-cluster deadline

[Copilot]

Summary

On a cold cluster the schema-load Job could not finish inside its own deadline, so it ended in a terminal DeadlineExceeded that Flux will not retry and spi-osdu-reference stayed blocked for the life of the cluster. This resizes the timeout budget to cover every delay that actually consumes it, and pins the relationships between the coupled values so they cannot drift apart again.

Changes

• Job deadline 30 min to 150 min (activeDeadlineSeconds: 9000), split into three named phases:

  Phase	Budget	Why
  Pod startup	1800s (30 min)	activeDeadlineSeconds starts when the Job becomes active, before the Pod is scheduled, so node provisioning, scheduling, and image pull all count against it. [bug] schema-load Job deadline is shorter than a cold-cluster load #83 observed ~28 min of node provisioning.
  Schema endpoint wait	2700s (45 min)	WAIT_DEADLINE_SECONDS in bootstrap.sh, which only starts once the Pod is running. Observed worst case in [bug] schema-load Job deadline is shorter than a cold-cluster load #83 was ~33 min.
  Schema load	4500s (75 min)	The throttled POST of ~1,386 schemas can exceed 30 min on its own while Cosmos autoscales up from its 400 RU/s floor (feat(platform): selectable AKS modes, pinned service images, Entra-only data plane, and cluster-side CI/CD onboarding #36).

  activeDeadlineSeconds is terminal and overrides backoffLimit, so it has to cover the combined case rather than any single phase.

• Flux timeout 35 min to 155 min on spi-osdu-schema-load. It must outlast the Job deadline, or Flux reports Ready=False while the loader is still running happily.

• Wait loop now bounded by elapsed time, not attempt count. The old 600 attempts * 2s budget claimed ~20 min, but each attempt can also spend up to 5s inside urlopen, so it could run closer to 70 min and burn the Job deadline while still polling. It now measures wall clock.

• Smoke gate raised to match (timeout-minutes: 50 to 270, --timeout 2700 to 13800). One Kustomization may now legitimately occupy 155 min, spi-osdu-reference only starts once it is Ready, and the job still needs room for checkout, tool install, login, and the HTTPS probe. The old gate would have killed a healthy --profile core run at 50 min and torn the cluster down.

• Docs: dated amendment on ADR-007, plus docs/design/flux-reconciliation.md and docs/design/ci-smoke.md synced to the new budgets. The stale claim that spi-osdu-services runs with wait: false is corrected in script.yaml.

Why the numbers are pinned, not just raised

Four coupled constants now live across four files, which is how the original 35 min and 1800s drifted out of agreement in the first place. Each relationship has a test:

Invariant	Test
Flux timeout > Job deadline	test_flux_timeout_exceeds_job_deadline
deadline - startup - wait >= 3600s	test_job_deadline_leaves_load_headroom_beyond_the_service_wait
smoke gate > schema-load + reference	test_verify_gate_outlives_the_schema_load_and_reference_timeouts

The first and third derive their expectations from the manifests, so a future tuning pass cannot satisfy them by editing the test. The pod-startup allowance is the exception: it exists only as a budget line, not a manifest field, so POD_STARTUP_ALLOWANCE_SECONDS is asserted against the value documented in job.yaml and the two must be kept in step by hand.

Tradeoffs worth knowing

A genuinely broken load now blocks for 150 min before failing, with spi-osdu-reference gated behind it, and a worst-case --profile core smoke run can occupy 270 min in verify.

The budget is also deliberately conservative in a way worth revisiting: the pod-startup and schema-endpoint phases are budgeted serially, but in the failure recorded in #83 they overlap, because both are waiting on the same osdu node pool to scale up. Sizing them independently is the safe direction, since over-budgeting only delays a failure rather than causing one, but it means 150 min is an upper bound built from correlated worst cases rather than a measured figure. Worth tuning once more than one cold-cluster run has been observed.

Not changed: healthChecks on spi-osdu-services. Adding them would fix the ordering race that lets the loader start before the schema pods are serving, but it is not a substitute for this change, because Cosmos throttling is independent of pod readiness. Left for a follow-up.

Validation

• CI is green (lint, typecheck, test, manifests, bicep-whatif)
• Review threads resolved
• Docs and ADRs updated if behavior changed

All 9 checks pass on a229fe9, and all 9 review threads are resolved. No live cold-cluster run has exercised the 150 min budget yet, so the sizing rests on the timelines recorded in #83 and #36.

Release label

release:patch - bug fix, no API change.

• Fixes [bug] schema-load Job deadline is shorter than a cold-cluster load #83

[Copilot]

Pull request overview

Extends schema loading timeouts for cold clusters and adds regression protection.

Changes:

• Raises the Job deadline to 60 minutes and Flux timeout to 65 minutes.
• Adds a timeout-ordering test.
• Updates readiness and timeout comments.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/test_profiles.py	Tests Flux timeout ordering.
software/stacks/osdu/schema-load/script.yaml	Updates readiness commentary.
software/stacks/osdu/schema-load/job.yaml	Extends the Job deadline.
software/stacks/osdu/profiles/core/stack.yaml	Extends the Flux timeout.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

software/stacks/osdu/schema-load/script.yaml:40

• This calculation omits the probe's urlopen(..., timeout=5). A failed iteration can consume about seven seconds including the sleep, so 600 attempts can run for roughly 70 minutes; the 60-minute Job deadline can terminate the pod before this loop finishes or schema loading starts. Use an elapsed-time polling budget that leaves time for the load, or document the actual deadline-driven behavior.

    # Wait up to ~20 min for the schema service to return 200 on /info.
    # spi-osdu-services waits for the HelmReleases to report Ready, which does
    # not guarantee the schema endpoint is serving yet. On a fresh cluster the
    # schema pod may still be pulling its image when this Job starts; 600
    # attempts * 2s remains bounded by the Job's 60 min deadline.

software/stacks/osdu/profiles/core/stack.yaml:423

• The repository's operational documentation still advertises the old value: docs/design/flux-reconciliation.md:66 and accepted ADR-007 line 37 both say schema-load has a 35-minute timeout. Update the living design document and record this tuning as a short dated ADR amendment rather than leaving two conflicting timeout contracts.

  # Matches the Job's activeDeadlineSeconds (3600s) plus headroom for
  # image pull and reconcile overhead.
  timeout: 65m

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Copilot The timeout arithmetic now closes and the headroom test is exactly right, but three things still need work, and one of them is a thread that was marked resolved without being fixed.

1. T...

Addressed all four points:

1. Raised the smoke gate: verify job timeout-minutes 50 -> 210 and wait_for_flux_ready.sh --timeout 2700 -> 12000 (200 min), sized for the ~30 min of preceding layers plus schema-load's full 125 min plus reference's 30 min. Added test_verify_gate_outlives_the_schema_load_and_reference_timeouts so this can't silently regress.
2. Fixed the job.yaml comment: the wait budget is 45 min (WAIT_DEADLINE_SECONDS=2700), not 35, so the halves now correctly sum to 120.
3. Added the sizing rationale to the PR description: 45 min wait cap (vs the ~33 min observed in [bug] schema-load Job deadline is shorter than a cold-cluster load #83, leaving headroom) + 75 min for token acquisition and the throttled load.
4. Updated the PR description to match the implemented 7200s/120m Job deadline and 125m Flux timeout, including the smoke gate change.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

software/stacks/osdu/profiles/core/stack.yaml:423

• The PR description still promises a 60-minute Job deadline and 65-minute Flux timeout, but the implementation now deploys 120 and 125 minutes. Update the description to the final timeout budget (or restore the documented values) so reviewers and release history accurately describe the operational change.

  timeout: 125m

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

.github/workflows/smoke.yml:205

• The 210-minute job limit leaves only 10 minutes after this step's 200-minute wait, but the HTTPS acceptance loop below can consume 15 minutes by itself (20 attempts × a 15-second curl plus a 30-second sleep), before checkout, tool installation, login, and the gateway probe. A healthy reconciliation can therefore finish and still have the job killed mid-verification. Raise the job timeout with explicit setup/probe headroom, and strengthen the new regression test to require that margin rather than merely checking job timeout > wait timeout.

    timeout-minutes: 210

software/stacks/osdu/schema-load/job.yaml:32

• The PR description still states that this deadline becomes 60 minutes and the Flux timeout becomes 65 minutes, while the implementation now uses 120 and 125 minutes. Update the description's timeout budget so the reviewed behavior and release context match the final code.

  activeDeadlineSeconds: 7200

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

software/stacks/osdu/schema-load/job.yaml:32

• activeDeadlineSeconds starts with the Job, while start_ts is created only after the loader container starts. Pod scheduling/image pull (and any retry) therefore consumes this 7200-second budget before WAIT_DEADLINE_SECONDS begins, so this does not actually reserve the stated 75 minutes for loading. The new test also subtracts only the script wait and can pass without preserving its claimed load headroom. Please include an explicit cold-start allowance in the Job budget and invariant, then update the coupled Flux and smoke timeouts.

  # 7200s covers the script's own WAIT_DEADLINE_SECONDS=2700 (the schema

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

software/stacks/osdu/schema-load/job.yaml:38

• The PR description still specifies a 7,200-second (120-minute) Job deadline and the older coupled 125-minute Flux, 12,000-second wait, and 210-minute verify budgets, while the proposed files now use 9,000/155/13,800/270. It also lists the smoke documentation as outstanding even though this diff updates it. Please update the PR description and its invariant/rationale text to describe the code reviewers are approving.

  activeDeadlineSeconds: 9000