Skip to content

ci(k8s): cap k8s deployment job runtime to stop multi-hour hangs - #15698

Merged
Maffooch merged 1 commit into
bugfixfrom
claude/k8s-tests-timeout
Aug 18, 2026
Merged

ci(k8s): cap k8s deployment job runtime to stop multi-hour hangs#15698
Maffooch merged 1 commit into
bugfixfrom
claude/k8s-tests-timeout

Conversation

@Maffooch

@Maffooch Maffooch commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

[sc-14593]

Problem

The test-k8s job (k8s-tests.yml) can run for hours. It had no job-level timeout-minutes, and several hang-prone steps were left untimed: Setup Minikube (cluster bring-up), the download-artifact step, and Configure HELM repos (helm dependency update). When any of those stalls, there is nothing to stop it, so the job falls through to GitHub's 360-minute (6-hour) default job timeout and burns runner time.

Fix

  • Job-level timeout-minutes: 30 as a hard backstop that catches a hang in any step, including the uses: steps that can't be caught by the existing per-step timeouts. A healthy run finishes well under 20 minutes, so 30 leaves comfortable headroom.
  • Per-step timeouts on the two untimed shell steps most likely to stall: Setup Minikube (10 min) and Configure HELM repos (5 min), so a failure attributes to the right step rather than only tripping the job cap.

Both k8s-version matrix cells are capped independently.

No behavior change for a healthy run; this only bounds the failure mode.

🤖 Generated with Claude Code

The k8s deployment job had no job-level timeout and left several
hang-prone steps untimed (minikube bring-up, artifact download, helm
dependency update). A stall in any of those fell through to GitHub's
360-minute default job timeout, letting the job burn hours of runner
time.

Add a job-level timeout-minutes: 30 backstop (a healthy run finishes
well under 20 minutes) plus per-step timeouts on the two untimed shell
steps most likely to stall.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Maffooch
Maffooch requested a review from blakeaowens as a code owner August 18, 2026 03:58
@Maffooch Maffooch added this to the 3.2.201 milestone Aug 18, 2026
@Maffooch
Maffooch added this pull request to the merge queue Aug 18, 2026
Merged via the queue into bugfix with commit 1f5a918 Aug 18, 2026
48 of 49 checks passed
@Maffooch
Maffooch deleted the claude/k8s-tests-timeout branch August 18, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants