Fail fast when OIDC credentials cannot be obtained - #31
Merged
Conversation
The `Configure AWS credentials` step took 82 minutes to fail on a cyber-dojo/dashboard run. Each of the twelve default retry attempts sat on a TCP connect to the STS endpoint that took ~6m45s to time out at the OS level. `connect ETIMEDOUT` on port 443 means no socket was ever opened, so nothing was going to succeed — but the runner was held and billed for 82 minutes, and the caller's `terraform-<name>` concurrency group was held with it, queueing every subsequent apply for that environment behind a job that was already doomed. Bound the step so it dies quickly instead: action-timeout-s: 45 bounds the action as a whole, retries included retry-max-attempts: 3 the default of 12 only helps if attempts are fast timeout-minutes: 2 backstop, independent of the action's behaviour 45 seconds is justified by measurement: across recent successful runs in kosli-dev and cyber-dojo the step took 0-2 seconds, worst case 6, so the timeout sits far above the real p99 and will not fail a slow-but-healthy authentication. `disable-retry` is deliberately not used — STS throttling is genuinely transient and a couple of quick retries is worth having now that their total cost is bounded. Every job also gets a `timeout-minutes` rather than silently inheriting GitHub's 360-minute default, which an 82-minute hang would never have tripped. For the plan/apply job the ceiling follows from `aws_role_duration` rather than from how long Terraform might run: the credentials are static environment variables that are never refreshed, so 20 minutes (the 1200s default) after the credentials step every AWS call fails with ExpiredToken and the work cannot usefully continue. Setup before that step measures 1-35s, so a job consuming its whole credential lifetime lands around 22-23 minutes — hence a default of 30, which is headroom over the real ceiling rather than an arbitrary number. Housekeeping jobs get 5-10. Because that ceiling is coupled to `aws_role_duration`, it is exposed as a `job_timeout_minutes` input on plan.yml, apply.yml and detect-drift.yml rather than hard-coded. A repository with a legitimately long apply raises both together, without needing a change here. Raising only the session duration would let the job be cancelled part-way through an apply, which can leave the state lock held. Beyond base.yml and apply.yml named in the ticket, detect-drift.yml has two `Configure AWS credentials` steps with the same failure mode, so they are covered here too. The values are recorded in the README as the standard for other repositories to adopt. Ticket: #1043
mbevc1
approved these changes
Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
Configure AWS credentialsstep took 82 minutes to fail on a cyber-dojo/dashboard run. Each of the twelve default retry attempts sat on a TCP connect to the STS endpoint that took ~6m45s to time out at the OS level.connect ETIMEDOUTon port 443 means no socket was ever opened, so nothing was going to succeed — but the runner was held and billed for 82 minutes, and the caller'sterraform-<name>concurrency group was held with it, queueing every subsequent apply for that environment behind a job that was already doomed.Bound the step so it dies quickly instead:
action-timeout-s: 45 bounds the action as a whole, retries included
retry-max-attempts: 3 the default of 12 only helps if attempts are fast
timeout-minutes: 2 backstop, independent of the action's behaviour
45 seconds is justified by measurement: across recent successful runs in kosli-dev and cyber-dojo the step took 0-2 seconds, worst case 6, so the timeout sits far above the real p99 and will not fail a slow-but-healthy authentication.
disable-retryis deliberately not used — STS throttling is genuinely transient and a couple of quick retries is worth having now that their total cost is bounded.Every job also gets a
timeout-minutesrather than silently inheriting GitHub's 360-minute default, which an 82-minute hang would never have tripped. For the plan/apply job the ceiling follows fromaws_role_durationrather than from how long Terraform might run: the credentials are static environment variables that are never refreshed, so 20 minutes (the 1200s default) after the credentials step every AWS call fails with ExpiredToken and the work cannot usefully continue. Setup before that step measures 1-35s, so a job consuming its whole credential lifetime lands around 22-23 minutes — hence a default of 30, which is headroom over the real ceiling rather than an arbitrary number. Housekeeping jobs get 5-10.Because that ceiling is coupled to
aws_role_duration, it is exposed as ajob_timeout_minutesinput on plan.yml, apply.yml and detect-drift.yml rather than hard-coded. A repository with a legitimately long apply raises both together, without needing a change here. Raising only the session duration would let the job be cancelled part-way through an apply, which can leave the state lock held.Beyond base.yml and apply.yml named in the ticket, detect-drift.yml has two
Configure AWS credentialssteps with the same failure mode, so they are covered here too. The values are recorded in the README as the standard for other repositories to adopt.