Skip to content

Fail fast when OIDC credentials cannot be obtained - #31

Merged
gsavage merged 1 commit into
mainfrom
oidc-fail-fast
Aug 4, 2026
Merged

Fail fast when OIDC credentials cannot be obtained#31
gsavage merged 1 commit into
mainfrom
oidc-fail-fast

Conversation

@gsavage

@gsavage gsavage commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The Configure AWS credentials step took 82 minutes to fail on a cyber-dojo/dashboard run. Each of the twelve default retry attempts sat on a TCP connect to the STS endpoint that took ~6m45s to time out at the OS level. connect ETIMEDOUT on port 443 means no socket was ever opened, so nothing was going to succeed — but the runner was held and billed for 82 minutes, and the caller's terraform-<name> concurrency group was held with it, queueing every subsequent apply for that environment behind a job that was already doomed.

Bound the step so it dies quickly instead:

action-timeout-s: 45 bounds the action as a whole, retries included
retry-max-attempts: 3 the default of 12 only helps if attempts are fast
timeout-minutes: 2 backstop, independent of the action's behaviour

45 seconds is justified by measurement: across recent successful runs in kosli-dev and cyber-dojo the step took 0-2 seconds, worst case 6, so the timeout sits far above the real p99 and will not fail a slow-but-healthy authentication. disable-retry is deliberately not used — STS throttling is genuinely transient and a couple of quick retries is worth having now that their total cost is bounded.

Every job also gets a timeout-minutes rather than silently inheriting GitHub's 360-minute default, which an 82-minute hang would never have tripped. For the plan/apply job the ceiling follows from aws_role_duration rather than from how long Terraform might run: the credentials are static environment variables that are never refreshed, so 20 minutes (the 1200s default) after the credentials step every AWS call fails with ExpiredToken and the work cannot usefully continue. Setup before that step measures 1-35s, so a job consuming its whole credential lifetime lands around 22-23 minutes — hence a default of 30, which is headroom over the real ceiling rather than an arbitrary number. Housekeeping jobs get 5-10.

Because that ceiling is coupled to aws_role_duration, it is exposed as a job_timeout_minutes input on plan.yml, apply.yml and detect-drift.yml rather than hard-coded. A repository with a legitimately long apply raises both together, without needing a change here. Raising only the session duration would let the job be cancelled part-way through an apply, which can leave the state lock held.

Beyond base.yml and apply.yml named in the ticket, detect-drift.yml has two Configure AWS credentials steps with the same failure mode, so they are covered here too. The values are recorded in the README as the standard for other repositories to adopt.

The `Configure AWS credentials` step took 82 minutes to fail on a
cyber-dojo/dashboard run. Each of the twelve default retry attempts sat
on a TCP connect to the STS endpoint that took ~6m45s to time out at the
OS level. `connect ETIMEDOUT` on port 443 means no socket was ever
opened, so nothing was going to succeed — but the runner was held and
billed for 82 minutes, and the caller's `terraform-<name>` concurrency
group was held with it, queueing every subsequent apply for that
environment behind a job that was already doomed.

Bound the step so it dies quickly instead:

  action-timeout-s: 45     bounds the action as a whole, retries included
  retry-max-attempts: 3    the default of 12 only helps if attempts are fast
  timeout-minutes: 2       backstop, independent of the action's behaviour

45 seconds is justified by measurement: across recent successful runs in
kosli-dev and cyber-dojo the step took 0-2 seconds, worst case 6, so the
timeout sits far above the real p99 and will not fail a slow-but-healthy
authentication. `disable-retry` is deliberately not used — STS
throttling is genuinely transient and a couple of quick retries is worth
having now that their total cost is bounded.

Every job also gets a `timeout-minutes` rather than silently inheriting
GitHub's 360-minute default, which an 82-minute hang would never have
tripped. For the plan/apply job the ceiling follows from
`aws_role_duration` rather than from how long Terraform might run: the
credentials are static environment variables that are never refreshed,
so 20 minutes (the 1200s default) after the credentials step every AWS
call fails with ExpiredToken and the work cannot usefully continue.
Setup before that step measures 1-35s, so a job consuming its whole
credential lifetime lands around 22-23 minutes — hence a default of 30,
which is headroom over the real ceiling rather than an arbitrary number.
Housekeeping jobs get 5-10.

Because that ceiling is coupled to `aws_role_duration`, it is exposed as
a `job_timeout_minutes` input on plan.yml, apply.yml and
detect-drift.yml rather than hard-coded. A repository with a
legitimately long apply raises both together, without needing a change
here. Raising only the session duration would let the job be cancelled
part-way through an apply, which can leave the state lock held.

Beyond base.yml and apply.yml named in the ticket, detect-drift.yml has
two `Configure AWS credentials` steps with the same failure mode, so
they are covered here too. The values are recorded in the README as the
standard for other repositories to adopt.

Ticket: #1043
@gsavage
gsavage requested a review from JonJagger August 4, 2026 14:43
@gsavage
gsavage merged commit d09aa0c into main Aug 4, 2026
1 check passed
@gsavage
gsavage deleted the oidc-fail-fast branch August 4, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants