Skip to content

feat(catalog): make tier requirements data, and hold every caller to them - #178

Merged
rldyourmnd merged 1 commit into
mainfrom
feat/validation-tier-contract
Aug 14, 2026
Merged

feat(catalog): make tier requirements data, and hold every caller to them#178
rldyourmnd merged 1 commit into
mainfrom
feat/validation-tier-contract

Conversation

@rldyourmnd

Copy link
Copy Markdown
Contributor

Closes the last of the three callers, and lands the contract that makes all three
permanent. Ordered deliberately: #170 and #172/#176 fixed ci.yml and
maintenance.yml first, so this contract arrives green rather than red.

The single root cause behind three findings

validate_all.py --tier scheduled needs a GitHub token, two external hosts, and
the tag refs. That requirement was written down once — as comments beside
maintenance.yml's egress allow-list — and enforced nowhere. Three jobs invoked the
tier; one granted what it needs.

Caller token googleapis download.qt.io tags
maintenance.yml yes yes yes yes (since #172)
release.yml::resolve no no no no
ci.yml no-base fallback no no no yes

The P1

release.yml::resolve ran validate_all.py -- — every tier — in a job with none
of the four. Every release would have stopped in preflight on four capability
failures: before its own promotion gate, and long before the missing promotion
evidence manifest (#157) that is the known blocker.

Nothing noticed because the graph had never run. At tag 0.13.3 the release was
resolve → publish and resolve did not invoke validate_all at all; the "Full
validation sweep" step arrived in 25b2514 on 2026-08-12, ten days after the last
release. Every green historical release run describes a different graph.

The fix keeps the intent

The old comment said a release "gets everything" because an immutable artifact must
not ship carrying an expired external fact. That intent is preserved, not dropped.

The advisory tier now splits by what a check reaches for:

  • CALENDAR — product-fact expiry, runtime-coverage waivers, release ledger vs
    tags, documentation links — reaches no further than the checkout.
  • EXTERNAL — the two SDK pins, and the two GitHub-API checks — does.

--tier release is core + CALENDAR. It passes with no token and no reachable
host
, which is the executable proof that the preflight is satisfiable by the job
that has to run it:

$ env -u GH_TOKEN -u GITHUB_TOKEN … --launch validate_all.py -- --tier release
validate_all (release): OK

resolve also now checks out with fetch-tags: true.

The contract

catalog/validation-tiers.yml declares what each check needs and which job may run
which tier. check_validation_tier_contract.py reads:

  • membership from validate_all itself, so the catalog cannot describe a tier
    that does not exist;
  • requirement from the catalog;
  • grant from the job — harden-runner allow-list, step and job env, checkout inputs.

Discovery is fail-closed both ways: a job running validate_all.py that the
catalog does not declare is a finding, and a declared caller that no longer exists is
a finding too. A closed allowlist that only checks what it was told is how this
problem stayed invisible in the first place.

Mutation Result
release.yml reverts to every tier (the original P1) RED (caught)
maintenance.yml loses its token RED (caught)
maintenance.yml loses the Qt host RED (caught)
maintenance.yml stops fetching tags RED (caught)
ci.yml validate-touched invokes the advisory tier RED (caught)
catalog declares a caller that does not exist RED (caught)

Not done here

The promotion gate is untouched — no role, signature, freshness or exact-SHA binding
is relaxed. #157 remains the blocker after this one, and this PR does not close it.

Verification

  • validate_all — all tiers OK with GH_TOKEN; --tier release OK without one
  • actionlint — clean · zizmor@1.26.1 --persona pedantic --min-severity low — no findings

…them

`--tier scheduled` needs a GitHub token, two external hosts and the tag refs.
That was recorded once, as comments beside maintenance.yml's egress
allow-list, and enforced nowhere. Three jobs invoked the tier; one granted
what it needs.

release.yml ran every tier with no token, no tags and an allow-list omitting
both SDK hosts, so a release stopped in preflight on four capability failures
-- before its own promotion gate, and long before the missing evidence
manifest that is the known blocker. It had never been noticed because the
graph had never run: at 0.13.3 `resolve` did not invoke validate_all at all,
and the step that does landed ten days after the last release.

The advisory tier now splits by what a check reaches for. CALENDAR reaches no
further than the checkout; EXTERNAL needs a token or a named host. `--tier
release` is core plus CALENDAR, so a release still refuses to ship an expired
external fact without a publishing graph reaching the network to find out. It
passes with no token and no reachable host.

catalog/validation-tiers.yml records what each check needs and which job may
run which tier. check_validation_tier_contract.py reads membership from
validate_all, the requirement from the catalog, and the grant from the job.
Discovery is fail-closed both ways: an undeclared caller is a finding, and so
is a declared caller that no longer exists.

Six mutations were each caught, including release.yml reverting to every tier.
@github-actions github-actions Bot added ci Continuous integration and removed ci Continuous integration labels Aug 14, 2026
@rldyourmnd

Copy link
Copy Markdown
Contributor Author

Note on a flaky check, so the record is accurate.

Between writing this PR's verification section and pushing, one local full-tier run
reported:

[FAIL] transitive-action-pins

It is not a regression from this branch — scripts/check_transitive_action_pins.py
is untouched here — and it is not reproducible: the same command passes now with
rate_limit.core.remaining = 4999/5000.

The cause is the defect tracked in #177, which maintenance.yml filed automatically on
its first successful run: the check fires ~44 sequential GitHub API requests with no
pacing and no retry, and reports a secondary rate limit as unreachable: HTTP Error 403.
It cannot tell throttling from a real permission failure.

That is scheduled-tier only, so it does not gate this PR, and it is the next piece of
work rather than something to paper over here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant