test(licensing): make the outage-is-not-a-cancellation test actually test that - #78
Conversation
…test that The test raised CloudSessionError(..., status=503, transient=True), but that class takes only status -- transient belongs to CloudFeatureError. So the construction raised TypeError, the caller's broad except Exception swallowed it, and getattr(exc, "status", None) was None rather than 503. The assertion therefore passed while never exercising a transport failure at all, and could not have caught the regression it exists to prevent: loosening the 402 gate so any error clears a paying customer's access. Verified by mutation -- with the gate changed to any-truthy-status the test now fails with "an outage revoked a paying customer"; before this change it passed even with that mutation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Coding-Dev-Tools has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.
Coding-Dev-Tools
left a comment
There was a problem hiding this comment.
Review — `test(licensing): make the outage-is-not-a-cancellation test actually test that`
✅ LGTM
Good catch on the `CloudSessionError`/`CloudFeatureError` conflation. The mutation-test evidence is exactly the right proof shape: before the fix the test passed even with the gate loosened to any-truthy-status, which is the regression this assertion was meant to prevent. A test that cannot fail on the regression it guards against is worse than no test — it creates the appearance of coverage while providing none.
The comment recording why `transient` is wrong here is worth its weight: the next person to touch this will not make the same mistake, and the reason is specific enough to be actionable (`transient belongs to CloudFeatureError`, not a generic note).
CI green across 3.10/3.11/3.12 + core floor + browser smoke. +6/-1 is proportional to the fix.
One minor note for the future: consider a lint rule or type narrowing that flags `CloudSessionError(..., transient=...)` at static-analysis time rather than relying on runtime `TypeError` discovery. Not blocking — the comment and mutation test close the loop for now.
🔍 Pre-PR Code Review — ✅ LGTM (approve blocked: own PR)Reviewer: Pre-PR Code Analyzer (automated) Test QualityCritical fix: the test was previously a no-op (TypeError from invalid Verdict: LGTM — ready to mergeMinimal, surgical fix that transforms a false-assurance test into a real guard. CI green. |
The defect
test_a_transport_failure_is_not_mistaken_for_a_billing_denialraised:CloudSessionError.__init__accepts onlymessageandstatus—transientbelongs toCloudFeatureError. So the construction raisedTypeError,_fetch_authoritative_entitlement's broadexcept Exceptionswallowed it, andgetattr(exc, "status", None)returnedNoneinstead of503.The assertion passed while never exercising a transport failure at all. It was proving "a
TypeErrordoesn't revoke access" — not "a 503 doesn't".This is mine: I introduced it earlier in this session while adding the 402 billing-denial handling.
Why it matters
That test guards a revenue-visible behaviour: only a
402may clear a paying customer's cached access; an outage must not. The gate is:If that were ever loosened to
if getattr(exc, "status", None):, a 503 would revoke a paying customer mid-outage. The test existed to catch exactly that — and could not, becauseTypeErrorcarries nostatusattribute at all, so the mutation was invisible to it.Verification
Mutation-tested both ways:
AssertionError: an outage revoked a paying customer, withcloud_access_active: False.Full offline gate on this branch: 1,397 passed / 0 failed (1,411 collected, 14 skipped),
ruff check .clean.One-line change plus a comment recording why, so the conflation is not reintroduced.
🤖 Generated with Claude Code