Skip to content

PAYMENTS-11727 Proof: the fork delivery specs fail on 0.8.3 as released - #43

Draft
WillemHoman wants to merge 5 commits into
mainfrom
PAYMENTS-11727-proof-0-8-3-metric-loss
Draft

PAYMENTS-11727 Proof: the fork delivery specs fail on 0.8.3 as released#43
WillemHoman wants to merge 5 commits into
mainfrom
PAYMENTS-11727-proof-0-8-3-metric-loss

Conversation

@WillemHoman

@WillemHoman WillemHoman commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Jira: PAYMENTS-11727

Not for merge. This exists to demonstrate that the regression specs added in
#45 would have caught the metric loss that 0.8.3 shipped.

What? Why?

main is 0.8.3, the release that dropped webhook metrics in bigpay. This branch adds nothing but the specs from #45,
cherry-picked, and the CI job that runs them.

The result is the completeness property failing and the overhead property passing, which is the signature of that
incident: job time stayed flat while the observations silently went nowhere.

1) metric delivery from Resque forked children completeness delivers one observation for every job that pushed one
   Failure/Error: expect(exporter.count_for(ForkDeliveryProbeJob::METRIC_NAME)).to eq JOB_COUNT

     expected: 100
          got: 0

2 examples, 1 failure

Zero of 100, not a partial loss. Production saw roughly 8% survive rather than 0% because a real webhook job does a
little work after its push and sometimes wins the race; the probe job pushes as its final statement, so it never does.

Commits

commit
209dd85 resque and sinatra >= 4.0 dev dependencies (cherry-pick)
48d0257 the completeness and overhead specs (cherry-pick)
1db4208 CI job on ruby 3.4 (cherry-pick)
db5261d monotonic clock so specs load on ruby 4.0 (cherry-pick)
3505a25 one-line adaptation, see below

The adaptation is the only deviation from #45. The spec's setup calls client.reset_after_fork! to make the run
independent of spec ordering, and that method is part of the fix, so it does not exist here. Replaced with a direct
@queue.clear, which is the only part of it the setup needs. Nothing in either assertion changed.

How was it tested?

redis-server --port 6399 --save '' --appendonly no --daemonize yes
FORK_INTEGRATION=1 REDIS_URL=redis://127.0.0.1:6399/15 bundle exec rspec spec/integration

ruby-3_4-rspec_fork_integration is expected to be red on this branch. That is the point of it.

See also #44, which does the same for the PAYMENTS-11567
latency regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant