Require a valid proof of work for a token, not just a good score - #29
Merged
Conversation
…ood score
A bare `curl` was issued a valid token. Ten times out of ten, on all three
servers:
$ curl -X POST /api/verify -d '{"siteKey":"demo-site-key","signals":{}}'
tokens minted: 10 / 10
No browser, no proof of work, a curl User-Agent, empty signals. Every token
passed siteverify. Confirmed against v1.21.0 too, so this predates the
siteverify work rather than being introduced by it.
The detectors were never the problem — they all fired. A forged solution
referencing a challenge that was never issued produced seven detections
(`PoW verification failed`, `User-Agent indicates bot/automation tool`, `Zero
mouse, touch, or keyboard events recorded`, `No mouse movement detected before
click`, ...) and still scored 0.4003, under the 0.5 success threshold.
The aggregation discarded them. calculateFinalScore is a weighted sum, so a
category contributes at most its own weight no matter how conclusive it is. The
bot category is weighted 0.13; every PoW failure firing at once reaches 0.9986
under noisy-OR, which is 0.1298 of the final score, 0.37 short of the threshold
and always will be. dispositiveFloor's own doc comment diagnoses exactly this,
but the mark had only ever been applied to navigator.webdriver and the
ChromeDriver/Puppeteer globals — never to the proof of work, whose own code
comment reads "hard fail".
Two changes, deliberately both:
- A proof of work is now a precondition, checked outside the score:
`success = score < 0.5 && hostnameAllowed && powSatisfied`. A score
threshold answers "how suspicious is this visitor", which is the wrong
question to ask of someone who never completed the challenge. Gating here
means no future reweighting can reopen this, and it holds even if the
dispositive floor is lowered or removed.
- The three PoW failures (absent, unverifiable, nonce-unbound) are marked
Dispositive, so the reported score is honest too — an integrator
risk-banding on the score now sees 0.9 rather than 0.4.
Refusals name the failed precondition (`pow_not_satisfied`,
`hostname_not_allowed`) via a new `reason` field, because a caller whose score is
comfortably under the threshold otherwise has no way to tell why no token came
back. Go's VerificationResult gained the field; Node and Python already reported
one for the hostname case.
No effect on legitimate traffic. The widget already solves a challenge on every
path and aborts rather than submit without one, so nothing about its flow
changes. Measured on the bench harness, which completes the real handshake:
human false-positive rate 0.00% (0/126), human median score unchanged at 0.097,
agent TPR 97.33%, gate exits 0.
The E2E suite posted synthetic signals with no PoW, so its eight "gets low score"
assertions floored at 0.9. Those now complete the real handshake via a
makeVerifiedRequest helper that reuses the bench solver rather than carrying a
second copy of the scheme. Detection-only assertions still use makeRequest: they
ask whether a payload trips a detector, which does not depend on the handshake.
104 -> 107 assertions, all passing on Node; the five Go and thirteen Python
failures are the documented pre-existing divergences, verified identical against
the pre-change build.
Regression tests in Go and Python cover a withheld token, the dispositive floor,
a forged solution, the reported reason, and — pinning why the gate must exist at
all — that the bot category alone cannot reach the threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A bare
curlwas issued a valid token — ten times out of ten, on all three servers:No browser, no proof of work, a curl User-Agent, empty signals. Every token passed
siteverify. Reproduced against v1.21.0, so it predates the siteverify work.Why the detectors did not stop it
They fired. A forged solution referencing a challenge that was never issued produced seven detections and still scored 0.4003, under the 0.5 success threshold.
calculateFinalScoreis a weighted sum, so a category contributes at most its own weight however conclusive it is.botis weighted 0.13; every PoW failure firing at once reaches 0.9986 under noisy-OR, which is 0.1298 of the final score — 0.37 short, permanently.dispositiveFloor's own doc comment diagnoses this exact failure, but the mark had only been applied tonavigator.webdriverand the ChromeDriver/Puppeteer globals, never to the proof of work — whose own code comment reads "hard fail".The fix
success = score < 0.5 && hostnameAllowed && powSatisfied. Gating here means no future reweighting can reopen it, and it holds even if the dispositive floor is lowered.Dispositive(absent, unverifiable, nonce-unbound), so the reported score is honest too — 0.9 rather than 0.4.pow_not_satisfied/hostname_not_allowed) via areasonfield.No effect on legitimate traffic
The widget already solves a challenge on every path and aborts rather than submit without one. Measured on the bench harness, which completes the real handshake:
curlbulk-mintBench gate exits 0.
Tests
E2E suite's eight "gets low score" assertions posted no PoW and so floored at 0.9; they now complete the real handshake via a helper that reuses the bench solver rather than duplicating the scheme. Detection-only assertions still skip it — whether a payload trips a detector does not depend on the handshake. 104 → 107 assertions, all passing on Node.
Go and Python regression tests cover a withheld token, the dispositive floor, a forged solution, the reported reason, and — pinning why the gate must exist — that the
botcategory alone cannot reach the threshold.The 5 Go / 13 Python E2E failures are the documented pre-existing divergences, verified identical against the pre-change build.
Breaking
Anyone calling
/api/verifydirectly from a custom client must now complete the handshake. There is no flag to restore the old behaviour — it was a bypass, not a feature. See Upgrading to 1.23.0.