ja*: Fix preserve logic to check for any header in fingerprint group #12811
Conversation
bneradt
added
ja3_fingerprint
ja4_fingerprint
Problem: When --preserve was enabled and a request passed through multiple proxies, each header was checked individually. This could result in mismatched fingerprint data - for example, x-ja3-raw being added by a downstream proxy while x-ja3-sig was preserved from an upstream proxy. Solution: The JA3 and JA4 fingerprint plugins now check if ANY header in a fingerprint group exists before adding headers. If any header in the group exists, ALL headers in that group are skipped. Changes: - ja3_fingerprint: Added group-level checks for JA3 headers - ja4_fingerprint: Added --preserve option and group-level check for JA4 headers - Updated tests to verify group-level preserve behavior
bneradt
force-pushed
the
ja_fix_preserve_across_headers_asf
branch
from
49e7f6f to
a7a1dd3
Compare
bneradt
changed the title
There was a problem hiding this comment.
Pull request overview
Updates the JA3 and JA4 fingerprint plugins’ --preserve behavior to operate at the “fingerprint group” level (skip adding all related headers if any already exist), preventing mixed upstream/downstream fingerprint header sets.
Changes:
- JA3 plugin: add group-level “any JA3 header exists” check before adding JA3 headers.
- JA4 plugin: add
--preserveoption plus group-level “any JA4 header exists” check before adding JA4 headers. - Tests/gold files updated to validate group-level preserve behavior for both plugins.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| plugins/ja3_fingerprint/ja3_fingerprint.cc | Implements group-level preserve logic for JA3 header set. |
| plugins/experimental/ja4_fingerprint/plugin.cc | Adds --preserve option parsing and group-level preserve logic for JA4 headers. |
| tests/gold_tests/pluginTest/ja4_fingerprint/ja4_fingerprint.test.py | Adds test coverage for JA4 --preserve behavior. |
| tests/gold_tests/pluginTest/ja4_fingerprint/ja4_fingerprint.replay.yaml | Extends replay scenarios to validate JA4 preserve cases. |
| tests/gold_tests/pluginTest/ja3_fingerprint/ja3_fingerprint.test.py | Adjusts JA3 test assertions for group-level preserve behavior. |
| tests/gold_tests/pluginTest/ja3_fingerprint/ja3_fingerprint_global.replay.yaml | Updates expected JA3 header presence/absence under preserve. |
| tests/gold_tests/pluginTest/ja3_fingerprint/modify-incoming-proxy.gold | Updates expected internal proxy header output for modify-incoming + preserve. |
| tests/gold_tests/pluginTest/ja3_fingerprint/modify-sent-proxy-remap.gold | Updates expected internal proxy header output for remap configuration. |
| tests/gold_tests/pluginTest/ja3_fingerprint/modify-sent-proxy-global.gold | New gold file for global-plugin internal proxy header output expectations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - [ uuid, no-existing-headers ] | ||
|
|
There was a problem hiding this comment.
The replay now requires a uuid header (e.g. uuid: no-existing-headers), but the ja4_fingerprint tests use curl and don’t set any uuid header. This will likely cause Proxy Verifier replay matching/validation failures. Either remove the uuid requirements from the replay or update the curl commands to send matching uuid values for each session.
| self._server.Streams.All += Testers.ExcludesExpression( | ||
| "x-ja3-sig:.*http2-request", | ||
| "Verify no JA3-Sig was added when other JA3 headers existed.", | ||
| reflags=re.IGNORECASE | re.DOTALL) |
There was a problem hiding this comment.
This ExcludesExpression regex can match across both requests in the server log because it uses re.DOTALL and looks for x-ja3-sig before http2-request. Since the first request legitimately contains x-ja3-sig, this check can produce false failures/flakiness depending on log ordering. Prefer relying on the replay.yaml per-transaction assertion (x-ja3-sig is absent for the http2 transaction) or scope the regex to the http2-request block (e.g., match http2-request first, then ensure x-ja3-sig does not appear within that block).
| self._server.Streams.All += Testers.ExcludesExpression( | |
| "x-ja3-sig:.*http2-request", | |
| "Verify no JA3-Sig was added when other JA3 headers existed.", | |
| reflags=re.IGNORECASE | re.DOTALL) |
| tr.MakeCurlCommand('-k -v "https://localhost:{0}/resource"'.format(params['port_one']), ts=params['ts']) | ||
| client.ReturnCode = 0 | ||
| client.Streams.stdout += Testers.ContainsExpression(r'Yay!', 'First request should succeed.') |
There was a problem hiding this comment.
The replay file now requires a uuid header (e.g. uuid: no-existing-headers), but the curl request built here doesn’t send one. Proxy Verifier will likely fail to match/validate the transaction. Either add a matching -H "uuid: ..." to each curl request (for /resource, /resource-with-headers, /resource-via-only) or remove the uuid requirements from the replay YAML.
| TSMBuffer bufp; | ||
| TSMLoc hdr_loc; | ||
| if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &bufp, &hdr_loc)) { | ||
| if (TS_SUCCESS != TSHttpTxnClientReqGet(txnp, &bufp, &hdr_loc)) { |
There was a problem hiding this comment.
Would it make more sense to say TS_ERROR == ? I think it might be easier to read the code that way
2 participants
Problem:
When --preserve was enabled and a request passed through multiple proxies, each header was checked individually. This could result in mismatched fingerprint data - for example, x-ja3-raw being added by a downstream proxy while x-ja3-sig was preserved from an upstream proxy.
Solution:
The JA3 and JA4 fingerprint plugins now check if ANY header in a fingerprint group exists before adding headers. If any header in the group exists, ALL headers in that group are skipped.
Changes: