Skip to content

fix(runner): add mutex to testAndSet to prevent race condition#2439

Open
usernametooshort wants to merge 49 commits intoprojectdiscovery:mainfrom
usernametooshort:fix/testAndSet-race
Open

fix(runner): add mutex to testAndSet to prevent race condition#2439
usernametooshort wants to merge 49 commits intoprojectdiscovery:mainfrom
usernametooshort:fix/testAndSet-race

Conversation

@usernametooshort
Copy link

Summary

Fixes a race condition in testAndSet where the read-check-then-write sequence is not atomic.

Problem

testAndSet performs:

  1. seen(k) - check if key exists
  2. setSeen(k) - set the key

Without synchronization, two concurrent goroutines can both pass the seen() check before either calls setSeen(), causing duplicate processing of the same target.

Impact

This race occurs in processTargets() where multiple goroutines call testAndSet() for discovered TLS SubjectAN/SubjectCN values (lines 1692, 1697, 1706, 1746, 1751).

When the race triggers:

  • Same target gets processed multiple times
  • Duplicate output entries
  • Wasted HTTP requests and resources

Fix

Add a sync.Mutex to the Runner struct to serialize testAndSet operations.

ayuxsec and others added 30 commits December 26, 2025 17:37
- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`).
- Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview.
- Add markdown escaping helper.
- Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout.
- Adjust option parsing and related logic to include markdown handling.
- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation.
- Remove body‑preview block from Markdown rows.
- Add guard in runner to write the header only once per run.
…d` flag

* Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable.
* Adjust file‑opening logic to add the “.md” suffix only when needed.
* Update related conditionals to respect the new combined flag handling.
* Ensure markdown output is correctly written without creating “.md.md” files.
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.

Supported databases:
- MongoDB
- PostgreSQL
- MySQL

Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)

New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data

Closes projectdiscovery#1973
Closes projectdiscovery#2360
Closes projectdiscovery#2361
Closes projectdiscovery#2362
- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names
- Fix SQL injection in mysql.go using custom quoteIdentifier function
- Fix race condition in writer.go by checking closed state before channel send
- Add missing idx_url index in mysql.go for parity with PostgreSQL schema
- Include RawHeaders in omitRaw check for consistency
…2383)

Bumps the modules group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.19` | `1.2.20` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.33` | `0.1.34` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.4` | `1.3.5` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.106` | `0.0.107` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.64` | `0.2.65` |


Updates `github.com/projectdiscovery/cdncheck` from 1.2.19 to 1.2.20
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.19...v1.2.20)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.33 to 0.1.34
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](projectdiscovery/networkpolicy@v0.1.33...v0.1.34)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.4 to 1.3.5
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](projectdiscovery/retryablehttp-go@v1.3.4...v1.3.5)

Updates `github.com/projectdiscovery/useragent` from 0.0.106 to 0.0.107
- [Release notes](https://github.com/projectdiscovery/useragent/releases)
- [Commits](projectdiscovery/useragent@v0.0.106...v0.0.107)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.64 to 0.2.65
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.64...v0.2.65)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
  dependency-version: 0.0.107
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.65
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.20` | `1.2.21` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.12` | `0.8.13` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.3` | `0.5.4` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.67` | `1.1.68` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.99` | `0.0.100` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.5` | `1.3.6` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.65` | `0.2.66` |


Updates `github.com/projectdiscovery/cdncheck` from 1.2.20 to 1.2.21
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.20...v1.2.21)

Updates `github.com/projectdiscovery/dsl` from 0.8.12 to 0.8.13
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](projectdiscovery/dsl@v0.8.12...v0.8.13)

Updates `github.com/projectdiscovery/fastdialer` from 0.5.3 to 0.5.4
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](projectdiscovery/fastdialer@v0.5.3...v0.5.4)

Updates `github.com/projectdiscovery/gologger` from 1.1.67 to 1.1.68
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](projectdiscovery/gologger@v1.1.67...v1.1.68)

Updates `github.com/projectdiscovery/hmap` from 0.0.99 to 0.0.100
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](projectdiscovery/hmap@v0.0.99...v0.0.100)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.5 to 1.3.6
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](projectdiscovery/retryablehttp-go@v1.3.5...v1.3.6)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.65 to 0.2.66
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.65...v0.2.66)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
  dependency-version: 0.8.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
  dependency-version: 1.1.68
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
  dependency-version: 0.0.100
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.66
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/dev/modules-60805faa73

chore(deps): bump the modules group with 7 updates
…ost-targets

fix: prevent data loss on interrupt by implementing graceful shutdown
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.49.0 to 0.50.0.
- [Commits](golang/net@v0.49.0...v0.50.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/refraction-networking/utls](https://github.com/refraction-networking/utls) from 1.7.1 to 1.8.2.
- [Release notes](https://github.com/refraction-networking/utls/releases)
- [Commits](refraction-networking/utls@v1.7.1...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/refraction-networking/utls
  dependency-version: 1.8.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/dev/golang.org/x/net-0.50.0

chore(deps): bump golang.org/x/net from 0.49.0 to 0.50.0
…abot/go_modules/github.com/refraction-networking/utls-1.8.2

chore(deps): bump github.com/refraction-networking/utls from 1.7.1 to 1.8.2
Bumps the modules group with 2 updates: [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) and [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo).


Updates `github.com/projectdiscovery/cdncheck` from 1.2.21 to 1.2.22
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.21...v1.2.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.66 to 0.2.67
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.66...v0.2.67)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.67
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
…e/database-output

feat: add database output support for storing scan results
dogancanbakir and others added 19 commits March 3, 2026 21:03
- Add Deprecated comment to OutputFilterErrorPage field
- Coerce PageType to string for safe type assertion
…it-page-classifier

Replace page classifier with dit, add -fpt flag
Bumps [filippo.io/edwards25519](https://github.com/FiloSottile/edwards25519) from 1.1.0 to 1.1.1.
- [Commits](FiloSottile/edwards25519@v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: filippo.io/edwards25519
  dependency-version: 1.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.50.0 to 0.51.0.
- [Commits](golang/net@v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/filippo.io/edwards25519-1.1.1

chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1
…abot/go_modules/dev/golang.org/x/net-0.51.0

chore(deps): bump golang.org/x/net from 0.50.0 to 0.51.0
…abot/go_modules/dev/modules-6b7541c889

chore(deps): bump the modules group with 2 updates
…ix_resolver_parsing

fix resolver parsing
testAndSet performs a non-atomic read-then-write sequence:
1. seen(k) - check if key exists
2. setSeen(k) - set the key

Without synchronization, two concurrent goroutines can both pass the
seen() check before either calls setSeen(), causing duplicate
processing of the same target.

This race can occur in processTargets() where multiple goroutines
call testAndSet() for discovered TLS SubjectAN/SubjectCN values.

The fix adds a sync.Mutex to serialize testAndSet operations.
@auto-assign auto-assign bot requested a review from dogancanbakir March 6, 2026 08:33
@neo-by-projectdiscovery-dev
Copy link

neo-by-projectdiscovery-dev bot commented Mar 6, 2026

Neo - PR Security Review

No security issues found

Highlights

  • Adds mutex synchronization to testAndSet() to prevent duplicate processing of TLS certificate domains
  • Race condition affected deduplication logic for SubjectAN, SubjectCN, and CSP-discovered targets
  • Mutex implementation is correct: lock at entry, defer unlock, protects entire check-then-set sequence
Hardening Notes
  • The race condition causes duplicate HTTP requests and output entries when multiple goroutines process the same discovered target simultaneously, but this is a reliability issue with no exploitable security impact.
  • httpx is a local reconnaissance tool run by security researchers, not a network service exposed to external attackers - the threat model doesn't include adversarial control over concurrent execution timing.
  • Duplicate processing cannot be weaponized for DoS: the race window is microseconds, worst-case impact is 2x requests instead of 1x for a brief moment, and rate limiting/resource controls are enforced elsewhere in the codebase.
  • No authentication, authorization, injection, or information disclosure vulnerabilities are introduced or fixed by this change - the deduplication logic is purely operational.

Comment @pdneo help for available commands. · Open in Neo

@coderabbitai
Copy link

coderabbitai bot commented Mar 6, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: adf11cd7-8b2a-4692-808c-817f0071b36f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Member

@Mzack9999 Mzack9999 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR should point to the dev branch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants