fix(runner): add mutex to testAndSet to prevent race condition#2439
fix(runner): add mutex to testAndSet to prevent race condition#2439usernametooshort wants to merge 49 commits intoprojectdiscovery:mainfrom
Conversation
- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`). - Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview. - Add markdown escaping helper. - Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout. - Adjust option parsing and related logic to include markdown handling.
- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation. - Remove body‑preview block from Markdown rows. - Add guard in runner to write the header only once per run.
…d` flag * Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable. * Adjust file‑opening logic to add the “.md” suffix only when needed. * Update related conditionals to respect the new combined flag handling. * Ensure markdown output is correctly written without creating “.md.md” files.
Add support for storing httpx scan results directly to databases with both CLI flags and YAML config file options. Supported databases: - MongoDB - PostgreSQL - MySQL Features: - Batched writes for performance (configurable batch size) - Auto-flush with configurable interval - Individual columns for each Result field (not JSON blob) - Support for environment variable HTTPX_DB_CONNECTION_STRING - Option to omit raw request/response data (-rdbor) New CLI flags under OUTPUT group: - -rdb, -result-db: enable database storage - -rdbc, -result-db-config: path to YAML config file - -rdbt, -result-db-type: database type (mongodb, postgres, mysql) - -rdbcs, -result-db-conn: connection string - -rdbn, -result-db-name: database name (default: httpx) - -rdbtb, -result-db-table: table/collection name (default: results) - -rdbbs, -result-db-batch-size: batch size (default: 100) - -rdbor, -result-db-omit-raw: omit raw data Closes projectdiscovery#1973 Closes projectdiscovery#2360 Closes projectdiscovery#2361 Closes projectdiscovery#2362
- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names - Fix SQL injection in mysql.go using custom quoteIdentifier function - Fix race condition in writer.go by checking closed state before channel send - Add missing idx_url index in mysql.go for parity with PostgreSQL schema - Include RawHeaders in omitRaw check for consistency
…2383) Bumps the modules group with 5 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.19` | `1.2.20` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.33` | `0.1.34` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.4` | `1.3.5` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.106` | `0.0.107` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.64` | `0.2.65` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.19 to 1.2.20 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.19...v1.2.20) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.33 to 0.1.34 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](projectdiscovery/networkpolicy@v0.1.33...v0.1.34) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.4 to 1.3.5 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.3.4...v1.3.5) Updates `github.com/projectdiscovery/useragent` from 0.0.106 to 0.0.107 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](projectdiscovery/useragent@v0.0.106...v0.0.107) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.64 to 0.2.65 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.64...v0.2.65) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.20 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.34 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.3.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.107 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.65 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the modules group with 7 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.20` | `1.2.21` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.12` | `0.8.13` | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.3` | `0.5.4` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.67` | `1.1.68` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.99` | `0.0.100` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.5` | `1.3.6` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.65` | `0.2.66` | Updates `github.com/projectdiscovery/cdncheck` from 1.2.20 to 1.2.21 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.20...v1.2.21) Updates `github.com/projectdiscovery/dsl` from 0.8.12 to 0.8.13 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](projectdiscovery/dsl@v0.8.12...v0.8.13) Updates `github.com/projectdiscovery/fastdialer` from 0.5.3 to 0.5.4 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](projectdiscovery/fastdialer@v0.5.3...v0.5.4) Updates `github.com/projectdiscovery/gologger` from 1.1.67 to 1.1.68 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](projectdiscovery/gologger@v1.1.67...v1.1.68) Updates `github.com/projectdiscovery/hmap` from 0.0.99 to 0.0.100 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](projectdiscovery/hmap@v0.0.99...v0.0.100) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.5 to 1.3.6 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](projectdiscovery/retryablehttp-go@v1.3.5...v1.3.6) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.65 to 0.2.66 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.65...v0.2.66) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.21 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.13 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.5.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.68 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.100 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.3.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.66 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/dev/modules-60805faa73 chore(deps): bump the modules group with 7 updates
…ost-targets fix: prevent data loss on interrupt by implementing graceful shutdown
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.49.0 to 0.50.0. - [Commits](golang/net@v0.49.0...v0.50.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.50.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/refraction-networking/utls](https://github.com/refraction-networking/utls) from 1.7.1 to 1.8.2. - [Release notes](https://github.com/refraction-networking/utls/releases) - [Commits](refraction-networking/utls@v1.7.1...v1.8.2) --- updated-dependencies: - dependency-name: github.com/refraction-networking/utls dependency-version: 1.8.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/dev/golang.org/x/net-0.50.0 chore(deps): bump golang.org/x/net from 0.49.0 to 0.50.0
…abot/go_modules/github.com/refraction-networking/utls-1.8.2 chore(deps): bump github.com/refraction-networking/utls from 1.7.1 to 1.8.2
Bumps the modules group with 2 updates: [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) and [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo). Updates `github.com/projectdiscovery/cdncheck` from 1.2.21 to 1.2.22 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](projectdiscovery/cdncheck@v1.2.21...v1.2.22) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.66 to 0.2.67 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](projectdiscovery/wappalyzergo@v0.2.66...v0.2.67) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.22 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.67 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com>
…e/database-output feat: add database output support for storing scan results
- Add Deprecated comment to OutputFilterErrorPage field - Coerce PageType to string for safe type assertion
…it-page-classifier Replace page classifier with dit, add -fpt flag
Bumps [filippo.io/edwards25519](https://github.com/FiloSottile/edwards25519) from 1.1.0 to 1.1.1. - [Commits](FiloSottile/edwards25519@v1.1.0...v1.1.1) --- updated-dependencies: - dependency-name: filippo.io/edwards25519 dependency-version: 1.1.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.50.0 to 0.51.0. - [Commits](golang/net@v0.50.0...v0.51.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.51.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…abot/go_modules/filippo.io/edwards25519-1.1.1 chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1
…abot/go_modules/dev/golang.org/x/net-0.51.0 chore(deps): bump golang.org/x/net from 0.50.0 to 0.51.0
…abot/go_modules/dev/modules-6b7541c889 chore(deps): bump the modules group with 2 updates
…ix_resolver_parsing fix resolver parsing
…put-support Add Markdown output support
testAndSet performs a non-atomic read-then-write sequence: 1. seen(k) - check if key exists 2. setSeen(k) - set the key Without synchronization, two concurrent goroutines can both pass the seen() check before either calls setSeen(), causing duplicate processing of the same target. This race can occur in processTargets() where multiple goroutines call testAndSet() for discovered TLS SubjectAN/SubjectCN values. The fix adds a sync.Mutex to serialize testAndSet operations.
Neo - PR Security ReviewNo security issues found Highlights
Hardening Notes
Comment |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Mzack9999
left a comment
There was a problem hiding this comment.
The PR should point to the dev branch
Summary
Fixes a race condition in
testAndSetwhere the read-check-then-write sequence is not atomic.Problem
testAndSetperforms:seen(k)- check if key existssetSeen(k)- set the keyWithout synchronization, two concurrent goroutines can both pass the
seen()check before either callssetSeen(), causing duplicate processing of the same target.Impact
This race occurs in
processTargets()where multiple goroutines calltestAndSet()for discovered TLS SubjectAN/SubjectCN values (lines 1692, 1697, 1706, 1746, 1751).When the race triggers:
Fix
Add a
sync.Mutexto the Runner struct to serializetestAndSetoperations.