Skip to content

release httpx v1.9.0#2435

Open
dogancanbakir wants to merge 48 commits intomainfrom
dev
Open

release httpx v1.9.0#2435
dogancanbakir wants to merge 48 commits intomainfrom
dev

Conversation

@dogancanbakir
Copy link
Member

closes #2434

ayuxsec and others added 30 commits December 26, 2025 17:37
- Introduce `MarkDownOutput` flag and CLI option (`-markdown` / `-md`).
- Implement `Result.MarkdownOutput` to generate a formatted Markdown table with optional title and CDN columns, and a response body preview.
- Add markdown escaping helper.
- Update runner to create `.md` output file, handle markdown generation per result, and write to file or stdout.
- Adjust option parsing and related logic to include markdown handling.
- Split Markdown generation into `MarkdownHeader` and `MarkdownRow` for clearer separation.
- Remove body‑preview block from Markdown rows.
- Add guard in runner to write the header only once per run.
…d` flag

* Consolidate output‑type checks into a single `jsonOrCsvOrMD` variable.
* Adjust file‑opening logic to add the “.md” suffix only when needed.
* Update related conditionals to respect the new combined flag handling.
* Ensure markdown output is correctly written without creating “.md.md” files.
Add support for storing httpx scan results directly to databases with
both CLI flags and YAML config file options.

Supported databases:
- MongoDB
- PostgreSQL
- MySQL

Features:
- Batched writes for performance (configurable batch size)
- Auto-flush with configurable interval
- Individual columns for each Result field (not JSON blob)
- Support for environment variable HTTPX_DB_CONNECTION_STRING
- Option to omit raw request/response data (-rdbor)

New CLI flags under OUTPUT group:
- -rdb, -result-db: enable database storage
- -rdbc, -result-db-config: path to YAML config file
- -rdbt, -result-db-type: database type (mongodb, postgres, mysql)
- -rdbcs, -result-db-conn: connection string
- -rdbn, -result-db-name: database name (default: httpx)
- -rdbtb, -result-db-table: table/collection name (default: results)
- -rdbbs, -result-db-batch-size: batch size (default: 100)
- -rdbor, -result-db-omit-raw: omit raw data

Closes #1973
Closes #2360
Closes #2361
Closes #2362
- Fix SQL injection in postgres.go using pq.QuoteIdentifier for table/index names
- Fix SQL injection in mysql.go using custom quoteIdentifier function
- Fix race condition in writer.go by checking closed state before channel send
- Add missing idx_url index in mysql.go for parity with PostgreSQL schema
- Include RawHeaders in omitRaw check for consistency
Bumps the modules group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.19` | `1.2.20` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.33` | `0.1.34` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.4` | `1.3.5` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.106` | `0.0.107` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.64` | `0.2.65` |


Updates `github.com/projectdiscovery/cdncheck` from 1.2.19 to 1.2.20
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.19...v1.2.20)

Updates `github.com/projectdiscovery/networkpolicy` from 0.1.33 to 0.1.34
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](projectdiscovery/networkpolicy@v0.1.33...v0.1.34)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.4 to 1.3.5
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](projectdiscovery/retryablehttp-go@v1.3.4...v1.3.5)

Updates `github.com/projectdiscovery/useragent` from 0.0.106 to 0.0.107
- [Release notes](https://github.com/projectdiscovery/useragent/releases)
- [Commits](projectdiscovery/useragent@v0.0.106...v0.0.107)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.64 to 0.2.65
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.64...v0.2.65)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.20
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
  dependency-version: 0.1.34
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
  dependency-version: 0.0.107
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.65
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the modules group with 7 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.20` | `1.2.21` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.12` | `0.8.13` |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.5.3` | `0.5.4` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.67` | `1.1.68` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.99` | `0.0.100` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.3.5` | `1.3.6` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.65` | `0.2.66` |


Updates `github.com/projectdiscovery/cdncheck` from 1.2.20 to 1.2.21
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.20...v1.2.21)

Updates `github.com/projectdiscovery/dsl` from 0.8.12 to 0.8.13
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](projectdiscovery/dsl@v0.8.12...v0.8.13)

Updates `github.com/projectdiscovery/fastdialer` from 0.5.3 to 0.5.4
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](projectdiscovery/fastdialer@v0.5.3...v0.5.4)

Updates `github.com/projectdiscovery/gologger` from 1.1.67 to 1.1.68
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](projectdiscovery/gologger@v1.1.67...v1.1.68)

Updates `github.com/projectdiscovery/hmap` from 0.0.99 to 0.0.100
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](projectdiscovery/hmap@v0.0.99...v0.0.100)

Updates `github.com/projectdiscovery/retryablehttp-go` from 1.3.5 to 1.3.6
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](projectdiscovery/retryablehttp-go@v1.3.5...v1.3.6)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.65 to 0.2.66
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.65...v0.2.66)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
  dependency-version: 0.8.13
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/fastdialer
  dependency-version: 0.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
  dependency-version: 1.1.68
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
  dependency-version: 0.0.100
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
  dependency-version: 1.3.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.66
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
…dev/modules-60805faa73

chore(deps): bump the modules group with 7 updates
fix: prevent data loss on interrupt by implementing graceful shutdown
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.49.0 to 0.50.0.
- [Commits](golang/net@v0.49.0...v0.50.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.50.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/refraction-networking/utls](https://github.com/refraction-networking/utls) from 1.7.1 to 1.8.2.
- [Release notes](https://github.com/refraction-networking/utls/releases)
- [Commits](refraction-networking/utls@v1.7.1...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/refraction-networking/utls
  dependency-version: 1.8.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
…dev/golang.org/x/net-0.50.0

chore(deps): bump golang.org/x/net from 0.49.0 to 0.50.0
…github.com/refraction-networking/utls-1.8.2

chore(deps): bump github.com/refraction-networking/utls from 1.7.1 to 1.8.2
Bumps the modules group with 2 updates: [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) and [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo).


Updates `github.com/projectdiscovery/cdncheck` from 1.2.21 to 1.2.22
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Commits](projectdiscovery/cdncheck@v1.2.21...v1.2.22)

Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.66 to 0.2.67
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](projectdiscovery/wappalyzergo@v0.2.66...v0.2.67)

---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/cdncheck
  dependency-version: 1.2.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
  dependency-version: 0.2.67
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: modules
...

Signed-off-by: dependabot[bot] <support@github.com>
feat: add database output support for storing scan results
dogancanbakir and others added 16 commits March 3, 2026 21:03
- Add Deprecated comment to OutputFilterErrorPage field
- Coerce PageType to string for safe type assertion
Replace page classifier with dit, add -fpt flag
Bumps [filippo.io/edwards25519](https://github.com/FiloSottile/edwards25519) from 1.1.0 to 1.1.1.
- [Commits](FiloSottile/edwards25519@v1.1.0...v1.1.1)

---
updated-dependencies:
- dependency-name: filippo.io/edwards25519
  dependency-version: 1.1.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.50.0 to 0.51.0.
- [Commits](golang/net@v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…filippo.io/edwards25519-1.1.1

chore(deps): bump filippo.io/edwards25519 from 1.1.0 to 1.1.1
…dev/golang.org/x/net-0.51.0

chore(deps): bump golang.org/x/net from 0.50.0 to 0.51.0
…dev/modules-6b7541c889

chore(deps): bump the modules group with 2 updates
@dogancanbakir dogancanbakir self-assigned this Mar 5, 2026
@auto-assign auto-assign bot requested a review from dwisiswant0 March 5, 2026 11:11
@neo-by-projectdiscovery-dev
Copy link

neo-by-projectdiscovery-dev bot commented Mar 5, 2026

Neo - PR Security Review

No security issues found

Highlights

  • Adds markdown table output format via new -md/--markdown flag
  • Implements proper markdown table escaping for pipe characters and newlines
  • Integrates markdown output into existing output pipeline alongside JSON/CSV formats
Hardening Notes
  • Consider adding unit tests for escapeMarkdown() in runner/md_output.go to verify edge cases like consecutive pipes, mixed special characters, and empty strings
  • The reflection-based field iteration in MarkdownHeader() and MarkdownRow() could benefit from caching the field list to avoid repeated reflection calls on every result

Comment @neo help for available commands. · Open in Neo

@dogancanbakir dogancanbakir requested a review from knakul853 March 5, 2026 11:11
@coderabbitai
Copy link

coderabbitai bot commented Mar 5, 2026

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 474d3fd6-47bd-4f8b-96c7-e25b12d90955

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch dev

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants