Skip to content

fix: improve CDN URL normalization and version extraction - #31

Merged
HC-ONLINE merged 1 commit into
mainfrom
fix/cdn-url-normalization
Jul 31, 2026
Merged

fix: improve CDN URL normalization and version extraction#31
HC-ONLINE merged 1 commit into
mainfrom
fix/cdn-url-normalization

Conversation

@HC-ONLINE

Copy link
Copy Markdown
Owner

Description

Fix CDN-hosted library version extraction that was failing for npm-style URLs and expand CDN provider coverage from 2 to 9.

Problem

The fingerprinter detected CDN-hosted libraries using regex patterns like jquery[.-]([\d\.]+)?, but:

  1. @ separator not handled: URLs like cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js used @ as version separator, but the regex only matched / and -. Version was silently dropped.
  2. Only 2 CDNs detected: Only cdnjs.cloudflare.com and ajax.googleapis.com were recognized. Popular CDNs like jsDelivr, unpkg, BootstrapCDN were ignored.
  3. No structured CDN parsing: No code parsed CDN URL structures (/ajax/libs/{lib}/{ver}/, /npm/{lib}@{ver}/) to extract library name + version reliably.

Changes

1. Regex @ separator

  • File: src/ciberwebscan/core/analyzers/fingerprint/html_analyzer.py
  • Changed [\/-]? to [/@-]? in both script (line 234) and CSS (line 287) analysis loops
  • Now captures versions from npm-style CDN URLs

2. CDN provider detection (2 → 9)

  • File: src/ciberwebscan/core/analyzers/fingerprint/html_analyzer.py
  • Replaced hardcoded if/if blocks with dict-based CDN_HOSTNAMES lookup
  • Added: jsDelivr, unpkg, BootstrapCDN, Fastly, jQuery CDN, Microsoft CDN, bootcdn, staticfile.org

3. Structured CDN URL parser

  • File: src/ciberwebscan/core/analyzers/fingerprint/html_analyzer.py
  • New _parse_cdn_url() function that extracts library + version from CDN URL patterns by hostname
  • Supports cdnjs (/ajax/libs/{lib}/{ver}/), jsDelivr (/npm/{lib}@{ver}/), unpkg (/{lib}@{ver}/)

Examples

# BEFORE (version silently dropped):
cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js → "jQuery" (no version)
unpkg.com/vue@2.7.14/dist/vue.js → "Vue.js" (no version)

# AFTER (version correctly extracted):
cdn.jsdelivr.net/npm/jquery@3.6.0/dist/jquery.min.js → "jQuery 3.6.0"
unpkg.com/vue@2.7.14/dist/vue.js → "Vue.js 2.7.14"
cdnjs.cloudflare.com/ajax/libs/angular/1.8.2/angular.min.js → "angular 1.8.2"
Testing
- All 1180 unit tests pass (no regressions)

Breaking Changes
None. Backward-compatible improvement — existing regex patterns still work, this adds @ support and new CDN detection.

- Add @ separator to regex character class for npm-style CDN URLs
  (jsDelivr, unpkg) like jquery@3.6.0
- Expand CDN detection from 2 to 9 providers (jsDelivr, unpkg,
  BootstrapCDN, Fastly, jQuery CDN, Microsoft CDN, bootcdn, staticfile)
- Add _parse_cdn_url() for structured CDN URL parsing by hostname
  and path patterns (/ajax/libs/{lib}/{ver}/, /npm/{lib}@{ver}/)
- Replace if/if CDN detection with dict-based CDN_HOSTNAMES lookup
@HC-ONLINE
HC-ONLINE merged commit 3ad02bb into main Jul 31, 2026
3 checks passed
@HC-ONLINE
HC-ONLINE deleted the fix/cdn-url-normalization branch July 31, 2026 19:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant