IntelliScrape

Scrape anything.

A Python web scraping library with anti-detection, TLS fingerprint impersonation, and stealth browsing.

What is IntelliScrape?

IntelliScrape is a Python web scraping library that scrapes any website out of the box. It uses a 5-tier engine system that automatically escalates from fast HTTP requests to full browser automation — you get the cheapest, fastest method that works, and heavier weapons only when needed.

No more switching between requests, playwright, and selenium. No more debugging why your scraper got blocked. Just scrape(url) and you're done.

Key capabilities:

5-tier engine escalation (static → Playwright → Camoufox → nodriver → DrissionPage)
TLS fingerprint impersonation (JA3/JA4 bypass)
Browser fingerprint randomization
Human-like behavioral simulation
CAPTCHA detection, automated solving, and manual solving
Anti-bot vendor detection (Cloudflare, Akamai, DataDome, PerimeterX) with smart false-positive prevention
Real-time progress reporting (see which engine is running)
Intelligent site analysis and auto-configuration
Proxy rotation with free proxy finder
Export to JSON, CSV, Excel, SQLite, Text, Markdown
Async support for concurrent scraping
Link checking (status verification, categorization, broken link detection)
Website mirroring (HTTrack-ported, WARC/ZIP export, offline browsing)
Markdown corpus (whole-site Markdown for LLM/RAG ingestion, llms.txt / llms-full.txt / index.md)
SEO auditing (0–100 score, 11 weighted checks, keyword density, readability, heading hierarchy, image alt, technical, performance)
Backlink discovery (Google/Bing link: queries, anchor-text and rel analysis)
Website intelligence (detect frameworks, CMS, analytics, CDN, hosting, and more)
API detection (REST/GraphQL/WebSocket endpoints, third-party services, exposed keys)

Installation

pip install intelliscrape

Optional Extras

Extra	Command	What it adds
`stealth`	`pip install intelliscrape[stealth]`	nodriver engine (anti-WebDriver detection)
`camoufox`	`pip install intelliscrape[camoufox]`	Camoufox engine (Firefox-based, C++ patches)
`captcha`	`pip install intelliscrape[captcha]`	CapSolver integration (reCAPTCHA, hCaptcha, Turnstile)
`async`	`pip install intelliscrape[async]`	Async/concurrent scraping
`all`	`pip install intelliscrape[all]`	Everything above
`dev`	`pip install intelliscrape[dev]`	pytest, ruff

Quick Start

One-liner (Python)

from intelliscrape import scrape

text = scrape("https://example.com")
print(text[:500])

CLI

intelliscrape https://example.com

Full-featured class

from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result)

With proxy

scraper = IntelliScrape(proxy="user:pass@proxy:8080")
result = scraper.scrape("https://protected-site.com")

Get structured data

scraper = IntelliScrape()
data = scraper.get_structured("https://github.com")

print(data.title)          # Page title
print(data.description)    # Meta description
print(data.og_data)        # OpenGraph tags
print(data.json_ld)        # JSON-LD structured data

Crawl entire website

from intelliscrape import crawl

result = crawl("https://docs.python.org", max_pages=100)
print(f"Scraped {result.total_pages} pages")

for page in result.pages:
    print(f"  {page.url}: {len(page.content)} chars")

CLI Reference

intelliscrape [URL] [OPTIONS]

Output

Flag	Description
`-o, --output FILE`	Save output to file
`--json`	Structured JSON (title, description, meta tags)
`--raw`	Raw HTML instead of extracted text

Intelligent Mode

Flag	Description
`--analyze`	Analyze site and show recommendations
`--no-intelligent`	Disable intelligent auto-detection

Engine

Flag	Description
`--engine ENGINE`	Force specific engine: `static`, `playwright_stealth`, `camoufox`, `nodriver`, `drissionpage`
`--force-browser`	Force browser engine for JS-heavy sites
`--manual-captcha`	Open visible browser for manual CAPTCHA solving
`-v, --verbose`	Show real-time progress (which engine is running, CAPTCHA solving, etc.)

Proxy

Flag	Description
`--use-free-proxies`	Use free proxies automatically
`--no-free-proxies`	Disable free proxy finder
`--find-proxies`	Find and test free proxies (no scraping)
`--brightdata-key KEY`	Bright Data API key
`--scraperapi-key KEY`	ScraperAPI key
`--oxylabs-key KEY`	Oxylabs API key
`--smartproxy-key KEY`	Smartproxy API key

Authentication

Flag	Description
`--login`	Login before scraping
`--username USER`	Username/email
`--password PASS`	Password
`--login-url URL`	Explicit login URL

Cookies

Flag	Description
`--save-cookies FILE`	Save cookies to JSON
`--load-cookies FILE`	Load cookies from JSON

Request Modification

Flag	Description
`--block PATTERNS`	Block URLs (comma-separated)
`--header "Key: Value"`	Add custom header (repeatable)

Pagination & Search

Flag	Description
`--paginate`	Auto-follow pagination
`--max-pages N`	Max pages (default: 50)
`--search QUERY`	Submit search query on a specific page

Web Search

Flag	Description
`--web-search QUERY`	Search the web (DuckDuckGo → Google News → Bing News) and return a list of results
`--search-limit N`	Max results for `--web-search` (default: 10)
`--fetch-content`	Also scrape the full text of each result page

Crawl

Flag	Description
`--crawl`	Crawl entire website

Link Checking

Flag	Description
`--check-links`	Check all links on the page and report status

SEO Audit

Flag	Description
`--seo`	Run a full SEO audit: score /100, 11 weighted checks (title, meta, headings, images, links, canonical, OG, Twitter Cards, schema, technical, content), keyword density, readability (Flesch-Kincaid), heading hierarchy, image alt coverage, technical checks, page performance, and prioritized fixes

Backlink Discovery

Flag	Description
`--backlinks`	Discover backlinks via Google/Bing `link:` queries
`--backlink-limit N`	Max backlinks to find (default: 50)
`--backlink-sources ENG`	Search engines: `google,bing` (default)
`--no-scrape-backlinks`	Return search-result URLs without visiting pages

Website Intelligence

Flag	Description
`--tech`	Detect technology stack: frameworks, CSS frameworks, JS libraries, analytics, CDN, hosting, CMS, payment, languages, email marketing

API Detection

Flag	Description
`--detect-api`	Detect REST/GraphQL/WebSocket endpoints, API documentation paths, third-party services, and exposed API keys

Downloads

Flag	Description
`--download`	Download linked files
`--download-images`	Download all images
`--download-dir DIR`	Download directory (default: downloads)

Mirror (HTTrack-style)

Flag	Description
`--mirror`	Mirror entire website for offline browsing
`--mirror-depth N`	Max recursion depth (default: 5)
`--mirror-output DIR`	Output directory (default: ./mirror)
`--mirror-zip FILE`	Also create ZIP archive
`--mirror-warc FILE`	Also create WARC archive
`--mirror-delay SEC`	Delay between requests (default: 0.5)
`--mirror-exclude PAT`	Exclude URL patterns (repeatable)
`--mirror-include PAT`	Include URL patterns (repeatable)
`--mirror-engine ENG`	Engine: static, playwright, camoufox, nodriver, auto
`--mirror-proxy URL`	Proxy for mirroring
`--mirror-update`	Resume/update existing mirror
`--no-robots`	Ignore robots.txt

Markdown (LLM ingestion)

Flag	Description
`--markdown`	Convert entire website to Markdown corpus (per-page `.md` + `llms.txt` + `llms-full.txt` + `index.md`)
`--md-depth N`	Max recursion depth (default: 5)
`--md-output DIR`	Output directory (default: ./markdown)
`--md-merge` / `--no-md-merge`	Write merged `llms.txt` / `llms-full.txt` / `index.md` (default: on)
`--md-frontmatter` / `--no-md-frontmatter`	Add YAML frontmatter to each page (default: on)
`--md-images`	Keep image references as Markdown images
`--md-keep-nav`	Keep `<nav>` / `<footer>` content

Reuses --mirror-delay, --mirror-exclude, --mirror-include, --mirror-engine, --mirror-proxy, --mirror-zip, --mirror-warc, --no-robots, --max-pages.

Export

Flag	Description
`--export FORMAT`	`json`, `csv`, `excel`, `sqlite`, `text`, `markdown`

Examples

# Basic
intelliscrape https://example.com -o output.txt

# Structured data
intelliscrape https://example.com --json

# Analyze protection
intelliscrape https://amazon.com --analyze

# Free proxies
intelliscrape https://amazon.com --use-free-proxies

# Login
intelliscrape https://site.com --login --username user --password pass

# Pagination
intelliscrape https://example.com/products --paginate --max-pages 10

# Crawl
intelliscrape https://docs.python.org --crawl --max-pages 50

# Check links
intelliscrape https://example.com --check-links
intelliscrape https://example.com --check-links --export json -o report.json

# Export
intelliscrape https://example.com --export csv -o data.csv

# Manual CAPTCHA
intelliscrape https://protected-site.com --manual-captcha

# Force browser
intelliscrape https://react-app.com --force-browser

# Mirror entire site
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./backup

# Mirror + ZIP
intelliscrape https://example.com --mirror --mirror-zip site.zip

# Mirror + WARC
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# Mirror with proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

# Mirror excluded patterns
intelliscrape https://example.com --mirror --mirror-exclude "*.pdf" --mirror-exclude "/admin/*"

# Web search (no URL needed)
intelliscrape --web-search "python web scraping"
intelliscrape --web-search "openai news" --search-limit 5
intelliscrape --web-search "site:github.com python scraper" --search-limit 20 --export json -o results.json

# Web search + scrape each result page
intelliscrape --web-search "best python libraries" --fetch-content
intelliscrape --web-search "fastapi tutorial" --fetch-content --export json -o results.json

# Markdown corpus for LLM ingestion
intelliscrape https://docs.example.com --markdown

# Shallow, no merged files
intelliscrape https://docs.example.com --markdown --md-depth 2 --no-md-merge --md-output ./docs-md

# Keep images and nav, add robots.txt exception
intelliscrape https://docs.example.com --markdown --md-images --md-keep-nav --no-robots

# SEO audit
intelliscrape https://example.com --seo

# SEO audit exported to JSON
intelliscrape https://example.com --seo --export json -o seo.json

# Discover backlinks
intelliscrape https://example.com --backlinks --backlink-limit 50

# Backlinks without visiting pages (search results only)
intelliscrape https://example.com --backlinks --no-scrape-backlinks

# Technology stack detection
intelliscrape https://example.com --tech

# API endpoint detection
intelliscrape https://example.com --detect-api

Library API Reference

`scrape()` — Quick One-liner

from intelliscrape import scrape

text = scrape(url, **kwargs)

Parameter	Type	Default	Description
`url`	str	required	Target URL
`engine`	str	None	Force engine: `static`, `playwright_stealth`, `camoufox`, `nodriver`
`extract`	bool	True	Extract text from HTML
`clean`	bool	True	Clean extracted text
`return_raw`	bool	False	Return raw HTML
`return_structured`	bool	False	Return `StructuredData`
`handle_consent`	bool	True	Handle cookie consent banners
`force_browser`	bool	False	Force browser engine

`IntelliScrape` — Main Class

Constructor

from intelliscrape import IntelliScrape

scraper = IntelliScrape(**kwargs)

Parameter	Type	Default	Description
`proxy`	str, ProxyConfig, list	None	Single proxy or list
`proxies`	list of str	None	Proxy strings
`brightdata_key`	str	None	Bright Data API key
`scraperapi_key`	str	None	ScraperAPI key
`oxylabs_key`	str	None	Oxylabs API key
`smartproxy_key`	str	None	Smartproxy API key
`prefer_residential`	bool	True	Prefer residential proxies
`use_free_proxies`	bool	True	Auto-find free proxies
`api_key`	str	None	CAPTCHA solving API key
`captcha_provider`	str	None	`2captcha` or `capsolver`
`headless`	bool	True	Headless browser mode
`simulate_behavior`	bool	True	Human-like behavior simulation
`manual_captcha`	bool	False	Manual CAPTCHA solving mode
`tls_profile`	str	`chrome131`	TLS fingerprint profile
`session_profile`	str	None	Persistent session name
`max_retries`	int	3	Max retry attempts
`min_delay`	float	0.5	Min delay between requests
`max_delay`	float	3.0	Max delay between requests
`requests_per_minute`	int	None	Rate limit
`intelligent`	bool	True	Enable intelligent mode
`log_level`	str	`WARNING`	Logging level

Methods

`scrape(url, **kwargs)`

Scrape a URL and return text content.

result = scraper.scrape(
    url="https://example.com",
    engine=None,
    extract=True,
    clean=True,
    return_raw=False,
    return_structured=False,
    handle_consent=True,
    force_browser=False,
    intelligent=None,
)

`get_structured(url, **kwargs)`

Get structured data (title, description, meta tags, JSON-LD).

data = scraper.get_structured("https://github.com")
print(data.title)
print(data.description)
print(data.og_data)
print(data.json_ld)

`analyze(url)`

Analyze a site and return recommendations.

analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type)           # "ecommerce"
print(analysis.protection_level)    # "high"
print(analysis.recommended_engine)  # "playwright_stealth"
print(analysis.recommended_delay)   # 3.0

`scrape_many(urls, **kwargs)`

Scrape multiple URLs with rate limiting.

results = scraper.scrape_many([
    "https://example.com/page1",
    "https://example.com/page2",
])
# Returns: [{"url": ..., "content": ..., "success": ..., "error": ...}, ...]

`check_captcha(url)`

Check if a URL has a CAPTCHA.

captcha = scraper.check_captcha("https://site.com")
if captcha:
    print(captcha.captcha_type)  # CaptchaType.RECAPTCHA_V2
    print(captcha.site_key)

`check_antibot(url)`

Check anti-bot protection on a URL.

info = scraper.check_antibot("https://site.com")
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95

`check_links(url, **kwargs)`

Check all links on a page and return a detailed report with status codes, categorization, and summary statistics.

report = scraper.check_links("https://example.com", ignore_external=True)

print(f"Total links: {report.summary.total}")
print(f"OK: {report.summary.ok}, Broken: {report.summary.broken}")
print(f"Success rate: {report.summary.success_rate:.1f}%")
print(f"Internal: {report.summary.internal}, External: {report.summary.external}")
print(f"By type: {report.summary.by_type}")

# Per-link details
for link in report.links:
    print(f"  {link.url} -> {link.status_code} ({link.status.value})")

Parameter	Type	Default	Description
`url`	str	required	Page URL to check
`timeout`	int/float	5	Per-request timeout in seconds
`ignore_external`	bool	False	Skip external links
`max_workers`	int	10	Concurrent threads for checking
`allowed_statuses`	sequence	200-399	HTTP codes considered "OK"

Returns LinkCheckReport with:

report.links — list of SingleLinkResult (url, status_code, status, link_type, is_external)
report.summary — LinkCheckSummary with aggregate stats
report.summary.by_type — breakdown by link type (page, image, video, etc.)

Standalone function:

from intelliscrape import check_links

report = check_links("https://example.com")
print(f"Broken: {report.summary.broken}")

`web_search()` — Web Search

Search DuckDuckGo, Google News, and Bing News with automatic engine fallback. Returns a structured list of results and optionally scrapes the full content of each result page in one call.

`web_search()` — Convenience Function

from intelliscrape import web_search

report = web_search("python web scraping", limit=10)
print(f"Engine: {report.engine_used}, Results: {report.total}")

for r in report.results:
    print(f"  {r.rank}. {r.title}")
    print(f"     {r.url}")
    print(f"     {r.snippet[:100]}")

Parameter	Type	Default	Description
`query`	str	required	Search query string
`limit`	int	10	Maximum number of results
`fetch_content`	bool	False	Scrape full text of each result URL
`max_concurrent`	int	3	Parallel workers for content fetching
`scraper`	IntelliScrape	None	Existing scraper instance to reuse

Returns WebSearchReport:

report.query — original query string
report.engine_used — which engine returned results (duckduckgo, google_news, bing_news)
report.total — number of results
report.results — list of SearchResult objects
report.to_dict() — JSON-serialisable dict

Each SearchResult:

result.rank — 1-based position
result.title — page title
result.url — result URL
result.snippet — short description from the SERP
result.content — full scraped page text (only when fetch_content=True, None otherwise)
result.source — engine that returned this result
result.to_dict() — JSON-serialisable dict

With full page content:

report = web_search("openai news", limit=5, fetch_content=True)
for r in report.results:
    if r.content:
        print(f"{r.title}: {r.content[:300]}")

Export results to JSON:

from intelliscrape import web_search, DataExporter

report = web_search("python scraping", limit=10)
DataExporter.to_json([r.to_dict() for r in report.results], file="results.json")

`WebSearch` — Class

from intelliscrape import WebSearch, IntelliScrape

# Reuse an existing scraper (proxies, settings, etc. carry over)
scraper = IntelliScrape(use_free_proxies=True)
ws = WebSearch(scraper=scraper)

report = ws.search("site:github.com python scraper", limit=20)

Parameter	Type	Default	Description
`scraper`	IntelliScrape	None	Existing scraper; creates one if not provided
`**scraper_kwargs`	—	—	Forwarded to `IntelliScrape()` when `scraper` is not given

IntelliScrape.search_web() method:

from intelliscrape import IntelliScrape

scraper = IntelliScrape()
report = scraper.search_web("python web scraping", limit=10, fetch_content=False)
for r in report.results:
    print(r.rank, r.title, r.url)

`find_free_proxies(test=True)`

Find and test free proxies.

proxies = scraper.find_free_proxies(test=True)
for p in proxies:
    print(f"{p['url']} - speed: {p['speed']:.2f}s")

`get_proxy_status()`

Get proxy manager status.

status = scraper.get_proxy_status()
print(status['user_proxies'])
print(status['healthy_proxies'])

`crawl()` — Website Crawler

from intelliscrape import crawl

result = crawl(
    url="https://docs.python.org",
    max_pages=50,
    delay=0.5,
    on_page=None,  # Callback: on_page(done, failed)
)

Parameter	Type	Default	Description
`url`	str	required	Starting URL
`max_pages`	int	50	Maximum pages to crawl
`delay`	float	0.5	Delay between requests
`on_page`	callable	None	Progress callback

Returns CrawlResult:

result.pages — list of ScrapeResult (url, content, status)
result.failed — list of failed pages
result.total_pages — total scraped
result.total_failed — total failed
result.to_text() — all content as single text string

`AsyncIntelliScrape` — Async Scraping

import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape() as scraper:
        urls = [
            "https://example.com",
            "https://python.org",
            "https://github.com",
        ]
        results = await scraper.scrape_many(urls, max_concurrent=5)
        for r in results:
            print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())

Standalone async functions:

from intelliscrape import scrape_async, scrape_many_async

result = await scrape_async("https://example.com")
results = await scrape_many_async(urls, max_concurrent=10)

`DataExporter` — Export Formats

from intelliscrape import DataExporter

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db", table="scraped_data")
DataExporter.to_text(data, file="output.txt")
DataExporter.to_markdown(data, file="output.md")
DataExporter.export(data, format="json", file="output.json")

`mirror()` — Website Mirroring

Download entire websites for offline browsing with URL rewriting, robots.txt compliance, and archive support.

from intelliscrape import SiteMirror, MirrorConfig

# Quick mirror
from intelliscrape import mirror_site
result = mirror_site("https://example.com", max_depth=3)

`mirror()` — Convenience Function

from intelliscrape import mirror_site

result = mirror_site(
    url="https://example.com",
    output_dir="./mirror",
    max_depth=5,
    save_zip="site.zip",
    save_warc="archive.warc.gz",
)

Parameter	Type	Default	Description
`url`	str	required	Starting URL
`output_dir`	str	`./mirror`	Output directory
`max_depth`	int	5	Max link-following depth
`save_zip`	str	None	Create ZIP archive at path
`save_warc`	str	None	Create WARC archive at path
`exclude_patterns`	list	[]	URL exclude patterns
`include_patterns`	list	[]	URL include patterns
`delay`	float	0.5	Delay between requests (seconds)
`respect_robots`	bool	True	Respect robots.txt
`engine`	str	`static`	Scraping engine
`proxy`	str	None	Proxy URL

Returns MirrorResult:

result.pages_downloaded — Number of HTML pages
result.assets_downloaded — Number of assets (CSS, JS, images)
result.total_bytes — Total bytes downloaded
result.elapsed_seconds — Time taken
result.errors — Number of errors
result.output_dir — Output directory path
result.zip_path — ZIP archive path (if created)
result.warc_path — WARC archive path (if created)

`SiteMirror` — Full Control

from intelliscrape.track import SiteMirror, MirrorConfig

config = MirrorConfig(
    url="https://example.com",
    max_depth=3,
    output_dir="./my-mirror",
    exclude_patterns=["*.pdf", "/admin/*"],
    engine="static",
    delay=0.5,
    respect_robots=True,
    url_mode="relative",  # relative | absolute | keep_original
)

m = SiteMirror(config)
result = m.run(save_zip="mirror.zip", save_warc="mirror.warc.gz")

`MirrorConfig` — All Options

from intelliscrape.track import MirrorConfig

config = MirrorConfig(
    # What to mirror
    url="https://example.com",
    max_depth=5,
    max_pages=10000,
    max_file_size=50 * 1024 * 1024,  # 50 MB

    # Scope
    travel="same_domain",  # same_address | same_domain | same_tld | everywhere

    # What to fetch
    fetch_html=True,
    fetch_css=True,
    fetch_js=True,
    fetch_images=True,
    fetch_fonts=True,
    fetch_media=True,
    fetch_documents=True,

    # Filtering
    include_patterns=[],
    exclude_patterns=["*.pdf"],

    # Output
    output_dir="./mirror",
    url_mode="relative",
    generate_index=True,

    # Resume
    use_cache=True,
    update_mode=False,

    # Politeness
    delay=0.5,
    max_concurrent=5,
    respect_robots=True,

    # Engine & proxy
    engine="static",
    proxy=None,
    cookies=None,
)

CLI Examples

# Basic mirror
intelliscrape https://example.com --mirror

# Depth 3, custom output
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./site

# With ZIP
intelliscrape https://example.com --mirror --mirror-zip backup.zip

# With WARC (web archive format)
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# With proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

# Exclude patterns
intelliscrape https://example.com --mirror --mirror-exclude "*.pdf" --mirror-exclude "/api/*"

# Resume interrupted mirror
intelliscrape https://example.com --mirror --mirror-update

`markdown_site()` — Markdown Corpus for LLMs

Convert an entire website to a Markdown corpus optimized for LLM / RAG ingestion (like an offline llms.txt export). Each page becomes a clean .md file with YAML frontmatter; the crawler skips CSS/JS/images, strips nav/footer/scripts, absolutizes links, and appends JSON-LD structured data.

from intelliscrape import markdown_site

result = markdown_site(
    url="https://docs.example.com",
    output_dir="./markdown",
    max_depth=5,
    save_zip="corpus.zip",
)
print(result.pages_converted)   # 42
print(result.total_chars)       # 1_234_567
print(result.llms_file)         # ./markdown/llms.txt
print(result.llms_full_file)    # ./markdown/llms-full.txt

Output layout:

markdown/
├── llms.txt                 # site map: per-page title, URL, summary
├── llms-full.txt            # every page merged into one corpus file
├── index.md                 # TOC linking to local .md files
└── docs.example.com/        # one .md per page, mirroring the URL tree
    ├── index.md
    ├── getting-started.md
    └── guides/
        └── proxy-setup.md

Parameter	Type	Default	Description
`url`	str	required	Starting URL
`output_dir`	str	`./markdown`	Output directory
`max_depth`	int	5	Max link-following depth
`save_zip` / `save_warc`	str	None	Also create ZIP / WARC archive
`progress_callback`	callable	None	Called with crawl progress
`markdown_merge`	bool	True	Write `llms.txt` / `llms-full.txt` / `index.md`
`markdown_frontmatter`	bool	True	Add YAML frontmatter (title, url, date, word_count, ...)
`markdown_keep_nav`	bool	False	Keep `<nav>` / `<footer>` content
`markdown_images`	bool	False	Keep image references as Markdown images
`markdown_json_ld`	bool	True	Append JSON-LD as Structured Data section
`respect_robots`	bool	True	Respect robots.txt
`exclude_patterns` / `include_patterns`	list	[]	URL filters
`engine`	str	`static`	Scraping engine
`proxy`	str	None	Proxy URL
`delay`	float	0.5	Delay between requests

Returns MarkdownResult:

result.pages_converted — Number of pages converted to Markdown
result.total_chars — Total characters written
result.errors — Number of failed pages
result.llms_file / result.llms_full_file / result.index_file — Paths to merged files (None when markdown_merge=False or no pages)
result.zip_path / result.warc_path — Archive paths (if created)

Single-page converter

from intelliscrape import html_to_markdown

md = html_to_markdown(html, url="https://example.com/page",
                      frontmatter=True, keep_nav=False)

`Downloader` — File Downloads

from intelliscrape import Downloader

downloader = Downloader()

# Download linked files
results = downloader.download_links(html, base_url, "downloads/")

# Download all images
results = downloader.download_images(html, base_url, "downloads/images/")

`Authenticator` — Login & Sessions

from intelliscrape import Authenticator, LoginCredentials

auth = Authenticator()
credentials = LoginCredentials(
    username="user@example.com",
    password="secret",
)
success = auth.login("https://site.com/login", credentials)

`FormSubmitter` — Form Interaction

from intelliscrape import FormSubmitter

form_submitter = FormSubmitter()
forms = form_submitter.find_forms(html, base_url="https://site.com")
result_html = form_submitter.search(html, "query", base_url="https://site.com")

`Paginator` — Auto-pagination

from intelliscrape import Paginator

paginator = Paginator()
next_url = paginator.find_next_page(html, current_url, current_page)

`RequestInterceptor` — Request/Response Modification

from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()
interceptor.block_urls(["analytics", "tracking"])
interceptor.modify_headers({"X-Custom": "value"})
interceptor.add_response_handler(my_handler)

`CookieManager` — Cookie Persistence

from intelliscrape import CookieManager

cookie_mgr = CookieManager()
cookie_mgr.save_cookies("https://site.com", {"session": "abc123"})
cookies = cookie_mgr.load_cookies("https://site.com")

`CaptchaDetector` & `CaptchaSolver`

from intelliscrape import CaptchaDetector, CaptchaSolver

# Detect
captcha = CaptchaDetector.detect(html, url="https://site.com")

# Solve (requires API key)
solver = CaptchaSolver(provider="capsolver", api_key="YOUR_KEY")
token = solver.solve_recaptcha_v2(site_key, page_url)
token = solver.solve_hcaptcha(site_key, page_url)
token = solver.solve_turnstile(site_key, page_url)

`AntiBotDetector` — Anti-bot Vendor Detection

from intelliscrape import AntiBotDetector

info = AntiBotDetector.detect(html=html, headers=headers, cookies=cookies)
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95

Smart detection uses a two-tier approach to avoid false positives:

Strong markers (only on actual challenge pages): 1 match = blocked
Weak markers (can appear in docs/blogs): require 3+ matches AND page <50KB
Real challenge pages are tiny (<50KB), content pages are large — size prevents false triggers on sites like cloudflare.com that mention their own products

Anti-bot Bypass Classes

from intelliscrape import (
    CloudflareTurnstileBypass,
    DataDomeBypass,
    PerimeterXBypass,
    AkamaiBypass,
)

Each bypass class provides detection, recommended settings, and automated token solving where possible.

Engine System

IntelliScrape uses a 5-tier engine escalation system. It tries the cheapest, fastest method first and escalates only when needed.

Tier 1: Static (curl_cffi)        → Sub-second, TLS impersonation
    ↓ if JS-only content
Tier 2: Playwright Stealth         → 2-5s, headless Chromium + patches
    ↓ if still blocked
Tier 3: Camoufox                   → 3-8s, custom Firefox (C++ patches)
    ↓ if still blocked
Tier 4: nodriver                   → 5-15s, raw CDP, no WebDriver traces
    ↓ if still blocked
Tier 5: DrissionPage               → 5-15s, hybrid HTTP+browser mode

Tier	Engine	Speed	Stealth	Best For
1	`static`	Sub-second	Low	Static sites, APIs
2	`playwright_stealth`	2-5s	Medium	JS-heavy sites, basic bot detection
3	`camoufox`	3-8s	High	Protected sites, fingerprint detection
4	`nodriver`	5-15s	Maximum	DataDome, PerimeterX, Akamai
5	`drissionpage`	5-15s	High	Hybrid HTTP+browser, fallback

# Auto-detect (default)
text = scraper.scrape("https://site.com")

# Force specific engine
text = scraper.scrape("https://site.com", engine="playwright_stealth")

# Force browser for known JS-heavy sites
text = scraper.scrape("https://react-app.com", force_browser=True)

# Force engine via CLI
intelliscrape https://amazon.com --engine camoufox -v

# Auto-detect with verbose progress
intelliscrape https://amazon.com -v

Intelligent Mode

Enabled by default (intelligent=True). Before scraping, IntelliScrape analyzes the URL to determine:

Site type — ecommerce, social, news, tech, education, etc.
Protection level — none, basic, moderate, high, extreme
Recommended engine — which tier to start with
Recommended delay — slower for protected sites
Residential proxy needed — auto-selects proxy type

analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type.value)         # "ecommerce"
print(analysis.protection_level.value)  # "high"
print(analysis.recommended_engine)      # "playwright_stealth"
print(analysis.requires_residential_proxy)  # True

Features

Anti-Detection

Feature	Description
TLS Fingerprinting	Impersonates Chrome, Firefox, Safari (JA3/JA4)
Header Rotation	Randomizes HTTP headers
Browser Fingerprinting	Randomizes viewport, timezone, WebGL, canvas
Human Simulation	Bezier mouse paths, natural scrolls, realistic delays
Cookie Consent	Auto-handles consent banners
Rate Limiting	Smart delays based on site protection
Retry with Backoff	Exponential backoff on failures

CAPTCHA Solving

Automated (requires API key):

scraper = IntelliScrape(api_key="YOUR_KEY", captcha_provider="capsolver")
result = scraper.scrape("https://protected-site.com")

CAPTCHA Type	2Captcha	CapSolver
reCAPTCHA v2	Yes	Yes
reCAPTCHA v3	No	Yes
hCaptcha	Yes	Yes
Cloudflare Turnstile	No	Yes

Manual (opens visible browser):

scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://site-with-captcha.com")
# Browser opens → solve CAPTCHA → press Enter in terminal

intelliscrape https://site.com --manual-captcha

Anti-bot challenge pages (Cloudflare, PerimeterX, Akamai, DataDome) are automatically detected during the engine fallback chain. When detected, a browser opens for manual solving (Press and Hold, Turnstile, etc.). No API key needed.

Proxy Configuration

# Single proxy
scraper = IntelliScrape(proxy="user:pass@proxy:8080")

# Multiple proxies
scraper = IntelliScrape(proxies=["proxy1:8080", "proxy2:8080"])

# Residential proxy
scraper = IntelliScrape(brightdata_key="YOUR_KEY")

# Free proxies (automatic)
scraper = IntelliScrape(use_free_proxies=True)

Export Formats

from intelliscrape import DataExporter

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db")
DataExporter.to_markdown(data, file="output.md")

intelliscrape https://site.com --export csv -o data.csv
intelliscrape https://site.com --export json -o data.json

Website Mirroring

Download complete websites for offline browsing with URL rewriting and archive support.

from intelliscrape import mirror_site

# Basic mirror
result = mirror_site("https://example.com", max_depth=3)

# With ZIP archive
result = mirror_site("https://example.com", save_zip="site.zip")

# Full options
from intelliscrape.track import SiteMirror, MirrorConfig

config = MirrorConfig(
    url="https://example.com",
    max_depth=3,
    output_dir="./mirror",
    exclude_patterns=["*.pdf", "/admin/*"],
    engine="static",
    delay=0.5,
)
m = SiteMirror(config)
result = m.run(save_zip="mirror.zip", save_warc="mirror.warc.gz")

# Mirror site
intelliscrape https://example.com --mirror

# Mirror with depth and output dir
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./backup

# Mirror + ZIP
intelliscrape https://example.com --mirror --mirror-zip backup.zip

# Mirror + WARC (web archive format)
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# Mirror with proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

Markdown Corpus (LLM / RAG)

from intelliscrape import markdown_site

# Convert whole site to Markdown
result = markdown_site("https://docs.example.com", max_depth=5)
print(f"Converted {result.pages_converted} pages -> {result.output_dir}")

# Convert site to Markdown corpus
intelliscrape https://docs.example.com --markdown

# With ZIP archive
intelliscrape https://docs.example.com --markdown --mirror-zip corpus.zip

SEO Auditing

Full on-page SEO analysis with a weighted 0–100 score across 11 checks: title, meta description, headings, images, links, canonical, Open Graph, Twitter Cards, schema, technical, and content quality. Every check returns a SEOCheck with score and pass/fail status, plus prioritized issues and suggestions for fixing what matters first.

from intelliscrape import analyze_seo, IntelliScrape

# Quick one-liner
report = analyze_seo("https://example.com")
print(f"Score: {report.overall_score}/100")

# With the scraper (reuses engine selection / proxy)
report = IntelliScrape().analyze_seo("https://example.com")

# Deep inspection
print(f"Word count: {report.content.word_count}")
print(f"Readability grade: {report.content.readability_grade}")
print(f"Top keyword: {report.content.top_keywords[0]}")
print(f"Internal links: {report.links.internal}, External: {report.links.external}")
print(f"H1 count: {report.headings.counts.get('h1', 0)}")
print(f"Images missing alt: {report.images.missing_alt}")
print(f"Has viewport: {report.technical.has_viewport}")
print(f"External scripts: {report.performance.external_scripts}")

# Every check with pass/fail
for check in report.checks:
    print(f"{check.name}: {check.score:.0%} ({'PASS' if check.passed else 'FAIL'})")

# Export everything
report.to_dict()

# CLI
intelliscrape https://example.com --seo

# Export the full audit
intelliscrape https://example.com --seo --export json -o seo.json

Backlink Discovery

Find pages linking to a target domain with Google/Bing link: queries, visit each referring page to extract the exact link, anchor text, and rel attributes, and analyze the anchor-text distribution.

Search engines (especially Bing's link: operator) can return pages that don't actually link to the target. Each result is marked verified (a direct link to the target was confirmed on the page) or unverified (search-engine hit only). unique_domains, dofollow, and nofollow counts reflect verified backlinks only.

from intelliscrape import find_backlinks

# Search + scrape referring pages
report = find_backlinks("https://example.com", limit=50)

# Search results only (no page visits)
report = find_backlinks("https://example.com", scrape_backlinks=False)

for bl in report.backlinks:
    if bl.verified:
        print(f"[verified] {bl.source_url} -> {bl.anchor_text or '[no anchor]'} ({bl.rel_type})")

# CLI
intelliscrape https://example.com --backlinks

# Results only from search engines, skip page visits
intelliscrape https://example.com --backlinks --no-scrape-backlinks

# Limit and export
intelliscrape https://example.com --backlinks --backlink-limit 100 --export json -o backlinks.json

Website Intelligence (Tech Stack)

Detect the technology stack from HTML, headers, cookies, and asset URLs with confidence-weighted signature matching.

from intelliscrape import IntelliScrape

stack = IntelliScrape().detect_tech("https://stripe.com")
print(stack.summary)
# {'frameworks': ['next.js'], 'css_frameworks': ['tailwind css'], ...}

intelliscrape https://example.com --tech

API Detection

Detect REST, GraphQL, and WebSocket endpoints, API documentation paths, third-party services, and exposed API keys from page source.

from intelliscrape import IntelliScrape

report = IntelliScrape().detect_apis("https://example.com")
for ep in report.endpoints:
    print(f"{ep.method} {ep.url} ({ep.category}, {ep.confidence:.0%})")

intelliscrape https://example.com --detect-api

Examples

Scrape React/Vue/Angular SPAs

result = scraper.scrape("https://react-app.com", force_browser=True)

Scrape with Custom Headers

result = scraper.scrape(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123"},
)

Persistent Sessions

scraper = IntelliScrape(session_profile="my_session")
scraper.scrape("https://site.com")       # Creates session
scraper.scrape("https://site.com/dashboard")  # Reuses session

Download Files

from intelliscrape import Downloader

downloader = Downloader()
html = scraper.scrape("https://example.com/downloads", return_raw=True)
results = downloader.download_links(html, "https://example.com", "downloads/")

Batch Scraping with Export

from intelliscrape import IntelliScrape, DataExporter

scraper = IntelliScrape()
urls = [f"https://example.com/page/{i}" for i in range(100)]

results = scraper.scrape_many(urls)
DataExporter.to_csv(
    [{"url": r["url"], "content": r["content"], "success": r["success"]} for r in results],
    file="results.csv",
)

Check Links on a Page

from intelliscrape import check_links

report = check_links("https://example.com")

# Summary
print(f"Total: {report.summary.total}")
print(f"OK: {report.summary.ok}")
print(f"Broken: {report.summary.broken}")
print(f"Success rate: {report.summary.success_rate:.1f}%")

# Only internal links
report = check_links("https://example.com", ignore_external=True)

# Export broken links
for link in report.links:
    if not link.is_ok:
        print(f"BROKEN: {link.url} -> {link.status_code}")

Web Search

from intelliscrape import web_search

# Basic search — returns list of results with title, URL, snippet
report = web_search("python web scraping", limit=10)
print(f"Engine: {report.engine_used}  Results: {report.total}")
for r in report.results:
    print(f"  {r.rank}. {r.title}")
    print(f"     {r.url}")

# With full page content (Firecrawl-style, one call)
report = web_search("fastapi tutorial", limit=5, fetch_content=True)
for r in report.results:
    if r.content:
        print(f"{r.title}: {r.content[:300]}")

# Export to JSON
from intelliscrape import DataExporter
DataExporter.to_json([r.to_dict() for r in report.results], file="results.json")

# Reuse an existing scraper instance (proxies, config carry over)
from intelliscrape import IntelliScrape
scraper = IntelliScrape(use_free_proxies=True)
report = scraper.search_web("site:github.com python scraper", limit=20)

Troubleshooting

Problem	Solution
Returns empty or widget text	Use `force_browser=True` — site is a JS SPA
CAPTCHA blocking	Use `manual_captcha=True` or `api_key` + `captcha_provider`
Blocked by anti-bot	Try `--engine camoufox` + residential proxy
Anti-bot challenge opens browser but no CAPTCHA visible	The challenge may require Press and Hold — hold the button for 3-5 seconds
False positive: "CAPTCHA detected" on normal site	v3.1.1+ uses smart detection — update with `pip install -U intelliscrape`
Playwright not installed	`pip install playwright && playwright install chromium`
Markdown mode says markdownify missing	`pip install markdownify`
Camoufox not installed	`pip install "camoufox[geoip]" && python -m camoufox fetch`
nodriver not installed	`pip install nodriver`
DrissionPage not installed	`pip install DrissionPage`
Want to see what's happening	Add `-v` flag for real-time progress output
Scraping too slow	Try `--engine static` for fastest results
Site requires login	Use `--login --username USER --password PASS`

Project Structure

intelliscrape/
    __init__.py             # Public API exports
    __main__.py             # python -m intelliscrape
    core.py                 # IntelliScrape class (main orchestrator)
    cli.py                  # CLI (argparse + rich)
    progress.py             # ProgressTracker, ScrapeProgress (real-time status)
    async_scraper.py        # AsyncIntelliScrape
    intelligent.py          # SiteAnalyzer, SmartRateLimiter
    auth.py                 # Authenticator, LoginCredentials
    forms.py                # FormSubmitter
    pagination.py           # Paginator
    export.py               # DataExporter
    downloader.py           # Downloader
    cookies.py              # CookieManager
    crawler.py              # crawl(), CrawlResult
    interceptor.py          # RequestInterceptor
    link_checker.py         # check_links, LinkCheckReport
    markdown.py             # html_to_markdown, markdown_site (LLM corpus)
    parser.py               # HTML DOM builder
    cleaner.py              # Text cleaning
    utils.py                # HTML analysis
    exceptions.py           # Exceptions

    engines/                # 5-tier scraping engines
        base.py             # BaseEngine, ScrapeResult
        static.py           # curl_cffi (Tier 1)
        playwright_stealth.py  # Playwright (Tier 2)
        camoufox.py         # Camoufox (Tier 3)
        stealth.py          # nodriver (Tier 4)
        drissionpage.py     # DrissionPage (Tier 5)

    anti_detection/         # Anti-detection subsystem
        antibot.py          # AntiBotDetector
        behavior.py         # HumanBehavior
        bypass.py           # Vendor-specific bypasses
        consent.py          # CookieConsentHandler
        fingerprint.py      # FingerprintGenerator
        headers.py          # HeaderManager
        throttle.py         # SmartThrottle, RateLimiter
        tls.py              # TLSConfig (JA3/JA4)

    challenges/             # Challenge handling
        captcha.py          # CaptchaDetector, CaptchaSolver
        manual.py           # ManualCaptchaSolver (opens visible browser)

    extractor/              # Content extraction
        structured.py       # StructuredExtractor, StructuredData

    proxy/                  # Proxy management
        __init__.py         # ProxyConfig, ProxyManager
        free_finder.py      # FreeProxyFinder
        manager.py          # IntelligentProxyManager
        providers.py        # BrightData, ScraperAPI, etc.

    session/                # Session persistence
        __init__.py         # SessionManager

    track/                  # Website mirroring (HTTrack port)
        __init__.py         # Package exports
        config.py           # MirrorConfig (30+ options)
        mirror.py           # SiteMirror engine (async workers, WARC/ZIP)
        parser.py           # AssetDiscovery (HTML/CSS/JS extraction)
        rewriter.py         # URLRewriter (relative/absolute paths)
        naming.py           # SaveNamer (URL→filesystem mapping)
        cache.py            # MirrorCache (resume support)
        filters.py          # URLFilter (include/exclude patterns)
        robots.py           # RobotsParser (RFC 9309 compliance)

Contributing

We welcome contributions! See CONTRIBUTING.md.

git clone https://github.com/GuixJoy/IntelliScrape.git
cd IntelliScrape/IntelliScrape_library
pip install -e ".[dev]"
pytest

License

LGPL-2.1 License — see LICENSE.

PyPI · GitHub · Report Issues

Name		Name	Last commit message	Last commit date
api		api
examples		examples
frontend		frontend
intelliscrape		intelliscrape
testing		testing
tests		tests
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Procfile		Procfile
README.md		README.md
demo.tape		demo.tape
main.py		main.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
scraped_content.txt		scraped_content.txt
setup.py		setup.py
vercel.json		vercel.json

Latest commit

History

Folders and files

Repository files navigation

IntelliScrape

What is IntelliScrape?

Installation

Optional Extras

Quick Start

One-liner (Python)

CLI

Full-featured class

With proxy

Get structured data

Crawl entire website

CLI Reference

Output

Intelligent Mode

Engine

Proxy

Authentication

Cookies

Request Modification

Pagination & Search

Web Search

Crawl

Link Checking

SEO Audit

Backlink Discovery

Website Intelligence

API Detection

Downloads

Mirror (HTTrack-style)

Markdown (LLM ingestion)

Export

Examples

Library API Reference

scrape() — Quick One-liner

IntelliScrape — Main Class

Constructor

Methods

scrape(url, **kwargs)

get_structured(url, **kwargs)

analyze(url)

scrape_many(urls, **kwargs)

check_captcha(url)

check_antibot(url)

check_links(url, **kwargs)

web_search() — Web Search

web_search() — Convenience Function

WebSearch — Class

find_free_proxies(test=True)

get_proxy_status()

crawl() — Website Crawler

AsyncIntelliScrape — Async Scraping

DataExporter — Export Formats

mirror() — Website Mirroring

mirror() — Convenience Function

SiteMirror — Full Control

MirrorConfig — All Options

CLI Examples

markdown_site() — Markdown Corpus for LLMs

Single-page converter

Downloader — File Downloads

Authenticator — Login & Sessions

FormSubmitter — Form Interaction

Paginator — Auto-pagination

RequestInterceptor — Request/Response Modification

CookieManager — Cookie Persistence

CaptchaDetector & CaptchaSolver

AntiBotDetector — Anti-bot Vendor Detection

Anti-bot Bypass Classes

Engine System

Intelligent Mode

Features

Anti-Detection

CAPTCHA Solving

Proxy Configuration

Export Formats

Website Mirroring

`scrape()` — Quick One-liner

`IntelliScrape` — Main Class

`scrape(url, **kwargs)`

`get_structured(url, **kwargs)`

`analyze(url)`

`scrape_many(urls, **kwargs)`

`check_captcha(url)`

`check_antibot(url)`

`check_links(url, **kwargs)`

`web_search()` — Web Search

`web_search()` — Convenience Function

`WebSearch` — Class

`find_free_proxies(test=True)`

`get_proxy_status()`

`crawl()` — Website Crawler

`AsyncIntelliScrape` — Async Scraping

`DataExporter` — Export Formats

`mirror()` — Website Mirroring

`mirror()` — Convenience Function

`SiteMirror` — Full Control

`MirrorConfig` — All Options

`markdown_site()` — Markdown Corpus for LLMs

`Downloader` — File Downloads

`Authenticator` — Login & Sessions

`FormSubmitter` — Form Interaction

`Paginator` — Auto-pagination

`RequestInterceptor` — Request/Response Modification

`CookieManager` — Cookie Persistence

`CaptchaDetector` & `CaptchaSolver`

`AntiBotDetector` — Anti-bot Vendor Detection