Skip to content

perf: comparison with competing packages - #304

Open
PeterStaar-IBM wants to merge 2 commits into
mainfrom
dev/add-comparison-with-competing-packages
Open

perf: comparison with competing packages#304
PeterStaar-IBM wants to merge 2 commits into
mainfrom
dev/add-comparison-with-competing-packages

Conversation

@PeterStaar-IBM

@PeterStaar-IBM PeterStaar-IBM commented Aug 3, 2026

Copy link
Copy Markdown
Member

docs/performance_benchmarks.md had an empty table under Performance comparison on parsing and rendering and no way
to fill it: the perf scripts could time docling-parse against pypdfium2 and pymupdf, but only reported wall time per
run — no per-page distribution, no hardware/version provenance, and no coverage of the other top-5 packages.

Filling that table meant touching most of ./perf, which turned out to hold five scripts with three mutually
incompatible CSV formats and three copies of the third-party backend adapters. This PR does both: adds the
comparison suite, and collapses the tooling to three entry points around one interchange format.

What changed

  1. Cross-package comparison suite (run_scaling.py --compare)

Six packages, two tasks, reported as a per-page time distribution (mean / median / p95 / p99) plus a wall-time
speedup grid.

┌───────────────┬───────────────────────────────────┬──────────────────────────┐
│    package    │               parse               │       parse+render       │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ docling-parse │ DoclingThreadedPdfParser          │ + RenderConfig           │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ pymupdf       │ page.get_text("rawdict")          │ + get_pixmap → pil_image │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ pypdfium2     │ textpage rects + get_text_bounded │ + page.render → to_pil   │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ pdfplumber    │ page.chars                        │ + page.to_image()        │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ pdfminer.six  │ extract_pages + LTChar walk       │ not supported            │
├───────────────┼───────────────────────────────────┼──────────────────────────┤
│ pypdf         │ extract_text(visitor_text=...)    │ not supported            │
└───────────────┴───────────────────────────────────┴──────────────────────────┘

Design points worth reviewing:

  • Every backend uses a position-bearing extraction API, so all rows measure the same task. pymupdf is deliberately
    timed on rawdict rather than get_text("text") — the latter returns a bare string with no geometry and is a
    materially cheaper task. Historical pymupdf numbers from the old run_perf.py are therefore not comparable to these.
  • docling-parse is run once per --threads value; everything else is single-threaded, because no other package
    exposes a thread-safe multi-page pipeline. That is the differentiator the tables are meant to show, so total time
    (wall clock over the corpus, including document opens) drops with threads while the per-page distribution stays
    flat.
  • Per-page numbers come from different clocks by necessity: third-party backends get a wall-clock timer around each
    page's work; docling-parse reports the C++ page timing, because under concurrency no wall-clock interval belongs to
    a single page. The table has a per-page source column stating which was used, and the two agree at threads=1.
  • Render size check. A render comparison is meaningless if the backends rasterised different canvases, so every
    rendered page's pixel size is recorded and compared against pypdfium2 (PDFium being the rasteriser three of the six
    packages ultimately rely on). It reports how many pages agree within 2 px and the worst offender.
  1. Consolidated ./perf from 5 scripts to 3
┌──────────────────┬───────────────────────────────────────────────────────────────────────┐
│                  │                                                                       │
├──────────────────┼───────────────────────────────────────────────────────────────────────┤
│ _common.py (new) │ PageRow, the per-page CSV schema, shared helpers                      │
├──────────────────┼───────────────────────────────────────────────────────────────────────┤
│ run_scaling.py   │ measure — sweeps and comparison; the only place backend adapters live │
├──────────────────┼───────────────────────────────────────────────────────────────────────┤
│ run_eval.py      │ plot — absorbed run_scaling_visualization.py                          │
├──────────────────┼───────────────────────────────────────────────────────────────────────┤
│ run_analysis.py  │ drill down — absorbed run_perf.py's aggregate timing breakdown        │
└──────────────────┴───────────────────────────────────────────────────────────────────────┘

Deleted: run_perf.py, run_scaling_visualization.py.

  • One CSV format. run_scaling.py --pages-csv writes it from both the sweep and --compare; every row carries backend,
    task and threads, so one file can hold several packages at several thread counts. --enable-timing/--timing-csv are
    gone, their C++ stage columns folded into the single schema. run_eval.py no longer infers the backend from the
    filename.
  • run_threaded is now a thin wrapper over cmp_docling — they were two near-identical loops that could silently
    drift.
  • Duplicated find_pdfs / percentile moved to _common.py.
  1. Reports are now self-contained; you only name a directory

uv run python ./perf/run_scaling.py --threads 1,4,8,12 --compare all --mode render
--output-dir ./docs/performance_benchmarks/

Writes .md and .csv (e.g. apple_m3_max_performance-dataset-bo767_render.md), so results from
different machines never overwrite each other. The report records the exact command, the dataset name/revision/size,
the machine and every benchmarked package version, the decode/content/render configs the run was driven with, and
all result tables — a published number is traceable without the terminal scrollback it came from. Default output dir
is ./scratch so exploratory runs don't touch the docs.

  1. Docs

docs/performance_benchmarks.md gains a methodology section (what each task means, which API each package is timed
on, how the statistics are defined and why the per-page clocks differ) and a reproduction section.

Bugs fixed along the way

  • run_perf.py --parser docling-threaded recorded elapsed_sec ≈ 0 — t0/t1 were taken back to back with nothing
    between them, and run_analysis.py/run_eval.py consumed that column without a guard. Fixed by deletion; the surviving
    path uses C++ page timings.
  • uv sync --group perf-test and pip install .[perf-tools], documented in perf/README.md and run_perf.py, never
    existed. The group is perf.
  • pypdf and an explicit pdfminer.six added to the perf dependency group — pypdf was silently skipped because it
    wasn't installed. Missing backends now print a visible skip line and are restated before the tables.

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
…solidate the perf tooling

Signed-off-by: Peter Staar <taa@zurich.ibm.com>
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

DCO Check Passed

Thanks @PeterStaar-IBM, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@PeterStaar-IBM PeterStaar-IBM changed the title Dev/add comparison with competing packages perf: comparison with competing packages Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant