perf: comparison with competing packages - #304
Open
PeterStaar-IBM wants to merge 2 commits into
Open
Conversation
Signed-off-by: Peter Staar <taa@zurich.ibm.com>
…solidate the perf tooling Signed-off-by: Peter Staar <taa@zurich.ibm.com>
Contributor
|
✅ DCO Check Passed Thanks @PeterStaar-IBM, all your commits are properly signed off. 🎉 |
Contributor
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/performance_benchmarks.md had an empty table under Performance comparison on parsing and rendering and no way
to fill it: the perf scripts could time docling-parse against pypdfium2 and pymupdf, but only reported wall time per
run — no per-page distribution, no hardware/version provenance, and no coverage of the other top-5 packages.
Filling that table meant touching most of ./perf, which turned out to hold five scripts with three mutually
incompatible CSV formats and three copies of the third-party backend adapters. This PR does both: adds the
comparison suite, and collapses the tooling to three entry points around one interchange format.
What changed
Six packages, two tasks, reported as a per-page time distribution (mean / median / p95 / p99) plus a wall-time
speedup grid.
Design points worth reviewing:
timed on rawdict rather than get_text("text") — the latter returns a bare string with no geometry and is a
materially cheaper task. Historical pymupdf numbers from the old run_perf.py are therefore not comparable to these.
exposes a thread-safe multi-page pipeline. That is the differentiator the tables are meant to show, so total time
(wall clock over the corpus, including document opens) drops with threads while the per-page distribution stays
flat.
page's work; docling-parse reports the C++ page timing, because under concurrency no wall-clock interval belongs to
a single page. The table has a per-page source column stating which was used, and the two agree at threads=1.
rendered page's pixel size is recorded and compared against pypdfium2 (PDFium being the rasteriser three of the six
packages ultimately rely on). It reports how many pages agree within 2 px and the worst offender.
Deleted: run_perf.py, run_scaling_visualization.py.
task and threads, so one file can hold several packages at several thread counts. --enable-timing/--timing-csv are
gone, their C++ stage columns folded into the single schema. run_eval.py no longer infers the backend from the
filename.
drift.
uv run python ./perf/run_scaling.py --threads 1,4,8,12 --compare all --mode render
--output-dir ./docs/performance_benchmarks/
Writes .md and .csv (e.g. apple_m3_max_performance-dataset-bo767_render.md), so results from
different machines never overwrite each other. The report records the exact command, the dataset name/revision/size,
the machine and every benchmarked package version, the decode/content/render configs the run was driven with, and
all result tables — a published number is traceable without the terminal scrollback it came from. Default output dir
is ./scratch so exploratory runs don't touch the docs.
docs/performance_benchmarks.md gains a methodology section (what each task means, which API each package is timed
on, how the statistics are defined and why the per-page clocks differ) and a reproduction section.
Bugs fixed along the way
between them, and run_analysis.py/run_eval.py consumed that column without a guard. Fixed by deletion; the surviving
path uses C++ page timings.
existed. The group is perf.
wasn't installed. Missing backends now print a visible skip line and are restated before the tables.