Add font-aware decoder for LaTeX math glyphs in PDFs

[jonasvq]

Problem

PDFs compiled with pdflatex embed Computer Modern math fonts (CMEX/CMSY/CMMI) without a ToUnicode CMap. pdfminer can't map their glyphs, so math comes out as unreadable (cid:12)(cid:104)… tokens — sums, integrals, roots, norms, delimiters, and Greek are all lost.

Fix

A font-aware CID decoder:

• build_cid_map makes one low-level pdfminer pass, reading each unmapped glyph's fontname so the correct per-font encoding table is used (cid:12 is | in CMEX10 but a different glyph elsewhere — resolution must be font-keyed).
• decode_cids substitutes the tokens, grouping them into clusters (≈ one formula). Each cluster gets a confidence = resolved/total; clusters below 0.6 are wrapped as <!-- FORMULA: ... --> instead of being half-mistranslated. Unknown (font, cid) pairs are never guessed.
• Tables verified glyph-by-glyph against the embedded font encodings; design-size and Latin Modern variants (CMSY8, LMMI10, …) map to the same family table.

Behavior

On by default; pass decode_cid=False to keep the raw tokens. This changes default PDF output for math documents.

Tests

test_pdf_cid.py + a generated test_math_cid.pdf fixture: default-on decoding, opt-out, prose-unaffected regression, clean-Unicode-not-corrupted, and CMSY/CMMI table coverage. Smoke vector added.

MarkItDown — LaTeX math PDF (before)

Default output without CID decoding. pdfminer cannot map the Computer Modern
math glyphs, so they leak through as raw (cid:N) tokens.

Induced norm

∥x∥ = (cid:112)⟨x, x⟩ = (cid:32) (cid:88) |x_j|2 (cid:33)1/2

Spectral theorem

A = (cid:90) λ dE(λ)

Variational problem

J[u] = (cid:90) ½ ∥(cid:114)u∥2 − f u dx, (cid:114)2u = f

Inner product of a tensor product

(cid:42) (cid:81) (v_i (cid:10) w_i) (cid:43) = (cid:34) (cid:83) S_k (cid:35) ∩ { z : |z| ≤ 1 }

MarkItDown — LaTeX math PDF (after)

Output with CID decoding (now on by default). Each (cid:N) is resolved
font-aware through the Computer Modern encoding tables to real Unicode.

Induced norm

∥x∥ = √⟨x, x⟩ = ( ∑ |x_j|2 )1/2

Spectral theorem

A = ∫ λ dE(λ)

Variational problem

J[u] = ∫ ½ ∥∇u∥2 − f u dx, ∇2u = f

Inner product of a tensor product

⟨ ∏ (v_i ⊗ w_i) ⟩ = [ ∪ S_k ] ∩ { z : |z| ≤ 1 }

🤖 Generated with Claude Code.

[jonasvq]

@microsoft-github-policy-service agree