clean_contents(sanitize=True) drops glyph positioning, duplicating an overprinted leading glyph (regression in 1.28.0 from 1.27.2.3)

[dmontaner-sme]

Description of the bug

Page.apply_redactions() corrupts unredacted text: the leading glyph of a
text span is duplicated when the page content stream is regenerated. It
reproduces even when the redaction removes nothing — a dummy annotation over an
empty area with every removal flag set to *_NONE — so it is the content-stream
regeneration itself, not the removal, that introduces the extra glyph.

The trigger is a span drawn with an "overprint" construct: a glyph is shown, then
0 0 TD moves back to the line-matrix origin, then a word is shown starting at
that same x. The two draws overlap exactly, so the page correctly reads faeces.
After apply_redactions() regenerates the stream, the draws no longer coincide
and both become visible, yielding ffaeces.

The duplication is not cosmetic: after redaction the two f glyphs render at
distinct x-origins (confirmed via get_text("rawdict")), so both the rendered
page and the extracted text show the extra letter.

This is a regression — correct in every release from 1.24.14 through
1.27.2.3, broken only in 1.28.0 (MuPDF 1.29.0):

PyMuPDF	MuPDF	text after no-op apply_redactions()
1.24.14	1.24.11	faeces (correct)
1.26.7	1.26.12	faeces (correct)
1.27.1	1.27.1	faeces (correct)
1.27.2	1.27.2	faeces (correct)
1.27.2.3	1.27.2	faeces (correct)
1.28.0	1.29.0	ffaeces (leading glyph duplicated)

Environment: PyMuPDF 1.28.0 / MuPDF 1.29.0 (broken), last good 1.27.2.3 /
MuPDF 1.27.2; Python 3.13.13; Linux x86-64. The construct is emitted by
real-world producers (first hit on an InDesign-generated PDF, where the extra
glyph landed inside a math formula). Looks related to the MuPDF 1.29.0 text/font
content-stream changes also behind the other 1.28.0 regressions (#5049, #5052).

How to reproduce the bug

Fully self-contained. Step 1 builds a ~600-byte PDF with the standard library
only (Helvetica base-14, no embedded fonts, no third-party content); step 2 needs
PyMuPDF.

Step 1 — make_repro_pdf.py (writes overprint.pdf):

"""Write a minimal, content-free PDF that triggers the redaction glyph-duplication bug.

The page draws the letter 'f' twice using the "overprint" construct:

    (f)Tj        # draw a lone 'f' at the line-matrix origin
    0 0 TD       # TD moves relative to the LINE matrix, not the pen -> back to origin
    (faeces)Tj   # draw 'faeces' starting at the SAME x -> its 'f' overprints the lone 'f'

Both f's paint at the same position, so the page shows a single, correct
"faeces" and text extraction returns "faeces".
"""

import sys

CONTENT = b"BT\n/F1 24 Tf\n1 0 0 1 100 700 Tm\n(f)Tj\n0 0 TD\n(faeces)Tj\nET\n"

OBJECTS = [
    b"<< /Type /Catalog /Pages 2 0 R >>",
    b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
    b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 300 800] "
    b"/Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >>",
    b"<< /Length %d >>\nstream\n%s\nendstream" % (len(CONTENT), CONTENT),
    b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
]


def build() -> bytes:
    pdf = b"%PDF-1.7\n"
    offsets = []
    for i, body in enumerate(OBJECTS, start=1):
        offsets.append(len(pdf))
        pdf += b"%d 0 obj\n%s\nendobj\n" % (i, body)
    xref_pos = len(pdf)
    pdf += b"xref\n0 %d\n" % (len(OBJECTS) + 1)
    pdf += b"0000000000 65535 f \n"
    for off in offsets:
        pdf += b"%010d 00000 n \n" % off
    pdf += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(OBJECTS) + 1, xref_pos)
    return pdf


if __name__ == "__main__":
    out = sys.argv[1] if len(sys.argv) > 1 else "overprint.pdf"
    with open(out, "wb") as fh:
        fh.write(build())
    print(f"wrote {out}")

Step 2 — repro.py (extract, no-op redact, extract again):

import pymupdf

doc = pymupdf.open("overprint.pdf")
page = doc[0]

before = page.get_text().strip()

# A redaction that targets nothing: empty corner, every removal flag NONE.
page.add_redact_annot(pymupdf.Rect(2, 2, 5, 5))
page.apply_redactions(
    text=pymupdf.PDF_REDACT_TEXT_NONE,
    images=pymupdf.PDF_REDACT_IMAGE_NONE,
    graphics=pymupdf.PDF_REDACT_LINE_ART_NONE,
)
after = page.get_text().strip()

print(f"PyMuPDF {pymupdf.version[0]} / MuPDF {pymupdf.version[1]}")
print(f"  before redaction: {before!r}")
print(f"  after  redaction: {after!r}")

Run:

python make_repro_pdf.py overprint.pdf
python repro.py

Observed on 1.28.0 (expected: after stays 'faeces'):

PyMuPDF 1.28.0 / MuPDF 1.29.0
  before redaction: 'faeces'
  after  redaction: 'ffaeces'

PyMuPDF version

1.28.0

Operating system

Linux

Python version

3.13

[JorjMcKie]

Thanks for submitting this. Confirming the behavior.
Please note:
The problem has nothing to do with redactions! I am not sure whether it is an actual problem.

import pymupdf
doc=pymupdf.open("overprint.pdf")
page=doc[0]
print(page.read_contents().decode())  # print the page's Contents stream
BT
/F1 24 Tf
1 0 0 1 100 700 Tm
(f)Tj
0 0 TD
(faeces)Tj
ET

page.clean_contents()  # clean / rebuild the Contents stream
print(page.read_contents().decode())  # print it again:
q BT/F1 24 Tf 100 700 TD (f)Tj (faeces)Tj ET Q

Applying redactions inevitably rebuilds a page's Contents stream - just like method .clean_contents(sanitize=True).
Rebuilding / sanitizing a Contents stream should not cause text extractions produce different results.
In its shortest form, the actual problem is this:

doc=pymupdf.open("overprint.pdf")
page=doc[0]
page.get_text()
'faeces\n'
page.clean_contents(sanitize=True)  # "sanitize" is the default
page.get_text()
'ffaeces\n'

[JorjMcKie]

This is the link to the connected MuPDF issue: https://bugs.ghostscript.com/show_bug.cgi?id=709556

[JorjMcKie]

Would you agree to change the issue title? Redactions are irrelevant, mentioning them blurs the actual cause. It should rather read something like
"Clean-Sanitize removes glyph positions ..."
This would help identifying the problem at a later time.

[dmontaner-sme]

Thanks for the quick response Jorj. Sure, I have renamed the title. Let me know if it makes sense now or if you would prefer any other.
And thanks for the package !!!