Fix #1582: fixes to reading PDF files larger than 2 GB#1591
Open
andreasrosdalw wants to merge 3 commits into
Open
Fix #1582: fixes to reading PDF files larger than 2 GB#1591andreasrosdalw wants to merge 3 commits into
andreasrosdalw wants to merge 3 commits into
Conversation
File positions were tracked as int in RandomAccessFileOrArray, PRTokeniser, PRStream and the PdfReader xref table, so offsets beyond Integer.MAX_VALUE overflowed and reading a >2 GB PDF failed with 'Rebuild failed: Position out of bounds; Original message: PDF startxref not found' (or scanned the file for a very long time before failing). Switch the read path to 64-bit offsets: - RandomAccessFileOrArray: length(), getFilePointer(), startOffset and skip() are now long-based; seek(long) is the real implementation. - PRTokeniser: seek/getFilePointer/length/getStartxref use long; new longValue() for parsing 10-digit offsets. - PdfReader: xref table is long[], lastXref/eofPos/fileLength are long, xref-stream type-1 offsets read as long, objStmToOffset maps to Long. - PRStream: stream offset is long. - PdfNumber: new longValue() accessor. - PdfWriter: prevxref is long so incremental updates of large files write a correct /Prev. - MappedRandomAccessFile was already long-based (LongMappedByteBuffer). Font-parsing call sites (TrueTypeFont, Type1Font, EnumerateTTC, CFFFont) updated for the new signatures; font files remain int-sized. The regression test builds a sparse PDF whose objects live above the 2 GB boundary (a few KB of real data), so it runs in under a second; a 30s timeout guards against regressions, which would manifest as long scans rather than fast failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
andreasrosdalw
changed the title
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 11 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
- RandomAccessFileOrArray.read()/read(byte[],int,int): call insureOpen() before accessing the backing file, fixing a potential NPE when reading after close() (Sonar blocker; getFilePointer/length/seek already did this). - PdfReader.readXRefStream: extract readXRefStreamEntry() and markObjStmEntry() helpers to reduce cognitive complexity. - PdfReader.checkPRStreamLength: rename local that shadowed the fileLength field. - LargeFilePdfReaderTest: build xref entries without String.format; the PDF xref format mandates LF, not the platform line separator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- RandomAccessFileOrArray: introduce openPlainFile()/openMappedFile() accessors that reopen and null-check the backing file, throwing a descriptive IOException instead of NPE when the object is closed and cannot be reopened. All backing-file accesses (read, length, seek, getFilePointer, getNioByteBuffer) go through them. - PdfReader.readXRefStream: extract readXRefStreamObject() so the method's cognitive complexity is within the allowed limit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1582.
Reading a PDF larger than ~2 GB failed with
InvalidPdfException: Rebuild failed: Position out of bounds; Original message: PDF startxref not found(after potentially scanning the file for a very long time). File positions were tracked asintinRandomAccessFileOrArray,PRTokeniser,PRStreamand thePdfReaderxref table, so any offset beyondInteger.MAX_VALUEoverflowed.Changes
The read path now uses 64-bit offsets end to end (the same migration iText 5 did for large-file support):
RandomAccessFileOrArray:length(),getFilePointer(),getStartOffset()/setStartOffset()andskip()arelong-based;seek(long)is the real implementation (seek(int)delegates). The backingMappedRandomAccessFilewas alreadylong-based viaLongMappedByteBuffer.PRTokeniser:seek/getFilePointer/length/getStartxrefuselong; newlongValue()parses 10-digit offsets.PdfReader: the xref table islong[],lastXref/eofPos/fileLengtharelong, xref-stream type-1 offsets are read aslong,objStmToOffsetis nowMap<Integer, Long>,rebuildXref()collectslongpositions.getLastXref(),getEofPos(),getFileLength()returnlong.PRStream: the stream offset islong.PdfNumber: newlongValue()accessor (used forstartxref,/Prev,/XRefStm).PdfWriter:prevxrefislong, so incremental updates (PdfStamperin append mode) of large files write a correct/Prev.TrueTypeFont,Type1Font,EnumerateTTC,CFFFont) updated for the new signatures; font files themselves remain int-sized.API note
Some
protectedfields and public getters changed frominttolong(PdfReader.xref,getLastXref(),getEofPos(),getFileLength(),RandomAccessFileOrArray.length()/getFilePointer(),PRStream.getOffset()). Callers passingintpositions still compile via widening; only code storing these values intointneeds a cast. This mirrors iText 5's large-file migration.Tests
LargeFilePdfReaderTestbuilds a sparse PDF whose objects live above the 2 GB boundary — only a few KB of real data are written, so the tests run in ~0.5 s:filePointerAndLengthBeyond2GB:length(),seek(),getFilePointer()andread()at offsets beyond 2^31.readPdfLargerThan2GB: fullPdfReaderopen of a >2 GB PDF (partial-read path with plain random access, as in the issue report), asserting page count, page size andgetLastXref().The class carries a 30 s
@Timeoutbecause a regression here manifests as a long file scan rather than a fast failure. A skip-guard (assumeTrue) avoids running on filesystems without sparse-file support / insufficient disk. Fullopenpdf-coresuite passes (2087 tests, 0 failures), and all modules compile.