Fix #1588: treat unresolvable /Encrypt trailer reference as unencrypted

[andreasrosdalw]

Summary

Fixes #1588.

PdfReader crashed on malformed PDFs whose trailer /Encrypt entry is an indirect reference to an object with no usable xref entry (produced by some home-grown PDF generators). The partial-read constructor (new PdfReader(RandomAccessFileOrArray, byte[])) surfaced a bare NullPointerException from readDecryptedDocObj(); the byte-array/filename constructors wrapped the same NPE in an InvalidPdfException after a failed rebuild.

Root cause

PdfDictionary enc = (PdfDictionary) getPdfObject(encDic);   // returns null for a dangling reference
...
PdfObject filter = getPdfObjectRelease(enc.get(PdfName.FILTER)); // NPE

getPdfObject resolves the /Encrypt reference through the xref table and returns null when the referenced object number has a free or missing entry. enc was never null-checked.

Fix

If the /Encrypt entry does not resolve to a dictionary, treat the document as unencrypted — matching poppler and qpdf, which open this class of file without error and report it as not encrypted. The instanceof check also covers /Encrypt resolving to a non-dictionary object, which previously died with a ClassCastException.

PdfObject encObj = getPdfObject(encDic);
if (!(encObj instanceof PdfDictionary)) {
    encrypted = false;
    encryptionError = false;
    return;
}

Tests

New DanglingEncryptReferenceTest builds the minimal malformed PDF from the issue report (/Size 5, /Encrypt 4 0 R, object 4 has only a free xref entry) and asserts that both the partial-read and full-read paths open the document, report isEncrypted() == false and see the correct page count. Both tests fail on master (2/2) and pass with the fix. Full openpdf-core suite passes.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 4 complexity · 2 duplication

Metric	Results
Complexity	4
Duplication	2

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud