Stream PDF from archives - #117
Open
lfoppiano wants to merge 3 commits into
Open
Conversation
Add process_archive(), used automatically when --input points to a
.zip/.tar/.tar.gz (.tgz/.tar.bz2/.tbz2) archive. Eligible entries are read out
of the archive in chunks of batch_size, each chunk is extracted to a temporary
directory, sent to GROBID via the existing process_batch (so concurrency,
TEI/JSON/Markdown output, --force/skip and error handling are reused), and the
temporary files are removed before the next chunk is extracted. The archive is
never fully decompressed, so disk usage stays bounded regardless of its size.
- zip via zipfile, tar/tar.gz/tar.bz2 via tarfile ('r:*')
- entries are streamed member-by-member (zipfile.open / tarfile.extractfile)
- archive paths are sanitized to prevent path traversal (zip slip)
- when --output is omitted, results go to a directory named after the archive
- directory-input eligibility check refactored into _is_eligible_input and shared
Documented in the Readme and covered by unit tests (zip, tar.gz, chunking,
cleanup, default output, traversal guard, delegation).
…rsal --input now accepts shell-style glob patterns (with recursive **), e.g. 'paper.zip' (one file), 'paper*.zip' (many), '**/paper*.zip' (subdirectories) or '**/*.pdf'. Each match is dispatched by type: archives are streamed, directories are recursed, eligible files are processed directly, and the results of all matches are aggregated into a single summary. - resolve --input via glob (has_magic + recursive=True, ~ expansion); a plain path is returned unchanged for backward compatibility - directory traversal refactored from os.walk to pathlib.Path.rglob - factor the batching loop, stats summary and archive streaming into reusable helpers (_run_file_batches, _print_processing_summary, _process_archive_core) so directory, loose-file and archive inputs share one code path - loose files matched by a glob are batched together under their common base Documented in the Readme and covered by unit tests (multi-archive glob, recursive **/*.pdf, mixed matches, no-match warning, common-base helper).
Add s3:// support to --input (and a new --input-list manifest). An s3 zip is range-streamed with smart_open (only the central directory and the requested entries are fetched - the object is never fully downloaded); loose remote PDFs are fetched a batch at a time to a temp dir. Mixed manifests (local + glob + s3) are supported and aggregated into one summary. - refactor process() -> process_paths(list of inputs); process() delegates - _resolve_input_paths handles s3 object / prefix / glob (list_objects_v2 + fnmatch) - _open_archive range-streams s3 zips (smart_open seekable stream); the stream is closed after use; s3 tar is rejected (not range-streamable) - new _process_remote_files streams loose s3 objects in bounded chunks - --input-list reads a file of paths (local/glob/s3, '#' comments) - s3 deps (smart_open[s3], boto3) are an optional 'pip install ...[s3]' extra, lazily imported with a clear install hint if missing - credentials use the standard AWS chain (env / ~/.aws / IAM) Documented in the Readme; covered by moto-backed tests (single object, prefix, glob, zip range-streaming, loose PDFs, mixed manifest, missing-extra error).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TBA