Skip to content

Split up doc collection and data extraction#447

Draft
ppinchuk wants to merge 88 commits into
mainfrom
pp/split_doc_collection
Draft

Split up doc collection and data extraction#447
ppinchuk wants to merge 88 commits into
mainfrom
pp/split_doc_collection

Conversation

@ppinchuk
Copy link
Copy Markdown
Collaborator

Add new CLI commands that run document collection and data extraction independently, allowing different parts of the pipeline to run in different places

@ppinchuk ppinchuk self-assigned this May 27, 2026
@ppinchuk ppinchuk added enhancement Update to logic or general code improvements breaking Breaks something in the API or config p-critical Priority: critical topic-python-async Issues/pull requests related to python async code labels May 27, 2026
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.69880% with 273 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.51%. Comparing base (eab9df2) to head (3c8c3da).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
compass/pipeline/collection/steps.py 42.55% 73 Missing and 8 partials ⚠️
compass/_cli/common.py 55.76% 43 Missing and 3 partials ⚠️
compass/pipeline/runtime.py 72.90% 34 Missing and 8 partials ⚠️
compass/pipeline/coordinator.py 81.92% 21 Missing and 9 partials ⚠️
compass/scripts/download.py 10.52% 17 Missing ⚠️
compass/pipeline/jurisdiction.py 86.58% 9 Missing and 2 partials ⚠️
compass/pipeline/collection/persistence.py 89.02% 7 Missing and 2 partials ⚠️
compass/pipeline/extraction.py 73.33% 4 Missing and 4 partials ⚠️
compass/pipeline/collection/dedupe.py 79.31% 4 Missing and 2 partials ⚠️
compass/services/threaded.py 86.48% 2 Missing and 3 partials ⚠️
... and 7 more

❌ Your patch status has failed because the patch coverage (74.69%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #447      +/-   ##
==========================================
+ Coverage   56.11%   60.51%   +4.39%     
==========================================
  Files          63       73      +10     
  Lines        6080     6595     +515     
  Branches      591      641      +50     
==========================================
+ Hits         3412     3991     +579     
+ Misses       2598     2496     -102     
- Partials       70      108      +38     
Flag Coverage Δ
unittests 60.51% <74.69%> (+4.39%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking Breaks something in the API or config enhancement Update to logic or general code improvements p-critical Priority: critical topic-python-async Issues/pull requests related to python async code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants