Tika 4771 composable document

[krickert]

TIKA-4771 PR draft

Title: TIKA-4771: composable Document design sketch (discussion only, no code)

Base: apache/tika:main ← Head: ai-pipestream:TIKA-4771-composable-document · Draft PR

---

This is the API design for tika-grpc/TIKA-4771-composable-document.md. It's the API for external parser results into the typed Document, as a starting point for discussion on TIKA-4771.

@dpol1's did this and I provided a few rounds of feedback - I like it. It's simple, flexible, and non-invasive. So I'm opening a PR for @dpol1 since he did it on my branch.

tldr

Shows a composer (a StormCrawler bolt today, a coordinator service later) sends the same bytes to Tika ParseBytes and to any external parsers.

Doing a claim-check pattern. Each external parser returns an ExtensionResult envelope: producer_id, status, a google.protobuf.Any payload with a producer-owned schema.

The composer merges everything into one Document via repeated ExtensionResult extensions = 40, under fixed rules: core/extension ownership is disjoint, input bytes are hash-checked against origin.sha256, duplicates resolve deterministically, and every planned slot is accounted for (OK | EMPTY | FAILED | SKIPPED).

Simple, can bring in multiple suppliers, and minimal grpc surface.

How this relates to the ticket as filed

TIKA-4771 proposes Tika brokering registered external parsers the way fetchers and emitters register today. Since it's grpc and a parser, this allows for parsers to return strongly typed results that are unknown to tika pipes.

ParseStream mode, payload mappings, and a schema registry are all called out as later rounds of work.

Dependencies

• TIKA-4766: minimal typed Document parse contract for tika-grpc (stage 1) #2961 (TIKA-4766) provides the Document contract this extends field 40.
• TIKA-4795 provides the exact-byte path and parity oracle the first demo would use (branch exists, no PR yet).

It's depending on #2961 which is why this is a draft.

Strong feelback encouraged. Thanks!

Thanks for your contribution to Apache Tika! Your help is appreciated!

Before opening the pull request, please verify that

• there is an open issue on the Tika issue tracker which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes.
• the issue ID (TIKA-XXXX)
  • is referenced in the title of the pull request
  • and placed in front of your commit messages surrounded by square brackets ([TIKA-XXXX] Issue or pull request title)
• commits are squashed into a single one (or few commits for larger changes)
• Tika is successfully built and unit tests pass by running ./mvnw clean test
• there should be no conflicts when merging the pull request branch into the recent main branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled main branch
• if you add new module that downstream users will depend upon add it to relevant group in tika-bom/pom.xml.

We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the Tika mailing list. Thanks!