Skip to content

@lde/pipeline core: transaction-aware Writer, Reader, generic Stage<In,Out> #535

Description

@ddeboer

Foundational core change for the search-as-pipeline design (epic: #534). Breaking; unblocks the rest.

Scope

  • Transaction-aware Writer: Writer.openRun(ctx) -> RunWriter { write, commit, abort } + RunContext { runId, startedAt, selectedSources(), provenance? }. Pipeline.run drives openRun -> write* -> commit/abort uniformly and never branches on mode.
  • Executor -> Reader rename (the quad-producing input unit). Input is a dataset description or a distribution; non-RDF backing via a SPARQL-Anything reader (still yields quads).
  • Generic Stage<In, Out> (small generalization of the executor-only Stage); typed composition so invalid pipelines (a quad stage after the projection) are compile errors.
  • Migrate fileWriter + sparqlUpdateWriter to the transactional Writer; re-touch the DKG (new Pipeline({...}) wiring).
  • A lifecycle-free writer uses a no-op helper, not a second interface.

Not in scope

Engine-specific sinks (follow-up), projection changes.

Acceptance

  • DKG + crawler run on the new interface; docker:smoke green.
  • Typed composition rejects a quad stage after the projection at compile time.

Design and rationale: #534.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions