Skip to content

docs: add a diagram to the fan-out ingestion explanation - #268

Merged
dimitri-yatsenko merged 1 commit into
mainfrom
docs/fanout-diagram
Aug 21, 2026
Merged

docs: add a diagram to the fan-out ingestion explanation#268
dimitri-yatsenko merged 1 commit into
mainfrom
docs/fanout-diagram

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

src/explanation/fan-out-ingestion.md explained the pattern in prose with no figure. Fan-out is the one shape in the model where a picture earns its place, because readers already know how to read a dependency graph and the whole point here is a write the dependency graph does not contain.

figure

Why it is hand-built

Every other diagram in these docs is dj.Diagram output, and this one is not. That needs justifying, so the generator script carries the reason: dj.Diagram cannot draw this figure. Pointed at the page's own schema it renders Subject, Session, and Recording as three unconnected nodes, because there is no foreign key for it to follow. The absence is the subject.

So the notation is borrowed rather than emitted, and the script reads its values from what dj.Diagram itself emits (datajoint-python #1544) rather than inventing any:

  • Tier by shape and color — Manual a green rounded box, Imported a blue ellipse.
  • Dependency edges navy, no arrowheads; direction follows the layout.
  • Edge weight carries cardinality.
  • An underlined name introduces a primary-key attribute of its own.

Two of those do real teaching work in this figure. RecordingFile → Ingest is thick because Ingest declares only -> RecordingFile, so that key covers its entire primary key. And Ingest is the only un-underlined name, because it introduces no key attribute of its own — which is exactly why it can be an ingestion step without distorting the domain model.

The one deliberate departure

Fan-out writes are drawn dashed and with arrowheads. A genuine dependency edge has no arrowhead, so the arrowhead is the tell that this is not one. I avoided the bronze reserved for renamed foreign keys — these are not foreign keys at all.

Tables and the source_file attribute match the code sample directly above the figure, so the two read against each other.

Notes

  • python scripts/gen_fanout_diagram.py regenerates it. No database and no graphviz needed, unlike gen_pipeline_diagrams.py.
  • Long descriptive alt text, following the convention in normalization.md and relational-workflow-model.md.
  • Dark palette follows prefers-color-scheme, consistent with the other committed figures — which means it tracks the reader's OS rather than the site's own light/dark toggle, since an <img>-embedded SVG cannot see the page's data-md-color-scheme. Pre-existing across all the diagrams here; noted in the script, not solved in this PR.

The page described the pattern in prose without showing it. Fan-out is the one
shape in the model where the picture carries information the prose has to work
hard for: readers know how to read a dependency graph, and the whole point here
is a write the dependency graph does not contain.

The figure is hand-built rather than dj.Diagram output, which is a departure from
every other diagram in the docs and needs the justification the script carries.
dj.Diagram cannot draw this: pointed at the page's own schema it renders Subject,
Session and Recording as three unconnected nodes, because there is no foreign key
to follow. The absence is the subject.

So the notation is borrowed exactly rather than emitted, and the script reads its
values from what dj.Diagram itself emits (datajoint-python #1544) rather than
picking them: tier by shape and color, navy dependency edges with no arrowheads,
edge weight carrying cardinality, an underlined name marking a table that
introduces a primary-key attribute of its own. Two details do real teaching work
here -- RecordingFile to Ingest is thick because Ingest declares only that one
foreign key, so it covers its entire primary key, and Ingest is the only unlined
name in the figure because it introduces no key of its own.

The one deliberate departure is the point of the figure: fan-out writes are drawn
dashed and with arrowheads, since a genuine dependency edge has none, so the
arrowhead is what tells the reader this is not one. The bronze reserved for
renamed foreign keys is avoided -- these are not foreign keys at all.

Tables and the source_file attribute match the code sample above the figure, so
the two can be read against each other.

Dark palette follows prefers-color-scheme, consistent with the other committed
figures, and so tracks the reader's OS rather than the site's own theme toggle --
an img-embedded SVG cannot see the page's data-md-color-scheme. Noted in the
script, not solved here.
@dimitri-yatsenko
dimitri-yatsenko merged commit 9183106 into main Aug 21, 2026
2 checks passed
@dimitri-yatsenko
dimitri-yatsenko deleted the docs/fanout-diagram branch August 21, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant