docs: add a diagram to the fan-out ingestion explanation - #268
Merged
Conversation
The page described the pattern in prose without showing it. Fan-out is the one shape in the model where the picture carries information the prose has to work hard for: readers know how to read a dependency graph, and the whole point here is a write the dependency graph does not contain. The figure is hand-built rather than dj.Diagram output, which is a departure from every other diagram in the docs and needs the justification the script carries. dj.Diagram cannot draw this: pointed at the page's own schema it renders Subject, Session and Recording as three unconnected nodes, because there is no foreign key to follow. The absence is the subject. So the notation is borrowed exactly rather than emitted, and the script reads its values from what dj.Diagram itself emits (datajoint-python #1544) rather than picking them: tier by shape and color, navy dependency edges with no arrowheads, edge weight carrying cardinality, an underlined name marking a table that introduces a primary-key attribute of its own. Two details do real teaching work here -- RecordingFile to Ingest is thick because Ingest declares only that one foreign key, so it covers its entire primary key, and Ingest is the only unlined name in the figure because it introduces no key of its own. The one deliberate departure is the point of the figure: fan-out writes are drawn dashed and with arrowheads, since a genuine dependency edge has none, so the arrowhead is what tells the reader this is not one. The bronze reserved for renamed foreign keys is avoided -- these are not foreign keys at all. Tables and the source_file attribute match the code sample above the figure, so the two can be read against each other. Dark palette follows prefers-color-scheme, consistent with the other committed figures, and so tracks the reader's OS rather than the site's own theme toggle -- an img-embedded SVG cannot see the page's data-md-color-scheme. Noted in the script, not solved here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
src/explanation/fan-out-ingestion.mdexplained the pattern in prose with no figure. Fan-out is the one shape in the model where a picture earns its place, because readers already know how to read a dependency graph and the whole point here is a write the dependency graph does not contain.Why it is hand-built
Every other diagram in these docs is
dj.Diagramoutput, and this one is not. That needs justifying, so the generator script carries the reason:dj.Diagramcannot draw this figure. Pointed at the page's own schema it rendersSubject,Session, andRecordingas three unconnected nodes, because there is no foreign key for it to follow. The absence is the subject.So the notation is borrowed rather than emitted, and the script reads its values from what
dj.Diagramitself emits (datajoint-python #1544) rather than inventing any:Manuala green rounded box,Importeda blue ellipse.Two of those do real teaching work in this figure.
RecordingFile → Ingestis thick becauseIngestdeclares only-> RecordingFile, so that key covers its entire primary key. AndIngestis the only un-underlined name, because it introduces no key attribute of its own — which is exactly why it can be an ingestion step without distorting the domain model.The one deliberate departure
Fan-out writes are drawn dashed and with arrowheads. A genuine dependency edge has no arrowhead, so the arrowhead is the tell that this is not one. I avoided the bronze reserved for renamed foreign keys — these are not foreign keys at all.
Tables and the
source_fileattribute match the code sample directly above the figure, so the two read against each other.Notes
python scripts/gen_fanout_diagram.pyregenerates it. No database and no graphviz needed, unlikegen_pipeline_diagrams.py.normalization.mdandrelational-workflow-model.md.prefers-color-scheme, consistent with the other committed figures — which means it tracks the reader's OS rather than the site's own light/dark toggle, since an<img>-embedded SVG cannot see the page'sdata-md-color-scheme. Pre-existing across all the diagrams here; noted in the script, not solved in this PR.