diff --git a/src/tutorials/advanced/sql-comparison.ipynb b/src/tutorials/advanced/sql-comparison.ipynb index 659cbca8..267a5cef 100644 --- a/src/tutorials/advanced/sql-comparison.ipynb +++ b/src/tutorials/advanced/sql-comparison.ipynb @@ -1256,19 +1256,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": [ - "### Algebraic Closure\n", - "\n", - "In standard SQL, query results are just \"bags of rows\" — they don't have a defined entity type. You cannot know what kind of thing each row represents without external context.\n", - "\n", - "DataJoint achieves **algebraic closure**: every query result is a valid entity set with a well-defined **entity type**. You always know what kind of entity the result represents, identified by a specific primary key. This means:\n", - "\n", - "1. **Every operator returns a valid relation** — not just rows, but a set of entities of a known type\n", - "2. **Operators compose indefinitely** — you can chain any sequence of operations\n", - "3. **Results remain queryable** — a query result can be used as an operand in further operations\n", - "\n", - "The entity type (and its primary key) is determined by precise rules based on the operator and the functional dependencies between operands. See the [Primary Keys specification](../../reference/specs/primary-keys.md) for details." - ] + "source": "### Algebraic Closure\n\nIn standard SQL, query results are just \"bags of rows\" — they don't have a defined entity type. You cannot know what kind of thing each row represents without external context.\n\nDataJoint achieves **algebraic closure**: every query result is a valid entity set with a well-defined **entity type**. You always know what kind of entity the result represents, identified by a specific primary key. This means:\n\n1. **Every operator returns a valid relation** — not just rows, but a set of entities of a known type\n2. **Operators compose indefinitely** — you can chain any sequence of operations\n3. **Results remain queryable** — a query result can be used as an operand in further operations\n\nThe entity type (and its primary key) is determined by precise rules based on the operator and the functional dependencies between operands. See the [Primary Keys specification](../../reference/specs/primary-keys) for details." }, { "cell_type": "markdown", @@ -3101,4 +3089,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} +} \ No newline at end of file diff --git a/src/tutorials/basics/01-first-pipeline.ipynb b/src/tutorials/basics/01-first-pipeline.ipynb index 2116c148..7f442ab8 100644 --- a/src/tutorials/basics/01-first-pipeline.ipynb +++ b/src/tutorials/basics/01-first-pipeline.ipynb @@ -3,7 +3,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "# A Simple Pipeline\n\nThis tutorial introduces DataJoint by building a simple research lab database. You'll learn to:\n\n- Define tables with primary keys and dependencies\n- Insert and query data\n- Use the four core operations: restriction, projection, join, aggregation\n- Understand the schema diagram\n\nWe'll work with **Manual tables** only—tables where you enter data directly. Later tutorials introduce automated computation.\n\nFor complete working examples, see:\n- [University Database](../examples/university/) — Academic records with complex queries\n- [Blob Detection](../examples/blob-detection/) — Image processing with computation" + "source": "# A Simple Pipeline\n\nThis tutorial introduces DataJoint by building a simple research lab database. You'll learn to:\n\n- Define tables with primary keys and dependencies\n- Insert and query data\n- Use the four core operations: restriction, projection, join, aggregation\n- Understand the schema diagram\n\nWe'll work with **Manual tables** only—tables where you enter data directly. Later tutorials introduce automated computation.\n\nFor complete working examples, see:\n- [University Database](../../examples/university) — Academic records with complex queries\n- [Blob Detection](../../examples/blob-detection) — Image processing with computation" }, { "cell_type": "markdown", @@ -2683,7 +2683,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "## Summary\n\nYou've learned the fundamentals of DataJoint:\n\n| Concept | Description |\n|---------|-------------|\n| **Tables** | Python classes with a `definition` string |\n| **Primary key** | Above `---`, uniquely identifies rows |\n| **Dependencies** | `->` creates foreign keys |\n| **Restriction** | `&` filters rows |\n| **Projection** | `.proj()` selects/computes columns |\n| **Join** | `*` combines tables |\n| **Aggregation** | `.aggr()` summarizes groups |\n\n### Next Steps\n\n- [Schema Design](02-schema-design/) — Primary keys, relationships, table tiers\n- [Queries](04-queries/) — Advanced query patterns\n- [Computation](05-computation/) — Automated processing with Imported/Computed tables\n\n### Complete Examples\n\n- [University Database](../examples/university/) — Complex queries on academic records\n- [Blob Detection](../examples/blob-detection/) — Image processing pipeline with computation" + "source": "## Summary\n\nYou've learned the fundamentals of DataJoint:\n\n| Concept | Description |\n|---------|-------------|\n| **Tables** | Python classes with a `definition` string |\n| **Primary key** | Above `---`, uniquely identifies rows |\n| **Dependencies** | `->` creates foreign keys |\n| **Restriction** | `&` filters rows |\n| **Projection** | `.proj()` selects/computes columns |\n| **Join** | `*` combines tables |\n| **Aggregation** | `.aggr()` summarizes groups |\n\n### Next Steps\n\n- [Schema Design](../02-schema-design) — Primary keys, relationships, table tiers\n- [Queries](../04-queries) — Advanced query patterns\n- [Computation](../05-computation) — Automated processing with Imported/Computed tables\n\n### Complete Examples\n\n- [University Database](../../examples/university) — Complex queries on academic records\n- [Blob Detection](../../examples/blob-detection) — Image processing pipeline with computation" }, { "cell_type": "code", diff --git a/src/tutorials/basics/02-schema-design.ipynb b/src/tutorials/basics/02-schema-design.ipynb index a2b7904b..c2e79172 100644 --- a/src/tutorials/basics/02-schema-design.ipynb +++ b/src/tutorials/basics/02-schema-design.ipynb @@ -1530,7 +1530,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "## Best Practices\n\n### 1. Choose Meaningful Primary Keys\n- Use natural identifiers when possible (`subject_id = 'M001'`)\n- Keep keys minimal but sufficient for uniqueness\n\n### 2. Use Appropriate Table Tiers\n- **Manual**: Data entered by operators or instruments\n- **Lookup**: Configuration, parameters, reference data\n- **Imported**: Data read from files (recordings, images)\n- **Computed**: Derived analyses and summaries\n\n### 3. Normalize Your Data\n- Don't repeat information across rows\n- Create separate tables for distinct entities\n- Use foreign keys to link related data\n\n### 4. Use Core DataJoint Types\n\nDataJoint has a three-layer type architecture (see [Type System Specification](../../reference/specs/type-system/)):\n\n1. **Native database types** (Layer 1): Backend-specific types like `INT`, `FLOAT`, `TINYINT UNSIGNED`. These are **discouraged** but allowed for backward compatibility.\n\n2. **Core DataJoint types** (Layer 2): Standardized, scientist-friendly types that work identically across MySQL and PostgreSQL. **Always prefer these.**\n\n3. **Codec types** (Layer 3): Types with `encode()`/`decode()` semantics like ``, ``, ``.\n\n**Core types used in this tutorial:**\n\n| Type | Description | Example |\n|------|-------------|---------|\n| `uint8`, `uint16`, `int32` | Sized integers | `session_idx : uint16` |\n| `float32`, `float64` | Sized floats | `reaction_time : float32` |\n| `varchar(n)` | Variable-length string | `name : varchar(100)` |\n| `bool` | Boolean | `correct : bool` |\n| `date` | Date only | `date_of_birth : date` |\n| `datetime` | Date and time (UTC) | `created_at : datetime` |\n| `enum(...)` | Enumeration | `sex : enum('M', 'F', 'U')` |\n| `json` | JSON document | `task_params : json` |\n| `uuid` | Universally unique ID | `experimenter_id : uuid` |\n\n**Why native types are allowed but discouraged:**\n\nNative types (like `int`, `float`, `tinyint`) are passed through to the database but generate a **warning at declaration time**. They are discouraged because:\n- They lack explicit size information\n- They are not portable across database backends\n- They are not recorded in field metadata for reconstruction\n\nIf you see a warning like `\"Native type 'int' used; consider 'int32' instead\"`, update your definition to use the corresponding core type.\n\n### 5. Document Your Tables\n- Add comments after `#` in definitions\n- Document units in attribute comments\n\n## Key Concepts Recap\n\n| Concept | Description |\n|---------|-------------|\n| **Primary Key** | Attributes above `---` that uniquely identify rows |\n| **Secondary Attributes** | Attributes below `---` that store additional data |\n| **Foreign Key** (`->`) | Reference to another table, imports its primary key |\n| **One-to-Many** | FK in primary key: parent has many children |\n| **One-to-One** | FK is entire primary key: exactly one child per parent |\n| **Master-Part** | Compositional integrity: master and parts inserted/deleted atomically |\n| **Nullable FK** | `[nullable]` makes the reference optional |\n| **Lookup Table** | Pre-populated reference data |\n\n## Next Steps\n\n- [Data Entry](03-data-entry/) — Inserting, updating, and deleting data\n- [Queries](04-queries/) — Filtering, joining, and projecting\n- [Computation](05-computation/) — Building computational pipelines" + "source": "## Best Practices\n\n### 1. Choose Meaningful Primary Keys\n- Use natural identifiers when possible (`subject_id = 'M001'`)\n- Keep keys minimal but sufficient for uniqueness\n\n### 2. Use Appropriate Table Tiers\n- **Manual**: Data entered by operators or instruments\n- **Lookup**: Configuration, parameters, reference data\n- **Imported**: Data read from files (recordings, images)\n- **Computed**: Derived analyses and summaries\n\n### 3. Normalize Your Data\n- Don't repeat information across rows\n- Create separate tables for distinct entities\n- Use foreign keys to link related data\n\n### 4. Use Core DataJoint Types\n\nDataJoint has a three-layer type architecture (see [Type System Specification](../../reference/specs/type-system/)):\n\n1. **Native database types** (Layer 1): Backend-specific types like `INT`, `FLOAT`, `TINYINT UNSIGNED`. These are **discouraged** but allowed for backward compatibility.\n\n2. **Core DataJoint types** (Layer 2): Standardized, scientist-friendly types that work identically across MySQL and PostgreSQL. **Always prefer these.**\n\n3. **Codec types** (Layer 3): Types with `encode()`/`decode()` semantics like ``, ``, ``.\n\n**Core types used in this tutorial:**\n\n| Type | Description | Example |\n|------|-------------|---------|\n| `uint8`, `uint16`, `int32` | Sized integers | `session_idx : uint16` |\n| `float32`, `float64` | Sized floats | `reaction_time : float32` |\n| `varchar(n)` | Variable-length string | `name : varchar(100)` |\n| `bool` | Boolean | `correct : bool` |\n| `date` | Date only | `date_of_birth : date` |\n| `datetime` | Date and time (UTC) | `created_at : datetime` |\n| `enum(...)` | Enumeration | `sex : enum('M', 'F', 'U')` |\n| `json` | JSON document | `task_params : json` |\n| `uuid` | Universally unique ID | `experimenter_id : uuid` |\n\n**Why native types are allowed but discouraged:**\n\nNative types (like `int`, `float`, `tinyint`) are passed through to the database but generate a **warning at declaration time**. They are discouraged because:\n- They lack explicit size information\n- They are not portable across database backends\n- They are not recorded in field metadata for reconstruction\n\nIf you see a warning like `\"Native type 'int' used; consider 'int32' instead\"`, update your definition to use the corresponding core type.\n\n### 5. Document Your Tables\n- Add comments after `#` in definitions\n- Document units in attribute comments\n\n## Key Concepts Recap\n\n| Concept | Description |\n|---------|-------------|\n| **Primary Key** | Attributes above `---` that uniquely identify rows |\n| **Secondary Attributes** | Attributes below `---` that store additional data |\n| **Foreign Key** (`->`) | Reference to another table, imports its primary key |\n| **One-to-Many** | FK in primary key: parent has many children |\n| **One-to-One** | FK is entire primary key: exactly one child per parent |\n| **Master-Part** | Compositional integrity: master and parts inserted/deleted atomically |\n| **Nullable FK** | `[nullable]` makes the reference optional |\n| **Lookup Table** | Pre-populated reference data |\n\n## Next Steps\n\n- [Data Entry](../03-data-entry) — Inserting, updating, and deleting data\n- [Queries](../04-queries) — Filtering, joining, and projecting\n- [Computation](../05-computation) — Building computational pipelines" }, { "cell_type": "code", diff --git a/src/tutorials/basics/03-data-entry.ipynb b/src/tutorials/basics/03-data-entry.ipynb index 4b76ae72..3525ad48 100644 --- a/src/tutorials/basics/03-data-entry.ipynb +++ b/src/tutorials/basics/03-data-entry.ipynb @@ -1588,7 +1588,7 @@ "cell_type": "markdown", "id": "cell-42", "metadata": {}, - "source": "## Quick Reference\n\n| Operation | Method | Use Case |\n|-----------|--------|----------|\n| Insert one | `insert1(row)` | Adding single entity |\n| Insert many | `insert(rows)` | Bulk data loading |\n| Update one | `update1(row)` | Surgical corrections only |\n| Delete | `delete()` | Removing entities (cascades) |\n| Delete quick | `delete_quick()` | Internal cleanup (no cascade) |\n| Validate | `validate(rows)` | Pre-insert check |\n\nSee the [Data Manipulation Specification](../../reference/specs/data-manipulation/) for complete details.\n\n## Next Steps\n\n- [Queries](04-queries/) — Filtering, joining, and projecting data\n- [Computation](05-computation/) — Building computational pipelines" + "source": "## Quick Reference\n\n| Operation | Method | Use Case |\n|-----------|--------|----------|\n| Insert one | `insert1(row)` | Adding single entity |\n| Insert many | `insert(rows)` | Bulk data loading |\n| Update one | `update1(row)` | Surgical corrections only |\n| Delete | `delete()` | Removing entities (cascades) |\n| Delete quick | `delete_quick()` | Internal cleanup (no cascade) |\n| Validate | `validate(rows)` | Pre-insert check |\n\nSee the [Data Manipulation Specification](../../reference/specs/data-manipulation) for complete details.\n\n## Next Steps\n\n- [Queries](../04-queries) — Filtering, joining, and projecting data\n- [Computation](../05-computation) — Building computational pipelines" }, { "cell_type": "code", diff --git a/src/tutorials/basics/04-queries.ipynb b/src/tutorials/basics/04-queries.ipynb index 12d4fcdf..2695ad73 100644 --- a/src/tutorials/basics/04-queries.ipynb +++ b/src/tutorials/basics/04-queries.ipynb @@ -965,13 +965,7 @@ "cell_type": "markdown", "id": "cell-11", "metadata": {}, - "source": [ - "### Restriction by Query Expression\n", - "\n", - "Restrict by another query expression. DataJoint uses **semantic matching**: attributes with the same name are matched only if they share the same origin through foreign key lineage. This prevents accidental matches on unrelated attributes that happen to share names (like generic `id` columns in unrelated tables).\n", - "\n", - "See [Semantic Matching](../reference/specs/semantic-matching.md) for the full specification." - ] + "source": "### Restriction by Query Expression\n\nRestrict by another query expression. DataJoint uses **semantic matching**: attributes with the same name are matched only if they share the same origin through foreign key lineage. This prevents accidental matches on unrelated attributes that happen to share names (like generic `id` columns in unrelated tables).\n\nSee [Semantic Matching](../../reference/specs/semantic-matching) for the full specification." }, { "cell_type": "code", @@ -3991,7 +3985,7 @@ "cell_type": "markdown", "id": "tt5h1lmim2", "metadata": {}, - "source": "### Primary Keys in Join Results\n\nEvery query result has a valid primary key. For joins, the result's primary key depends on **functional dependencies** between the operands:\n\n| Condition | Result Primary Key |\n|-----------|-------------------|\n| `A → B` (A determines B) | PK(A) |\n| `B → A` (B determines A) | PK(B) |\n| Both | PK(A) |\n| Neither | PK(A) ∪ PK(B) |\n\n**\"A determines B\"** means all of B's primary key attributes exist in A (as primary or secondary attributes).\n\nIn our example:\n- `Session` has PK: `(subject_id, session_idx)`\n- `Trial` has PK: `(subject_id, session_idx, trial_idx)`\n\nSince Session's PK is a subset of Trial's PK, `Session → Trial`. The join `Session * Trial` has the same primary key as Session.\n\nSee the [Query Algebra Specification](../../reference/specs/query-algebra/) for the complete functional dependency rules." + "source": "### Primary Keys in Join Results\n\nEvery query result has a valid primary key. For joins, the result's primary key depends on **functional dependencies** between the operands:\n\n| Condition | Result Primary Key |\n|-----------|-------------------|\n| `A → B` (A determines B) | PK(A) |\n| `B → A` (B determines A) | PK(B) |\n| Both | PK(A) |\n| Neither | PK(A) ∪ PK(B) |\n\n**\"A determines B\"** means all of B's primary key attributes exist in A (as primary or secondary attributes).\n\nIn our example:\n- `Session` has PK: `(subject_id, session_idx)`\n- `Trial` has PK: `(subject_id, session_idx, trial_idx)`\n\nSince Session's PK is a subset of Trial's PK, `Session → Trial`. The join `Session * Trial` has the same primary key as Session.\n\nSee the [Query Algebra Specification](../../reference/specs/query-algebra) for the complete functional dependency rules." }, { "cell_type": "markdown", @@ -6379,7 +6373,7 @@ "cell_type": "markdown", "id": "cell-63", "metadata": {}, - "source": "## Quick Reference\n\n### Operators\n\n| Operation | Syntax | Description |\n|-----------|--------|-------------|\n| Restrict | `A & cond` | Select matching rows |\n| Anti-restrict | `A - cond` | Select non-matching rows |\n| Top | `A & dj.Top(limit, order_by)` | Limit/order results |\n| Project | `A.proj(...)` | Select/compute columns |\n| Join | `A * B` | Combine tables |\n| Extend | `A.extend(B)` | Add B's attributes, keep all A rows |\n| Aggregate | `A.aggr(B, ...)` | Group and summarize |\n| Union | `A + B` | Combine entity sets |\n\n### Fetch Methods\n\n| Method | Returns | Use Case |\n|--------|---------|----------|\n| `to_dicts()` | `list[dict]` | JSON, iteration |\n| `to_pandas()` | `DataFrame` | Data analysis |\n| `to_arrays()` | `np.ndarray` | Numeric computation |\n| `to_arrays('a', 'b')` | `tuple[array, ...]` | Specific columns |\n| `keys()` | `list[dict]` | Primary keys |\n| `fetch1()` | `dict` | Single row |\n\nSee the [Query Algebra Specification](../../reference/specs/query-algebra/) and [Fetch API](../../reference/specs/fetch-api/) for complete details.\n\n## Next Steps\n\n- [Computation](05-computation/) — Building computational pipelines" + "source": "## Quick Reference\n\n### Operators\n\n| Operation | Syntax | Description |\n|-----------|--------|-------------|\n| Restrict | `A & cond` | Select matching rows |\n| Anti-restrict | `A - cond` | Select non-matching rows |\n| Top | `A & dj.Top(limit, order_by)` | Limit/order results |\n| Project | `A.proj(...)` | Select/compute columns |\n| Join | `A * B` | Combine tables |\n| Extend | `A.extend(B)` | Add B's attributes, keep all A rows |\n| Aggregate | `A.aggr(B, ...)` | Group and summarize |\n| Union | `A + B` | Combine entity sets |\n\n### Fetch Methods\n\n| Method | Returns | Use Case |\n|--------|---------|----------|\n| `to_dicts()` | `list[dict]` | JSON, iteration |\n| `to_pandas()` | `DataFrame` | Data analysis |\n| `to_arrays()` | `np.ndarray` | Numeric computation |\n| `to_arrays('a', 'b')` | `tuple[array, ...]` | Specific columns |\n| `keys()` | `list[dict]` | Primary keys |\n| `fetch1()` | `dict` | Single row |\n\nSee the [Query Algebra Specification](../../reference/specs/query-algebra) and [Fetch API](../../reference/specs/fetch-api) for complete details.\n\n## Next Steps\n\n- [Computation](../05-computation) — Building computational pipelines" }, { "cell_type": "code", diff --git a/src/tutorials/basics/05-computation.ipynb b/src/tutorials/basics/05-computation.ipynb index 1da2da33..6cfdad82 100644 --- a/src/tutorials/basics/05-computation.ipynb +++ b/src/tutorials/basics/05-computation.ipynb @@ -2049,78 +2049,7 @@ "cell_type": "markdown", "id": "cell-35", "metadata": {}, - "source": [ - "## Best Practices\n", - "\n", - "### 1. Keep `make()` Simple and Idempotent\n", - "\n", - "```python\n", - "def make(self, key):\n", - " # 1. Fetch source data\n", - " data = (SourceTable & key).fetch1()\n", - " \n", - " # 2. Compute result\n", - " result = compute(data)\n", - " \n", - " # 3. Insert result\n", - " self.insert1({**key, **result})\n", - "```\n", - "\n", - "### 2. Use Part Tables for Detailed Results\n", - "\n", - "Store summary in master, details in parts:\n", - "\n", - "```python\n", - "def make(self, key):\n", - " self.insert1({**key, 'summary': s}) # Master\n", - " self.Detail.insert(details) # Parts\n", - "```\n", - "\n", - "### 3. Re-populate After Data Changes\n", - "\n", - "```python\n", - "# Delete affected entries (cascades automatically)\n", - "(SourceTable & key).delete()\n", - "\n", - "# Reinsert corrected data\n", - "SourceTable.insert1(corrected)\n", - "\n", - "# Re-populate\n", - "ComputedTable.populate()\n", - "```\n", - "\n", - "### 4. Use Lookup Tables for Parameters\n", - "\n", - "```python\n", - "@schema\n", - "class Method(dj.Lookup):\n", - " definition = \"...\"\n", - " contents = [...] # Pre-defined methods\n", - "\n", - "@schema\n", - "class Analysis(dj.Computed):\n", - " definition = \"\"\"\n", - " -> Session\n", - " -> Method # Parameter combinations\n", - " ---\n", - " result : float64\n", - " \"\"\"\n", - "```\n", - "\n", - "See the [AutoPopulate Specification](../reference/specs/autopopulate.md) for complete details.\n", - "\n", - "## Quick Reference\n", - "\n", - "| Method | Description |\n", - "|--------|-------------|\n", - "| `populate()` | Compute all pending entries |\n", - "| `populate(restriction)` | Compute subset of entries |\n", - "| `populate(max_calls=N)` | Compute at most N entries |\n", - "| `populate(display_progress=True)` | Show progress bar |\n", - "| `populate(suppress_errors=True)` | Continue on errors |\n", - "| `progress()` | Check completion status |\n", - "| `key_source` | Entries that should be computed |" - ] + "source": "## Best Practices\n\n### 1. Keep `make()` Simple and Idempotent\n\n```python\ndef make(self, key):\n # 1. Fetch source data\n data = (SourceTable & key).fetch1()\n \n # 2. Compute result\n result = compute(data)\n \n # 3. Insert result\n self.insert1({**key, **result})\n```\n\n### 2. Use Part Tables for Detailed Results\n\nStore summary in master, details in parts:\n\n```python\ndef make(self, key):\n self.insert1({**key, 'summary': s}) # Master\n self.Detail.insert(details) # Parts\n```\n\n### 3. Re-populate After Data Changes\n\n```python\n# Delete affected entries (cascades automatically)\n(SourceTable & key).delete()\n\n# Reinsert corrected data\nSourceTable.insert1(corrected)\n\n# Re-populate\nComputedTable.populate()\n```\n\n### 4. Use Lookup Tables for Parameters\n\n```python\n@schema\nclass Method(dj.Lookup):\n definition = \"...\"\n contents = [...] # Pre-defined methods\n\n@schema\nclass Analysis(dj.Computed):\n definition = \"\"\"\n -> Session\n -> Method # Parameter combinations\n ---\n result : float64\n \"\"\"\n```\n\nSee the [AutoPopulate Specification](../../reference/specs/autopopulate) for complete details.\n\n## Quick Reference\n\n| Method | Description |\n|--------|-------------|\n| `populate()` | Compute all pending entries |\n| `populate(restriction)` | Compute subset of entries |\n| `populate(max_calls=N)` | Compute at most N entries |\n| `populate(display_progress=True)` | Show progress bar |\n| `populate(suppress_errors=True)` | Continue on errors |\n| `progress()` | Check completion status |\n| `key_source` | Entries that should be computed |" }, { "cell_type": "code", @@ -2162,4 +2091,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/src/tutorials/basics/06-object-storage.ipynb b/src/tutorials/basics/06-object-storage.ipynb index af05d260..b3d27227 100644 --- a/src/tutorials/basics/06-object-storage.ipynb +++ b/src/tutorials/basics/06-object-storage.ipynb @@ -1712,7 +1712,7 @@ "cell_type": "markdown", "id": "cell-32", "metadata": {}, - "source": "## Quick Reference\n\n| Pattern | Use Case |\n|---------|----------|\n| `` | Small Python objects |\n| `` | Large arrays with deduplication |\n| `` | Large arrays in specific store |\n| `` | Files preserving names |\n| `` | Schema-addressed data (Zarr, HDF5) |\n\n## Next Steps\n\n- [Configure Object Storage](../../how-to/configure-storage/) — Set up S3, MinIO, or filesystem stores\n- [Clean Up Storage](../../how-to/garbage-collection/) — Garbage collection for hash-addressed storage\n- [Custom Codecs](../../advanced/custom-codecs/) — Define domain-specific types\n- [Manage Large Data](../../how-to/manage-large-data/) — Performance optimization" + "source": "## Quick Reference\n\n| Pattern | Use Case |\n|---------|----------|\n| `` | Small Python objects |\n| `` | Large arrays with deduplication |\n| `` | Large arrays in specific store |\n| `` | Files preserving names |\n| `` | Schema-addressed data (Zarr, HDF5) |\n\n## Next Steps\n\n- [Configure Object Storage](../../how-to/configure-storage) — Set up S3, MinIO, or filesystem stores\n- [Clean Up Storage](../../how-to/garbage-collection) — Garbage collection for hash-addressed storage\n- [Custom Codecs](../../advanced/custom-codecs) — Define domain-specific types\n- [Manage Large Data](../../how-to/manage-large-data) — Performance optimization" }, { "cell_type": "code", diff --git a/src/tutorials/examples/blob-detection.ipynb b/src/tutorials/examples/blob-detection.ipynb index 9f5b06d1..236a859c 100644 --- a/src/tutorials/examples/blob-detection.ipynb +++ b/src/tutorials/examples/blob-detection.ipynb @@ -1589,7 +1589,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "## Key Concepts Recap\n\n| Concept | What It Does | Example |\n|---------|--------------|--------|\n| **Schema** | Groups related tables | `schema = dj.Schema('tutorial_blobs')` |\n| **Manual Table** | Stores user-entered data | `Image`, `SelectedDetection` |\n| **Lookup Table** | Stores reference/config data | `DetectionParams` |\n| **Computed Table** | Derives data automatically | `Detection` |\n| **Part Table** | Stores detailed results with master | `Detection.Blob` |\n| **Foreign Key** (`->`) | Creates dependency | `-> Image` |\n| **`populate()`** | Runs pending computations | `Detection.populate()` |\n| **Restriction** (`&`) | Filters rows | `Detection & 'num_blobs < 300'` |\n| **Join** (`*`) | Combines tables | `Image * Detection` |\n\n## Next Steps\n\n- [Schema Design](../basics/02-schema-design/) — Learn table types and relationships in depth\n- [Queries](../basics/04-queries/) — Master DataJoint's query operators\n- [Computation](../basics/05-computation/) — Build complex computational workflows" + "source": "## Key Concepts Recap\n\n| Concept | What It Does | Example |\n|---------|--------------|--------|\n| **Schema** | Groups related tables | `schema = dj.Schema('tutorial_blobs')` |\n| **Manual Table** | Stores user-entered data | `Image`, `SelectedDetection` |\n| **Lookup Table** | Stores reference/config data | `DetectionParams` |\n| **Computed Table** | Derives data automatically | `Detection` |\n| **Part Table** | Stores detailed results with master | `Detection.Blob` |\n| **Foreign Key** (`->`) | Creates dependency | `-> Image` |\n| **`populate()`** | Runs pending computations | `Detection.populate()` |\n| **Restriction** (`&`) | Filters rows | `Detection & 'num_blobs < 300'` |\n| **Join** (`*`) | Combines tables | `Image * Detection` |\n\n## Next Steps\n\n- [Schema Design](../../basics/02-schema-design) — Learn table types and relationships in depth\n- [Queries](../../basics/04-queries) — Master DataJoint's query operators\n- [Computation](../../basics/05-computation) — Build complex computational workflows" }, { "cell_type": "code", diff --git a/src/tutorials/examples/hotel-reservations.ipynb b/src/tutorials/examples/hotel-reservations.ipynb index 2f054590..a4757210 100644 --- a/src/tutorials/examples/hotel-reservations.ipynb +++ b/src/tutorials/examples/hotel-reservations.ipynb @@ -1913,7 +1913,7 @@ "cell_type": "markdown", "id": "cell-summary-md", "metadata": {}, - "source": "## Key Concepts\n\n| Concept | How It's Used |\n|---------|---------------|\n| **Workflow Dependencies** | `CheckOut -> CheckIn -> Reservation -> RoomAvailable` |\n| **Unique Constraints** | One reservation per room/night (primary key) |\n| **Referential Integrity** | Can't reserve unavailable room, can't check in without reservation |\n| **Error Translation** | Database exceptions → domain-specific errors |\n\nThe schema **is** the business logic. Application code just translates errors.\n\n## Next Steps\n\n- [University Database](university/) — Academic records with many-to-many relationships\n- [Languages & Proficiency](languages/) — International standards and lookup tables\n- [Data Entry](../basics/03-data-entry/) — Insert patterns and transactions" + "source": "## Key Concepts\n\n| Concept | How It's Used |\n|---------|---------------|\n| **Workflow Dependencies** | `CheckOut -> CheckIn -> Reservation -> RoomAvailable` |\n| **Unique Constraints** | One reservation per room/night (primary key) |\n| **Referential Integrity** | Can't reserve unavailable room, can't check in without reservation |\n| **Error Translation** | Database exceptions → domain-specific errors |\n\nThe schema **is** the business logic. Application code just translates errors.\n\n## Next Steps\n\n- [University Database](../university) — Academic records with many-to-many relationships\n- [Languages & Proficiency](../languages) — International standards and lookup tables\n- [Data Entry](../../basics/03-data-entry) — Insert patterns and transactions" }, { "cell_type": "code", diff --git a/src/tutorials/examples/languages.ipynb b/src/tutorials/examples/languages.ipynb index 0db7d019..695814b5 100644 --- a/src/tutorials/examples/languages.ipynb +++ b/src/tutorials/examples/languages.ipynb @@ -2267,7 +2267,7 @@ "cell_type": "markdown", "id": "cell-summary-md", "metadata": {}, - "source": "## Key Concepts\n\n| Pattern | Implementation |\n|---------|----------------|\n| **Many-to-many** | `Proficiency` links `Person` and `Language` |\n| **Lookup tables** | `Language` and `CEFRLevel` with `contents` |\n| **Association data** | `cefr_level` stored in the association table |\n| **Standards** | ISO 639-1 codes, CEFR levels |\n\n### Benefits of Lookup Tables\n\n1. **Data consistency** — Only valid codes can be used\n2. **Rich metadata** — Full names, descriptions stored once\n3. **Easy updates** — Change \"Español\" to \"Spanish\" in one place\n4. **Self-documenting** — `Language()` shows all valid options\n\n## Next Steps\n\n- [University Database](university/) — Academic records\n- [Hotel Reservations](hotel-reservations/) — Workflow dependencies\n- [Queries Tutorial](../basics/04-queries/) — Query operators in depth" + "source": "## Key Concepts\n\n| Pattern | Implementation |\n|---------|----------------|\n| **Many-to-many** | `Proficiency` links `Person` and `Language` |\n| **Lookup tables** | `Language` and `CEFRLevel` with `contents` |\n| **Association data** | `cefr_level` stored in the association table |\n| **Standards** | ISO 639-1 codes, CEFR levels |\n\n### Benefits of Lookup Tables\n\n1. **Data consistency** — Only valid codes can be used\n2. **Rich metadata** — Full names, descriptions stored once\n3. **Easy updates** — Change \"Español\" to \"Spanish\" in one place\n4. **Self-documenting** — `Language()` shows all valid options\n\n## Next Steps\n\n- [University Database](../university) — Academic records\n- [Hotel Reservations](../hotel-reservations) — Workflow dependencies\n- [Queries Tutorial](../../basics/04-queries) — Query operators in depth" }, { "cell_type": "code",