Feature/notebook docs polish

README.md

@@ -146,13 +146,13 @@ The same feature definitions are used at training and serving, so there is no tr
 
 ### Proposed models (progression)
 
-We deliberately keep models simple so the focus stays on the **MLOps lifecycle**, not
-hyper-tuning. We train a progression and let MLflow pick a champion:
+The models are kept deliberately simple so the focus stays on the **MLOps lifecycle**, not
+hyper-tuning. A short progression is trained and MLflow selects a champion:
 
 | Stage | Model | Why |
 |---|---|---|
-| Trap | **Random Forest** (raw imbalanced data) | Shows the imbalance trap: ~99.8% accuracy but recall ≈ 0 (flags no fraud) |
-| Fixed / registered | **Random Forest** (balanced training data) | Keep all fraud + down-sample non-fraud so the model actually catches fraud; evaluated on the real imbalanced test set. This is the version registered + promoted |
+| Baseline | **Random Forest** (raw imbalanced data) | Establishes the imbalance problem: ~99.8% accuracy but recall ≈ 0 (flags no fraud) |
+| Balanced (registered) | **Random Forest** (balanced training data) | Keeps all fraud and down-samples non-fraud so the model detects fraud; evaluated on the real imbalanced test set. This version is registered and promoted |
 
 Class imbalance is handled by **balancing the training data** (keep every fraud row,
 down-sample non-fraud to a 1:1 ratio), not by class weights. The model is still evaluated on
@@ -161,7 +161,7 @@ trade-off this creates and why the decision threshold is a business choice.
 
 ### Evaluation metrics
 
-Accuracy is misleading on imbalanced data, so we track:
+Accuracy is misleading on imbalanced data, so the tracked metrics are:
 
 - **ROC-AUC**, the metric the promotion gate scores the model on.
 - **precision**, **recall**, **F1** (macro), and accuracy for context.
@@ -170,7 +170,7 @@ Accuracy is misleading on imbalanced data, so we track:
 
 ### How it maps to the workshop
 
-| Session | What we do with the model |
+| Session | Model activity |
 |---|---|
 | Model Development | Build features (Feature Store), train the progression, log to MLflow |
 | Governance | Govern the model registered during training; aliases, versioning + lineage |
@@ -181,18 +181,18 @@ Accuracy is misleading on imbalanced data, so we track:
 > **The Serving step is slow.** The deployment job (`fraud_deployment_job`) provisions a
 > **Lakebase online feature store**, publishes the `card_features`/`client_features` tables to
 > it, and builds a first-time serving endpoint, which typically takes **15-25 minutes**
-> end-to-end. It runs automatically when a new model version is registered; let it run while
-> you talk through the concepts. Once it's ready, run `deployment/model_deployment/notebooks/query_endpoint.py`
+> end-to-end. It runs automatically when a new model version is registered; allow it to
+> complete before running `deployment/model_deployment/notebooks/query_endpoint.py`
 > to score live transactions against the endpoint (this also seeds the online monitor's traffic).
 
 ---
 
-## How you work with this repo (developer workflow)
+## Developer workflow
 
 This repo is a **Declarative Automation Bundle** (formerly a Databricks Asset Bundle / DAB) with a
-`databricks.yml` at the root. You
-develop either **in the Databricks UI** or **in a local IDE**, commit to Git, and let
-**CI/CD** promote the bundle through environments. You don't edit prod directly.
+`databricks.yml` at the root. Development happens either **in the Databricks UI** or **in a
+local IDE**; changes are committed to Git, and **CI/CD** promotes the bundle through
+environments. Prod is never edited directly.
 
 ```mermaid
 flowchart LR
@@ -399,15 +399,15 @@ targets. CI/CD just picks the target, and the bundle does the rest.
 ## Data isolation: shared reads, personal writes
 
 Attendees **read** from shared medallion data but **write** to their **own** schema, so 20
-people never overwrite each other. This is wired through the bundle so it "just works" when
+people never overwrite each other. The bundle wires this up, so it applies automatically when
 each attendee clones the repo and deploys.
 
 > **Why a personal schema and not more medallion layers?** Medallion (bronze/silver/gold) and
 > personal schemas are two *different, orthogonal* Databricks best practices, and both apply
 > here:
 > - **Medallion** organises the *shared enterprise data* by quality. It's "a recommended best
 >   practice but not a requirement" and each layer lives in its own schema of a shared catalog
->   ([medallion docs](https://docs.databricks.com/aws/en/lakehouse/medallion)). Our shared
+>   ([medallion docs](https://docs.databricks.com/aws/en/lakehouse/medallion)). The shared
 >   `fraud_bronze/silver/gold` (including the gold `transactions_enriched`) is unchanged.
 > - **Personal schemas** isolate each *developer's own outputs* in dev. Databricks explicitly
 >   recommends "a personal schema per user, for example `dev_${user_name}`… This prevents
@@ -416,8 +416,8 @@ each attendee clones the repo and deploys.
 >
 > So each attendee still *reads* the shared medallion; their feature tables / model / predictions
 > are dev outputs that go in their personal `dev_<you>_fraud` schema. In **production** one
-> pipeline would instead write the feature tables to the shared schema. We don't sub-layer
-> the personal schema into bronze/silver/gold, since that's a shared-data concept, and doing so
+> pipeline would instead write the feature tables to the shared schema. The personal schema is
+> not sub-layered into bronze/silver/gold, since that is a shared-data concept, and doing so
 > per user would explode the schema count.
 
 **Shared, read-only** (instructor provisions once):

solution/data_ingestion/notebooks/load_bronze_to_silver.py

@@ -1,10 +1,15 @@
 # Databricks notebook source
 # MAGIC %md
 # MAGIC # Bronze -> Silver
-# MAGIC Clean, type, dedupe and join the Bronze tables into conformed Silver tables,
-# MAGIC including the enriched transaction table used for feature engineering.
 # MAGIC
-# MAGIC **Setup notebook** run by the instructor before the workshop. Not a lab exercise.
+# MAGIC Instructor setup notebook, run before the workshop. Not a lab exercise.
+# MAGIC
+# MAGIC Turns the Bronze tables into conformed Silver tables (and the enriched Gold
+# MAGIC transaction table that feature engineering reads) by:
+# MAGIC
+# MAGIC - cleaning and casting columns to their real types;
+# MAGIC - deduplicating to one row per entity;
+# MAGIC - joining the sources into a single labelled transaction table.
 # MAGIC
 # MAGIC | Layer | Location |
 # MAGIC |---|---|
@@ -59,9 +64,10 @@
 
 # MAGIC %md
 # MAGIC ## Target schemas
-# MAGIC The shared `fraud_silver` / `fraud_gold` schemas are declared as bundle resources
-# MAGIC (the `shared_infra` block in `databricks.yml`) and created by
-# MAGIC `databricks bundle deploy -t dev` before this job runs.
+# MAGIC
+# MAGIC - The `fraud_silver` / `fraud_gold` schemas are declared as bundle resources (the
+# MAGIC   `shared_infra` block in `databricks.yml`).
+# MAGIC - `databricks bundle deploy -t dev` creates them before this job runs.
 
 # COMMAND ----------
 

solution/data_ingestion/notebooks/load_landing_to_bronze.py

@@ -1,10 +1,14 @@
 # Databricks notebook source
 # MAGIC %md
 # MAGIC # Landing -> Bronze
-# MAGIC Ingest the raw fraud source files from the landing volume into Bronze Delta
-# MAGIC tables, applying explicit schemas and audit columns. Minimal transformation.
 # MAGIC
-# MAGIC **Setup notebook** run by the instructor before the workshop. Not a lab exercise.
+# MAGIC Instructor setup notebook, run before the workshop. Not a lab exercise.
+# MAGIC
+# MAGIC Reads the raw fraud source files from the landing volume into Bronze Delta tables:
+# MAGIC
+# MAGIC - explicit schemas, no inference;
+# MAGIC - audit columns (ingest timestamp, ingest date, source file);
+# MAGIC - minimal transformation, so the shape stays close to the source.
 # MAGIC
 # MAGIC | Layer | Location |
 # MAGIC |---|---|
@@ -59,10 +63,11 @@
 
 # MAGIC %md
 # MAGIC ## Target schemas and volume
-# MAGIC The shared `fraud_landing` / `fraud_bronze` schemas and the `raw_data` volume are
-# MAGIC declared as bundle resources (the `shared_infra` block in `databricks.yml`) and
-# MAGIC created by `databricks bundle deploy -t dev` before this job runs. The raw files are
-# MAGIC uploaded into the volume after the deploy, and this notebook just reads them.
+# MAGIC
+# MAGIC - The `fraud_landing` / `fraud_bronze` schemas and the `raw_data` volume are declared as
+# MAGIC   bundle resources (the `shared_infra` block in `databricks.yml`).
+# MAGIC - `databricks bundle deploy -t dev` creates them before this job runs.
+# MAGIC - The raw files are uploaded into the volume after the deploy; this notebook only reads them.
 
 # COMMAND ----------
 
@@ -221,7 +226,7 @@ def read_csv_with_audit(path: str, schema: StructType):
 # COMMAND ----------
 
 # Governance: table-level comments so the raw layer is self-documenting. Bronze keeps
-# source fidelity, so we do not comment individual columns here (that happens in silver).
+# source fidelity, so individual columns are not commented here; that happens in silver.
 bronze_table_comments = {
     transactions_fqn: "Bronze: raw card transactions ingested as-is from the landing volume, plus audit columns.",
     cards_fqn: "Bronze: raw card metadata ingested as-is from the landing volume, plus audit columns.",

solution/deployment/README.md

@@ -3,4 +3,4 @@
 | Folder | Description |
 |---|---|
 | `model_deployment/` | The MLflow deployment job (evaluate → approve → promote to `@champion` and update the serving endpoint) plus a manual endpoint-test lab. |
-| `batch_inference/` | Batch scoring with the `@champion` model (the weekly fraud report). |
+| `batch_inference/` | Batch scoring with the `@champion` model (the scheduled batch prediction path). |

solution/deployment/batch_inference/README.md

@@ -2,4 +2,4 @@
 
 | File | Description |
 |---|---|
-| `notebooks/batch_inference.py` | Score a batch of transactions with the `@champion` model and append the results to the inference table (the weekly fraud report). |
+| `notebooks/batch_inference.py` | Score a batch of transactions with the `@champion` model and append the results to the inference table. |

solution/deployment/batch_inference/notebooks/batch_inference.py

@@ -1,21 +1,23 @@
 # Databricks notebook source
 # MAGIC %md
-# MAGIC # Batch inference: the weekly fraud report
+# MAGIC # Batch inference
 # MAGIC
 # MAGIC **Session:** Model Serving & Consumption / Monitoring & Retraining
 # MAGIC
 # MAGIC Score a batch of transactions with the governed `@champion` model and append the
-# MAGIC results to an inference table. This is the batch prediction path (a scheduled "weekly
-# MAGIC report"), and the table it writes is what the batch monitor profiles in the
-# MAGIC Monitoring session.
+# MAGIC results to an inference table. This is the scheduled batch prediction path, and the
+# MAGIC table it writes is what the batch monitor profiles in the Monitoring session.
 # MAGIC
 # MAGIC - **Model (personal):** `dev.<you>_fraud.fraud_detection@champion`
 # MAGIC - **Inference table (personal):** `dev.<you>_fraud.fraud_predictions`
 # MAGIC
-# MAGIC To be usable by Lakehouse Monitoring's InferenceLog analysis, every row needs four
-# MAGIC things beyond the features: a prediction, the model version that produced it, a
-# MAGIC timestamp, and (when it arrives) the ground-truth label. We write all four so the
-# MAGIC monitor can track both drift and quality over time.
+# MAGIC Lakehouse Monitoring's InferenceLog analysis needs four columns beyond the features on
+# MAGIC every row, all written here so the monitor can track drift and quality:
+# MAGIC
+# MAGIC - the prediction;
+# MAGIC - the model version that produced it;
+# MAGIC - a scoring timestamp;
+# MAGIC - the ground-truth label, when available.
 # MAGIC
 # MAGIC Docs: [Batch inference with Feature Engineering](https://learn.microsoft.com/azure/databricks/machine-learning/feature-store/score-batch)
 
@@ -44,7 +46,7 @@
 # The batch-inference job passes the full three-level model name; fall back for interactive runs.
 model_name = dbutils.widgets.get("model_name") or f"{catalog_name}.{user_schema}.fraud_detection"
 
-# Read the transactions to score from the shared gold source; write predictions to your schema.
+# Read the transactions to score from the shared gold source; write predictions to the personal schema.
 source_table = f"{catalog_name}.{gold_schema}.transactions_enriched"
 predictions_table = f"{catalog_name}.{user_schema}.fraud_predictions"
 
@@ -72,8 +74,8 @@
 mlflow.set_registry_uri("databricks-uc")
 fe = FeatureEngineeringClient()
 
-# Resolve the champion alias to a concrete version so we can stamp every scored row with
-# the model version that produced it (the monitor's model_id_col groups metrics by this).
+# Resolve the champion alias to a concrete version, to stamp every scored row with the model
+# version that produced it (the monitor's model_id_col groups metrics by this).
 client = MlflowClient()
 champion = client.get_model_version_by_alias(model_name, model_alias)
 model_version = str(champion.version)
@@ -86,15 +88,14 @@
 # MAGIC In production this would be the newly-arrived, not-yet-scored transactions. Build the
 # MAGIC same request spine the model was trained on: the raw transaction keys and fields
 # MAGIC (`card_id`, `client_id`, `amount`, `transaction_hour`, `mcc`, `use_chip`). The Feature
-# MAGIC Store looks up the card/client entity features and runs the on-demand functions, so we
-# MAGIC pass exactly what a live caller would send. `amount` is cast to double to match the
+# MAGIC Store looks up the card/client entity features and runs the on-demand functions, so
+# MAGIC the input matches what a live caller would send. `amount` is cast to double to match the
 # MAGIC feature tables and the on-demand function signatures (no train/serve skew).
 
 # COMMAND ----------
 
 # The request spine mirrors the training spine (raw transaction keys and fields, no label).
-# Building it is plain Spark, so it is provided; scoring with the champion is the step you
-# fill in below.
+# Building it is plain Spark and is provided; scoring with the champion is the exercise below.
 to_score = (
     spark.read.table(source_table)
     .select(
@@ -138,9 +139,9 @@
 # MAGIC not just prediction drift (a monitor can only drift columns that are in the table).
 # MAGIC Then append.
 # MAGIC
-# MAGIC > The label (`is_fraud`) is joined here for the workshop so quality metrics compute
-# MAGIC > immediately. In the real world fraud is confirmed later (chargebacks), so you would
-# MAGIC > backfill the label into this table when it arrives; drift is watched in the meantime.
+# MAGIC > The label (`is_fraud`) is joined here so quality metrics compute immediately. In
+# MAGIC > production, fraud is confirmed later (chargebacks), so the label would be backfilled
+# MAGIC > into this table when it arrives; drift is tracked in the meantime.
 
 # COMMAND ----------
 
@@ -153,9 +154,9 @@
 )
 
 # fe.score_batch already returns the looked-up entity features (credit_score, credit_limit,
-# yearly_income, current_age) plus the input `amount`, so we log those columns straight from
-# `scored`. Logging them is what makes FEATURE (data) drift monitorable downstream. (Re-joining
-# them from the source would duplicate the columns and make the reference ambiguous.)
+# yearly_income, current_age) plus the input `amount`, so those columns are logged straight
+# from `scored`. Logging them is what makes feature (data) drift monitorable downstream.
+# (Re-joining them from the source would duplicate the columns and make the reference ambiguous.)
 
 scored_at = F.to_timestamp(
     F.date_sub(F.current_date(), (F.col("transaction_id") % F.lit(14)).cast("int"))

solution/deployment/model_deployment/notebooks/approval.py

@@ -9,7 +9,7 @@
 # MAGIC as the signal (the mechanism MLflow 3 deployment jobs use):
 # MAGIC
 # MAGIC - **dev** (`auto_approve=true`): the task sets the approval tag itself and passes, so
-# MAGIC   the whole loop runs unattended in class.
+# MAGIC   the whole loop runs unattended.
 # MAGIC - **staging / prod** (`auto_approve=false`): the task fails until a human reviews the
 # MAGIC   evaluation metrics on the model-version page and clicks **Approve** (which sets the
 # MAGIC   tag `<task-name>=Approved` and repairs the run). The task then passes and the run
@@ -44,8 +44,8 @@
 
 client = MlflowClient(registry_uri="databricks-uc")
 
-# In dev we auto-approve: set the tag so this run (and the UI) reflect an approved state,
-# then pass. In staging/prod we require a human to have set the tag to "Approved".
+# In dev, auto-approve: set the tag so this run (and the UI) reflect an approved state, then
+# pass. In staging/prod, a human must have set the tag to "Approved".
 if auto_approve:
     client.set_model_version_tag(model_name, model_version, tag_name, "Approved")
     print(f"Auto-approved (dev): set {tag_name}=Approved on v{model_version}.")

solution/deployment/model_deployment/notebooks/evaluation.py

@@ -15,7 +15,7 @@
 # MAGIC 3. **Deployment**: promote the version to `@champion` and serve it.
 # MAGIC
 # MAGIC The version being evaluated is the one that triggered the job, so there is no
-# MAGIC champion/challenger bookkeeping here. We evaluate this version and gate it against a
+# MAGIC champion/challenger bookkeeping here. The version is evaluated and gated against a
 # MAGIC metric floor (and the current champion, if any).
 # MAGIC See [MLflow deployment jobs](https://learn.microsoft.com/azure/databricks/mlflow/deployment-job).
 
@@ -52,8 +52,8 @@
 # Independent evaluation: score the REGISTERED artifact on a fresh, labelled holdout rather
 # than trusting the metrics training self-reported. fe.score_batch replays the exact feature
 # lookups + on-demand functions recorded at training, and the served model returns a fraud
-# probability, so we can compute a real ROC AUC. (In production this holdout would be a
-# curated, leakage-controlled evaluation table; here we sample the shared gold source.)
+# probability, so a real ROC AUC can be computed. In production this holdout would be a
+# curated, leakage-controlled evaluation table; here it is sampled from the shared gold source.
 catalog_name = model_name.split(".")[0]
 gold_table = f"{catalog_name}.fraud_gold.transactions_enriched"
 
@@ -105,10 +105,11 @@ def score_holdout(version):
 # COMMAND ----------
 
 # MAGIC %md
-# MAGIC ### Confusion matrix for the approver
+# MAGIC ## Confusion matrix for the approver
+# MAGIC
 # MAGIC Threshold the candidate's fraud score at 0.5 and show the confusion matrix on the
-# MAGIC holdout, logged to an MLflow run so it sits alongside the metrics an approver reviews
-# MAGIC before clicking Approve.
+# MAGIC holdout, logged to an MLflow run so it sits alongside the metrics the approver reviews
+# MAGIC before approving.
 
 # COMMAND ----------
 
@@ -133,12 +134,16 @@ def score_holdout(version):
 # COMMAND ----------
 
 # MAGIC %md
-# MAGIC ### ROC and precision-recall curves
-# MAGIC The confusion matrix is a single operating point; these curves show the whole
-# MAGIC threshold trade-off. The **ROC** curve's area is the gate metric (threshold-independent),
-# MAGIC and on rare-fraud data the **precision-recall** curve (with its average precision) is the
-# MAGIC more honest read of how much precision you give up as you chase recall. Both are logged
-# MAGIC to the same evaluation run, alongside the confusion matrix.
+# MAGIC ## ROC and precision-recall curves
+# MAGIC
+# MAGIC The confusion matrix is a single operating point; these curves show the full threshold
+# MAGIC trade-off:
+# MAGIC
+# MAGIC - the ROC curve's area is the gate metric (threshold-independent);
+# MAGIC - on rare-fraud data, the precision-recall curve (and its average precision) shows how
+# MAGIC   much precision is given up as recall increases.
+# MAGIC
+# MAGIC Both are logged to the same evaluation run, alongside the confusion matrix.
 
 # COMMAND ----------