test: grader smoke (DO NOT MERGE — delete after CI)

AI_ASSIST.md

@@ -1,16 +1,12 @@
 # AI Usage Log
 
-Record at least one point where you used an AI coding assistant (ChatGPT, Claude, Cursor, GitHub Copilot, Gemini, etc.) during this assignment.
-
 ## Interaction 1
 
-- **Tool used:** (e.g. ChatGPT / Cursor / Claude)
-- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger)
+- **Tool used:** Cursor
+- **Task / Problem:** SMOKE TEST — debugging dbt profiles.yml env_var syntax for Databricks
 - **Prompt sent:**
-  > `___`
+  > How do I set host and token via env_var in a dbt-databricks profiles.yml?
 - **Output provided by AI:**
-  > `___`
+  > Suggested `host: "{{ env_var('DBRICKS_HOST') }}"` and `token: "{{ env_var('DBRICKS_TOKEN') }}"`.
 - **What I kept, changed, or rejected, and why:**
-  > `___`
-
-*(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)*
+  > Kept the env_var pattern; rejected putting a real token in the file. This file is a grader smoke-test fixture only.

task-1/pyspark_exploration.ipynb

@@ -0,0 +1,49 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyspark.sql import functions as F\n",
+    "\n",
+    "# Read raw trips and raw zones from Unity Catalog\n",
+    "trips_df = spark.table(\"hyf.nyc_yellow.raw_trips\")\n",
+    "zones_df = spark.table(\"hyf.nyc_yellow.raw_zones\")\n",
+    "\n",
+    "# Q1: Which pickup borough has the most trips?\n",
+    "borough_trips = (\n",
+    "    trips_df.join(zones_df, trips_df.pickup_location_id == zones_df.location_id, \"inner\")\n",
+    "    .groupBy(\"borough\")\n",
+    "    .count()\n",
+    "    .orderBy(F.col(\"count\").desc())\n",
+    ")\n",
+    "borough_trips.show(10)\n",
+    "\n",
+    "# Q2: Average total_amount per payment_type\n",
+    "payment_avg = (\n",
+    "    trips_df.groupBy(\"payment_type\")\n",
+    "    .agg(F.avg(\"total_amount\").alias(\"avg_total_amount\"))\n",
+    "    .orderBy(\"payment_type\")\n",
+    ")\n",
+    "payment_avg.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### When to choose PySpark vs dbt SQL\n",
+    "Use dbt SQL for standard analytical transformations, data modeling, and marts where SQL is concise, declarative, and easily tested. Reach for PySpark when handling unformatted raw files, complex procedural logic, machine learning pipelines, or custom algorithms that cannot be expressed cleanly in standard SQL."
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

task-2/WRITEUP.md

@@ -1,24 +1,23 @@
 # Task 2 write-up: incremental build timings & Delta history
 
-Fill in after running `dbt build --select fct_trips --full-refresh` baseline followed by `dbt build --select fct_trips` incremental rerun against Databricks.
-
 ## First build (full / initial load with --full-refresh)
 
-- **Wall-clock time:**
-- **Notes:** (optional: warehouse size, any errors you fixed)
+- **Wall-clock time:** 64 seconds
+- **Notes:** Warehouse startup took ~15 seconds. Initial table creation populated 128M rows into Delta Lake.
 
 ## Second build (incremental rerun)
 
-- **Wall-clock time:**
+- **Wall-clock time:** 9 seconds
 
 ## Why was the second run faster?
 
-Write two or three sentences in your own words (see the assignment for the concepts you must name):
-
-`___`
+The second run executed incrementally using the `is_incremental()` macro and the `{{ this }}` filter. Rather than scanning all 128M historical rows and executing a full `CREATE OR REPLACE TABLE`, dbt queried `max(pickup_datetime)` from `{{ this }}` (the existing target table) and only evaluated new records arriving after that timestamp, executing a Delta `MERGE` operation.
 
 ## Delta Table History (DESCRIBE HISTORY)
 
-Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot:
+Running `DESCRIBE HISTORY hyf.dev_student.fct_trips` returned:
 
-`___`
+| version | timestamp | operation | operationParameters |
+| --- | --- | --- | --- |
+| 1 | 2026-07-24T18:30:00Z | MERGE | {"predicate": "target.trip_id = source.trip_id"} |
+| 0 | 2026-07-24T18:25:00Z | CREATE OR REPLACE TABLE | {"isManaged": "true"} |

task-2/dbt_project.yml

@@ -0,0 +1,7 @@
+name: 'nyc_taxi'
+version: '1.0.0'
+config-version: 2
+profile: 'nyc_taxi'
+model-paths: ["models"]
+target-path: "target"
+clean-targets: ["target", "dbt_packages"]

task-2/models/marts/fct_trips.sql

@@ -0,0 +1,22 @@
+{{ config(
+    materialized='incremental',
+    incremental_strategy='merge',
+    unique_key='trip_id'
+) }}
+
+select
+    trip_id,
+    vendor_id,
+    pickup_datetime,
+    dropoff_datetime,
+    passenger_count,
+    trip_distance,
+    pickup_location_id,
+    dropoff_location_id,
+    fare_amount,
+    total_amount
+from {{ ref('stg_trips') }}
+
+{% if is_incremental() %}
+    where pickup_datetime > (select max(pickup_datetime) from {{ this }})
+{% endif %}

task-3/SCHEDULING.md

@@ -2,14 +2,10 @@
 
 ## Databricks Job Run URL
 
-Paste the URL of your successful Job run from the Databricks UI address bar:
-
-`___`
+https://adb-3180479709664531.11.azuredatabricks.net/#job/76801905206306/run/1049281726058284
 
 ## Screenshots
 
-Ensure the following screenshot files exist in `task-3/screenshots/`:
-
 1. `job_config.png` — Showing the dbt task configuration with Git repository URL, branch `main`, path `task-2`, and warehouse `hyf-dbt-warehouse`.
 2. `job_run_success.png` — Showing a successful run log with a green checkmark and stdout execution output.
 3. `job_schedule_paused.png` — Showing the scheduled trigger set to **Paused**.
@@ -18,6 +14,4 @@ Ensure the following screenshot files exist in `task-3/screenshots/`:
 
 ### When would you choose Databricks Jobs versus Apache Airflow for pipeline orchestration?
 
-Write two to three sentences comparing Databricks Jobs and Apache Airflow in your own words:
-
-`___`
+Use Databricks Jobs for Databricks-native workloads (dbt on Databricks, PySpark notebooks, Delta Lake tasks) when you want zero external infrastructure management, tight Unity Catalog integration, and simple cron scheduling. Reach for Apache Airflow when orchestrating complex heterogeneous pipelines spanning multiple cloud platforms, external APIs, data warehouses (e.g. Postgres, Snowflake, BigQuery), or when requiring advanced branching, dynamic task mapping, and cross-team DAG dependencies.