Slowly Changing Dimension Type 2 (scd2) custom materialization

SCD2 implementation using pyspark

A modern banking data pipeline built with Dagster and DBT!

SCD2 implementation using pyspark

Pipeline 100% Open Source

Advanced Healthcare Claims Pipeline using Snowflake, Snowpipe, Streams, Tasks, SCD Type 2, and AWS S3. Automates ingestion, CDC, dimensional modeling, and data quality checks for healthcare patient and claims data.

Production-grade parameterized ETL pipeline implementing SCD Type 2 for travel booking data using Databricks, Delta Lake, and ADLS — includes data quality checks, incremental fact table build, Z-Order optimization, and SQL reporting.

Implementation of SCD2 for employee relocation data

This is a data engineering pipeline built on Databricks + Delta Lake + PySpark that ingests travel booking and customer master data, applies SCD Type 2 logic, and delivers analytics-ready tables. It includes data quality enforcement, dimension versioning, fact aggregation, and performance tuning.

This repo contains details about travel booking project executed on Databricks, Thanks

Production-grade CDC pipeline: MySQL → Debezium → Kinesis → S3 → AWS Glue (PySpark) → Redshift + Postgres + OpenSearch. Multi-sink fanout with SCD2, idempotency tracking, and 13 modular Terraform modules.

Multi-tenant IoT telemetry Lakehouse on Databricks + Delta Lake. PySpark, Auto Loader, DLT, medallion architecture, Terraform IaC.

Modern data stack reference: dbt + BigQuery + Airflow (Cloud Composer) with medallion layering, SCD2 snapshots, exposures, freshness SLAs, and 45× cost reduction via partition + cluster + incremental tuning.

End-to-end ETL and data warehouse pipeline implementing star schema design, SCD Type 2 dimensions, and fact tables for analytical reporting. Built with SQL and structured for scalable analytics.

Batch retail data lakehouse on Databricks: Delta Live Tables (bronze → silver → gold), Unity Catalog, synthetic data generator, and an executive analytics dashboard.

Implements a data pipeline using DLT in Databricks (Delta Lake) and uses medallion layering in Delta Lake

Vijay works in an IT company for last 5 years, he always needs extra money to spend on his monthly expenses so he decided to apply for the credit card in icici bank. The bank does a background check of vijay to know if he is elligible for the credit card or not.

🏥 Streamline healthcare claims processing with this Snowflake pipeline, featuring auto-ingestion, CDC, SCD Type 2, and data quality checks.