TBD Workshop 1.

Workshop goals

1. Learn how to provision computing resources for running Big Data analyses using the Infrastructure as Code (IaC) approach.
2. Learn how to set up opinionated CI/CD pipelines to deploy cloud infrastructure.
3. Learn how to utilize linters for detecting security vulnerabilities in cloud infrastructure.
4. Learn how to run Apache Spark code in a distributed way on Hadoop cluster using Vertex AI notebooks and Dataproc services on GCP.
5. Learn how to use Workload Identity Federation for a secure authentication from GitHub Actions to Google Cloud.

High level architecture

Prerequisites

Software

• Google Cloud SDK
• terraform ~> 1.11.0
• gsutil
• pre-commit
• Terraform ( Requirements )
• Python ~>3.8
• Linux/MacOS
• pre-commit-terraform dependencies

GCP

• Redeem a GCP coupon to create a billing account
• Authenticate to GCP to obtain the default credentials used for running the code

# first remove the stored credentials if exist
gcloud auth application-default revoke
# login and get the new application credentials
gcloud auth application-default login

Project setup

0. Fork this repository to your own Github account.
1. Export shared environment variables

export TF_VAR_tbd_semester=2025L
# format: 20xx for teachers, student ID number for students 
export TF_VAR_user_id=9900
# use your own billing account id
export TF_VAR_billing_account=01F44C-CA9C7E-587C25
# for budget creation
export GOOGLE_BILLING_PROJECT=$(echo "tbd-${TF_VAR_tbd_semester}-${TF_VAR_user_id}" | tr '[:upper:]' '[:lower:]')

2. Enter bootstrap folder then init project and Terraform state bucket

cd bootstrap
terraform init
terraform apply
cd ..

3. CI/CD (Github Actions setup using Workload Identity Federation)

• Edit env/backend.tfvars file and set bucket variable with the Terraform state bucket
• Edit env/project.tfvars file and set project_name, iac_service_account variables using the output from the bootstrap phase, e.g.:
• Edit cicd_bootstrap/conf/github_actions.tfvars to set github_org and github_repo, e.g.:

  github_org  = "mwiewior"
  github_repo = "tbd-workshop-1"

• Init state file and set env variables

cd cicd_bootstrap
terraform init -backend-config=../env/backend.tfvars

• Apply

# authenticate Docker backend with GCP
gcloud auth configure-docker
# create CI/CD integration using Workload Identity
terraform apply -var-file ../env/project.tfvars -var-file conf/github_actions.tfvars -compact-warnings
cd ..

4. Use output variables for configuring Github Actions workflow: .github/workflows/pull-request.yml,e.g. : Please do not edit and hardcode these values in a YAML but set the Github Actions secrets instead while preserving the secret names, i.e. GCP_WORKLOAD_IDENTITY_PROVIDER_NAME and GCP_WORKLOAD_IDENTITY_SA_EMAIL.
5. Install and configure pre-commit

pre-commit install

6. Commit changes, push to a branch and open a PR to YOUR repository main/master branch. If you see a warning like this -- please enable the workflows: ...and repush your changes!

Once all Pull Requests checks have passed please merge your PR and wait until your release job finishes.

7. IMPORTANT ❗ ❗ ❗ Please remember to destroy all the resources after the workshop:

terraform init -backend-config=env/backend.tfvars
terraform destroy -no-color -var-file env/project.tfvars 

Requirements

Name	Version
terraform	~> 1.11.0
docker	3.0.2
google	~> 5.44.0
kubernetes	2.24.0

Providers

Name	Version
google	5.44.2
kubernetes	2.24.0

Modules

Name	Source	Version
composer	./modules/composer	n/a
data-pipelines	./modules/data-pipeline	n/a
dataproc	./modules/dataproc	n/a
dbt_docker_image	./modules/dbt_docker_image	n/a
gcr	./modules/gcr	n/a
jupyter_docker_image	./modules/jupyter_docker_image	n/a
vertex_ai_workbench	./modules/vertex-ai-workbench	n/a
vpc	./modules/vpc	n/a

Resources

Name	Type
google_compute_firewall.allow-all-internal	resource
kubernetes_service.dbt-task-service	resource
google_client_config.provider	data source
google_container_cluster.composer-gke-cluster	data source

Inputs

Name	Description	Type	Default	Required
ai_notebook_instance_owner	Vertex AI workbench owner	string	n/a	yes
project_name	Project name	string	n/a	yes
region	GCP region	string	"europe-west1"	no

Outputs

Name	Description
project_name	Project identifier
terraform_service_account	Terraform service account
terraform_state_bucket	Terraform state bucket