A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

This repo is the implementation of the paper A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning.

Preparation

• Spark Setup: spark-setup/README.md

• Python Setup (python 3.9+), pip install -r requirements.txt

• Trace Collection Scripts udao_trace_examples/README.md

Compile Time/Runtime Optimization

See details in playground/README.md