Large Scale Data Analysis

A collection of Large Scale Data Analysis projects completed as part of the Large Scale Data Analysis course at AGH University of Science and Technology.

The repository covers distributed data processing, Apache Spark, SQL, graph analytics and scalable data analysis techniques.

Academic Context

The exercise topics and task requirements were provided as part of the course. All implementations, experiments and solutions presented in this repository are my own work.

Topics

1. RDD
2. SQL
3. Apache Spark Cluster
4. Structured Data
5. GraphFrames

Technology Stack

• Python
• Apache Spark
• Spark SQL
• Spark RDD
• Spark DataFrames
• GraphFrames
• PySpark
• Jupyter Notebook

Setup

Clone the repository:

git clone https://github.com/bjamiolkowski/agh-large-scale-data-analysis.git
cd agh-large-scale-data-analysis

Create and activate a virtual environment:

python -m venv .venv

Windows:

.venv\Scripts\activate

Linux / macOS:

source .venv/bin/activate

Install the required dependencies:

pip install -r requirements.txt

Repository Structure

agh-large-scale-data-analysis/
├── 01-rdd/
├── 02-sql/
├── 03-spark-cluster/
├── 04-structured-data/
├── 05-graphframes/
├── .gitignore
├── LICENSE
├── README.md
└── requirements.txt

License

This repository is available under the MIT License.