🚀 GitHub Archive Live Dashboard

A real-time data pipeline and dashboard that visualizes live GitHub activity from the GitHub Archive dataset.

🎯 Features

• Real-time Data Collection: Continuously fetches GitHub Archive data
• Live Dashboard: Auto-refreshing Streamlit dashboard
• Redis Backend: Fast in-memory data storage
• Multiple Visualizations:
  • Programming language popularity
  • Trending repositories
  • Event type breakdown
  • Top active users
  • Live event stream

🏗️ Architecture

┌─────────────────┐
│ GitHub Archive  │
│   (gharchive)   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Collector     │ ◄── Fetches hourly data
│  (collector.py) │     Processes events
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│     Redis       │ ◄── Stores aggregated stats
│   (in-memory)   │     Sorted sets & hashes
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Dashboard     │ ◄── Reads from Redis
│   (main.py)     │     Real-time updates
└─────────────────┘

🚀 Quick Start

Choose the method that works best for you:

🐳 Method 1: Docker (Recommended - Easiest)

Prerequisites: Docker & Docker Compose (Install Docker)

# 1. Clone the repository
git clone <your-repo-url>
cd github-live-dashboard

# 2. Setup environment
cp .env.example .env

# 3. Start everything
docker compose up --build

# 4. Open browser
# http://localhost:8501

That's it! The collector will automatically start fetching GitHub data.

Useful commands:

# Run in background
docker compose up -d

# View logs
docker compose logs -f

# Stop services
docker compose down

---

🐍 Method 2: Standalone (Without Docker)

Prerequisites: Python 3.8+, pip, Redis (Install Redis)

Quick Setup (Linux/Mac)

# 1. Clone the repository
git clone <your-repo-url>
cd github-live-dashboard

# 2. Run automated setup
chmod +x setup-standalone.sh
./setup-standalone.sh

# 3. Start services
./run-all.sh

# 4. Open browser
# http://localhost:8501

Manual Setup (All Platforms)

# 1. Install Redis
# Ubuntu/Debian: sudo apt install redis-server
# Mac: brew install redis
# Windows: See WINDOWS_SETUP.md

# 2. Start Redis
sudo systemctl start redis-server  # Linux
brew services start redis           # Mac

# 3. Setup Python
python3 -m venv .venv
source .venv/bin/activate           # Linux/Mac
pip install -r requirements.txt

# 4. Configure environment
cp .env.example .env
# Edit .env: Change REDIS_HOST=redis to REDIS_HOST=localhost

# 5. Run services (in separate terminals)
python src/collector.py      # Terminal 1
streamlit run src/main.py    # Terminal 2

---

💡 Which Method Should I Use?

Choose Docker if...	Choose Standalone if...
✅ You want easiest setup	✅ You can't install Docker
✅ You're okay installing Docker	✅ You already have Python/Redis
✅ You want isolated environment	✅ You want more control
✅ Works on any OS	✅ Lower resource usage

---

🔧 VS Code Dev Containers (Optional)

For development with VS Code:

1. Install "Dev Containers" extension
2. Open project in VS Code
3. Press F1 → "Dev Containers: Reopen in Container"
4. Dashboard auto-starts at http://localhost:8501

📁 Project Structure

github-live-dashboard/
├── .devcontainer/
│   └── devcontainer.json          # VS Code dev container config
├── src/
│   ├── collector.py               # Data collection service
│   └── main.py                    # Streamlit dashboard
├── docker-compose.yml             # Service orchestration
├── Dockerfile                     # Application container
├── requirements.txt               # Python dependencies
├── .env                          # Environment variables
└── README.md                     # This file

🔧 Configuration

Environment Variables

Edit .env to customize:

# How many hours of historical data to fetch initially
LOOKBACK_HOURS=2

# Redis connection
REDIS_HOST=redis
REDIS_PORT=6379

# Streamlit configuration
STREAMLIT_SERVER_PORT=8501
STREAMLIT_SERVER_ADDRESS=0.0.0.0

Dashboard Settings

The dashboard includes controls for:

• Auto Refresh: Enable/disable automatic updates
• Refresh Interval: Set update frequency (5-60 seconds)
• Manual Refresh: Force immediate data reload

📊 Data Flow

1. Collector (collector.py):

   • Fetches compressed JSON files from https://data.gharchive.org
   • Processes each event (Push, Watch, Fork, Issues, etc.)
   • Updates Redis sorted sets and counters
   • Maintains hourly statistics

2. Redis Storage:

   • trending_repos: Sorted set of repository activity
   • language_popularity: Sorted set of language counts
   • event_types: Sorted set of event type counts
   • top_users: Sorted set of user activity
   • recent_events: List of last 100 events
   • hourly_stats:{hour}: Hash maps for hourly breakdowns
   • total_events: Counter for all processed events

3. Dashboard (main.py):

   • Reads from Redis every N seconds
   • Generates interactive Plotly visualizations
   • Displays real-time metrics and charts

🐛 Troubleshooting

Dashboard shows "Waiting for data"

• Check if collector is running: docker-compose ps
• View collector logs: docker-compose logs ingestor
• The collector fetches data from GitHub Archive, which may take 1-2 minutes initially

Redis connection failed

• Ensure Redis is running: docker-compose ps redis
• Check Redis logs: docker-compose logs redis
• Verify port 6379 is not in use: lsof -i :6379

Collector not processing events

• GitHub Archive may be temporarily unavailable
• Check network connectivity
• Try reducing LOOKBACK_HOURS in .env

📈 Scaling for Production

For production deployments, consider:

1. Persistent Redis: Use Redis with persistence (RDB/AOF)
2. Database: Store historical data in PostgreSQL/TimescaleDB
3. Caching: Add Redis caching layer for dashboard queries
4. Load Balancing: Use multiple dashboard instances behind nginx
5. Monitoring: Add Prometheus/Grafana for system metrics
6. Error Handling: Implement retry logic and dead letter queues

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Data provided by GitHub Archive
• Built with Streamlit
• Charts powered by Plotly

📧 Contact

Created by @IamIremIdil

---

⭐ If you found this project useful, please consider giving it a star! ⭐