A real-time analytics tool for law enforcement to detect suspicious patterns in massive IPDR logs using Kafka and Flink. Developed for a cybersecurity hackathon to aid in digital forensic investigations.
In digital forensics, investigators often face the monumental task of sifting through millions of Internet Protocol Detail Record (IPDR) logs to find evidence. This process is manual, time-consuming, and prone to error.
IPDR FlowAnalyzer was built to solve this problem. It is an intelligent, real-time data processing pipeline that ingests raw IPDR logs and automatically maps communication flows between an initiator (A-Party) and a recipient (B-Party). By identifying connections and flagging anomalies in real-time, our tool provides law enforcement with actionable intelligence, significantly reducing investigation time and increasing the efficiency of digital forensic support by over 30%.
- Real-Time Log Ingestion: Ingests high-volume IPDR streams using Apache Kafka without data loss.
- A-Party to B-Party Mapping: Intelligently parses complex logs to identify and map initiator vs. recipient communications.
- Automated Anomaly Detection: Utilizes Apache Flink for complex event processing to flag suspicious patterns, such as connections to known malicious IPs or unusual data transfer volumes.
- Interactive Visualization: A user-friendly dashboard featuring:
- Geographical Map View: To visualize the physical locations of communicating parties.
- Graph-Based Network View: To explore relationships and connections between different entities.
- Advanced Query & Search: A powerful search interface for investigators to filter and query specific sessions, IPs, or timeframes.
- Data Normalization: Handles various IPDR formats by normalizing them into a unified schema for consistent analysis.
The system is designed as a distributed, scalable pipeline to handle massive data loads.
This project leverages a modern, high-performance technology stack for real-time data processing and analysis.
| Component | Technology |
|---|---|
| Data Streaming | |
| Stream Processing |  |
| Backend & Logic | |
| Frontend |  |
| Database |  /  |
| Containerization |  |
To get a local copy up and running, follow these simple steps.
- Docker and Docker Compose
- Python 3.9+
- Node.js and npm (for the frontend)
-
Clone the repository:
git clone [https://github.com/your_username/ipdr-flowanalyzer.git](https://github.com/your_username/ipdr-flowanalyzer.git) cd ipdr-flowanalyzer -
Set up environment variables:
- Create a
.envfile from theenv.example. - Fill in the necessary configuration details.
- Create a
-
Launch services with Docker Compose:
- This will start Kafka, Flink, and the required databases.
docker-compose up -d
-
Set up the Backend:
cd backend pip install -r requirements.txt python app.py -
Set up the Frontend:
cd frontend npm install npm start
Once all services are running:
- Open the application: Navigate to
http://localhost:3000in your browser. - Feed data into Kafka: Use the provided Python scripts in the
/scriptsdirectory to simulate an IPDR log stream.python scripts/produce_logs.py --file data/sample_ipdr.csv
- View the Dashboard: Watch as the dashboard populates with real-time connections, and see anomalies being flagged as they are detected by the Flink job.
