FTP XML InfluxDB Bridge

A Python service that connects to an FTP server, downloads XML files, parses metrics, and writes them into an InfluxDB database. Also provides a REST API for querying metrics.

---

🔧 Features

• Connects to an FTP server and downloads files.
• Parses XML files to extract metrics.
• Converts metrics into InfluxDB line protocol format.
• Sends metrics to an InfluxDB server.
• REST API for querying metrics.
• Uses environment variables for configuration.
• Includes logging and error handling.

---

📁 Folder Structure

nexusacs-file-ingest/
│
├── main.py # Entry point of the application
├── .env # Environment variables
├── app/
│ ├── api.py # API server entry point
│ ├── services/
│ │ ├── file_ingest_service.py # Handles FTP connection, file ingestion and threading
│ │ ├── data_store_service.py # Handles storage of metrics into InfluxDB
│ │ └── api_service.py # FastAPI service implementation
│ ├── utils/
│ │ ├── metric_adapter.py # XML parsing and data adaptation
│ │ └── utils.py # Utility functions (e.g., name extractor)

---

⚙️ Installation

git clone <your-repo-url>
cd <your-project-folder>
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

---

🔐 Environment Variables

Create a .env file with the following keys:

FTP_HOST=your_ftp_host
FTP_PORT=21
FTP_USER=username
FTP_PASS=password
FTP_FOLDER_PATH=/remote/path
LOCAL_STORAGE_FOLDER=./downloads

INFLUX_URL=http://localhost:8086
INFLUX_TOKEN=your_token
INFLUX_ORG=your_org
INFLUX_BUCKET=your_bucket

LOG_LEVEL=INFO

---

▶️ Running the Project

File Ingestion Service

python app/main.py --mode ingest

When you run this command:

• It connects to the FTP server
• Downloads and stores XML files
• Parses them to extract metric data
• Sends metrics to InfluxDB

API Service

python app/main.py --mode api
# or
python app/api.py

Additional options:

python app/api.py --host 0.0.0.0 --port 8080

API Documentation will be available at:

• http://127.0.0.1:8000/docs (Swagger UI)
• http://127.0.0.1:8000/redoc (ReDoc)

Both Services

python app/main.py --mode both

This will start both the file ingestion service and the API service simultaneously. You can also specify host and port for the API service:

python app/main.py --mode both --host 0.0.0.0 --port 8080

---

🔄 Process Flow

1. main.py starts either the file ingestion service or the API service based on mode.
2. file_ingest_service.py handles:
   • FTP connection
   • XML file download
   • Calling the parser
3. metric_adapter.py:
   • Parses XML
   • Converts metrics to Influx line protocol
4. Data is sent to InfluxDB
5. api_service.py provides REST endpoints for querying data

---

📦 Requirements

• Python 3.8+
• Required packages in requirements.txt

---

🌐 API Endpoints

Health Check

• GET / - Health check endpoint

Query Metrics

• POST /api/v1/query - Generic query endpoint
• POST /api/v1/query/daily - Daily metrics query
• POST /api/v1/query/weekly - Weekly metrics query
• POST /api/v1/query/monthly - Monthly metrics query
• POST /api/v1/query/yearly - Yearly metrics query
• POST /api/v1/query/csv - Query metrics and return results in CSV format with KPI names and Min, Mean, Max values

Grafana Integration

• POST /api/ds/query - Grafana data source query endpoint (compatible with InfluxDB data source)

Each query endpoint accepts a JSON body with:

• query_json: Object defining the query parameters
• range: Object with either:
  • from_date and to_date for custom ranges, or
  • range_type for predefined ranges (daily/weekly/monthly/yearly)
• grafana_url: String with the Grafana URL

The CSV endpoint returns a downloadable CSV file with KPI names and their Min, Mean, and Max values, ignoring data points with value 0.

---

👨‍💻 Author

Shaikh Faisal Hossain, Saad Mohammad Minhaz

---

📝 License

Private Project. All rights reserved.