XML Job Importer

This project implements a scalable job import system that fetches jobs from multiple RSS/XML feeds, queues them using Redis and BullMQ, imports them into MongoDB using worker processes, and tracks the import history in an Admin UI built with Next.js.

Features

• Fetch jobs from multiple RSS/XML feeds
• XML-to-JSON conversion
• Queue-based background processing with Redis and BullMQ
• Worker processes with concurrency and retry/backoff
• Import History Tracking stored in MongoDB
• Admin UI with Next.js to view import logs
• Real-time updates with WebSockets
• Modular, scalable, microservice-friendly design

Repository Structure

/client # Next.js Admin UI /server # Express API, Queue, Worker /docs # Documentation and Architecture diagram

Technologies Used

• Frontend: Next.js
• Backend: Node.js with Express
• Queue: BullMQ + Redis
• Database: MongoDB (Mongoose)
• Real-time: Socket.IO
• Deployment ready for Docker/Vercel/Render

🛠️ Setup Instructions

1. Clone the repo

git clone https://github.com/i-niranjan/job-import-system.git
cd job-import-system

2. Setup Server

cd server
npm install

--- Create .env [local-example]

PORT=3012
MONGO_URI=mongodb://localhost:27017/jobImporter
REDIS_URL=redis://localhost:6379
QUEUE_CONCURRENCY=5
CRON_SCHEDULE=0 * * * *

Run Server

npm run dev

3 Setup Client

cd ../client
npm install
npm run dev

--- Create .env [local-example]

NEXT_PUBLIC_API_URL=http://localhost:3012

Run Server

npm run dev

Documentation

For system architecture and design decisions, see:

/docs/architecure.md