PDFFULLSEARCH

Description.

Dockerized rest API to index pdf files into elasticsearch database. To search using Ollama embeddings with vectorial index or full text search with inverted index.

Components

• Python
• Nginx
• Django
• Ollama
• Elasticsearch
• Apache Kafka
• Apache Tika
• Postgres

Requirements

1. Docker and compose Docker installation script

Installation

1. Clone the repository:

git clone https://github.com/fluques/pdffullsearch.git

2. Enter directory:

cd pdffullsearch

3. Set default settings:

cp env.default .env

4. Run docker-compose file:

docker compose -f .\compose.yaml  up --build --force-recreate

Running service on:

http://127.0.0.1:8989

Curl usage

Upload a pdf file

curl -XPUT --data-binary "@filename.pdf" http://127.0.0.1:8989/api/pdffile/upload/filename.pdf/

Search knn with embeddings

curl -XGET http://127.0.0.1:8989/api/pdffile/knn_search/ \
  -H "Content-Type: application/json" \
  -d '{"query": "query text", "k": "3", "candidates":"100"}'

Search full text with query text

curl -XGET http://127.0.0.1:8989/api/pdffile/fulltext_search/ \
  -H "Content-Type: application/json" \
  -d '{"query": "query text"}'

Python usage

Upload a pdf file

import requests
url = 'http://127.0.0.1:8989/api/pdffile/upload/filename.pdf/' 
# Open the file in binary read mode ('rb')
with open('filename.pdf', 'rb') as f:
    files = {'file': f} 
    response = requests.put(url, files=files)

Search knn with embeddings

import requests
url = 'http://127.0.0.1:8989/api/pdffile/knn_search/'
payload ={
    'query':'query text',
    'k' : '3',
    'candidates': '100'
}
response = requests.get(url, json=payload)

Search full text with query text

import requests
url = 'http://127.0.0.1:8989/api/pdffile/fulltext_search/' 
payload ={
    'query':'query text'
}
response = requests.get(url, json=payload)