Rust Search Engine

A search engine built from scratch in Rust.

The goal of this project is to explore how modern search systems work by implementing core concepts such as document processing, text tokenization, indexing, and search algorithms while focusing on Rust's performance and safety features.

📑 Features:

• Document processing: Reads and processes text documents from the filesystem.
• Text tokenization: Converts raw text into normalized searchable tokens.
• Inverted indexing: Builds an index mapping terms to the documents where they appear.
• Search capabilities: Designed to provide efficient document lookup based on user queries.
• High-performance architecture: Designed to explore Rust concepts such as ownership, lifetimes, concurrency, and async programming.

Architecture

The project follows a modular architecture:

Documents
    |
    v
Parser
    |
    v
Tokenizer
    |
    v
Indexer
    |
    v
Search Engine
    |
    v
Ranking
    |
    v
Search Results

Each component is responsible for a specific part of the search pipeline, allowing the system to evolve with features such as ranking algorithms, parallel processing, and persistent storage.

🛠️ Requirements

• Rust (latest stable version)
• Cargo (Rust's package manager and build system)

🔧 Project Setup

Clone the repository:

git clone https://github.com/drammos/search-engine.git
cd search-engine

Build the project:

cargo build

Run tests:

cargo test

How to Run

Create a directory containing text documents:

documents/
├── rust.txt
├── tokio.txt
└── ownership.txt

Run the application:

cargo run -- ./documents

The application will process the documents and build the search index.

Future Direction

The project aims to explore more advanced search engine concepts, including:

• Relevance ranking algorithms
• Query optimization
• Concurrent indexing
• Async document crawling
• Persistent and distributed storage

👤 Author

Rammos Dimitrios

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.