Develop

.dockerignore

@@ -0,0 +1,31 @@
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+.Python
+env/
+venv/
+.venv
+pip-log.txt
+pip-delete-this-directory.txt
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.log
+.git
+.gitignore
+.mypy_cache
+.pytest_cache
+.hypothesis
+*.egg-info/
+dist/
+build/
+*.egg
+.DS_Store
+tests/
+docs/
+*.md

.github/workflows/ci.yml

@@ -0,0 +1,47 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r requirements.txt
+        pip install pytest flake8 black
+        playwright install chromium
+
+    - name: Run tests
+      run: pytest tests/ -v
+
+    - name: Lint
+      run: |
+        flake8 src/ --max-line-length=120
+        black --check src/
+
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Build Docker image
+      run: docker build -t surenav:test .
+
+    - name: Test Docker image
+      run: |
+        docker run -d -p 8000:8000 surenav:test
+        sleep 10
+        curl -f http://localhost:8000/health || exit 1

Dockerfile

@@ -0,0 +1,26 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    wget \
+    gnupg \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Install Playwright browsers
+RUN playwright install chromium
+RUN playwright install-deps chromium
+
+# Copy application code
+COPY src/ ./src/
+
+# Expose port
+EXPOSE 8000
+
+# Run the server
+CMD ["python", "-m", "uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8000"]

README.md

@@ -1,5 +1,3 @@
-# surenav
-"Open-source web unblocker for AI agents - no API keys required"
 # 🧭 SureNav
 
 [![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
@@ -8,7 +6,7 @@
 
 **SureNav** is a fully open-source web navigation and scraping toolkit for AI agents that requires **zero API keys**. It provides anti-detection browser automation, intelligent proxy rotation, and web unblocking capabilities—completely free.
 
-> 🚀 **Drop-in replacement for paid services** like Massive's ClawPod, but open source and free forever.
+> 🚀 **Drop-in replacement for paid services** like Massive's ClawPod, but open source and free forever.
 
 ## ✨ Features
 
@@ -25,4 +23,83 @@
 ### Docker (Easiest)
 
 ```bash
-docker run -p 8000:8000 ghcr.io/YOUR_USERNAME/surenav:latest
+docker run -p 8000:8000 ghcr.io/colomboai-com/surenav:latest
+```
+
+### Local Installation
+
+```bash
+pip install surenav
+playwright install chromium
+surenav-server
+```
+
+## 📖 Usage
+
+### Fetch a Web Page
+
+```bash
+curl "http://localhost:8000/browser?url=https://example.com"
+```
+
+### Search Google
+
+```bash
+# JSON output
+curl "http://localhost:8000/search?terms=open+source+ai&format=json"
+
+# HTML output
+curl "http://localhost:8000/search?terms=python+tutorial"
+```
+
+### Python SDK
+
+```python
+from surenav import StealthBrowser
+
+browser = StealthBrowser()
+result = await browser.fetch_page("https://example.com")
+print(result["content"])
+```
+
+## 🏗️ Architecture
+
+```mermaid
+graph TD
+    A[Client Request] --> B[FastAPI Server]
+    B --> C[Proxy Manager]
+    C --> D[Free Proxy Lists]
+    B --> E[Stealth Browser]
+    E --> F[Playwright + Stealth]
+    F --> G[Target Website]
+    G --> H[Clean Content]
+```
+
+## ⚙️ Configuration
+
+Environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SURENAV_PORT` | 8000 | Server port |
+| `SURENAV_PROXY_REFRESH` | 300 | Proxy refresh interval (seconds) |
+| `SURENAV_HEADLESS` | true | Run browser headless |
+| `SURENAV_MAX_RETRIES` | 3 | Retry attempts per request |
+
+## 🛡️ Anti-Detection Features
+
+- ✅ User agent rotation
+- ✅ Viewport fingerprint randomization
+- ✅ WebGL/Canvas noise injection
+- ✅ Automation flag removal
+- ✅ Mouse movement humanization
+- ✅ Cookie/session handling
+- ✅ TLS/JA3 fingerprint randomization (planned)
+
+## 🤝 Contributing
+
+Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md)
+
+## 📜 License
+
+MIT - Free for personal and commercial use.

docs/ARCHITECTURE.md

@@ -0,0 +1,31 @@
+# SureNav Architecture
+
+## Overview
+SureNav is built with a modular architecture for web scraping and browser automation.
+
+## Components
+
+### 1. StealthBrowser
+- Playwright-based browser automation
+- Anti-detection fingerprinting
+- JavaScript rendering
+
+### 2. FreeProxyManager
+- Aggregates free proxies from multiple sources
+- Validates and rotates proxies
+- Handles proxy failures
+
+### 3. GoogleSearcher
+- Scrapes Google search results
+- Bypasses rate limiting
+- Returns structured data
+
+### 4. FastAPI Server
+- RESTful API endpoints
+- Async request handling
+- Health monitoring
+
+## Data Flow
+```
+Client → FastAPI → ProxyManager → StealthBrowser → Target Site → Response
+```

docs/DEPLOYMENT.md

@@ -0,0 +1,32 @@
+# SureNav Deployment Guide
+
+## Docker Deployment
+
+### Build Image
+```bash
+docker build -t surenav:latest .
+```
+
+### Run Container
+```bash
+docker run -d -p 8000:8000 surenav:latest
+```
+
+## Local Deployment
+
+### Install Dependencies
+```bash
+pip install -r requirements.txt
+playwright install chromium
+```
+
+### Run Server
+```bash
+python -m uvicorn src.server:app --host 0.0.0.0 --port 8000
+```
+
+## Environment Variables
+- `SURENAV_PORT`: Server port (default: 8000)
+- `SURENAV_PROXY_REFRESH`: Proxy refresh interval in seconds (default: 300)
+- `SURENAV_HEADLESS`: Run browser headless (default: true)
+- `SURENAV_MAX_RETRIES`: Max retry attempts (default: 3)

pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "surenav"
+version = "1.0.0"
+description = "Open-source web unblocker for AI agents"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Your Name", email = "your.email@example.com"}
+]
+keywords = ["web scraping", "browser automation", "ai agents", "anti-detection", "proxies"]
+
+[project.urls]
+Homepage = "https://github.com/ColomboAI-com/surenav"
+Documentation = "https://github.com/ColomboAI-com/surenav#readme"
+Repository = "https://github.com/ColomboAI-com/surenav.git"
+Issues = "https://github.com/ColomboAI-com/surenav/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]

requirements.txt

@@ -0,0 +1,12 @@
+fastapi>=0.109.0
+uvicorn[standard]>=0.27.0
+playwright>=1.41.0
+playwright-stealth>=1.0.6
+beautifulsoup4>=4.12.3
+markdownify>=0.11.6
+aiohttp>=3.9.3
+aiofiles>=23.2.1
+fake-useragent>=1.4.0
+pydantic>=2.6.0
+pyyaml>=6.0.1
+httpx>=0.26.0

scripts/install-playwright.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+# Install Playwright and browsers
+
+echo "Installing Playwright..."
+pip install playwright
+
+echo "Installing Chromium browser..."
+playwright install chromium
+
+echo "Installing system dependencies..."
+playwright install-deps chromium
+
+echo "✅ Playwright installation complete!"

setup.py

@@ -0,0 +1,50 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+setup(
+    name="surenav",
+    version="1.0.0",
+    author="Your Name",
+    author_email="your.email@example.com",
+    description="Open-source web unblocker for AI agents",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/ColomboAI-com/surenav",
+    packages=find_packages(where="src"),
+    package_dir={"": "src"},
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
+        "Programming Language :: Python :: 3.12",
+        "Topic :: Internet :: WWW/HTTP",
+        "Topic :: Software Development :: Libraries :: Python Modules",
+    ],
+    python_requires=">=3.9",
+    install_requires=[
+        "fastapi>=0.109.0",
+        "uvicorn[standard]>=0.27.0",
+        "playwright>=1.41.0",
+        "playwright-stealth>=1.0.6",
+        "beautifulsoup4>=4.12.3",
+        "markdownify>=0.11.6",
+        "aiohttp>=3.9.3",
+        "aiofiles>=23.2.1",
+        "fake-useragent>=1.4.0",
+        "pydantic>=2.6.0",
+        "pyyaml>=6.0.1",
+        "httpx>=0.26.0",
+    ],
+    entry_points={
+        "console_scripts": [
+            "surenav-server=surenav.server:main",
+        ],
+    },
+)

src/__init__.py

@@ -0,0 +1,12 @@
+"""
+SureNav - Open-source web unblocker for AI agents
+"""
+
+__version__ = "1.0.0"
+__author__ = "Your Name"
+
+from .browser import StealthBrowser
+from .proxy_manager import FreeProxyManager
+from .search import GoogleSearcher
+
+__all__ = ["StealthBrowser", "FreeProxyManager", "GoogleSearcher"]