diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..8d4e9eb --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,81 @@ +# Contributing to Loreo + +Thanks for helping improve Loreo. This guide captures the repo's current contribution flow and quality gates. + +## Before You Start + +- Read `README.md` for setup and repo structure. +- Check whether your change is already covered by an active plan under `docs/plans/` or `context/features/`. +- Keep secrets, credentials, and local environment values out of commits. + +## Branches + +- Work on a dedicated branch, not `main`. +- Prefer descriptive branch names such as `feature/`, `fix/`, `chore/`, or `refactor/`. +- Keep branch names short and specific to the change. + +## Local Checks + +Run the relevant checks before asking for review: + +```bash +pnpm lint +pnpm typecheck +pnpm test +pnpm build +``` + +For focused changes, run the smallest useful subset first, then the broader checks that cover your files. + +## Testing Strategy + +- Prefer tests for hooks, utilities, library code, and API/query wrappers. +- Cover happy paths and meaningful error cases. +- Add component or page tests only when the behavior cannot be covered cleanly at a lower layer. +- Skip tests that only restate markup or implementation details. +- Use judgment: a test should add confidence or protect a real regression risk. + +## Formatting and Hooks + +- Pre-commit hooks run `lint-staged`. +- TypeScript, TSX, JSON, Markdown, YAML, and YML files are formatted through `oxfmt`. +- TypeScript and TSX files are also checked with `oxlint`. +- Commit messages are validated with `commitlint`. + +## Commit Messages + +- Use conventional commits. +- The repo includes Commitizen, so `pnpm commit` is the easiest way to create a compliant message. +- Keep commits small and logically grouped. + +Examples: + +```text +feat(web): add article tag filter +fix(server): handle empty import rows +docs: update notes +``` + +## Pull Requests + +- Include a short summary of what changed and why. +- Mention any commands you ran to verify the change. +- Call out known risks, tradeoffs, or follow-up work. +- Reference the related issue or plan when one exists. + +## Docs and Release Notes + +- Update `README.md` when setup, scripts, or deployment steps change. +- Add or update tests when behavior changes. + +## Security + +- Do not commit secrets, tokens, passwords, or private URLs. +- Use example environment files as the source of truth for required config. +- Review changes that touch auth, storage, networking, or deployment carefully. + +## If You're Unsure + +- Ask before making large structural changes. +- Prefer the smallest correct change. +- If a repository convention conflicts with this guide, follow the repo convention. diff --git a/README.md b/README.md index f6e85f5..20ea961 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,85 @@ # Loreo - +![Cover](./docs/images/cover.png) + +Loreo is a read-it-later app for saving articles worth revisiting, built with self-hosting in mind + +## Demo + +[Demo](https://loreo-demo.onrender.com) + +> The demo runs on a free tier Render instance, expect a cold start on first visit (~30s). Self-hosted instances don't have this limitation +> The demo is read-only, you can't save or edit articles + +## Features + +### Reading Experience + +- Focused reader view +- Customize theme, font, spacing, and alignment +- Highlight and annotate text +- Paragraph focus indicator +- Full keyboard navigation (`j/k`, arrow keys) +- Automatic reading progress saves + +### Saving & Organization + +- Save links with automatic article extraction +- Favorite articles for quick access +- Archive articles to hide them from your reading list +- Organize with tag groups and tags +- Filter by status, priority, read length, and search +- Priority system for intentional reading + +### Utilities + +- CSV import with field mapping +- Docker Compose support, run locally or as separate web/server processes + +## Why Loreo? + +Most read-later apps eventually become cluttered: too many features, too much noise, and endless accumulation. Loreo aims to be different by focusing on the core functionality of saving and reading articles, while providing a clean and distraction-free interface. + +Loreo is built around a simpler idea: you shouldn't have to consume everything immediately. + +Save something once, trust it will be there later, and come back when you actually have the time and focus for it. + +The goal is not productivity maximization, but creating a calmer relationship with information. + +## Why this exists + +I was a [Pocket](https://getpocket.com) user for a decade, but they decided to [shut down their service](https://blog.mozilla.org/en/mozilla/building-whats-next/). I tried finding alternatives, but most of them either just bookmark links without actually saving the content, have cluttered dashboards, or have too many features that I don't need. So I decided to build a calm, focused read-it-later app that I could self-host myself + +## Behind the Name + +The name "Loreo" is derived from "Lore" (the story or content). It represents the idea of letting things flow naturally: saving it now and coming back to it later, without pressure. It's a small reflection of the app itself: calm, intentional, and designed around revisiting things when the time feels right ## Stack -**Frontend** (`apps/web`) +This is a monorepo project using pnpm workspaces based on my [monorepo template](https://github.com/technowizard/monorepo-template). + +**Frontend** - [React 19](https://react.dev) + [TypeScript](https://www.typescriptlang.org) - [Vite 8](https://vite.dev) - build tool and dev server -- [TanStack Router](https://tanstack.com/router) - file-based routing -- [TanStack Query](https://tanstack.com/query) - server state and data fetching +- [TanStack Router](https://tanstack.com/router) - file-based routing with typed URL search params +- [TanStack Query](https://tanstack.com/query) - server state, centralized mutation invalidation via `MutationCache` +- [ky](https://github.com/sindresorhus/ky) HTTP client based on native Fetch API - [Tailwind CSS v4](https://tailwindcss.com) + [shadcn/ui](https://ui.shadcn.com) (Base UI variant) -- [i18next](https://www.i18next.com) - i18n with EN/ID support out of the box +- [i18next](https://www.i18next.com) - i18n with EN/ID support (can be extended to other languages) +- [Zustand](https://zustand-demo.pmnd.rs) + [Immer](https://immerjs.github.io/immer) - client-only state (ephemeral UI state that doesn't belong in the URL or server cache) - [Vitest](https://vitest.dev) + [Testing Library](https://testing-library.com) + [MSW](https://mswjs.io) - unit and integration tests -**Backend** (`apps/server`) +**Backend** - [Hono](https://hono.dev) - lightweight HTTP framework - [Drizzle ORM](https://orm.drizzle.team) + PostgreSQL - type-safe database access +- [Redis](https://redis.io) + [BullMQ](https://github.com/taskforcesh/bullmq) - background jobs processing +- [Playwright](https://playwright.dev) + [Camoufox](https://github.com/daijiro/camoufox) - browser service for crawling and article extraction +- [Mozilla Readability](https://github.com/mozilla/readability) - content extraction from web pages - [Zod](https://zod.dev) - request/response validation - OpenAPI docs via `@hono/zod-openapi` + Scalar UI -- [Vitest](https://vitest.dev) - integration tests against a real database +- [Vitest](https://vitest.dev) - handler tests with centralized in-memory adapters, no database required **Monorepo** @@ -30,48 +89,87 @@ - [oxlint](https://oxc.rs/docs/guide/usage/linter) + [oxfmt](https://oxc.rs) + [@fdhl/oxlint-config](https://github.com/technowizard/oxlint-config) - fast linting and formatting - [Commitizen](https://commitizen-tools.github.io/commitizen) + [commitlint](https://commitlint.js.org) - conventional commits -## Project structure +## Note About Development -``` +This project is built as a personal tool I use daily, with a focus on intentional design decisions over feature accumulation. I'm open to suggestions and contributions that align with that philosophy + +## Acknowledgements + +Open-source software and transparent architecture discussions helped Loreo tremendously during development, so proper attribution feels important + +- [Karakeep](https://github.com/karakeep-app/karakeep) — inspiration for how scraping works and self-hosting patterns + +## Project Structure + +```text loreo/ ├── apps/ -│ ├── server/ # Hono API server -│ │ └── src/ -│ │ ├── db/ # Drizzle schema and migrations -│ │ ├── routes/ # Route definitions and handlers -│ │ ├── services/ -│ │ ├── repositories/ -│ │ └── tests/ # Integration tests -│ └── web/ # React frontend -│ └── src/ -│ ├── features/ # Feature-scoped components and API hooks -│ ├── pages/ # Page-level components -│ ├── routes/ # TanStack Router file-based routes -│ ├── components/ # Shared UI components -│ ├── lib/ # API client, query client, i18n, env -│ ├── locales/ # en / id translation files -│ └── tests/ # Test utilities, MSW handlers -└── packages/ - └── config/ # Shared TypeScript configs +│ ├── server/ # Hono API, jobs, storage, DB schema, migrations +│ └── web/ # React app, routes, features, shared UI +├── packages/ +│ └── config/ # Shared TypeScript configs +├── docs/ # Docs, self-hosting templates +├── docker-compose.yml # Local development stack +└── docker-compose.prod.yml ``` ## Prerequisites -- [Node.js](https://nodejs.org) 22+ -- [pnpm](https://pnpm.io) 10+ -- [Docker](https://docs.docker.com/get-started) (for the recommended dev setup) +- Node.js 22+ +- pnpm 10+ +- Docker, for the recommended local setup + +## Getting Started -## Getting started +Install dependencies: -### With Docker (recommended) +```bash +pnpm install +``` -Copy the root env file and fill in the required values: +Copy the root environment example: ```bash cp .env.example .env ``` -`.env` at the root is used by Docker Compose to configure local infrastructure and production compose values: +Start the Docker stack: + +```bash +pnpm dev +``` + +This starts Postgres, Redis, the browser extraction service, the API, and the web app. + +- Web app: http://localhost:3001 +- API: http://localhost:3000 +- API docs: http://localhost:3000/reference + +## Local Development Without Docker + +Run Postgres, Redis, and a Camoufox-compatible browser service yourself, then configure app env files: + +```bash +cp apps/server/.env.example apps/server/.env +cp apps/web/.env.example apps/web/.env +``` + +Start web and server directly: + +```bash +pnpm dev:local +``` + +Or run one app at a time: + +```bash +pnpm dev:web +pnpm dev:server +``` + +## Environment + +Root `.env` is used by Docker Compose: ```env NODE_ENV=development @@ -79,253 +177,172 @@ APP_PORT=3000 FRONTEND_PORT=3001 HOST_IP=localhost ORIGIN=http://localhost:3001 +PUBLIC_URL=http://localhost:3000 POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres POSTGRES_DB=app POSTGRES_PORT=5433 REDIS_PORT=6379 CORS_ORIGINS=http://localhost:3001 +JWT_SECRET=change-me-to-a-random-secret-at-least-32-chars ``` -Start everything: +Server-only local env lives in `apps/server/.env`. Web-only local env lives in `apps/web/.env`: -```bash -pnpm dev -# or: docker compose up +```env +VITE_API_URL=http://localhost:3000 ``` -This starts Postgres, runs migrations, and serves: - -- Frontend → http://localhost:3001 -- Backend → http://localhost:3000 -- API docs → http://localhost:3000/reference - -### Without Docker (local) - -1. Start a PostgreSQL instance manually +For production, see the [self-hosting section](./README.md#self-hosting). -2. Copy and configure the server env file: +## Commands ```bash -cp apps/server/.env.example apps/server/.env -``` - -3. Install dependencies and run: - -```bash -pnpm install +# Development +pnpm dev pnpm dev:local -``` +pnpm dev:web +pnpm dev:server -## Environment variables - -### `apps/server/.env` - -| Variable | Default | Description | -| ---------------------- | ------------------------------ | --------------------------------------- | -| `NODE_ENV` | `production` | `development` \| `production` \| `test` | -| `PORT` | `3000` | Server port | -| `HOST` | `localhost` | Server hostname | -| `APP_PORT` | `3000` | Docker/local helper for public app URL | -| `FRONTEND_PORT` | `3001` | Docker/local helper for frontend port | -| `HOST_IP` | `localhost` | Docker/local helper for public host | -| `DATABASE_USER` | - | **Required.** Postgres user | -| `DATABASE_PASSWORD` | - | **Required.** Postgres password | -| `DATABASE_DB` | `postgres` | Database name | -| `DATABASE_HOST` | `localhost` | Database host | -| `DATABASE_PORT` | `5432` | Database port | -| `DATABASE_POOL_MAX` | `10` | Max Postgres pool connections | -| `CORS_ORIGINS` | `http://localhost:3001` | Allowed origins, comma-separated | -| `JWT_SECRET` | - | **Required.** JWT signing secret | -| `REDIS_HOST` | `localhost` | Redis host | -| `REDIS_PORT` | `6379` | Redis port | -| `RATE_LIMIT_WINDOW_MS` | `60000` | Rate limit window in ms | -| `RATE_LIMIT_MAX` | `50` | Max requests per window | -| `BODY_SIZE_LIMIT` | `4194304` | Request body size limit in bytes | -| `PUBLIC_URL` | - | Public server URL for local file links | -| `STORAGE_PROVIDER` | `local` | `local` \| `local-docker` \| `s3` | -| `STORAGE_PATH` | - | Local storage path | -| `S3_ENDPOINT` | - | Optional S3-compatible endpoint | -| `S3_REGION` | `auto` | S3 region | -| `S3_ACCESS_KEY_ID` | - | S3 access key | -| `S3_SECRET_ACCESS_KEY` | - | S3 secret key | -| `S3_BUCKET_NAME` | - | S3 bucket name | -| `S3_PUBLIC_URL` | - | Public S3/CDN URL | -| `BROWSER_URL` | `ws://localhost:4444/camoufox` | Browser automation websocket URL | - -### `apps/web/.env` - -| Variable | Description | -| -------------- | ---------------------------------------------- | -| `VITE_API_URL` | Backend base URL, e.g. `http://localhost:3000` | - -## Development commands +# Build +pnpm build +pnpm build:web +pnpm build:server -```bash -# Start everything via Docker -pnpm dev +# Test +pnpm test +pnpm test:web +pnpm test:server -# Start frontend and backend directly (no Docker) -pnpm dev:local +# Quality +pnpm lint +pnpm lint:fix +pnpm typecheck -# Start individually -pnpm dev:web -pnpm dev:server +# Database +pnpm db:push +pnpm db:migrate +pnpm db:studio + +# Commit helper +pnpm commit ``` ## Database +The server uses Drizzle migrations under `apps/server/src/db/migrations`. + ```bash -# Push schema changes to the database (no migration file) +# Push schema changes directly pnpm db:push -# Generate and apply a migration +# Apply migrations pnpm db:migrate -# Open Drizzle Studio +# Inspect data pnpm db:studio ``` ## Testing -```bash -# Run all tests -pnpm test +Frontend tests use Vitest, Testing Library, and MSW. -# Run per app +```bash pnpm test:web -pnpm test:server ``` -**Frontend tests** use Vitest + Testing Library + MSW. MSW intercepts `fetch` at the network layer, so tests exercise the full component → hook → API client chain without a running server - -**Backend tests** are integration tests that require a real Postgres database. The test runner automatically applies migrations before the suite runs (`pnpm db:migrate:test`). Set up `apps/server/.env.test` with a separate test database before running - -Use the committed test example as a starting point: +Backend tests are integration tests and require a separate Postgres test database. Start from the example env: ```bash cp apps/server/.env.test.example apps/server/.env.test +pnpm test:server ``` -## Code quality - -```bash -# Lint and format all files -pnpm lint +The server test setup applies test migrations before running the suite. -# Type-check all packages -pnpm typecheck -``` +## Deployment -Linting and formatting run automatically on staged files via Husky pre-commit hooks. Commit messages are enforced to follow [Conventional Commits](https://www.conventionalcommits.org) +### Self-Hosting -```bash -# Interactive commit prompt -pnpm commit -``` +See [self-hosting guide](docs/SELF_HOSTING.md) for a complete guide covering environment setup, Docker Compose, reverse proxy, SSL, storage, backups, and upgrades. -## Deployment +Use the template files in `docs/self-hosting-templates/` as a starting point for your deployment: -### 1. Name your images +- `base.yml` - Base configuration with shared services and networks +- `neon_with_r2.yml` - Complete setup with Neon PostgreSQL and R2 storage (should be configured with your own credentials) -Production Compose is set up to use GHCR images by default. Update `docker-compose.prod.yml` if you publish under a different GitHub user or organization: +Production Compose expects published GHCR images. The Docker publish GitHub Actions workflow builds and publishes these images from `main`, version tags, or manual dispatch: -```yaml -loreo-browser: - image: ghcr.io/your-username/loreo-browser:latest +- `ghcr.io/technowizard/loreo-browser:latest` +- `ghcr.io/technowizard/loreo-server:latest` +- `ghcr.io/technowizard/loreo-web:latest` -loreo-server: - image: ghcr.io/your-username/loreo-server:latest +Build images from the repository root: -loreo-web: - image: ghcr.io/your-username/loreo-web:latest +```bash +docker build -f apps/server/Dockerfile.browser -t loreo-browser:latest . +docker build -f apps/server/Dockerfile.prod -t loreo-server:latest . +docker build -f apps/web/Dockerfile.prod -t loreo-web:latest . ``` -### 2. Build the images +The production web image is portable by default. Browser API calls use the same origin as the web app, and nginx proxies API routes to `API_UPSTREAM` at container startup. -Run the build commands from the **monorepo root**. The web image requires `VITE_API_URL` at build time - Vite bakes it into the static bundle. +Run production Compose locally: ```bash -docker build \ - -f apps/server/Dockerfile.browser \ - -t ghcr.io/your-username/loreo-browser:latest \ - . - -docker build \ - -f apps/server/Dockerfile.prod \ - -t ghcr.io/your-username/loreo-server:latest \ - . - -docker build \ - -f apps/web/Dockerfile.prod \ - --build-arg VITE_API_URL=https://api.yourdomain.com \ - -t ghcr.io/your-username/loreo-web:latest \ - . +cp .env.prod.example .env.prod +docker compose --env-file .env.prod -f docker-compose.prod.yml up -d ``` -### Browser service / Camoufox +The production stack serves: -Loreo uses Camoufox for server-side article extraction. There is no official Camoufox runtime Docker image, so this repo includes `apps/server/Dockerfile.browser` as part of the self-hosting bundle. The image installs `camoufox[geoip]`, fetches the browser runtime, and exposes a Playwright-compatible websocket at `ws://loreo-browser:4444/camoufox`. +- Web app: http://localhost:3001 +- API: http://localhost:3000 by default, or `SERVER_PUBLIC_PORT` when set -The browser websocket is internal-only in Compose. The server connects to it over the Docker network through `BROWSER_URL`; it should not be published directly to the internet. +> [!IMPORTANT] +> Replace `JWT_SECRET`, database credentials, CORS origins, public URLs, and any storage credentials before an internet-facing deployment. +> The values in `.env.example` are development placeholders, not production secrets. Start hosted deployments from `.env.prod.example`. -### 3. Test locally before pushing +## Using External Services -You can run the full production stack on your machine without pushing to any registry. Docker Compose uses locally built images if the tag already exists: +Loreo supports decoupling services for better scalability and reliability. You can use external services for: -```bash -# Create a local .env with production-like values -cp .env.example .env +- **Database**: PostgreSQL (e.g., Neon) +- **Redis**: For job queue and caching (e.g., Upstash) +- **Storage**: S3-compatible storage for images and other assets (e.g., Cloudflare R2) -docker compose -f docker-compose.prod.yml up -d -``` +I have tested this with Neon as the database provider, Upstash as the Redis provider, and Cloudflare R2 as the storage provider. To get the best performance, make sure each service's region is close to your server location -- Frontend → http://localhost:80 -- Backend → http://localhost:3000 +### Examples -Tear down when done: +If you want to use Neon, you can modify the following environment variables: -```bash -docker compose -f docker-compose.prod.yml down -v -``` - -### 4. Push and deploy - -Push the images to your registry: +Remove `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB`, then add `DATABASE_URL` to your `.env` file: -```bash -docker push ghcr.io/your-username/loreo-browser:latest -docker push ghcr.io/your-username/loreo-server:latest -docker push ghcr.io/your-username/loreo-web:latest +``` +DATABASE_URL= ``` -On your production server, create an `.env` file and bring the stack up: +For Upstash, you can use the following environment variables: -```bash -# Copy docker-compose.prod.yml and .env.example to the server, then: -cp .env.example .env -# Edit .env with real credentials - -docker compose -f docker-compose.prod.yml up -d +``` +REDIS_URL= # e.g., rediss://default:password@upstash-redis.example.com:6379 ``` -The server container runs database migrations automatically on startup before accepting traffic - -## Adding shadcn/ui components +For Cloudflare R2, you can use the following environment variables: -```bash -pnpm --filter web shadcn:add -# e.g.: pnpm --filter web shadcn:add dialog +``` +STORAGE_PROVIDER=s3 +STORAGE_ENDPOINT= +STORAGE_ACCESS_KEY_ID= +STORAGE_SECRET_ACCESS_KEY= ``` -## Adding a new feature +## Operational Notes -The frontend follows a feature-slice pattern. Each feature lives under `src/features//`: +- Article extraction uses a Playwright-compatible Camoufox websocket configured by `BROWSER_URL`. +- Local storage is available by default; S3-compatible storage is supported through server env vars. -``` -features/tasks/ -├── api/ # React Query hooks (get, create, update, delete) -└── components/ # Feature-specific UI components -``` +## License -Page-level components go in `src/pages/` and are referenced from `src/routes/` +Loreo is licensed under the [AGPL-3.0](LICENSE) diff --git a/docs/SELF_HOSTING.md b/docs/SELF_HOSTING.md new file mode 100644 index 0000000..ed03047 --- /dev/null +++ b/docs/SELF_HOSTING.md @@ -0,0 +1,320 @@ +# Self-Hosting Loreo + +This guide walks through deploying Loreo with Docker Compose, including domain setup, SSL, and ongoing maintenance. + +## Architecture + +![Loreo Architecture](./images/architecture.png) + +Loreo's production stack consists of four containers: + +| Service | Image | Role | +| ---------------- | ------------------------------------ | --------------------------------------- | +| `loreo-postgres` | `postgres:17-alpine` | Database | +| `loreo-redis` | `redis:7-alpine` | Job queue (BullMQ) | +| `loreo-browser` | `ghcr.io/technowizard/loreo-browser` | Headless browser for article extraction | +| `loreo-server` | `ghcr.io/technowizard/loreo-server` | Hono API, background jobs | +| `loreo-web` | `ghcr.io/technowizard/loreo-web` | Nginx + static React app | + +The web container serves the React app through nginx and proxies API requests to the server. The server connects to Postgres, Redis, and the browser service internally. No services other than the web and server containers are exposed on host ports. + +## Prerequisites + +- A Linux server with Docker and Docker Compose installed +- A domain name pointing to your server +- Git (to clone the repository) + +## Quick Start + +```bash +# Clone the repository +git clone https://github.com/technowizard/loreo.git +cd loreo + +# Create and edit your production environment +cp .env.prod.example .env.prod +nano .env.prod # or use text editor of your choice + +# Start the stack +docker compose --env-file .env.prod -f docker-compose.prod.yml up -d +``` + +Your Loreo instance will be available at `http://YOUR_SERVER_IP:3001` (web) and `http://YOUR_SERVER_IP:3000` (API). + +## Environment Configuration + +All production configuration lives in `.env.prod`. Copy the example and fill in every value: + +```bash +cp .env.prod.example .env.prod +``` + +### Required Variables + +| Variable | Description | +| ------------------- | ------------------------------------------------------------------------------ | +| `POSTGRES_USER` | Database user (e.g. `loreo`) | +| `POSTGRES_PASSWORD` | Database password — generate a strong random value at least 16 characters long | +| `POSTGRES_DB` | Database name (e.g. `loreo`) | +| `CORS_ORIGINS` | Your deployment domain (e.g. `https://loreo.example.com`) | +| `JWT_SECRET` | Random string of at least 32 characters | +| `PUBLIC_URL` | Public URL where Loreo is accessed (e.g. `https://loreo.example.com`) | + +### Port Variables + +| Variable | Default | Description | +| -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `SERVER_PUBLIC_PORT` | `3000` | Host port for the API container. Change if port 3000 is in use. The server always listens on port 3000 internally; this only controls the host mapping. | +| `WEB_PUBLIC_PORT` | `3001` | Host port for the web container. Change if port 3001 is in use. | + +### Generating Secrets + +```bash +# Generate a JWT secret +openssl rand -hex 32 + +# Generate a database password +openssl rand -base64 24 +``` + +### Complete Example + +```env +POSTGRES_USER=loreo +POSTGRES_PASSWORD= # minimum 16 characters +POSTGRES_DB=loreo + +SERVER_PUBLIC_PORT=3002 +WEB_PUBLIC_PORT=3001 + +CORS_ORIGINS=https://loreo.example.com +JWT_SECRET= # minimum 32 characters + +PUBLIC_URL=https://loreo.example.com +``` + +## Service Details + +### PostgreSQL + +Data is persisted in the `postgres_data` Docker volume. The database runs on an internal network and is not exposed to the host. Backups must be done through `docker exec`. + +### Redis + +Used by BullMQ for background job processing (article extraction, image downloads). Data is persisted in the `redis_data` volume. + +### Browser Service + +The browser container runs a Camoufox-compatible Playwright server. It requires shared memory (`shm_size: 1g`) and is isolated on its own Docker network. Port 4444 is not exposed to the host. + +### Server + +The API server handles authentication, article management, and background job scheduling. It automatically runs database migrations on startup via `docker-entrypoint.sh`. + +The server uses `STORAGE_PROVIDER: local-docker` by default, storing uploaded files in the `storage_data` volume. See the Storage section for S3 configuration. + +### Web + +The web container is **portable** — it does not require rebuilding when your deployment URL or server port changes. The browser calls the API on the same origin (through nginx's reverse proxy), and the nginx template resolves `${API_UPSTREAM}` at container startup. + +The default `API_UPSTREAM=http://loreo-server:3000` works for standard Docker Compose deployments. You only need to change it if you rename the server service or run containers on different Docker networks. + +## Reverse Proxy + +Place Loreo behind a reverse proxy for SSL termination. Here are examples for common proxies. + +### Nginx + +```nginx +server { + listen 443 ssl http2; + server_name loreo.example.com; + + ssl_certificate /etc/ssl/loreo.example.com.crt; + ssl_certificate_key /etc/ssl/loreo.example.com.key; + + client_max_body_size 50M; + + location / { + proxy_pass http://127.0.0.1:3001; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +### Caddy + +``` +loreo.example.com { + reverse_proxy localhost:3001 +} +``` + +### Traefik + +Add these labels to the `loreo-web` service in `docker-compose.prod.yml`: + +```yaml +labels: + - 'traefik.enable=true' + - 'traefik.http.routers.loreo.rule=Host(`loreo.example.com`)' + - 'traefik.http.routers.loreo.tls.certresolver=letsencrypt' +``` + +### Configuration with Reverse Proxy + +When using a reverse proxy: + +1. Set `WEB_PUBLIC_PORT` to match the port your proxy forwards to +2. Set `PUBLIC_URL` to your HTTPS domain (e.g. `https://loreo.example.com`) +3. Set `CORS_ORIGINS` to the same HTTPS domain +4. The server port (`SERVER_PUBLIC_PORT`) does not need to be exposed through the proxy — nginx inside the web container handles API routing + +## Storage + +### Local Storage (Default) + +The production compose uses `STORAGE_PROVIDER: local-docker` with data persisted in the `storage_data` volume. This works well for single-server deployments. + +Back up the volume: + +```bash +docker run --rm -v loreo-prod_storage_data:/data -v $(pwd):/backup alpine \ + tar czf /backup/loreo-storage-backup.tar.gz -C /data . +``` + +### S3-Compatible Storage + +For multi-server or cloud deployments, configure S3-compatible storage. Add these variables to `.env.prod` and pass them as environment variables to the `loreo-server` service: + +| Variable | Description | +| ---------------------- | --------------------------------------------------- | +| `STORAGE_PROVIDER` | Set to `s3` | +| `S3_ENDPOINT` | S3 endpoint URL (omit for AWS S3, set for R2/MinIO) | +| `S3_REGION` | Bucket region (default: `auto`) | +| `S3_ACCESS_KEY_ID` | Access key | +| `S3_SECRET_ACCESS_KEY` | Secret key | +| `S3_BUCKET_NAME` | Bucket name | +| `S3_PUBLIC_URL` | Public URL for the bucket (CDN, R2 domain, etc.) | + +Add these to the `loreo-server` environment block in `docker-compose.prod.yml`. + +## Database Backups + +### Backup + +```bash +docker exec loreo-prod-loreo-postgres-1 pg_dump -U loreo loreo > loreo-backup-$(date +%Y%m%d).sql +``` + +### Restore + +```bash +docker exec -i loreo-prod-loreo-postgres-1 psql -U loreo loreo < loreo-backup-YYYYMMDD.sql +``` + +### Automated Backups + +Add a cron job for daily backups: + +```bash +0 2 * * * docker exec loreo-prod-loreo-postgres-1 pg_dump -U loreo loreo > /backups/loreo-$(date +\%Y\%m\%d).sql +``` + +## Updating + +Loreo uses GitHub Container Registry images tagged `latest` for the main branch. + +```bash +# Pull new images +docker compose --env-file .env.prod -f docker-compose.prod.yml pull + +# Recreate containers with new images +docker compose --env-file .env.prod -f docker-compose.prod.yml up -d +``` + +Database migrations run automatically when the server container starts. The server's `docker-entrypoint.sh` applies any pending migrations before starting the API. + +### Pinning Versions + +For production stability, consider pinning to specific version tags instead of `latest`: + +```yaml +# In docker-compose.prod.yml, change: +image: ghcr.io/technowizard/loreo-server:latest +# To: +image: ghcr.io/technowizard/loreo-server:v0.1.0 +``` + +Check available tags at https://github.com/technowizard/loreo/pkgs/container/loreo-server. + +## Building From Source + +If you prefer building images locally instead of pulling from GHCR: + +```bash +docker build -f apps/server/Dockerfile.browser -t ghcr.io/technowizard/loreo-browser:latest . +docker build -f apps/server/Dockerfile.prod -t ghcr.io/technowizard/loreo-server:latest . +docker build -f apps/web/Dockerfile.prod -t ghcr.io/technowizard/loreo-web:latest . +``` + +The web image build does not require any build arguments — it is portable by default. + +## Troubleshooting + +### Web app loads but API calls fail + +Check that the web container can reach the server: + +```bash +docker exec loreo-prod-loreo-web-1 wget -qO- http://loreo-server:3000/health +``` + +If this fails, verify the `API_UPSTREAM` value on the web container matches the server's internal address and port. + +### Server can't reach the database + +```bash +docker exec loreo-prod-loreo-server-1 wget -qO- http://loreo-postgres:5432 2>&1 || echo "Postgres unreachable" +``` + +Verify the database credentials in `.env.prod` match the `loreo-postgres` environment. + +### Article extraction fails + +The browser service may need more time to start on first run: + +```bash +docker logs loreo-prod-loreo-browser-1 +``` + +Look for a line indicating Camoufox is ready. The server will retry failed extraction jobs automatically. Something like: + +``` +browser-1 | Browser server listening on port 4444 +browser-1 | WebSocket endpoint: ws://0.0.0.0:4444/camoufox +``` + +### Port already in use + +If port 3000 or 3001 is in use, set `SERVER_PUBLIC_PORT` and `WEB_PUBLIC_PORT` in `.env.prod` to different values. The internal container ports remain unchanged. + +### Viewing logs + +```bash +# All services +docker compose --env-file .env.prod -f docker-compose.prod.yml logs -f + +# Specific service +docker compose --env-file .env.prod -f docker-compose.prod.yml logs -f loreo-server +``` + +## Security Notes + +- Always set strong, unique values for `JWT_SECRET` and `POSTGRES_PASSWORD` +- Place the stack behind a reverse proxy with SSL +- The browser service is on an isolated Docker network — do not expose port 4444 to the host +- Keep the server and web containers updated for security patches diff --git a/docs/images/architecture.png b/docs/images/architecture.png new file mode 100644 index 0000000..f08c2b1 Binary files /dev/null and b/docs/images/architecture.png differ diff --git a/docs/images/cover.png b/docs/images/cover.png new file mode 100644 index 0000000..b59f7b3 Binary files /dev/null and b/docs/images/cover.png differ diff --git a/docs/self-hosting-templates/base.yml b/docs/self-hosting-templates/base.yml new file mode 100644 index 0000000..b8b0ad8 --- /dev/null +++ b/docs/self-hosting-templates/base.yml @@ -0,0 +1,96 @@ +services: + loreo-postgres: + image: postgres:17-alpine + restart: always + environment: + POSTGRES_USER: ${POSTGRES_USER} + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} + POSTGRES_DB: ${POSTGRES_DB} + volumes: + - postgres_data:/var/lib/postgresql/data + networks: + - backend + healthcheck: + test: ['CMD-SHELL', 'pg_isready -U ${POSTGRES_USER}'] + interval: 5s + timeout: 5s + retries: 5 + + loreo-redis: + image: redis:7-alpine + restart: always + volumes: + - redis_data:/data + networks: + - backend + healthcheck: + test: [CMD, redis-cli, ping] + interval: 10s + timeout: 5s + retries: 5 + + loreo-browser: + image: ghcr.io/technowizard/loreo-browser:latest + restart: always + expose: + - '4444' + shm_size: '1g' + networks: + - browser + + loreo-server: + image: ghcr.io/technowizard/loreo-server:latest + restart: always + ports: + - '${SERVER_PUBLIC_PORT:-3000}:3000' + environment: + NODE_ENV: production + DATABASE_HOST: loreo-postgres + DATABASE_PORT: 5432 + DATABASE_USER: ${POSTGRES_USER} + DATABASE_PASSWORD: ${POSTGRES_PASSWORD} + DATABASE_DB: ${POSTGRES_DB} + JWT_SECRET: ${JWT_SECRET} + CORS_ORIGINS: ${CORS_ORIGINS} + REDIS_HOST: loreo-redis + REDIS_PORT: 6379 + STORAGE_PROVIDER: local-docker + STORAGE_PATH: /app/data/storage + PUBLIC_URL: ${PUBLIC_URL:-http://localhost:3001} + BROWSER_URL: ws://loreo-browser:4444/camoufox + volumes: + - storage_data:/app/data/storage + networks: + - backend + - browser + - frontend + depends_on: + loreo-postgres: + condition: service_healthy + loreo-redis: + condition: service_healthy + loreo-browser: + condition: service_started + + loreo-web: + image: ghcr.io/technowizard/loreo-web:latest + restart: always + ports: + - '${WEB_PUBLIC_PORT:-3001}:80' + environment: + API_UPSTREAM: http://loreo-server:3000 + networks: + - frontend + depends_on: + - loreo-server + +volumes: + postgres_data: + redis_data: + storage_data: + +networks: + backend: + internal: true + frontend: + browser: diff --git a/docs/self-hosting-templates/neon_with_r2.yml b/docs/self-hosting-templates/neon_with_r2.yml new file mode 100644 index 0000000..e1665d7 --- /dev/null +++ b/docs/self-hosting-templates/neon_with_r2.yml @@ -0,0 +1,78 @@ +services: + loreo-redis: + image: redis:7-alpine + restart: always + volumes: + - redis_data:/data + networks: + - backend + healthcheck: + test: [CMD, redis-cli, ping] + interval: 10s + timeout: 5s + retries: 5 + + loreo-browser: + image: ghcr.io/technowizard/loreo-browser:latest + restart: always + expose: + - '4444' + shm_size: '1g' + networks: + - browser + + loreo-server: + image: ghcr.io/technowizard/loreo-server:latest + restart: always + ports: + - '${SERVER_PUBLIC_PORT:-3000}:3000' + environment: + NODE_ENV: production + DATABASE_URL: ${DATABASE_URL} # connection string + JWT_SECRET: ${JWT_SECRET} + CORS_ORIGINS: ${CORS_ORIGINS} + REDIS_HOST: loreo-redis + REDIS_PORT: 6379 + STORAGE_PROVIDER: s3 + PUBLIC_URL: ${PUBLIC_URL:-http://localhost:3001} + BROWSER_URL: ws://loreo-browser:4444/camoufox + S3_ENDPOINT: ${S3_ENDPOINT} + S3_REGION: ${S3_REGION} + S3_ACCOUNT_ID: ${S3_ACCOUNT_ID} + S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID} + S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY} + S3_BUCKET_NAME: ${S3_BUCKET_NAME} + S3_PUBLIC_URL: ${S3_PUBLIC_URL} + volumes: + - storage_data:/app/data/storage + networks: + - backend + - browser + - frontend + depends_on: + loreo-redis: + condition: service_healthy + loreo-browser: + condition: service_started + + loreo-web: + image: ghcr.io/technowizard/loreo-web:latest + restart: always + ports: + - '${WEB_PUBLIC_PORT:-3001}:80' + environment: + API_UPSTREAM: http://loreo-server:3000 + networks: + - frontend + depends_on: + - loreo-server + +volumes: + redis_data: + storage_data: + +networks: + backend: + internal: true + frontend: + browser: