Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ pnpm dev:down

## Prod

Production releases and server updates are documented in [docs/deployment.md](docs/deployment.md).

Create the local production environment file once:

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0002-single-host-compose-image-release.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ Central production runs on one Linux host with Docker Compose as the deployment

The core production stack is Cockpit, Backend, PostgreSQL, migrations, and an Nginx gateway exposed through Tailscale. Assistant, voice, STT, TTS, and LLM services are excluded from the baseline until they are reliable enough to ship as an optional profile. Major SemVer releases signal incompatible changes that may require planned downtime.

Deployments coordinate with backend work through a DB-backed maintenance mode, a deploy advisory lock, and bounded task draining before migrations run. This keeps old services serving while images are pulled, stops new mutating/background work before schema changes, and only restarts the app stack after migrations pass.
Future deployment hardening may add DB-backed maintenance mode, a deploy advisory lock, and bounded task draining before migrations run. The current `central-update` script pulls images, starts PostgreSQL, optionally backs up the database, runs migrations, starts Backend/Cockpit/Gateway, and checks health.
40 changes: 22 additions & 18 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
The repository is organized as a multi-project Nx workspace:

- `apps/*`: user-facing applications (currently `apps/cockpit`)
- `services/*`: backend runtime services (`services/backend`, `services/assistant`)
- `i12e/*`: infrastructure and orchestration projects (`postgres`, `orchestrator`)
- `services/*`: backend runtime services (`backend`, `assistant`, `stt`, `tts`, `llm`)
- `i12e/*`: infrastructure and orchestration projects (`postgres`, `orchestrator`, `gateway`)
- `libs/*`: shared reusable libraries (currently `ts-log` for cross-cutting TypeScript logging)
- `docs/*`: cross-cutting repository documentation

Expand All @@ -15,14 +15,14 @@ The repository is organized as a multi-project Nx workspace:
### Cockpit (`apps/cockpit`)

- TanStack Start + React frontend.
- Fetches data on the cockpit server via TanStack Start loaders/server functions, then sends the result to the browser.
- Cockpit reaches the backend service over HTTP for weather (`/api/v1/weather/current` and `/api/v1/weather/forecast`).
- Cockpit reaches assistant-service over HTTP (`POST /api/v1/assistant/turn` and `POST /api/v1/assistant/turn/stream`).
- Fetches backend data on the cockpit server via TanStack Start server functions, then sends the result to the browser.
- Backend exposes weather current and forecast endpoints; Cockpit currently calls `/api/v1/weather/current`.
- Cockpit has assistant client code for `POST /api/v1/assistant/turn` and `POST /api/v1/assistant/turn/stream`, but the default UI does not start turns while browser VAD is disabled.
- Configuration:
- Runtime backend base URL is configured via `BACKEND_BASE_URL` with `VITE_BACKEND_API_BASE_URL` as a browser/build-time fallback.
- Runtime assistant service base URL is configured via `ASSISTANT_SERVICE_BASE_URL` with `VITE_ASSISTANT_API_BASE_URL` as a browser/build-time fallback.
- If neither backend variable is set, cockpit defaults to `http://localhost:3010` for local orchestrator-driven development.
- If neither assistant variable is set, cockpit defaults to `http://localhost:3020` for local orchestrator-driven development.
- If neither assistant variable is set, cockpit defaults to `http://localhost:3020`, matching the standalone assistant container run target.

### Backend Service (`services/backend`)

Expand All @@ -48,7 +48,7 @@ The repository is organized as a multi-project Nx workspace:
- `llm-proxy` mode to call an external LLM while keeping mock STT / TTS boundaries.
- `openai` mode to use OpenAI-native STT / LLM / TTS endpoints.
- `proxy` mode to call external STT / LLM / TTS runtimes.
- The orchestrated dev and prod stacks use `proxy` mode by default, backed by faster-whisper STT, Qwen3-TTS voice cloning, and an Ollama-based LLM wrapper.
- The standalone service supports `proxy` mode with faster-whisper STT, Qwen3-TTS voice cloning, and an Ollama-based LLM wrapper. The orchestrator compose definitions for those support services are currently commented out, so the active dev/prod stack does not start assistant services by default.

### Persistence (`i12e/postgres`)

Expand All @@ -57,8 +57,8 @@ The repository is organized as a multi-project Nx workspace:

### Orchestration (`i12e/orchestrator`)

- Docker Compose project used to start the full local stack.
- Separate environment files define dev and prod default port mappings and assistant model settings.
- Docker Compose project used to start the active local stack.
- Separate environment files define dev and prod default port mappings. Assistant model settings remain in env templates for the commented assistant support services.
- Production releases are code-free on the server: CI publishes a tested core image set to GHCR plus a deploy bundle with `docker-compose.prod.yml`, `central-update`, and `.env.prod.example`.
- The production baseline runs PostgreSQL, a migration job, Backend, Cockpit, and an Nginx gateway on one Docker Compose host. Assistant, voice, STT, TTS, and LLM services are optional future production profiles, not part of the baseline.

Expand All @@ -82,27 +82,31 @@ The repository is organized as a multi-project Nx workspace:
### Weather

1. Browser requests cockpit.
2. Cockpit server requests the backend weather API.
3. The backend weather domain checks the PostgreSQL cache first.
4. If the cache is stale or missing, the backend fetches fresh data from Open-Meteo.
5. Fresh responses are returned immediately; persistence writes happen asynchronously.
6. Cockpit serializes widget data to the client and can refresh through server functions without exposing backend directly to the browser.
2. Cockpit widget calls a TanStack Start server function.
3. The server function requests the backend weather API.
4. The backend weather domain checks the PostgreSQL cache first.
5. If the cache is stale or missing, the backend fetches fresh data from Open-Meteo.
6. Fresh responses are returned immediately; persistence writes happen asynchronously.
7. Cockpit serializes widget data to the client and can refresh through server functions without exposing backend directly to the browser.

### Finance Cash

1. Browser opens `/finance/cash`.
1. Browser opens `/finance/transactions`.
2. Cockpit server functions call backend finance APIs.
3. Backend persists manual income and expense transactions in PostgreSQL.
4. Monthly summaries are computed from transactions filtered by transaction date.
4. Summaries are computed from transactions filtered by transaction date.
5. Cockpit currently lists and creates transactions end to end; edit and delete UI controls still need route-id wiring in the server function helpers.

### Voice

1. Browser VAD cuts a speech segment locally.
2. Browser posts the segment directly to assistant-service's streaming endpoint.
1. Browser VAD is currently disabled, so Cockpit does not capture speech segments in the default UI.
2. When the implemented turn client is invoked, Cockpit posts audio to assistant-service's streaming endpoint.
3. Assistant-service performs `STT -> streamed LLM -> chunked TTS`.
4. Browser plays synthesized chunks as they arrive.
5. Cockpit still exposes a non-streaming server function path for fallback flows.

This flow exists in the Cockpit and assistant-service code, but the assistant service and support runtimes are not active in the default orchestrator compose stack while their service blocks remain commented out.

## Boundary Rules

- Keep UI and presentation concerns in `apps/*`.
Expand Down
141 changes: 141 additions & 0 deletions docs/deployment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Release and Production Deployment

Central production deploys from tested container images, not from source code on the server.

## Release flow

CI in `.github/workflows/ci.yml` owns release creation:

1. Pull requests run validation and build disposable GHCR images tagged `pr-<number>-<sha12>`.
2. Pushes to `main` run validation, build GHCR images tagged `sha-<sha12>`, and smoke-test that exact image set with the production Compose file.
3. Git tags matching `v*.*.*` also run validation, build `sha-<sha12>` images, and smoke-test the image set.
4. After smoke tests pass on a tag push, CI publishes version tags from the tested SHA images.
5. Exact stable tags such as `v1.2.3` also move the `stable` tag.
6. Prerelease tags such as `v1.3.0-rc.1` publish only that version tag and do not move `stable`.
7. Tag releases package a deploy bundle artifact containing:
- `docker-compose.prod.yml`
- `central-update`
- `.env.prod.example`

The release tag must point at a commit reachable from `main`; CI rejects release tags outside `main`.

## Images

Production Compose pulls these images from GHCR:

- `ghcr.io/themattcode/central/app-cockpit:${CENTRAL_VERSION}`
- `ghcr.io/themattcode/central/service-backend:${CENTRAL_VERSION}`
- `ghcr.io/themattcode/central/i12e-postgres:${CENTRAL_VERSION}`
- `ghcr.io/themattcode/central/i12e-gateway:${CENTRAL_VERSION}`

`CENTRAL_VERSION` defaults to `stable`. Use exact release tags for pinned deployments and rollbacks.

## Create a release

From a clean `main` commit:

```bash
git tag v1.2.3
git push origin v1.2.3
```

Then wait for CI to pass. The tag push publishes the versioned image set, updates `stable` for exact stable SemVer tags, and uploads the deploy bundle artifact.

For a prerelease:

```bash
git tag v1.3.0-rc.1
git push origin v1.3.0-rc.1
```

Deploy prereleases by exact version. `stable` remains unchanged.

## Server setup

Install Docker and Tailscale on the production host. Tailscale is host-managed; the Compose stack binds the gateway to `127.0.0.1:4000` by default so Tailscale Serve can expose it over the tailnet.

Unpack the deploy bundle on the server, then create the environment file once:

```bash
cp .env.prod.example .env.prod
```

Set production values in `.env.prod`:

- `POSTGRES_PASSWORD`
- `BACKEND_DATABASE_URL`
- `BACKEND_CORS_ALLOW_ORIGIN`
- `CENTRAL_ORIGIN`
- optional `CENTRAL_VERSION`

If GHCR packages are private, log in on the host:

```bash
docker login ghcr.io
```

## Deploy or update

Run from the unpacked deploy bundle:

```bash
./central-update
```

The script prompts for a version and defaults to `stable`.

Deploy an exact version:

```bash
./central-update v1.2.3
```

Deploy a prerelease:

```bash
./central-update v1.3.0-rc.1
```

Major version jumps require explicit approval:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Don't promise major-upgrade approval from stable

For a fresh deploy bundle, .env.prod.example sets CENTRAL_VERSION=stable, and central-update only marks major_jump when both the current and target versions parse as vX.Y.Z; stable yields an empty current major. In that default production setup, ./central-update v2.0.0 proceeds without --allow-major and without the major-jump backup path, so this guide gives operators a safety guarantee the script does not enforce.

Useful? React with 👍 / 👎.


```bash
./central-update v2.0.0 --allow-major
```

The update script:

1. writes `CENTRAL_VERSION` to `.env.prod`,
2. pulls the selected image set,
3. starts PostgreSQL and waits for health,
4. creates a PostgreSQL backup for major jumps or when `--backup` is passed,
5. runs migrations,
6. starts `service-backend`, `app-cockpit`, and `i12e-gateway`,
7. checks gateway and backend health,
8. prints Compose status.

Backups are written to `backups/` next to the deploy bundle unless `BACKUP_DIR` overrides it. Use `--skip-backup` only when an external backup already exists.

## Rollback

Rollback by deploying an older exact release tag:

```bash
./central-update v1.2.2
```

Database rollback is not automatic. If migrations are not backward-compatible, restore from a backup before starting the older version.

## Inspect production

Use the same env and Compose file as the update script:

```bash
docker compose --env-file .env.prod --file docker-compose.prod.yml ps
docker compose --env-file .env.prod --file docker-compose.prod.yml logs
```

Stop the stack:

```bash
docker compose --env-file .env.prod --file docker-compose.prod.yml down
```
Loading