Skip to content

Conversation

@ezekg
Copy link
Member

@ezekg ezekg commented Jan 8, 2026

First pass. Commit history is a mess, but I'll clean it up later (probably). Introduces a new Clickhouse database, which we'll eventually use as the primary source for logs and analytics, and soon, meter consumption ingest.

Introduces a new gem called dual_writes that replicates writes from Postgres to Clickhouse, which we're going to be using for request and event logs. Planning on running dual-writes for a few weeks — enough to collect a sufficient backlog in order to switch over analytical queries to Clickhouse without breaking everybody's charts. After the cutover, we can backfill historical log data for customers with longer retention periods than the default. Right now, the gem only has a Clickhouse strategy, but additional strategies can be implemented to e.g. replicate from Postgres to MySQL.

In addition, we're also introducing a new gem called read_your_own_writes, or ryow, which lays the groundwork for a read replica database with a non-session-based "read-your-own-writes" resolver — since our API doesn't rely on sessions for most API requests (except Portal), clients are identified via a combination of account, session, auth, and IP.

Both of these gems will eventually be open sourced when time allows.

@ezekg ezekg force-pushed the feature/dual-writes-gem branch from 3eded0e to ebf3642 Compare January 9, 2026 16:35
@ezekg ezekg force-pushed the feature/dual-writes-gem branch from 99ec50a to 012d4a6 Compare January 9, 2026 17:25
@ezekg ezekg force-pushed the feature/dual-writes-gem branch from fed7ae0 to b9ecec0 Compare January 9, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants