Skip to content

Latest commit

 

History

History
675 lines (550 loc) · 24.1 KB

File metadata and controls

675 lines (550 loc) · 24.1 KB

Patchwork Technical Guide

Patchwork is an agent-native version control system designed from scratch for AI-assisted and multi-agent development workflows. It replaces Git's snapshot-and-merge model with an append-only operation log, composable views, and a built-in speculative submit queue. This document covers architecture, data model, and usage at a level of detail sufficient for a senior engineer evaluating adoption.


1. Architecture Overview

                           +-----------+
                           |    CLI    |
                           +-----+-----+
                                 |
                    +------------+-------------+
                    |     HTTP / gRPC API      |
                    +------------+-------------+
                                 |
                    +------------+-------------+
                    |         Server           |
                    |                          |
                    |  +--------+  +--------+  |
                    |  | OpLog  |  | Blobs  |  |
                    |  +--------+  +--------+  |
                    |                          |
                    |  +-----------+  +-----+  |
                    |  |ChangeStore|  |Views|  |
                    |  +-----------+  +-----+  |
                    |                          |
                    |  +--------+  +--------+  |
                    |  |EventBus|  |Submit  |  |
                    |  |        |  |Queue   |  |
                    |  +--------+  +--------+  |
                    +------+----------+--------+
                           |          |
                    +------+---+ +----+-----+
                    |  SQLite  | |Filesystem|
                    |(oplog.db)| |  (blobs) |
                    +----------+ +----------+

OpLog -- Append-only log of file-level operations stored in SQLite. Every operation receives a monotonically increasing sequence number. This is the single source of truth.

BlobStore -- Content-addressable filesystem store for file contents. Blobs are referenced by SHA-256 hash.

ChangeStore -- SQLite-backed storage for changes and their versions. Manages lifecycle transitions (draft, ready, testing, landed, abandoned).

ViewEngine -- Computes composable projections of the codebase by replaying the operation log through a base plus overlays. Handles 3-way merge for conflict detection during materialization.

EventBus -- In-process pub/sub that broadcasts system events (op_appended, change_landed, conflict_detected, test_result, queue_position) to SSE clients and workspace watchers.

SubmitQueue -- Zuul-style speculative testing pipeline. Tests multiple queued changes in parallel by assuming all preceding entries pass.


2. Git vs Patchwork

Concept Git Patchwork
Unit of change Commit (tree snapshot) Operation (single file op)
Grouping Branch Change (with versions)
Workspace Working directory + checkout View (composable projection)
History DAG of commits Append-only operation log
Collaboration Clone + merge Shared server, real-time events
CI External (GitHub Actions etc.) Built-in submit queue
Conflict detection At merge time Continuous, during materialization
Identity SHA-1 hash of tree Global sequence number (uint64)
Branching model Lightweight refs into DAG Changes with named targets
Atomicity Whole-tree snapshot per commit Per-file operation granularity

The core difference is structural. Git records state (tree snapshots). Patchwork records intent (file-level operations). A Git commit bundles all changed files into a single node in a DAG. A Patchwork operation records exactly one file-level action -- create, modify, delete, rename, or copy -- with a global sequence number that provides total ordering across all concurrent agents.

This has consequences:

  • No merge ambiguity. Total ordering means the server always knows which operation came first. There is no need to find a common ancestor.
  • Fine-grained history. You can query "every operation that ever touched pkg/server/server.go" without walking a commit graph.
  • Composable workspaces. Views can layer arbitrary combinations of in-progress changes on top of main, something Git cannot express without octopus merges or stacked branch tools.

3. Key Innovations

3a. Operation Log vs Commits

Every file-level change in Patchwork is an Operation -- a single row in the append-only log:

type Operation struct {
    ID          uint64            `json:"id"`
    Timestamp   time.Time         `json:"timestamp"`
    AgentID     string            `json:"agent_id"`
    ChangeID    string            `json:"change_id"`
    Version     uint32            `json:"version"`
    OpType      OpType            `json:"op_type"`
    Path        string            `json:"path"`
    ContentHash string            `json:"content_hash,omitempty"`
    DiffHash    string            `json:"diff_hash,omitempty"`
    OldPath     string            `json:"old_path,omitempty"`
    ParentSeq   uint64            `json:"parent_seq"`
    Metadata    map[string]any    `json:"metadata,omitempty"`
}

OpType is one of five values:

const (
    OpFileCreate OpType = "create"
    OpFileModify OpType = "modify"
    OpFileDelete OpType = "delete"
    OpFileRename OpType = "rename"
    OpFileCopy   OpType = "copy"
)

The ID field is a globally unique, monotonically increasing sequence number assigned at append time. This is the key property: total ordering across all agents and all changes. When agent A writes operation 42 and agent B writes operation 43, the server knows unambiguously that 42 happened before 43, regardless of wall-clock skew.

ContentHash points to a blob in the content-addressable store. DiffHash optionally stores a precomputed unified diff. ParentSeq links back to the previous operation on the same path, forming a per-file chain that can be traversed without scanning the full log.

Compare this to Git, where a commit captures the entire tree state. If two developers modify different files in parallel, Git represents this as two commits that must be merged. Patchwork represents this as two independent operations that simply have adjacent sequence numbers -- no merge required.

3b. Changes with Versions

A Change groups related operations into a reviewable, submittable unit (analogous to a pull request or Gerrit change):

type Change struct {
    ID             string            `json:"id"`
    Title          string            `json:"title"`
    Author         string            `json:"author"`
    Target         string            `json:"target"`
    Status         ChangeStatus      `json:"status"`
    CurrentVersion uint32            `json:"current_version"`
    Versions       []Version         `json:"versions"`
    Dependencies   []string          `json:"dependencies,omitempty"`
    Metadata       map[string]any    `json:"metadata,omitempty"`
    CreatedAt      time.Time         `json:"created_at"`
    UpdatedAt      time.Time         `json:"updated_at"`
}

Changes have a lifecycle expressed by ChangeStatus:

const (
    ChangeStatusDraft     ChangeStatus = "draft"
    ChangeStatusReady     ChangeStatus = "ready"
    ChangeStatusTesting   ChangeStatus = "testing"
    ChangeStatusLanded    ChangeStatus = "landed"
    ChangeStatusAbandoned ChangeStatus = "abandoned"
)

The lifecycle flows:

draft --> ready --> testing --> landed
  |                              ^
  +----- (submit shortcut) ------+
  |
  +--> abandoned

Each change can have explicit Dependencies -- other change IDs that must be landed before this change can be submitted. The submit queue enforces this constraint at enqueue time.

Versions are snapshots of a change at a point in time:

type Version struct {
    Number     uint32      `json:"number"`
    Operations []uint64    `json:"operations"`
    BaseSeq    uint64      `json:"base_seq"`
    CreatedAt  time.Time   `json:"created_at"`
    Message    string      `json:"message"`
    TestResult *TestResult `json:"test_result,omitempty"`
}

Operations is a list of operation sequence numbers that belong to this version. BaseSeq records which point in the main log this version was created against.

This is the versioning model: you work on a change, accumulate operations, then create a version to snapshot that state. If you need to rebase (because main has advanced), you create a new version with new operations replayed against the current head. Version 1 might have ops [1, 2, 3] based at seq 10. After rebase, version 2 has ops [14, 15] based at seq 13.

Each version independently tracks its TestResult:

type TestResult struct {
    Status  TestStatus `json:"status"`
    Output  string     `json:"output,omitempty"`
    StartAt time.Time  `json:"start_at"`
    EndAt   time.Time  `json:"end_at"`
}

3c. Views vs Checkouts

A Git checkout writes one branch to disk. A Patchwork View is a composable projection that says "show me the codebase as if these changes were applied":

type View struct {
    ID             string         `json:"id"`
    Owner          string         `json:"owner"`
    Base           ViewBase       `json:"base"`
    Overlays       []Overlay      `json:"overlays"`
    ConflictPolicy ConflictPolicy `json:"conflict_policy"`
}

type ViewBase struct {
    Type ViewBaseType `json:"type"`
    Seq  uint64       `json:"seq,omitempty"`
}

type Overlay struct {
    ChangeID string `json:"change_id"`
    Version  uint32 `json:"version"`
    Pin      bool   `json:"pin"`
}

ViewBase determines the starting point:

const (
    ViewBaseMainAtSeq    ViewBaseType = "main_at_seq"     // main at a specific seq
    ViewBaseMainAtLatest ViewBaseType = "main_at_latest"  // main at current head
)

Each Overlay layers a specific change version on top. The Pin field, when true, locks the overlay to that exact version rather than floating to latest.

A view like this:

{
  "id": "agent-workspace",
  "base": { "type": "main_at_latest" },
  "overlays": [
    { "change_id": "add-auth", "version": 2 },
    { "change_id": "fix-logging", "version": 1 }
  ],
  "conflict_policy": "warn"
}

...means "show me main at its current head, then apply version 2 of add-auth, then apply version 1 of fix-logging." Materialization replays the relevant operations from the log, applying 3-way merge logic when overlays touch the same files.

The ConflictPolicy controls what happens during materialization when files conflict:

const (
    ConflictPolicyWarn      ConflictPolicy = "warn"       // mark conflicts, continue
    ConflictPolicyLastWrite ConflictPolicy = "last_write" // last overlay wins
    ConflictPolicyManual    ConflictPolicy = "manual"     // halt and require resolution
)

When a conflict is detected, the view engine produces a ConflictEvent:

type ConflictEvent struct {
    Type            string `json:"type"`
    Path            string `json:"path"`
    YourChange      string `json:"your_change"`
    YourVersion     uint32 `json:"your_version"`
    ConflictingOp   uint64 `json:"conflicting_op"`
    BaseContent     string `json:"base_content"`
    YourContent     string `json:"your_content"`
    TheirContent    string `json:"their_content"`
    ConflictMarkers string `json:"conflict_markers,omitempty"`
}

This is broadcast over the event bus in real time. An agent subscribed to conflict_detected events can immediately react to conflicts as they emerge, rather than discovering them at merge time.

Materialization produces a FileTree:

type FileTree struct {
    ViewID  string      `json:"view_id"`
    Seq     uint64      `json:"seq"`
    Entries []FileEntry `json:"entries"`
}

type FileEntry struct {
    Path         string         `json:"path"`
    ContentHash  string         `json:"content_hash"`
    IsConflict   bool           `json:"is_conflict,omitempty"`
    ConflictInfo *ConflictEvent `json:"conflict_info,omitempty"`
}

Views are cheap to create and destroy. An agent can spin up a view to "what if I combine these three in-flight changes?" without touching disk. The server also auto-resolves dependency overlays: if change B depends on change A, creating a view with B automatically prepends A as an overlay (unless A is already landed).

3d. Speculative Submit Queue

The submit queue implements Zuul-style speculative parallel testing. The core idea: when multiple changes are queued, don't test them sequentially -- test them in parallel, each assuming all preceding entries will pass.

Example: Changes A, B, C, D submitted in order.

Position 0: A tests against [main]
Position 1: B tests against [main + A]
Position 2: C tests against [main + A + B]
Position 3: D tests against [main + A + B + C]

All four test runs execute in parallel. The queue materializes speculative views for each position by stacking the preceding changes as overlays.

If all pass: A, B, C, D land sequentially. Total time = max(test_time), not sum(test_time).

If B fails: A lands normally. B is ejected (status reverts to draft). C and D are reset to pending and retested:

Position 0: C retests against [main + A]
Position 1: D retests against [main + A + C]

This is implemented in SubmitQueue.processSpeculative:

func (q *SubmitQueue) processSpeculative(entries []*types.QueueEntry, indices []int) {
    // ...
    for i, entry := range entries {
        wg.Add(1)
        go func(i int, entry *types.QueueEntry) {
            defer wg.Done()
            preceding := entries[:i]
            passed, output := q.runTest(entry, preceding)
            results[i] = result{idx: i, passed: passed, output: output}
        }(i, entry)
    }
    wg.Wait()

    // Process in order: if an entry fails, reset all subsequent to pending
    for i, r := range results {
        if r.passed {
            q.handleResult(entries[i], indices[i]-i, true, r.output)
        } else {
            q.handleResult(entries[i], indices[i]-i, false, r.output)
            for j := i + 1; j < len(entries); j++ {
                entries[j].TestStatus = types.TestStatusPending
            }
            break
        }
    }
}

Each test run materializes a view with the appropriate overlays and writes the result to a temporary directory:

func (q *SubmitQueue) runTest(entry *types.QueueEntry, preceding []*types.QueueEntry) (bool, string) {
    var overlays []types.Overlay
    for _, prev := range preceding {
        overlays = append(overlays, types.Overlay{
            ChangeID: prev.ChangeID,
            Version:  prev.Version,
        })
    }
    overlays = append(overlays, types.Overlay{
        ChangeID: entry.ChangeID,
        Version:  ch.CurrentVersion,
    })

    v := &types.View{
        ID:             "queue-test-" + entry.ChangeID,
        Owner:          "submit-queue",
        Base:           types.ViewBase{Type: types.ViewBaseMainAtLatest},
        Overlays:       overlays,
        ConflictPolicy: types.ConflictPolicyLastWrite,
    }
    ft, _ := q.server.Views.MaterializeView(v)
    // Write ft to testDir, then exec test command...
}

Configure the queue via CLI:

patchwork config set test-cmd "go test ./..."

The QueueConfig controls parallelism:

type QueueConfig struct {
    TestCmd        string        // e.g., "go test ./..."
    TestDir        string        // working directory for tests
    MaxSpeculative int           // max entries to test in parallel (default 1 = linear)
    PollInterval   time.Duration // how often to check the queue
}

With MaxSpeculative: 4, up to 4 changes test simultaneously. With MaxSpeculative: 1 (the default), the queue degrades to linear FIFO.


4. Agent Workflow Walkthrough

A step-by-step example of an agent (or developer) making a change, from creation through landing.

Step 1: Initialize repository

patchwork init
# Initialized patchwork repository in /project/.patchwork

Step 2: Create a change

patchwork change create "Add user authentication" --author agent-1 --target main
# Created change: add-user-authentication
#   Title:  Add user authentication
#   Author: agent-1
#   Target: main
#   Status: draft

Step 3: Create a workspace

The workspace creates a view (main + change overlay) and materializes it to a directory the agent can read and write.

patchwork workspace create add-user-authentication ./ws-auth --agent agent-1
# Created workspace: ws-add-user-authentication
#   Change:    add-user-authentication
#   Directory: /project/ws-auth
#   Agent:     agent-1

Step 4: Edit files

The agent writes files directly into the workspace directory using any tool -- text editors, sed, code generation, etc. Patchwork does not intercept file writes; it detects changes by scanning.

# Agent edits files in ./ws-auth/
echo 'package auth' > ./ws-auth/pkg/auth/auth.go
echo 'package auth' > ./ws-auth/pkg/auth/middleware.go

Step 5: Scan for changes

Scanning diffs the workspace directory against the view and automatically creates operations in the log for any new, modified, or deleted files.

patchwork workspace scan ws-add-user-authentication
# Captured 2 operations:
#   #1 create pkg/auth/auth.go
#   #2 create pkg/auth/middleware.go

Step 6: Create a version

Snapshotting the current work into a named version:

patchwork version create add-user-authentication "Initial auth implementation"
# Created version 1 of add-user-authentication: Initial auth implementation

Step 7: Mark as ready and submit

patchwork change ready add-user-authentication
# Change add-user-authentication marked as ready.

patchwork submit add-user-authentication
# Submitted change add-user-authentication (version 1) to the queue.
#   Position: 0
#   Status:   pending

Step 8: Monitor the queue

patchwork queue
# POS  CHANGE                    VER  STATUS   ENTERED
# 0    add-user-authentication   v1   running  2026-02-24T10:30:00Z

Step 9: Watch events (optional)

patchwork watch
# [2026-02-24T10:30:05Z] test_result: {"change_id":"add-user-authentication","status":"passed"}
# [2026-02-24T10:30:05Z] change_landed: {"change_id":"add-user-authentication","version":1}

Step 10: Clean up

patchwork workspace destroy ws-add-user-authentication
# Workspace ws-add-user-authentication destroyed.

5. API Overview

REST API

The HTTP server starts on port 8080 by default (patchwork serve). All endpoints are under /api/ and accept/return JSON unless noted.

Method Path Purpose
GET /api/status Server status summary
POST /api/ops Append an operation to the log
GET /api/ops List operations (filter: ?change_id=, ?path=, ?since=)
GET /api/ops/{seq} Get a single operation by sequence number
GET /api/head Current log head sequence number
POST /api/changes Create a new change
GET /api/changes List changes (filter: ?status=)
GET /api/changes/{id} Get a single change
PATCH /api/changes/{id}/status Update change status
PATCH /api/changes/{id}/metadata Update change metadata
POST /api/changes/{id}/versions Create a new version
POST /api/changes/{id}/rebase Rebase a change onto current head
GET /api/changes/{id}/conflicts Check for path conflicts
POST /api/views Create a view
GET /api/views List views (filter: ?owner=)
GET /api/views/{id} Get a single view
DELETE /api/views/{id} Delete a view
GET /api/views/{id}/tree Materialize a view into a file tree
GET /api/views/{id}/files/{path} Read a single file from a materialized view
POST /api/workspaces Create a workspace
GET /api/workspaces List active workspaces
GET /api/workspaces/{id} Get workspace status
DELETE /api/workspaces/{id} Destroy a workspace
POST /api/workspaces/{id}/sync Sync workspace with its view
POST /api/workspaces/{id}/scan Scan workspace for file changes
POST /api/queue Submit a change to the queue
GET /api/queue Get current queue status
POST /api/blobs Upload a blob (raw body, 100MB max)
GET /api/blobs/{hash} Download a blob by hash
GET /api/git/refs List Git-compatible refs
GET /api/git/summary Git bridge status summary
GET /api/git/export/{id} Export a change as Git-compatible patches
POST /api/git/import/{id} Import files into a change

All endpoints include CORS headers (Access-Control-Allow-Origin: *) for browser-based tooling.

gRPC API

The same functionality is available via gRPC on port 9090. The protobuf service definitions mirror the REST endpoints above.

Server-Sent Events (SSE)

Real-time event streaming is available at GET /api/events. Supports optional type filtering via the types query parameter:

GET /api/events?types=op_appended,conflict_detected

Event types:

Event Type Payload Trigger
op_appended Operation details Any operation appended to the log
change_landed Change ID, version A change transitions to landed
conflict_detected Path, changes, content View materialization detects conflict
test_result Change ID, status, output Submit queue test completes
queue_position Change ID, position Queue position changes

The SSE stream sends heartbeat comments every 15 seconds to keep connections alive. Example client:

curl -N "http://localhost:8080/api/events?types=change_landed,test_result"

6. CLI Reference

Command Description
patchwork init Initialize a new patchwork repository in the current (or specified) directory
patchwork status Show repository status: log head, change/view/workspace counts, queue depth
patchwork change create <title> Create a new change (options: --author, --target, --depends-on)
patchwork change list List all changes (option: --status <filter>)
patchwork change show <id> Show change details including all versions and test results
patchwork change ready <id> Mark a change as ready for testing
patchwork change abandon <id> Mark a change as abandoned
patchwork version create <change> <msg> Snapshot current work as a new version of the change
patchwork version list <change> List all versions of a change with operation counts
patchwork workspace create <change> [dir] Create a workspace for a change (option: --agent <name>)
patchwork workspace list List all active workspaces
patchwork workspace sync <id> Sync workspace directory with its view
patchwork workspace scan <id> Detect file changes in workspace and capture as operations
patchwork workspace destroy <id> Destroy a workspace and clean up its view
patchwork submit <change> Submit a change to the testing queue
patchwork queue Show current submit queue status
patchwork log Show recent operation log (options: --change <id>, --path <path>)
patchwork cat <view> <path> Read a file from a materialized view
patchwork diff <change> Show diff for a change (option: --version <n>)
patchwork land <change> Directly land a change, bypassing the submit queue
patchwork rebase <change> Rebase a change onto the current log head
patchwork conflicts <change> Check for path conflicts against landed operations
patchwork serve Start the HTTP API server (option: --addr <host:port>, default :8080)
patchwork watch Stream all events to stdout in real time
patchwork config show Show current configuration
patchwork config set <key> <value> Set a configuration value (e.g., test-cmd)
patchwork git-bridge refs List Git-compatible refs generated from changes
patchwork git-bridge export <change> Export a change as Git-compatible patches
patchwork git-bridge import <change> <files...> Import files from disk into a change
patchwork git-bridge summary Show Git bridge status