GitOps Reverser architecture

GitOps Reverser is a Kubernetes operator that observes cluster mutations and writes the resulting desired object state to Git. It reverses the traditional GitOps direction: instead of Git driving the cluster, the Kubernetes API drives Git. The repository becomes a continuously updated mirror of live cluster state.

This document describes how the operator works today. If you are reading it for the first time, keep one sentence in mind: Kubernetes watch supplies state, Git stores the mirror, and Redis keeps the operator's short-lived coordination state. Read the Ground Rules, Mental Model, and Data Sources for the shape of the system, then the Configuration Model and Common Flows for how the pieces move together. The later sections give the reference detail behind each piece. If a detail here ever disagrees with the source, the source wins; deeper design records live under docs/design/.

Source and destination connections deliberately have different scopes. A namespaced GitProvider is a team's Git write boundary: its credential, branch policy, and targets usually belong together. ClusterProvider is cluster-scoped because it represents one shared logical source identity whose client, discovery surface, watch state, and attribution partition must remain consistent across namespaces. allowedNamespaces then explicitly controls which control-cluster namespaces may reference that shared source; it does not grant source-cluster RBAC or select source namespaces. The source identity is the ClusterProvider name alone, with no API-server identity probe: two providers configured for the same server deliberately remain separate source partitions.

Ground rules

These are the design decisions to keep in your head while reading or changing the code:

The Kubernetes API is the source of truth. Git is a materialized mirror of desired state from the API. State is ingested by watch; these paths never treat Git as authority. When a push conflicts with a newer remote commit, the operator fetches the new remote state, resets its local clone, and replays its retained writes from the API.

Watch is the only object-state source. Each GitTarget names one ClusterProvider and opens one Kubernetes watch per claimed (GVR, scope) against that source, with sendInitialEvents=true. For a namespaced type, a scope is one concrete source namespace; cluster-scoped types use the cluster-wide scope. Every Git write derives from persisted state the watch observed. Audit never defines what changed; it only, optionally, explains who caused it.

Sensitive resources are never written in plaintext. Core Secrets and configured sensitive resource types must be encrypted before they touch the Git worktree. If encryption cannot be configured, the write fails; there is no plaintext opt-out.

Writes are serialized per Git branch. One BranchWorker owns each (GitProvider namespace, GitProvider name, branch) tuple. Multiple GitTargets may share one branch. Every write to that branch goes through the worker's single event loop and commit window.

Redis/Valkey is optional but advised. The default configured-author mode runs without it: a plain helm install comes up healthy and watches cold-replay on restart. When an endpoint is configured, Redis stores watch resume cursors (warm restarts) and the small coordination records used by CommitRequest author capture and HA. Attribution no longer requires it on its own: its facts travel on a selectable transport, Redis Streams by default and an in-process ring with --author-attribution-transport=memory, which is refused with more than one replica. HA will require Redis as the shared store across replicas.

Audit is an optional attribution lookup. When attribution is enabled, kube-apiserver posts audit events to /audit-webhook/<audit-route> (or a configured annotation-routed shared endpoint). The route is ClusterProvider.spec.attribution.auditRoute, which defaults to the provider's own name. The operator appends a minimal fact to that route's per-type fact stream; every process watching the type follows that stream into a bounded, TTL'd in-process index, and a watch event joins against the index within a bounded grace window. A missing, late, or absent fact never blocks state capture; it only changes the author.

Behavior is deterministic and proven by tests. Given the same observed Kubernetes state, configuration, and Git base, the operator makes the same materialization decisions. Ordering, attribution fallbacks, conflict replay, path refusal, encryption behavior, and commit semantics are pinned by unit and e2e tests.

Mental model

The easy part is "write YAML to Git." The hard parts shape the whole architecture:

Ordering. Two updates to the same object must land in Git in cluster order.
Not losing changes. A dropped or late event must never leave Git permanently wrong. In particular, a delete that happens while no watch is running must still be reconciled.
Secrets. Sensitive objects must never touch disk in plaintext.
Scale. A cluster holds thousands of objects across hundreds of types; you cannot keep a live watch open on all of them.

The solution, in the vocabulary used throughout this document:

Watch is the only object-state source. Each GitTarget opens one Kubernetes watch per claimed (GVR, scope) with sendInitialEvents=true. The apiserver delivers a watch's events already ordered by resourceVersion for that type, so there is nothing to re-order. Every Git write derives from persisted state the watch observed.
Watches are per GitTarget and scaled by claims. A watch opens only for the claimed ∩ followable (GVR, scope) set, so cost scales with what GitTargets claim, not with cluster type count.
Recovery prefers watch. A new watch normally starts with sendInitialEvents, establishes a current snapshot boundary, and runs a mark-and-sweep: any Git file whose object is no longer present is deleted. When Redis has a fresh per-type cursor, the operator skips the snapshot and resumes a normal watch from that resourceVersion. Cursors are keyed by GitTarget UID and carry a TTL refreshed on every watch event and bookmark, so a live watch keeps its cursor warm while a deleted one's cursor expires, and a stale resourceVersion (410 Gone) rebuilds from a fresh replay. Older APIs that reject sendInitialEvents fall back to LIST plus buffered WATCH. The sweep fires on snapshot establishment, never on a timer.
Audit, when enabled, only names the author. It is an optional attribution lookup; a missing or late fact costs author fidelity, never correctness. With attribution disabled the product commits as the configured committer. With attribution enabled, an unresolved live change is visibly authored as unknown (attribution unresolved), so an installation that expects attribution can investigate its audit policy, ingress, Redis connection, or source identity instead of mistaking the result for configured-author mode.
One BranchWorker per Git branch serializes all writes. Every write to a branch funnels through a single worker and a single commit window, which keeps concurrent GitTargets and authors from racing each other into a corrupt tree.

Data sources

GitOps Reverser reads Kubernetes through four mechanisms. Two are required to function; the other two are optional and only add who did something:

Source	Answers	Required	What it carries
Discovery (CRD/APIService)	what types exist	yes	the API surface: served GVRs, scope, preferred version, subresources; rules resolve against it
Watch (per claimed `(GVR, scope)`)	what changed	yes	the object body, ordered per type; the only object-state source, with deletes reconciled via `sendInitialEvents` replay + mark-and-sweep
Audit webhook (`/audit-webhook/<provider>`)	who changed mirrored state	no	a post-persist attribution fact, partitioned by source provider and joined to the watch event by resourceVersion (Optional Attribution)
Validating admission webhook (`/validate-operator-types`)	who issued a command	no	the submitter of a `CommitRequest`, captured at admission and keyed 1:1 by UID (CommitRequest Finalize)

With both optional sources off, the product still mirrors state correctly, and every commit is authored by the committer.

Why audit, not admission, attributes a mirrored change

For a mirrored change, taking the author from a validating admission webhook is tempting, because the request already carries userInfo. We tried it; the edge cases are not fixable, because admission runs before the write reaches etcd:

Admission sees attempts, not persistence. A request that passes our webhook can still be rejected by a later webhook or fail an optimistic-concurrency conflict at storage, so the change never becomes state, yet we would already have recorded an author for it (a dry-run is the same shape: admission fires, nothing persists).
There is nothing to join on yet. Mirrored attribution matches an author to a persisted watch event by identity + resourceVersion; at admission the resourceVersion does not exist, and for a generateName create even the final name/UID is unassigned, so an admission record cannot be coupled to the watch or audit stream.

Audit avoids both: kube-apiserver posts a ResponseComplete event after the write persisted, carrying the resourceVersion that joins it to the watch event.

The one place admission is the right source is the CommitRequest, and only for its command meaning (who issued the save) and not for mirroring the object. That holds because it is our own type with a memory model we control: the controller reconciles off its own watch, which only delivers persisted objects, so the author is captured at admission and read back 1:1 by UID: no resourceVersion join, no persistence proof from audit needed. (We still can't guarantee a create is never blocked, but it holds across our edge cases.) If a WatchRule also selects CommitRequests, that mirroring runs through the normal pipeline and is audit-attributed like any other resource. See CommitRequest Finalize.

Proving it on new Kubernetes versions

How Kubernetes emits these structures is not something to guess at. A separate, standalone project. The mutation-capture lab (design, cmd/mutation-capture-lab/) records the exact watch, audit, and admission output a real apiserver produces for each interesting scenario and commits it as normalized example YAML (a versioned corpus). It doubles as a regression harness: point it at a new Kubernetes release, regenerate, and any change in verb naming, event ordering, body presence, or deletecollection fan-out surfaces as a reviewable corpus diff before it can surprise the operator. The "admission sees attempts, not persistence" claim above is read straight out of that corpus: its dry-run, record-and-reject, and conflict scenarios capture audit and admission with no resulting watch event.

Configuration model

You configure GitOps Reverser through six CRDs (group configbutler.ai, version v1alpha3). WatchRule and ClusterWatchRule choose which Kubernetes resources enter the pipeline. CommitRequest can ask for the current window to be saved. GitTarget joins one source cluster to one Git destination. ClusterProvider supplies the source connection and authorization boundary; GitProvider supplies the repository, credentials, commit settings, and push policy.

graph LR
    subgraph CONTROL["Control plane: operator cluster"]
        WR[WatchRule] -->|targetRef| GT[GitTarget]
        CWR[ClusterWatchRule] -->|targetRef| GT
        CR[CommitRequest] -->|targetRef| GT
        GT -->|clusterProviderRef| CP[ClusterProvider]
        GT -->|providerRef| GP[GitProvider]
        KCFG[("Remote kubeconfig Secret\noperator namespace")]
        WM["Watch manager\nper-provider source context"]
    end

    subgraph SOURCE["Source cluster selected by ClusterProvider"]
        API["Kubernetes API\ndiscovery · Namespace policy · watches"]
    end

    CP -. "optional kubeconfig reference" .-> KCFG
    CP -->|"source identity + config"| WM
    WM -->|"source client"| API

    style GP fill:#e8f4fd,stroke:#2196f3
    style CP fill:#e8f4fd,stroke:#2196f3
    style GT fill:#e8f4fd,stroke:#2196f3
    style WR fill:#fff3e0,stroke:#ff9800
    style CWR fill:#fff3e0,stroke:#ff9800
    style CR fill:#f3e5f5,stroke:#8e24aa

The diagram follows one GitTarget. A provider with no spec.kubeConfig selects the operator's own cluster as its source; any provider name, including default, can instead select a remote cluster through a kubeconfig Secret in the operator namespace. Each referenced ClusterProvider gets an independent source context: client, discovery catalog, followability registry, source-namespace snapshot, and reachability state. Its GitTargets open their source watches through that context. The contexts are keyed by provider name and are released when no GitTarget references them.

CRD	Scope	One line role
`WatchRule`	namespaced	which NAMESPACED resources, in which source namespaces, route to a GitTarget
`ClusterWatchRule`	cluster	which CLUSTER SCOPED resources route to a GitTarget
`CommitRequest`	namespaced	a one shot "save the open window now" signal
`GitTarget`	namespaced	one materialization from a source provider to `(provider, branch, path)`
`ClusterProvider`	cluster	one source-cluster connection plus namespace access policy
`GitProvider`	namespaced	a Git repo + credentials + commit/signing config

WatchRule / ClusterWatchRule

Sources: watchrule_types.go, clusterwatchrule_types.go
Controllers: watchrule_controller.go, clusterwatchrule_controller.go

Scope is carried by the rule KIND. A WatchRule selects NAMESPACED resources and routes matching events to a same namespace GitTarget; each of its rule items names the source namespace it watches. A ClusterWatchRule selects CLUSTER SCOPED resources, with an explicit namespace targetRef and no namespace selection of its own. Both share the rule model:

spec.rules[]: OR resource rules (MinItems=1).
rules[].operations: CREATE / UPDATE / DELETE / *; omitted means all.
rules[].apiGroups: omitted resolves the named resource across served groups; "" is the core group; * is all.
rules[].apiVersions: omitted means the preferred served version.
rules[].resources: plural resource names or *.
WatchRule adds rules[].sourceNamespace: omitted for the rule's own namespace, an exact name, or * for every namespace GitTarget.spec.allowedSourceNamespaces admits. Anything but the rule's own namespace passes the source-namespace gate (SourceNamespaceAuthorized); the resolved set is expanded to concrete names at compile time, so no wildcard reaches the data plane. A wildcard opens one stream for each admitted namespace.
ClusterWatchRule has no scope or namespace choice. rules[].scope is deprecated, accepts only Cluster, and a stored Namespaced value is refused at compile time.

Subresources are rejected in rule resources. Mirroring operates on top level resources; the selected /scale subresource effect is translated separately into a parent spec.replicas field patch.

The two namespace policies intentionally live in different planes:

ClusterProvider.spec.allowedNamespaces authorizes control-cluster namespaces to reference the provider. It is a tenant/export boundary; it neither selects nor grants access to source namespaces.
GitTarget.spec.allowedSourceNamespaces bounds what a target may mirror from its source cluster. Exact names need no Namespace read; selector and sourceNamespace: "*" policies are evaluated from a per-source-cluster Namespace-label snapshot. The source credential needs permission to list Namespaces for that selector path; otherwise exact-name policies still work but selector policies are held unevaluatable rather than widened or treated as empty.

CommitRequest

Source: api/v1alpha3/commitrequest_types.go
Controller: internal/controller/commitrequest_controller.go

A one shot "save now" signal that finalizes the open commit window for a same namespace GitTarget instead of waiting for the silence timer. The entire spec is immutable. Key fields:

spec.targetRef.name: target whose open window should be finalized.
spec.message: optional verbatim commit message (1–1024 chars, no control characters).
spec.closeDelaySeconds: optional 0–300s delay before the window is closed, so the author's own in flight changes can join the window before it closes.
status.conditions: kstatus-compatible. Ready is the summary (True once the request reached a terminal outcome that is not an error: a pushed commit, or a benign no-commit); Reconciling/Stalled are the kstatus progress/blocked pair; AuthorAttributed reports whether the /validate-operator-types validating admission webhook named the submitter; an absent record means the request claims no actor and can attach only to an unnamed window. Pushed reports whether the commit reached the remote. The Ready condition's reason carries Committed, NoWindowInGrace, WindowMismatch, AlreadyPresent, or FinalizeFailed. A benign no-commit (e.g. NoWindowInGrace) is Ready=True, Stalled=False, a correct non-error outcome, whereas a FinalizeFailed is Ready=False, Stalled=True.
status.branch / status.sha: set when the commit was pushed (Pushed=True).

How attribution and finalization interact is described under CommitRequest Finalize.

GitTarget

Source: api/v1alpha3/gittarget_types.go
Controller: internal/controller/gittarget_controller.go

One materialization from a source provider to a Git destination: (cluster provider, provider, branch, path). Key fields:

spec.providerRef: a GitProvider in the same namespace (group/kind default to configbutler.ai/GitProvider, the only accepted values).
spec.clusterProviderRef: a cluster-scoped source ClusterProvider; it defaults to {name: default}.
spec.branch: immutable branch, validated against GitProvider.spec.allowedBranches.
spec.path: immutable, required path under the repo (MinLength=1; . means repo root and must be chosen explicitly).
spec.encryption: optional SOPS/age encryption settings for sensitive resources.

providerRef, clusterProviderRef, branch, and path are immutable so a target cannot silently orphan an old materialization or change its source cluster. The controller also rejects path overlaps between GitTargets sharing a provider and branch.

Status has a kstatus-compatible summary layer plus domain conditions:

Ready, Reconciling, and Stalled are the generic conditions used by GitOps tooling. Ready=True means the latest observed generation is valid, the Git path is accepted, and source streams are running. Initial replay reports Reconciling=True. A human-fixable block reports Stalled=True.
Validated and EncryptionConfigured explain control-plane health.
StreamsRunning explains the source side: every tracked type is past initial replay and routing live events.
ClusterProviderReady and SourceClusterReachable distinguish valid source configuration from a source API the data plane can currently reach.
GitPathAccepted explains the target side: the selected Git path is safe for the operator to materialize.
status.streams is a bounded count summary, not a per-type list.

WatchRule and ClusterWatchRule add ResourcesResolved and GitTargetReady. ResourcesResolved explains the source selector. GitTargetReady mirrors the referenced GitTarget's write readiness. This keeps StreamsRunning honest: it only says source watches are running, not that Git writes can succeed.

GitProvider

Source: api/v1alpha3/gitprovider_types.go
Controller: internal/controller/gitprovider_controller.go

Represents a Git repository and the credentials/configuration used to write it. Key fields:

spec.url: immutable repository URL.
spec.secretRef: optional Secret in the same namespace for HTTP/SSH authentication.
spec.knownHostsRef: optional SSH known hosts source.
spec.allowedBranches: glob patterns that gate writable branches.
spec.push.commitWindow: rolling silence window for grouped commits, defaulting to 5s.
spec.commit.committer: committer identity (defaults to GitOps Reverser / noreply@configbutler.ai).
spec.commit.message: eventTemplate / reconcileTemplate / groupTemplate Go templates.
spec.commit.signing: SSH signing key reference and optional key generation.
status.signingPublicKey: populated when signing is configured and key material is available.

The controller verifies repository reachability and manages the signing key lifecycle. It generates an ed25519 keypair when signing.generateWhenMissing is set. The portable artifact across GitOps ecosystems is the credentials Secret, not a foreign repository object. The credentials reader accepts the Kubernetes native, Flux, and Argo CD Secret key dialects (see design/git-credentials-interop.md).

ClusterProvider

ClusterProvider is the read-side peer of GitProvider. A GitTarget references it by the immutable spec.clusterProviderRef; omission defaults to the conventional name default. The name is only a defaulting convention: a provider without spec.kubeConfig uses the operator's in-cluster client, while any name (including default) may instead carry a remote kubeconfig resolved from the operator namespace.

Its cluster scope is intentional. Several namespaces may mirror through one provider, but that provider's source identity must not vary by target because it keys source clients, discovery, watches, and attribution. The identity is the provider name rather than a deduplicated physical-cluster identity, so two providers pointing at the same API server still have separate contexts and authorization boundaries. spec.allowedNamespaces is therefore a deny-by-default control-cluster policy, enforced on every reconcile before watches start, so tightening it also stops an already-existing GitTarget, which an admission-time check could not do. It guards which tenant may cause the operator to export a shared source; it does not expand that source credential's Kubernetes permissions. The ClusterProvider validates its connection inputs, while a GitTarget projects provider readiness and live source reachability.

Remote credentials are re-read on the catalog-refresh cadence, not on every watch reconnect. A rotated Secret or a qps/burst change rebuilds that provider's clients and restarts only the watches using that provider; a definitively invalid or missing credential fails closed and stops those watches. An unreachable remote similarly makes only its dependent targets unready; it does not block discovery or watches for other source contexts.

Common flows

Say a GitTarget in control-cluster namespace team-a watches ConfigMaps, and a user runs kubectl apply to edit the ConfigMap team-a/app-config in that target's selected source cluster. Here is the path that change takes.

flowchart LR
    subgraph CONTROL["Control plane: operator cluster"]
        CP[ClusterProvider]
        KCFG[("Remote kubeconfig Secret\noperator namespace")]
        OWN["GitTarget watch set\nclaimed ∩ followable (GVR, scope)"]
        RELEVANCE["Relevance filter\nsanitize · followability · no-op diff"]
        RESOLVE["Resolver\nbounded grace window"]
        GTES[GitTargetEventStream]
        AFACTS["Audit fact extractor"]
        ASTREAM[("Per-type fact stream\nRedis Streams or in-process ring")]
        AINDEX[("In-process fact index\nbounded, TTL, keyed for join")]
    end

    subgraph SOURCE["Source cluster selected by ClusterProvider\n(the control cluster when kubeConfig is omitted)"]
        WATCH["WATCH + sendInitialEvents replay<br/>(per claimed GVR + scope)"]
        DISC["Discovery: CRDs / APIServices"]
        NS["Namespace labels\nfor allowedSourceNamespaces selectors"]
        AUDIT["/audit-webhook/&lt;provider&gt; (optional)"]
    end

    subgraph GIT["Git writes: internal/git"]
        BW[BranchWorker]
        PLAN["Manifest aware plan/flush"]
        PUSH["Atomic push"]
    end

    CP -. "optional Secret reference" .-> KCFG
    CP -->|"source client"| DISC
    CP -->|"source client"| WATCH
    DISC --> OWN
    NS -. "selector snapshot" .-> OWN
    WATCH --> OWN
    OWN --> RELEVANCE
    RELEVANCE --> RESOLVE
    RESOLVE --> GTES
    GTES --> BW
    BW --> PLAN
    PLAN --> PUSH

    AUDIT -. configured .-> AFACTS
    AFACTS -->|append| ASTREAM
    ASTREAM -->|follow| AINDEX
    AINDEX -. lookup .-> RESOLVE

Following the ConfigMap edit:

Watch delivers it. The API server applies the change and bumps the object's resourceVersion. The GitTarget's watch on core/configmaps in team-a delivers a MODIFIED event carrying the new object body. (On a cold start or after 410 Gone, the same object instead arrives as an ADDED during the sendInitialEvents replay.)
Relevance filter. The event is sanitized (status, managedFields, and volatile metadata stripped), checked for followability, and diffed against current Git content. A no-op (e.g. a */status bump whose desired-state projection is unchanged) is dropped here.
Resolve the author. When attribution is enabled, the resolver waits a bounded grace window (--author-attribution-grace, default 3s) for a matching audit fact in the attribution index, joining by resourceVersion/UID. On a strong match the real user or named service account becomes the author. With attribution disabled, the configured committer is the author; with attribution enabled but no usable fact, the explicit unknown (attribution unresolved) author marks the unresolved change. This wait is per-event and never reorders a watch. See Watch Event Ordering.
Route + window. The event flows into the GitTargetEventStream, and the BranchWorker appends it to the open commit window for (author, GitTarget).
Commit + push. When the window closes (5s of silence, a CommitRequest, or a buffer limit), the manifest aware writer patches team-a/.../app-config.yaml in place, commits as the resolved author, and pushes via PushAtomic (retrying with fetch/reset/replay if the remote moved).

Separately, the audit path (only when attribution is enabled): kube-apiserver POSTs audit events to /audit-webhook/<provider>; AuditHandler extracts the minimal attribution facts and appends them to that route's per-type fact stream. A follower on the watch side reads the streams for the types it watches into a bounded, TTL'd in-process index. That index is read only by the resolver in step 3; it never creates or repairs object state.

And if the watch had been lost? A delete that happened while no watch was running is reconciled on the next watch (re)connect: the sendInitialEvents replay plus mark-and-sweep removes any Git file whose object no longer exists. See State Ingestion and Not Losing Deletes. No path silently drops a delete.

Normal commit flow

Most writes are closed by the commit window timer. A CommitRequest is the explicit "save now" path: it does not create new state, and it does not bypass watch. It only asks the branch worker to close a matching open window early.

flowchart TD
    CHANGE[Persisted Kubernetes change] --> WATCH[Per-GitTarget watch event]
    WATCH --> FILTER[Sanitize + relevance filter]
    FILTER --> AUTHOR{Attribution enabled?}
    AUTHOR -->|yes| AUDIT[Wait bounded grace for audit fact]
    AUTHOR -->|no| COMMITTER[Use configured committer as author]
    AUDIT -->|fact matched| RESOLVED[Use resolved actor]
    AUDIT -->|no usable fact| UNRESOLVED[Use explicit unresolved author]
    COMMITTER --> WINDOW[BranchWorker open window<br/>one author + one GitTarget]
    RESOLVED --> WINDOW
    UNRESOLVED --> WINDOW

    WINDOW --> TIMER{Close trigger}
    TIMER -->|silence timer| FLUSH[Plan YAML edits + commit]
    TIMER -->|buffer limit or resync boundary| FLUSH

    CR[CommitRequest persisted] --> ADMISSION[Admission submitter lookup<br/>named or unnamed]
    ADMISSION --> ATTACH[Attach to matching open window]
    ATTACH -->|same author + GitTarget| FLUSH
    ATTACH -->|no matching window| BENIGN[Ready=True, Pushed=False]

    FLUSH --> PUSH[PushAtomic]
    PUSH --> DONE[Remote branch updated]

The important boundary is the open window: it accepts only the same author and GitTarget. A CommitRequest from another author never closes someone else's window; it resolves as a benign no-commit instead.

Remote moved while we were writing

If someone else pushes to the same remote branch before GitOps Reverser's push lands, the operator does not treat its local clone as authoritative. It keeps the finalized pending writes, fetches the new remote tip, resets the local clone, replays those writes, and pushes again.

flowchart TD
    LOCAL[Window finalized<br/>pending write retained] --> CHECK[PushAtomic checks remote ref]
    CHECK --> MATCH{Remote still at expected SHA?}
    MATCH -->|yes| ACCEPT[Push accepted]
    ACCEPT --> CLEAR[Clear retained pending writes]

    MATCH -->|no| FETCH[Fetch latest remote tip]
    FETCH --> RESET[Hard reset local clone to remote]
    RESET --> REPLAY[Replay retained pending writes<br/>from sanitized API state]
    REPLAY --> REFRESH[Create fresh commit hashes]
    REFRESH --> CHECK

    CHECK -->|attempts exhausted or fetch fails| RETAIN[Keep pending writes for later retry]

For a CommitRequest, success is reported only after the push reaches the remote. If the push had to replay on top of someone else's commit, status.sha is the refreshed post-replay commit SHA, not the stale local SHA from before the retry.

What it writes to Git

A GitTarget owns one subtree (spec.path) on one branch. A new object follows the layout the folder already has (or a declared policy); once a document exists it is edited in place wherever it already lives. A populated target seeded from empty looks like:

team-a-config/                              # GitTarget spec.path
├── README.md                               # operator-managed bootstrap file
├── .sops.yaml                              # present only when encryption is configured
├── team-a/
│   ├── configmaps/app-config.yaml
│   ├── secrets/db-creds.sops.yaml          # sensitive types are SOPS/age encrypted
│   └── apps/deployments/api.yaml

The built-in default path is {spec.path}/{namespace}/{group}/{resource}/{name}.yaml, namespace first, the API group omitted for core resources, no version segment, and a .sops.yaml suffix for sensitive resources; a cluster-scoped resource uses the literal _cluster/ in place of the namespace (an illegal Kubernetes namespace name, so it can never clash with a real one). That default is what a new resource gets unless something more specific applies: a GitTarget's own declared placement policy, or a folder that kustomize builds from a single root. Details are in File Placement.

State ingestion and not losing deletes

This is the heart of the system. Object state is ingested by watch, and the guarantee is: every persisted mutation observed while watching reaches Git, and no delete is ever silently dropped across a gap.

Watch is already ordered by resource version

The apiserver delivers a watch's events already ordered by resourceVersion for that type, so there is nothing to re-order: a MODIFIED always lands after the create it modifies. Each event carries GVR, scope, event type (ADDED / MODIFIED / DELETED, plus the transport BOOKMARK and ERROR), namespace/name/UID/resourceVersion/deletionTimestamp, and the sanitized object body. The initial-events-end bookmark marks the end of a replay, and an ERROR such as 410 Gone triggers a fresh sendInitialEvents reconnect.

Recovery: resume, replay, or list plus mark-and-sweep

Losing a watch (pod eviction, rollout, crash, 410 Gone) is normal. When Redis has a cursor for the watch shard, the next session first opens a normal watch from that resourceVersion. If the apiserver can supply all events since that cursor, the watch continues from there. If the cursor is expired, or no cursor exists, the watch opens with sendInitialEvents=true and ResourceVersionMatch=NotOlderThan, so the apiserver streams current state as a replay of ADDED events terminated by the initial-events-end bookmark. The operator runs a mark-and-sweep over that replay:

every replayed object is marked ADDED, up to initial-events-end;
at the bookmark, any Git file under that GitTarget whose object was not marked no longer exists, so a DELETED is emitted for it (committer-authored, because the actual delete was never witnessed);
then the watch streams live events.

This mark-and-sweep is load-bearing and fires only on watch re-establishment, never on a timer: there is no periodic object LIST or hourly object-drift sweep. (A target using a selector-based allowedSourceNamespaces policy periodically lists only Namespace labels to maintain that authorization scope; it never uses that list to infer object state or sweep Git.) The sweep is the only thing that reconciles a delete that happened while no watch was running, so it is what makes the watch safe to lose and restart. It is applied through the same per-type reconcile/writer machinery as live writes (see Mark and Sweep Resync).

If the apiserver forbids sendInitialEvents for a type, the operator logs an explicit warning, starts a normal watch, buffers its events, performs a LIST snapshot, runs the same scoped mark-and-sweep from that list, and only then lets the buffered watch events through. This is the compatibility path for older aggregated API servers that do not implement streaming lists.

Relevance filtering is product code

Watch has no audit policy, so it delivers every persisted MODIFIED, including controller status churn. The relevance filter reproduces that filter in product code, on the hot path:

Sanitization (internal/sanitize) strips status, managedFields, and volatile metadata before diffing, so runtime churn never masquerades as a desired-state change.
Followability (internal/typeset) encodes "controller-owned → don't mirror" A type that is not followable never gets a watch.
No-op suppression. A */status write bumps resourceVersion but its sanitized projection equals the prior commit; the writer diffs it to a no-op and discards it.

History granularity

Watch carries only the versions it observes. While connected it sees each MODIFIED; across a replay after 410, a compaction, or downtime it collapses every intermediate version into current state. The product is therefore a state mirror with opportunistic per-mutation history, not a guaranteed per-mutation change log.

Optional attribution

Handler / fact extractor: internal/webhook/audit_handler.go
Fact transport (per-type stream): internal/queue/fact_stream.go
In-process fact index: internal/queue/fact_index.go
Resolver (grace window join): internal/watch/author_resolver.go
Spec: spec/attribution.md (the single reference)
Design record: attribution-fact-stream.md

Attribution runs only when --author-attribution=true. A normal source posts audit EventList payloads to /audit-webhook/<audit-route>, where the route is ClusterProvider.spec.attribution.auditRoute and defaults to the provider's own name. The bare /audit-webhook endpoint is enabled only with --author-attribution-audit-route-annotation-key, for a trusted control plane that puts an audit route in each event. There is no supplementary body endpoint or body joiner, because watch (not audit) carries the object body. The handler applies an intrinsic accept gate (StageResponseComplete, a mutating verb, success, non-dry-run, a changed resourceVersion, and the /scale subresource only), then appends the accepted events' facts to that route's per-type fact stream: one append per type per request, not one per event.

The two halves never call each other. The receiver publishes and returns; the watch side follows the streams for the types it watches and keeps their facts in a bounded, TTL'd in-process index. They meet only through the keys a fact was filed under. That split is what lets the audit endpoint answer fast during a rollout, and what lets a fact published by one replica serve a watch running on another once the streams are shared (the Redis transport below).

The transport is selectable, because the index is what does the work and the stream is only how facts travel:

`--author-attribution-transport`	Store	When
`redis` (default)	Redis Streams, per type, with a retention window a restarting process replays	any install; required for more than one replica
`memory`	an in-process ring	single replica, no Redis; every fact is lost on restart, by design

Endpoint	Role
`/audit-webhook/<audit-route>`	One audit route's stream; some ClusterProvider must carry that route
`/audit-webhook`	Shared stream only when annotation routing is configured; each event names its route

The handler accepts a client certificate signed by the audit CA, but it does not yet bind that certificate to a named provider. Do not treat named routes as an isolation boundary for independently administered remote sources that share a client credential; provider-bound ingress authentication remains outstanding.

Optional, but never casual

Attribution being optional does not make it casual or take-it-or-leave-it. When it is enabled the operator does everything it can to name the real actor, but it values high certainty over a plausible guess. Attaching a name to a change is a consequential, sometimes politically charged claim ("this person did that"), so it must not be made lightly. The design therefore fails toward honesty: a weak, conflicting, late, or absent fact produces an explicit unresolved author rather than a guessed person or a misleading committer identity. It is better to say that attribution did not resolve than to assert an actor we are not sure of. In an installation that expects live mutations to be attributable, this outcome is a concrete signal to check audit delivery and attribution configuration.

Two engineering choices follow directly from that stance:

mTLS on the audit ingress is on by default. The audit server requires and verifies a client certificate (tls.RequireAndVerifyClientCert against a configured CA; --audit-insecure defaults to false). This authenticates membership in that CA's client set. It is not yet a sender-to-provider binding, so multi-source deployments must not share a credential across independently trusted sources.
Tests pin the behavior. Because a misattribution is a real harm, the attribution and resolver paths carry unit and e2e tests that prove the concrete cases: strong match, weak/last-key match, deletes whose audit RV differs from the watch RV, a collection delete joined by uid set and by scope, an aggregated API whose facts carry only a name, missing/late/expired facts, service-account vs human actor, impersonation, and the explicit unresolved outcome, so these certainty guarantees cannot silently regress.

Attribution fact shape

The fact is the smallest thing needed to name an author, not an object log:

Field	Purpose
`auditID`	diagnostics / dedupe
`user` / `impersonatedUser`	author candidate (human or service account)
`verb`, `subresource`	explain the write
`responseStatus.code`, `dryRun`	reject failures and non-persistent requests (at the handler gate)
source provider, GVR, namespace, name, UID	source partition plus exact join keys
response object resourceVersion	exact watch-event match
stage timestamp	recency

A deletecollection produces one collection fact instead: it names no object, so it keeps the request's selector and whatever uid set the API server returned, and every removal in its scope joins against it.

The index files each fact under the strongest key it has, first match wins, within a scope of (audit route, group/resource): (uid, rv) and (uid) together when it has both, else (uid), else (rv), else (namespace, name). A fact keeps every field it recovered but is not filed under weaker keys it could never be read by. A watch event always knows its object's uid, so a uid-keyed fact is never looked up by name, and a second copy would cost memory on every replica for the whole TTL and answer nothing. The one branch that files twice is (uid, rv) plus (uid): the first serves creates and updates, the second serves removals, one fact answering two different questions.

Entries carry a short TTL (minutes, not hours) and are bounded per type and in total; expiry is checked on read, so an aged-out fact is never joined merely because the sweep has not run. Old facts are never needed for correctness, because watch owns state.

The resolver and its grace window

A watch event waits a bounded grace window (--author-attribution-grace, default 3s) for matching evidence, then ships regardless. It does not poll: it registers a waiter under every key its query could match, looks once, and then sleeps until either a fact that could answer it is applied or the grace expires. Registering before the first lookup is what closes the race a fact landing in the gap would otherwise win. There is no Redis call on this path: the fast case is a map read.

The resolver takes the strongest evidence available, and the tier it used is the tier label on attribution_resolutions_total. Who that evidence named is the separate actor_kind label, in the same user/serviceaccount/none vocabulary commits_total{author_kind} uses:

Tier	Key	What it asserts
`delete_sticky`	uid (a delete fact, sticky)	who asked for this object's deletion
`exact`	uid + rv	this actor produced this exact version
`deletecollection_body_uid`	uid in a collection fact's set	the API server said this request deleted this object
`latest`	uid (the object's own delete fact)	who removed it
`name`	namespace + name	the same, for a fact with no uid (an aggregated API's usual shape)
`latest`	uid (a write fact)	who last wrote it; a fallback for a removal
`deletecollection_scope`	namespace + selector + window	a collection request covering it was made
`resource_version`	rv (the escape hatch for a fact with no uid)	a fact for this exact version, unidentified
`absent`	none	nothing usable arrived in time

Three rules carry most of the behavior. A fact about a deletion may not be replaced by a fact about a write: a removal fact takes a sticky, uid-keyed slot that the ordinary last-writer-wins structures cannot reach, and a removal asks that slot first. That slot is the one structure the fact TTL does not bound: a uid is unique across space and time, so the statement cannot be superseded, and its horizon is the index's caps instead. It is still in-memory: a restart re-warms the index from one TTL of stream retention like everything else. Without the slot, a finalizer patch's fact overwrites the deleter's, because both carry the resourceVersion the deletion stamped (spec/attribution.md). A removal never returns on a write fact without looking further: the per-object tiers are last-writer-wins, so for a removal they hold whoever last edited the object, which is not who deleted it; such a match is held as a fallback while the wait continues for evidence about the deletion itself. And an exact-capable event may not fall through to the removal tiers: a create or update presents the resourceVersion its own write produced, so if the exact tier misses, the uid pointer may name an older, different author.

No branch on this path depends on the type. Every decision is made on the verb, on whether the event is a removal, and on which fields the event happens to carry. That is why an aggregated API needed no special case to attribute.

On a resolved tier the actor becomes the author: a human or a service account alike, always named by its own username (e.g. system:serviceaccount:flux-system:kustomize-controller). An absent resolution produces unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid> instead of a guessed actor. A late fact that arrives after a commit has shipped never rewrites it. With attribution disabled the resolver is absent and every commit is committer-authored.

The wait is head-of-line on its watch shard, so a resolution that sits out the grace delays the events queued behind it on the same (GitTarget, GVR, scope) goroutine. That cost, and what it once broke, is in Watch Event Ordering and attribution-removal-wait-options.md.

The CommitRequest controller does not read this audit index. A CommitRequest's submitter is named by the /validate-operator-types validating admission webhook instead, captured synchronously at admission, never joined from an audit fact (see CommitRequest Finalize).

Author and committer identity in Git

Git natively carries two identities on every commit, the committer (who created the commit object) and the author (who wrote the change), and GitOps Reverser uses both on purpose (commit.go):

The committer is always the operator: the configured GitProvider.spec.commit.committer, defaulting to GitOps Reverser <noreply@configbutler.ai>. Every commit, attributed or not, is committed by the operator, because the operator is what wrote it to Git.
The author is the real actor, but only when we are sure. On a strong attribution match the author is set to that actor. Git always carries an author, so it is never left blank, and it is never a guessed person. What fills it when we are not sure depends on WHY:
- Attribution is switched off (configured-author mode, and reconcile/resync writes that have no actor at all): the author is the operator, identical to the committer. That is honest, because the operator is the author.
- Attribution ran and did not resolve an actor: the author is the explicit sentinel unknown (attribution unresolved) <attribution-unresolved@gitops-reverser.invalid>. It is deliberately NOT the operator, because authoring it as the operator made a lost actor byte-identical to a configured-author commit, so the gap was invisible in Git history and only countable in a metric.
A commit whose author is a person is therefore a positive statement that the operator is confident who made the change; one authored by the sentinel is a positive statement that it tried and could not tell.

The author identity is never fabricated as a person: a real author is always taken from the authenticated request, and an unresolved one is labeled as unresolved rather than attributed to anybody. The name and email come from the OIDC claims when the apiserver maps them; otherwise they come from the actor's own Kubernetes identity: the username, which for a controller is its service account (system:serviceaccount:<namespace>:<name>), with a stable derived email under noreply.cluster.local. The two user.extra keys the operator reads for the OIDC name and email, and the apiserver config that fills them, are in Wiring OIDC author claims.

A confidently attributed commit shows distinct author and committer lines, which git surfaces with --format=fuller. A human carries the OIDC display name and email:

$ git show --no-patch --format=fuller HEAD
Author:     alice <alice@example.com>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

A service-account actor (no OIDC claims) is named by its own Kubernetes identity, with a complete derived email and the operator still the committer:

$ git show --no-patch --format=fuller HEAD
Author:     system:serviceaccount:flux-system:kustomize-controller <systemserviceaccountflux-systemkustomize-controller@noreply.cluster.local>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

When attribution is off, or when a reconcile/resync write has no actor because attribution was never attempted, the author is set to the operator as well, so both lines are identical. A Git UI will show this as a single "author". This is configured-author mode: the operator created the Git commit and makes no claim about a Kubernetes actor.

$ git show --no-patch --format=fuller HEAD
Author:     GitOps Reverser <noreply@configbutler.ai>
AuthorDate: Mon Jun 30 12:00:05 2026 +0000
Commit:     GitOps Reverser <noreply@configbutler.ai>
CommitDate: Mon Jun 30 12:00:05 2026 +0000

Wiring OIDC author claims

Audit-path extractor: internal/queue/audit_event_parsing.go
Admission-path extractor: internal/webhook/validate_operator_types_handler.go
Signature construction: internal/git/commit.go (authorName / authorEmail)

The operator never talks to the identity provider. It reads two fixed keys from the request's user.extra map: the audit attribution path and the CommitRequest admission path resolve the same two keys:

`user.extra` key	OIDC claim it carries	Used for
`configbutler.ai/claims/display-name`	`name`	git author `Name`
`configbutler.ai/claims/email`	`email`	git author `Email`

So the ID token the provider issues must carry the standard name and email claims:

{
  "iss": "https://idp.example.com",
  "sub": "248289761001",
  "name": "alice",
  "email": "alice@example.com",
  "aud": "kubernetes",
  "exp": 1751284805
}

The cluster operator maps those claims into the two keys with a structured AuthenticationConfiguration (apiserver.config.k8s.io, beta from Kubernetes 1.30):

apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://idp.example.com
      audiences: [kubernetes]
    claimMappings:
      username:
        claim: email          # the Kubernetes username; `sub` also works
      extra:
        - key: "configbutler.ai/claims/display-name"
          valueExpression: "claims.name"
        - key: "configbutler.ai/claims/email"
          valueExpression: "claims.email"

Both mappings are optional: a missing or unusable name falls back to the username, and a missing or invalid email falls back to a derived address under noreply.cluster.local.

Watch event ordering

The grace window is a per-event wait on a single-threaded watch goroutine (one goroutine per (GitTarget, GVR, scope)), and the downstream GitTargetEventStream → BranchWorker path is a synchronous FIFO. So:

Same object / same type order is strictly preserved. An object's events all flow through one watch goroutine in resourceVersion order; the grace wait is head-of-line (it delays the next event, never lets it overtake), so an older mutation can never overwrite a newer one.
The cost is throughput, not ordering. A long wait stalls its own watch up to the grace window.
Unrelated objects on different types (separate concurrent watches) may interleave or be grouped into commits differently than wall-clock, and resourceVersion is not comparable across types anyway. They are usually different files, but they can share one, as different documents of a multi-document YAML, e.g. when a human or kustomize placed them together, so "different files" is not guaranteed. It still does not affect the materialized state: every write carries its object's current state, all writes to a branch funnel through one BranchWorker, and the editor patches each document in place without disturbing its siblings. Only which commit each lands in and when can differ, and only within the commit window. a few seconds at most.

The full analysis, worked examples, and the future non-blocking option are in Watch event ordering under the attribution grace window.

Rule and type resolution

How a user's WatchRule becomes "this GitTarget follows these concrete types in these namespaces."

RuleStore

Source: internal/rulestore/store.go

An in memory cache populated through the shared compile paths (watch.CompileWatchRule / watch.CompileClusterWatchRule) that both the controllers and the startup bootstrap call, so a restart cannot seed an ungated rule. Compiled rules carry the full chain from rule to GitTarget, GitProvider, branch, and path, plus each WatchRule item's RESOLVED source-namespace set. It is read by the watch manager (to build watched type tables for GitTargets) and the rule change reconcile path.

APIResourceCatalog

Source: internal/watch/api_resource_catalog.go

A thin normalizer for each scan: it turns one discovery result into a policy annotated typeset.Scan and keeps only mechanical bookkeeping. All judgement across scans lives in the typeset registry (Registry.UpdateFromScan): a failed group/version keeps serving last known facts instead of looking like an empty API surface, and a group/version that vanishes from a complete scan rides a removal grace rather than being pruned. Both protect against accidental Git deletions on a discovery blink (see typeset-owns-discovery-grace.md). Every active source context has its own catalog and registry, so a CRD served only by prod-eu-1 is never resolved for a target mirroring prod-us-1. Catalogs refresh at startup and on the 30-second reconciliation cadence. CRD/APIService trigger informers run only in the control plane, where the operator's own CRDs live; a remote source's API-surface change is discovered on that periodic refresh or when a rule/target reconciliation explicitly refreshes that source.

TypeRegistry and followability

Source: internal/typeset/
Design: type-followability.md

internal/typeset is the single decision surface for "can this type be followed?" Each TypeRecord carries GVK/GVR identity, scope and preferred version facts, origin classification, subresource facts (including usable /scale bindings), sensitivity policy, and one Followability verdict. Manifest analysis and delete/scale resolution with only GVR all read it. The registry also owns the second, demand axis via the Materializer: a type is materialized only when it is Followable ∩ claimed.

WatchedTypeTable

Source: internal/watch/watched_type_table.go

A projection for each GitTarget from the type registry, filtered by that target's rules, recording resolved GVK/GVR/scope plus namespace and operation coverage. This is where rule matching effectively happens: it resolves the set of (GVR, scope) a GitTarget claims, so the watch manager opens one watch per claimed ∩ followable (GVR, scope) and scopes each watch's events back to that GitTarget's source namespaces. A sourceNamespace: "*" rule is already expanded here, so each admitted namespace has its own stream and mark-and-sweep boundary.

Watch ingestion and reconcile

Desired state comes from one raw watch per (GitTarget, GVR, scope). Each event is sanitized, diffed against current Git content, and applied. There is no separate per-type object store to reconstruct, because Git already holds current state. This section is the authoritative contract; the watch-first design record is historical context.

Manager: internal/watch/manager.go
Watch / replay / sweep: internal/watch/target_watch.go
Author resolver (attribution join + grace window): internal/watch/author_resolver.go
Event router: internal/watch/event_router.go
Worker resync apply (mark-and-sweep): internal/git/resync_flush.go

The watch manager

The watch manager is a controller runtime Runnable (NeedLeaderElection). It owns type level discovery, the per-GitTarget watch sets, and the watched type tables for GitTargets. Its object-state intake is the watches themselves. Every active ClusterProvider has an independent source context; its catalog, registry, dynamic client, and reachability can fail or recover without borrowing facts from another source. Discovery trigger informers (CRDs / APIServices) run only in the control plane and accelerate its catalog refresh; remote source catalogs use the periodic refresh.

On Start it bootstraps the RuleStore from existing rules, refreshes the API catalog, updates the TypeRegistry, builds watched type tables, and opens one watch per claimed ∩ followable (GVR, scope).

Opening watches

On each GitTarget reconcile the controller resolves the GitTarget's claimed ∩ followable (GVR, scope) set. Fully specified GVRs are claimed unconditionally; wildcard rules and rules without a version are resolved fail closed against discovery. For each (GVR, scope) the manager runs one watch goroutine that opens the watch with sendInitialEvents=true, folds the replay into a desired set, runs the mark-and-sweep at initial-events-end, then streams live events through the relevance filter and the author resolver into the GitTargetEventStream. On disconnect or 410 Gone the goroutine reconnects and repeats the replay + sweep. See Recovery: resume, replay, or list plus mark-and-sweep.

There is no periodic object LIST, object checkpoint, or timer-driven object-drift sweep. The sweep fires only on watch re-establishment. Selector-based source-namespace authorization is separate: it may periodically list Namespace labels in the relevant source cluster, but that list never materializes objects or triggers a Git sweep.

Rule change reconcile

A WatchRule / ClusterWatchRule / GitTarget change, or a control-plane CRD / APIService change, reaches a GitTarget through the GitTarget controller, which Watches those objects (generation change predicates) and queues the affected GitTarget again. On reconcile the GitTarget refreshes its own source catalog and resolves its claimed (GVR, scope) set again; a type a new rule starts watching gets a new watch opened (with a sendInitialEvents replay) and a type no longer claimed has its watch closed. A remote CRD / APIService change has no trigger informer; the next periodic source-catalog refresh performs the same re-resolution. The watch manager then refreshes the watched type tables.

Mark and sweep resync

The BranchWorker applies a reconcile by scanning the GitTarget subtree and building a manifest plan (this write side is shared with live writes):

before anything is planned, a structure-only acceptance gate runs over the scanned subtree; if it finds content the operator cannot safely manage: a kustomization using an unsupported feature (generators / inline or JSON6902 patches / components / helm / replacements / transformers / name(pre|suf)fix / remote bases) or malformed images:/replicas: overrides, a duplicate manifest identity, an impure managed file, or a standalone non-KRM / invalid YAML; (a path-based strategic-merge patches: entry is tolerated as read-only build context, and an overlay reading ../../base is rendered by reading that base, and neither refuses the folder) the whole apply is refused: nothing is committed, GitPathAccepted=False, Stalled=True, and Ready=False with reason UnsupportedContent until a human cleans the path;
desired resources are upserted through the same content derived path as live writes;
existing managed documents that are watched but absent from the desired set are deleted;
the operator's own build directives (kustomization.yaml, .sops.yaml) and other allowlisted auxiliary YAML are retained, not materialised and not refused;
nothing is committed if the apply cannot complete safely.

The acceptance gate is structure-only on purpose: it never refuses on a discovery-derived followability fact (unwatched / out-of-scope), which can blink on a discovery wobble; only facts that are true from the path's structure alone block a GitTarget. The same gate runs on the live write path, so a refused path is never written into by a racing live event either. See unsupported-folder-refusal-plan.md.

A reconcile is type scoped (ScopeGVR): the sweep is restricted to one (group, resource), so anchoring one type again never disturbs another's manifests. The desired set for the sweep is the sendInitialEvents replay (everything marked ADDED up to initial-events-end), so a delete is reconciled only after the replay completes.

Git write architecture

BranchWorker

Source: internal/git/branch_worker.go
Worker manager: internal/git/worker_manager.go

BranchWorker owns a local clone and a single FIFO event loop for its (provider namespace, provider, branch) tuple. Events accumulate in one open commit window, which accepts only one (author, GitTarget) pair at a time:

same author + same GitTarget: append to the window;
different author or GitTarget: finalize the current window first;
repeated writes to the same Git path inside a window use last write wins.

The window finalizes when spec.push.commitWindow passes with no new matching event, the retained buffer reaches --branch-buffer-max-size (default 8Mi), a CommitRequest finalize deadline matches the open author and GitTarget, or a resync request that is not a heal or shutdown arrives. Successful local commits are retained until a fixed push cooldown (5s) allows a push, which prevents remote push storms during bursts. Heal resyncs that arrive during a window are deferred and drained at the next idle boundary.

Local clones and conflict retry

Local clones live under /tmp/gitops-reverser-workers/{provider namespace}/{provider}/{branch}/repos/{url-digest}. The leading segments are exactly the (provider namespace, provider, branch) tuple that identifies the BranchWorker, and the final {url-digest} segment is a short digest of GitProvider.spec.url (a truncated SHA-256 of the remote URL, repoCacheKey), not a commit hash. It only disambiguates distinct remote URLs under the same branch and is stable across commits, so the worker reuses one clone for the life of the branch. PushAtomic checks the remote ref before pushing. If the remote diverged it smart fetches the latest tip, hard resets the local clone, replays the retained pending writes against the fresh tip (refreshing commit hashes), and retries up to the attempt limit. This is valid because every pending write is rebuilt from sanitized API state; nothing depends on locally edited files.

Durability of the write queue (planned)

A BranchWorker's queue (the open commit window's retained writes plus any local commits not yet pushed) lives only in process memory today. It is about to be materialized into Redis, for two reasons:

High availability. Multi-pod HA needs a durable, cross-pod write queue so a branch's accepted-but-unpushed work survives a failover: the pod that takes the branch-shard lease resumes that queue instead of starting from an empty buffer. This is part of the HA / GitTarget distribution plan.
Crash safety that watch replay cannot guarantee. A crash drops the in-memory queue, and those writes cannot be reliably re-derived by replaying the watch, because Kubernetes does not guarantee it can replay events from every resourceVersion. The apiserver compacts old versions, so resuming from a stored cursor can return 410 Gone; recovery then falls back to a full sendInitialEvents replay plus mark-and-sweep, which rebuilds current state but collapses the intermediate versions (see History granularity). Persisting the queue lets accepted-but-unpushed writes survive a restart on their own, independent of whether the watch can resume from where it left off.

Manifest aware writer

Steady state: internal/git/plan_flush.go
Resync apply: internal/git/resync_flush.go
YAML editor: internal/git/manifestedit/
Analyzer/planner: internal/manifestanalyzer/
Object projection: internal/manifestreport/

For each commit the writer scans YAML files under the GitTarget path, builds a manifest store without bytes keyed by resource identity, resolves each event (or each desired resource, for resync) to one action, hydrates only touched files into buffers for the commit, and flushes only changed or deleted files.

Upserts: if a managed document for the resource already exists, patch it in place (preserving siblings in a multi document file); if it is sensitive, encrypt the whole document again at its existing path; if no document exists, place a new file per File Placement (declared policy, then the folder's one kustomize root, then the canonical default).
Kustomize override edit-through: a live value produced by a well-formed images: or replicas: entry in the document's kustomization chain is written back to that entry (comment-preserving, only fields the entry already declares); the source manifest keeps its bytes. Anything the inversion cannot express falls back to the plain in-place patch. See images-and-replicas edit-through design.
Deletes: use the manifest identity index, so a moved manifest can still be deleted even when it is not at the canonical path.
Field patches (currently /scale → parent spec.replicas) are intentionally narrow: they only patch an existing parent manifest and never fabricate a parent object from partial subresource data; a spec.replicas assignment governed by a replicas: override is routed to the entry instead.

File placement

Placement runs only for a resource with no existing document in the target. Existing resources are match first: once a document exists in Git, updates and deletes use its current location, found by manifest identity rather than by path, instead of recomputing placement. So a change to how new files are placed never moves a file already in Git. A new resource is placed by the first of these that applies (internal/manifestanalyzer/placement.go, design):

Declared policy (spec.placement). A GitTarget can declare a byType map (exact [group/]version/resource → path template) plus a default template, rendered from a small brace-variable path language ({namespace}, {group}, {resource}, {name}, …).
The folder's one kustomize root. When the whole writable subtree is governed by exactly one supported kustomization.yaml, the file lands beside it and gets a resources: entry in the same commit. This step is a structural fact rather than a reading of the folder's conventions: the canonical path below is a tree a resources: graph cannot reach, so a file written there would never be rendered. Two supported kustomizations is ambiguous and declines.
Canonical fallback. Otherwise the built-in default {spec.path}/{namespace}/{group}/{resource}/{name}.yaml: namespace-first, group omitted for core, no version, _cluster/ for cluster-scoped, .sops.yaml for sensitive.

The layout of the folder's other documents is not an input. An earlier release followed it (sibling inference), which made a human's edit to the repository change where the operator wrote next, with no Kubernetes object changing and nothing in status recording the move. It was removed; a layout the ladder cannot derive is declared in spec.placement, and gitopsreverser_placements_total{source="canonical"} names the target and type that needs the line.

Sensitivity is a write-safety classifier, not a placement input: whatever path is chosen, a sensitive resource is written encrypted, is never appended to an existing file, and is never co-mingled with a plaintext document. When those guarantees cannot be honoured (e.g. a bundling default would route a sensitive resource into a shared file), the resource is refused fail-safe rather than written unsafely: logged per-resource, counted in the resync summary as placementSkipped, and counted by gitopsreverser_placement_refusals_total{reason}.

Bootstrap, encryption, and signing

Bootstrap (bootstrapped_repo_template.go): the first write to a GitTarget path stages a README.md and, when encryption is configured, a .sops.yaml with age recipient rules. Existing files are preserved.
Encryption (encryption.go, sops_encryptor.go, sensitivity policy): core Secrets are sensitive by default; --additional-sensitive-resources adds more. Sensitive resources are never written in plaintext. If encryption is required and unavailable, the write fails before any plaintext file is created. Encrypted output is cached by metadata + plaintext digest to avoid redundant SOPS work.
Signing (signing.go, sshsig/): commits use OpenSSH signatures, with the key read from GitProvider.spec.commit.signing.secretRef or generated when configured.

CommitRequest finalize

Controller: internal/controller/commitrequest_controller.go
Admission handler: internal/webhook/validate_operator_types_handler.go
Admission author cache: internal/queue/command_author_store.go

A CommitRequest finalizes the open commit window for its GitTarget. The request author is resolved from the /validate-operator-types validating admission webhook, which captures the authenticated submitter at admission (keyed by the object's UID) before the object is visible. The lookup is present-or-never: if the record is missing, the request is marked AuthorAttributed=False, claims no actor, and still attaches immediately. That is not a failure:

The controller stamps the in-progress conditions (Reconciling=True) and settles AuthorAttributed synchronously from the admission author cache. There is no audit wait on this path.
The controller eagerly attaches the request to the worker (AttachCommitRequest), anchoring the finalize at receipt + closeDelaySeconds. The worker binds it to an open window only when the author state and GitTarget match. It never finalizes another author's window; a window carries at most one request.
The window finalizes on the deadline (or when it closes for any other reason). If a finalize closes an open window, the worker always schedules a push, so a window closed by an otherwise no-op resync is not stranded.
Outcomes resolve on push and are reported as conditions: a pushed commit sets Ready=True / Pushed=True with branch/sha; a benign no-commit sets Ready=True with the reason on Ready and Pushed=False; a failure sets Ready=False / Stalled=True with a message.

The CommitRequest submitter is not recoverable from object state alone, so without an admission record the request cannot claim an actor. The final Git author remains the attached watch window's author: configured committer when attribution was disabled, or the explicit unresolved author when live attribution ran but could not resolve. There is no audit-fact join for the request itself: its submitter is captured at admission by the validating webhook, not derived from the audit attribution index.

Controller wiring

Controllers watch their dependencies so dependents reconcile quickly after spec changes:

GitTargetReconciler watches GitProvider, ClusterProvider, Namespace, WatchRule, and ClusterWatchRule. Provider readiness/spec changes and namespace-label changes promptly re-check source authorization; rules re-declare the claimed (GVR, scope) set. It deliberately does not watch encryption Secrets, so their recovery is picked up by periodic reconciliation without retaining every Secret value in the control-plane cache.
WatchRuleReconciler / ClusterWatchRuleReconciler watch GitTarget and GitProvider, populate the RuleStore, and trigger the rule-change reconcile.
GitProviderReconciler validates reachability and manages the signing key lifecycle.
ClusterProviderReconciler validates source-connection inputs and projects its readiness to dependent targets. It also removes the retired fact-purge finalizer from pre-release objects during an upgrade.
CommitRequestReconciler runs with MaxConcurrentReconciles=1 and attributes/attaches as above; its optional AuthorLookup is the command-author cache populated by the /validate-operator-types webhook (wired whenever the admission webhook is enabled, independent of --author-attribution, and nil otherwise, so the request claims no actor).

Dependency watches use narrow predicates to avoid status-only heartbeat churn: a ClusterProvider also admits a Ready transition, and a Namespace admits only label changes. GitProvider, GitTarget, and CommitRequest carry immutability constraints where a spec change would orphan a materialized subtree or invalidate an in-flight finalize.

Startup sequence

Defined in cmd/main.go:

flowchart TD
    A[Parse flags + logger + build info] --> B[Init telemetry / metrics server]
    B --> C[Create controller runtime manager]
    C --> D[Create RuleStore]
    D --> E[Create WorkerManager + register Runnable]
    E --> F[Create Watch Manager + EventRouter; inject TypeRegistry]
    F --> G[Register WatchRule + ClusterWatchRule controllers]
    G --> H{redis-addr set?}
    H -->|yes| Hi[Create Redis cursor store + wire WatchCursorStore + readiness gate]
    H -->|no| Hj[Skip Redis: WatchCursorStore nil; watches cold-replay on restart]
    Hi --> Hq{author-attribution?}
    Hj --> Hq
    Hq -->|yes| I[Select fact transport; start fact index + follower;<br/>wire audit fact extractor + audit HTTP server + resolver]
    Hq -->|no| J[Configured-author: no fact streams or index; audit webhook skipped]
    I --> K[Setup + register Watch Manager]
    J --> K
    K --> L[Register GitProvider + ClusterProvider + GitTarget + CommitRequest controllers]
    L --> M[Add cert watchers + health checks]
    M --> N[mgr.Start]

Redis is optional in configured-author mode. When --redis-addr is set, the cursor store is wired and a Redis readiness gate keeps the pod not-ready until Redis is reachable; watches resume from their last stored resourceVersion after a restart. When --redis-addr is empty, the cursor store is skipped and watches cold-replay from scratch on restart instead. The binary's --author-attribution flag defaults to on: the fact transport is constructed, the fact index and its follower are started, the audit HTTP handler is wired with the fact extractor, the watch manager gets the author resolver, and the audit ingress is added to /readyz. Which transport it builds is --author-attribution-transport: redis (the default) requires a non-empty --redis-addr, while memory runs the streams in process and is refused unless --replica-count is 1. The Helm chart deliberately passes --author-attribution=false by default, so a first install runs configured-author with no fact streams or audit webhook and every commit is committer-authored.

Observability

Source: internal/telemetry/exporter.go

Metrics are exported over OTLP / the metrics server. The reader's guide to every live family, with copy-pasteable PromQL, is interpreting-metrics.md.

The pipeline is one sentence: watch events arrive and are processed into commits. The coverage follows those stages:

Audit ingress. gitopsreverser_audit_events_total{outcome,category,group,version,resource,verb} gives one terminal outcome per audit event (queued, stage, read_only_or_unknown_verb, failed_request, dry_run, unchanged_resource_version, non_scale_subresource, no_attribution_fact, write_error), and gitopsreverser_audit_eventlists_total / _eventlist_events_total / _eventlist_duration_seconds {outcome} cover the /audit-webhook request boundary.
Attribution join. gitopsreverser_attribution_resolutions_total{tier,actor_kind,group,version,resource} says which tier of evidence named the author and who it named, per type; _resolution_wait_seconds{tier,event_kind,…} says how long the grace wait cost, split by write and removal. Splitting the wait by tier is what turned a head-of-line stall from a mystery into a measurement, and event_kind is what the removal wait (the number the grace is tuned from) is read through. Match coverage is tier!="absent".
Fact pipeline. _attribution_facts_total{op} (written/matched, not subtractable), _attribution_fact_index_entries, _attribution_fact_index_evictions_total{reason}, _attribution_fact_stream_gaps_total{stream} (facts lost for good to a trim; should be zero), _attribution_fact_stream_decode_errors_total{transport} (an entry skipped because it could not be decoded: the loss path with no other symptom), _attribution_fact_follower_errors_total{transport} with _attribution_fact_follower_last_success_timestamp_seconds (a wedged follower degrades attribution cluster-wide), _attribution_transport_info{transport}, and _attribution_collection_without_uidset_total{reason}.
Git write and reconcile. gitopsreverser_commits_total{provider_*,branch,author_kind} is the bottom line: unresolved means attribution ran and could not name an actor. Alongside it, _branch_worker_queue_depth, _objects_written_total, _resync_sweep_deletes_total, gitopsreverser_target_reconcile_completed_total{gittarget_*} (read by the restart-reconcile guarantee), and resync/background-apply failure counters so a silently-recovered fault stays visible.
Discovery and encryption. The API resource catalog and Secret-encryption families.

Watch ingestion itself is not instrumented. Per-type event volume, restarts and 410 rebuilds, replay cost, recovery mode, and the delay between an event arriving on a shard and being processed are all designed in the metrics observability plan and not yet emitted. The attribution half of that plan (the label taxonomy and the silent loss paths) has shipped; the migration for the label break is in UPGRADING.md. See Operational Boundaries.

Operational boundaries

Current limitations:

Single active replica. The watch manager and worker manager declare NeedLeaderElection, so all object-state work runs on one elected pod; multi-pod HA is not finished. The target design is the HA / GitTarget distribution plan, which needs Redis for resume cursors, branch-shard leases, and durable write queues.
Resume cursors are best-effort. Each watch shard stores its last processed resourceVersion in Redis, so short reconnects resume a normal watch from that cursor. Kubernetes does not guarantee replay from an arbitrary resourceVersion, so if the apiserver has expired the cursor (410 Gone) recovery falls back to sendInitialEvents replay or LIST + mark-and-sweep.
Watch-ingestion metrics are not yet emitted. The attribution join is instrumented, but per-type watch volume, restarts, replay cost, recovery mode, and shard queue delay are not, so a stalled or thrashing watch is visible only in logs and in its downstream effects (see Observability).
The in-process attribution transport is single-replica. --author-attribution-transport=memory is refused with more than one replica, and it loses every unjoined fact on restart by design; a multi-replica install must use the Redis transport.
No pull request creation; the operator writes directly to branches.
Audit ingress uses a shared-CA trust boundary. Named /audit-webhook/<provider> routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one provider. This is an accepted privileged-control-plane assumption, not tenant isolation; see SECURITY.md.
deletecollection is reconciled by the watch (each item arrives as its own DELETED, or the mark-and-sweep reconciles them on replay).
A fail-safe placement skip has no dedicated status condition. A resource the writer refuses to place unsafely is logged per-resource and counted in the resync summary (placementSkipped), but is not (yet) surfaced as a distinct GitTarget status condition.

Package map

Package	Role
api/v1alpha3/	CRD types
cmd/	operator entry point and server setup
internal/auditutil/	audit identity, objectRef, and subresource helpers feeding attribution facts
internal/controller/	Kubernetes reconcilers
internal/git/	branch workers, Git ops, commit/signing/encryption, manifest writer
internal/git/manifestedit/	YAML document editor
internal/giteaclient/	Gitea helper client
internal/manifestanalyzer/	manifest inventory, acceptance, and resync planning
internal/manifestreport/	projection of Kubernetes objects into comparable manifest reports
internal/queue/	attribution fact streams (Redis or in-process), the in-process fact index and its follower, and per-watch resume cursors
internal/reconcile/	per-GitTarget event stream (watch event → branch worker)
internal/rulestore/	compiled rule cache
internal/sanitize/	Kubernetes object sanitization and stable YAML marshal
internal/ssh/	SSH authentication helpers
internal/sshsig/	SSH signature implementation
internal/telemetry/	metrics and OTLP setup
internal/types/	shared resource identity/reference and sensitivity policy
internal/typeset/	type followability registry, lookup model, and the relevance filter (controller-owned → don't mirror)
internal/watch/	discovery catalog, watch manager, per-`(GVR, scope)` raw watches with `sendInitialEvents` replay + mark-and-sweep, the author resolver, watched type tables, event router
internal/webhook/	`/audit-webhook` ingress and attribution fact extraction (no body joiner)

Design documents

Deeper dives live under docs/design/:

Ingestion, attribution, and reconcile:

Watch-first ingestion design record: historical context for the current watch-only object-state model and optional audit attribution.
Attribution: how a commit gets its author: the two halves, the tier ladder, and the wait.
Attribution facts as a stream, not a keyspace: the shipped transport seam, the in-process index, and what running without Redis costs.
Watch event ordering under the attribution grace window
Reconcile via watchlist mark and sweep
CommitRequest design

Types, discovery, status, and the Git write side:

Future direction:

HA / GitTarget distribution plan

Uh oh!

FilesExpand file tree

architecture.md

Latest commit

History