Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .docs-lint-scope
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,4 @@
README.md
docs/architecture.md
docs/configuration.md
docs/design/attribution-branch-findings.md
docs/design/attribution-publish-and-join.md
docs/design/attribution-metrics-proposal.md
docs/spec/attribution.md
17 changes: 11 additions & 6 deletions api/v1alpha3/gittarget_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,10 +155,14 @@ type GitTargetSpec struct {
// path identity-complete, never append or co-mingle), not a second placement
// namespace the user has to configure. A user routes Secrets the same way they
// route anything else — by naming their type in ByType. When a resource's type
// has no ByType entry and no Default, placement falls back to following the layout
// already established by sibling resources in the repository, and finally to the
// canonical, versionless {namespaceOrCluster}/{group}/{resource}/{name}.yaml path
// when there is nothing to follow. Because that fallback omits the API version,
// has no ByType entry and no Default, the new document goes beside the folder's
// kustomization when the whole folder is governed by exactly one supported
// kustomization (so the file is reachable from a render root instead of being
// written where kustomize would never build it), and otherwise at the built-in
// canonical, versionless {namespaceOrCluster}/{group}/{resource}/{name}.yaml path.
// Nothing infers a destination from where the repository keeps other resources of
// the same type: a layout this operator cannot derive from one root is declared
// here or it is canonical. Because the canonical path omits the API version,
// objects that differ only by version share a file; a target that watches several
// versions of the same group/resource and wants them separated must use a
// ByType/Default template that includes {version}.
Expand All @@ -173,8 +177,9 @@ type GitTargetPlacementSpec struct {
ByType map[string]string `json:"byType,omitempty"`

// Default is the path template used for a new resource whose type has no ByType
// entry. Omitted, it falls through to sibling-layout inference and then the
// built-in canonical path. A bundling default (one that is not identity-complete,
// entry. Omitted, it falls through to the folder's one supported kustomization
// root, if it has exactly one, and then to the built-in canonical path.
// A bundling default (one that is not identity-complete,
// such as "all.yaml") is only valid when a sensitive resource can never reach it
// — give every sensitive type an explicit identity-complete ByType entry.
// +optional
Expand Down
5 changes: 3 additions & 2 deletions config/crd/bases/configbutler.ai_gittargets.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -308,8 +308,9 @@ spec:
default:
description: |-
Default is the path template used for a new resource whose type has no ByType
entry. Omitted, it falls through to sibling-layout inference and then the
built-in canonical path. A bundling default (one that is not identity-complete,
entry. Omitted, it falls through to the folder's one supported kustomization
root, if it has exactly one, and then to the built-in canonical path.
A bundling default (one that is not identity-complete,
such as "all.yaml") is only valid when a sensitive resource can never reach it
— give every sensitive type an explicit identity-complete ByType entry.
type: string
Expand Down
18 changes: 8 additions & 10 deletions docs/INDEX.md

Large diffs are not rendered by default.

11 changes: 8 additions & 3 deletions docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,14 @@ This file is meant to track the smaller current backlog, not historical notes.
Secrets and CozyStack `tenantsecrets`; resources with sensitive fields under shapes such as
`spec.credentials` need an explicit field policy or full-file encryption decision.

- [ ] Revisit output layout.
Think about better control over target folders and whether some use cases should support multiple
resources per file.
- [ ] Revisit output layout. **Now designed and postponed, not open-ended.** The answer is to declare
what the folder *is* rather than a path template: `GitTarget.spec.layout`, designed in
[gittarget-layout-model.md](design/gittarget-layout-model.md) and filed as
[#293](https://github.com/ConfigButler/gitops-reverser/issues/293), sequenced with the rest of the
breaking `GitTarget` work as [#294](https://github.com/ConfigButler/gitops-reverser/issues/294).
Deliberately **not** in 0.41.0, which already carries the new attribution model and the
sibling-inference removal. Multiple resources per file is bundle support, which exists for
match-first today and is a separate question from where a *new* file goes.

- [ ] Reduce duplication between `WatchRule` and `ClusterWatchRule` code paths where it makes sense.

Expand Down
75 changes: 75 additions & 0 deletions docs/UPGRADING.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,81 @@ guidance that the changelog's breaking-change entries link to.
We are pre-1.0, so breaking changes bump the **minor** version (release-please is configured with
`bump-minor-pre-major`) rather than the major. Read the relevant entry before upgrading across it.

## New resources land where you declare, not where the folder's other documents live

Sibling inference is gone. A resource with no document in Git yet is placed by the first of three
things that applies, and nothing else:

1. the GitTarget's `spec.placement.byType` entry for its type, or `spec.placement.default`;
2. the folder's one supported `kustomization.yaml`, when the whole folder has exactly one — the file
lands beside it and joins its `resources:` list;
3. the built-in canonical path, `{namespaceOrCluster}/{groupPath}/{resource}/{name}{sensitiveSuffix}`:
a cluster-scoped resource uses the literal `_cluster/` in place of the namespace, a core resource
omits the group segment, there is no version segment, and a sensitive resource gets `.sops.yaml`
instead of `.yaml`.

Before this, a folder with no declared placement was read for its layout: a new ConfigMap was appended
to the bundle the other ConfigMaps shared, or written beside them one-per-file. That is what changes.

**Who is affected.** A target whose repository this operator created is **unaffected** — such a folder
already used canonical paths, which inference also produced. A target pointed at a **hand-authored**
folder with a layout of its own is affected: a resource of a type that folder already holds, in a
namespace or with a name it has never held, now gets the canonical path instead of joining the
existing file or directory. Nothing already in Git moves — an existing document is still edited in
place at its current location, forever.

**What to do about it.** Declare the layout the folder means:

```yaml
spec:
placement:
byType:
v1/configmaps: "all.yaml" # keep bundling ConfigMaps into one file
v1/secrets: "team-a/secrets/{name}.sops.yaml" # one encrypted file per Secret
```

A declared template does everything inference did and says so on the page. A **kustomize** folder needs
no declaration: step 2 already places new files where that folder builds them.

**One shape worth expecting.** In a folder that kustomize builds, a bundle file is no longer extended
by default. A new resource gets a file of its own beside the `kustomization.yaml` and an entry added to
its `resources:` list, so the build file changes where it previously did not (the bundle was already
listed, so extending it needed no entry). Both outcomes mirror the resource and both render; the new
one keeps a resource the operator placed out of a file a human curated, and it is undone by declaring
the bundle in `byType`.

**How to tell whether it affects you**, before or after upgrading — every placement is counted, by
GitTarget and by type:

```promql
sum by (gittarget_namespace, gittarget_name, group, version, resource) (
increase(gitopsreverser_placements_total{source="canonical"}[24h]))
```

Each series is a type that took the built-in path. For a canonical-layout folder that is simply the
layout. For a folder with a convention of its own it is the `byType` line to add. `source="declared"`
and `source="kustomize_root"` need no attention. Two companions ship with it:
`gitopsreverser_placement_refusals_total{reason}` (resources the writer declined to place — each one
is absent from the mirror) and `gitopsreverser_placement_kustomization_entries_total{outcome}`, whose
`failed` value is a new file committed outside every render. See
[interpreting-metrics.md](interpreting-metrics.md).

**Why the feature was removed rather than made switchable.** It let an edit to the *repository* change
where the operator writes, with no Kubernetes object changing and nothing in status recording the move
— delete enough of one namespace's documents from a shared bundle and the next new one takes a
different path. Its namespace-safety guard had also failed once by cascading: one wrong append made a
per-namespace file look namespace-agnostic, which legitimized it for every later resource, which
collapsed a whole type into one file. There is deliberately **no** `spec.placement.mode` flag to turn
inference back on: an off-switch for a removed feature is a permanent API field bought to solve a
temporary problem. The full argument is in
[`open-asks-priority.md`](design/open-asks-priority.md).

**One related fix.** A new document in a directory whose kustomization sets `namespace:` omits
`metadata.namespace` only when the transformer names the resource's **own** namespace. When it names a
different one the namespace is now written explicitly — omitting it would have handed the namespace to
kustomize and rendered a different object than the one being mirrored. This also now applies to a path
you declared, which previously wrote a `namespace:` line the rest of that folder omits.

## 0.41.0 — attribution facts travel on a selectable transport, and Redis is no longer implied

Attribution stopped meaning Redis. The audit receiver appends its facts to a per-type **stream**,
Expand Down
42 changes: 24 additions & 18 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -538,8 +538,8 @@ The **built-in default** path is `{spec.path}/{namespace}/{group}/{resource}/{na
first, the API group omitted for core resources, no version segment, and a `.sops.yaml` suffix for
sensitive resources; a cluster-scoped resource uses the literal `_cluster/` in place of the namespace
(an illegal Kubernetes namespace name, so it can never clash with a real one).
But that default is only the cold-start seed: a new resource first follows its **siblings'** existing
layout, and a `GitTarget` can declare its own placement policy. Details and the placement policy are in
That default is what a new resource gets unless something more specific applies: a `GitTarget`'s own
declared placement policy, or a folder that kustomize builds from a single root. Details are in
[File Placement](#file-placement).

***
Expand Down Expand Up @@ -614,8 +614,8 @@ per-mutation change log.
- **Fact transport (per-type stream)**: [internal/queue/fact_stream.go](../internal/queue/fact_stream.go)
- **In-process fact index**: [internal/queue/fact_index.go](../internal/queue/fact_index.go)
- **Resolver (grace window join)**: [internal/watch/author_resolver.go](../internal/watch/author_resolver.go)
- **Design**: [attribution-publish-and-join.md](design/attribution-publish-and-join.md),
[attribution-fact-stream.md](finished/attribution-fact-stream.md)
- **Spec**: [spec/attribution.md](spec/attribution.md) (the single reference)
- **Design record**: [attribution-fact-stream.md](finished/attribution-fact-stream.md)

Attribution runs only when `--author-attribution=true`. A normal
source posts audit `EventList` payloads to `/audit-webhook/<audit-route>`, where the route is
Expand Down Expand Up @@ -734,8 +734,8 @@ TTL does not bound: a uid is unique across space and time, so the statement cann
and its horizon is the index's caps instead. It is still in-memory: a restart re-warms the index
from one TTL of stream retention like everything else. Without the slot, a finalizer patch's fact
overwrites the deleter's, because both carry the resourceVersion the *deletion* stamped
([attribution-deletion-intent-actor.md](design/attribution-deletion-intent-actor.md)). **A removal
never returns on a write fact without looking further:** the per-object tiers are last-writer-wins,
([spec/attribution.md](spec/attribution.md)). **A removal never returns on a write fact
without looking further:** the per-object tiers are last-writer-wins,
so for a removal they hold whoever last *edited* the object, which is not who deleted it; such a
match is held as a fallback while the wait continues for evidence about the deletion itself. And
**an exact-capable event may not fall through to the removal tiers:** a create or update presents
Expand Down Expand Up @@ -1111,7 +1111,7 @@ hydrates only touched files into buffers for the commit, and flushes only change
- **Upserts:** if a managed document for the resource already exists, patch it in place (preserving
siblings in a multi document file); if it is sensitive, encrypt the whole document again at its existing
path; if no document exists, place a new file per [File Placement](#file-placement) (declared policy,
then sibling inference, then the canonical default).
then the folder's one kustomize root, then the canonical default).
- **Kustomize override edit-through:** a live value produced by a well-formed `images:` or `replicas:`
entry in the document's kustomization chain is written back to that entry (comment-preserving, only
fields the entry already declares); the source manifest keeps its bytes. Anything the inversion cannot
Expand All @@ -1135,21 +1135,27 @@ placed never moves a file already in Git. A new resource is placed by the first
1. **Declared policy (`spec.placement`).** A `GitTarget` can declare a `byType` map (exact
`[group/]version/resource` → path template) plus a `default` template, rendered from a small
brace-variable path language (`{namespace}`, `{group}`, `{resource}`, `{name}`, …).
2. **Sibling inference.** With no matching declared template, the new resource follows the layout its
siblings already use: appended to the bundle its type shares, or placed one-per-file beside them.
so pointing a target at an existing folder continues that folder's convention. When the whole
subtree is governed by one supported kustomization and the type is brand new, the file lands beside
that kustomization and gets a `resources:` entry.
3. **Canonical fallback.** With nothing to follow (an empty repo, a brand-new type), the built-in default
2. **The folder's one kustomize root.** When the whole writable subtree is governed by exactly one
supported `kustomization.yaml`, the file lands beside it and gets a `resources:` entry in the same
commit. This step is a structural fact rather than a reading of the folder's conventions: the
canonical path below is a tree a `resources:` graph cannot reach, so a file written there would never
be rendered. Two supported kustomizations is ambiguous and declines.
3. **Canonical fallback.** Otherwise the built-in default
`{spec.path}/{namespace}/{group}/{resource}/{name}.yaml`: namespace-first, group omitted for core, no
version, `_cluster/` for cluster-scoped, `.sops.yaml` for sensitive, so a fresh target is deterministic
and self-propagating.
version, `_cluster/` for cluster-scoped, `.sops.yaml` for sensitive.

**The layout of the folder's other documents is not an input.** An earlier release followed it (sibling
inference), which made a human's edit to the repository change where the operator wrote next, with no
Kubernetes object changing and nothing in status recording the move. It was removed; a layout the ladder
cannot derive is declared in `spec.placement`, and
`gitopsreverser_placements_total{source="canonical"}` names the target and type that needs the line.

Sensitivity is a write-safety classifier, not a placement input: whatever path is chosen, a sensitive
resource is written encrypted, is never appended to an existing file, and is never co-mingled with a
plaintext document. When those guarantees cannot be honoured (e.g. a bundling `default` would route a
sensitive resource into a shared file), the resource is **skipped fail-safe** (logged per-resource and
counted in the resync summary as `placementSkipped`) rather than written unsafely.
sensitive resource into a shared file), the resource is **refused fail-safe** rather than written
unsafely: logged per-resource, counted in the resync summary as `placementSkipped`, and counted by
`gitopsreverser_placement_refusals_total{reason}`.

### Bootstrap, encryption, and signing

Expand Down Expand Up @@ -1382,7 +1388,7 @@ Deeper dives live under [docs/design/](design/):

- [Watch-first ingestion design record](finished/watch-first-ingestion-architecture.md): historical context
for the current watch-only object-state model and optional audit attribution.
- [How attribution works: the publish side and the join side](design/attribution-publish-and-join.md):
- [Attribution: how a commit gets its author](spec/attribution.md):
the two halves, the tier ladder, and the wait.
- [Attribution facts as a stream, not a keyspace](finished/attribution-fact-stream.md): the shipped
transport seam, the in-process index, and what running without Redis costs.
Expand Down
Loading