Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
---
title: "The Lineage Controller Webhook"
linkTitle: "Lineage Controller Webhook"
description: "What the lineage-controller-webhook does, how it's deployed, and the one knob worth knowing about."
weight: 40
---

The **lineage controller webhook** is a mutating admission webhook shipped as
part of the `cozystack.cozystack-engine` Package. On every CREATE and UPDATE
of a tenant `Pod`, `Secret`, `Service`, `PersistentVolumeClaim`,
`Ingress`, or `WorkloadMonitor` it walks up the ownership graph and stamps the
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The brace expansion notation {group,kind,name} is a shell-ism that might be ambiguous to some readers. It is clearer to list the literal label keys explicitly.

Suggested change
`Ingress`, or `WorkloadMonitor` it walks up the ownership graph and stamps the
(`apps.cozystack.io/application.group`, `apps.cozystack.io/application.kind`, and `apps.cozystack.io/application.name`).

owning Cozystack `Application`'s identity onto the resource as labels
(`apps.cozystack.io/application.{group,kind,name}`). The Cozystack dashboard,
the aggregated API server, and the SchedulingClass mechanism all rely on those
labels.

The webhook is registered with `failurePolicy: Fail`, so the kube-apiserver
must be able to reach a healthy webhook pod for tenant CREATE/UPDATE traffic
to succeed.

## Default deployment shape

The chart deploys a single `Deployment` modelled on the `cozystack-api` shape:

- **2 replicas** (override via `replicas`).
- **Soft `nodeAffinity`** preferring `node-role.kubernetes.io/control-plane`
(`Exists` matches both Talos's empty value and k3s/kubeadm's `"true"`). The
preference is *soft* — pods land on a control-plane node when one is
reachable, and on any worker otherwise. No override is needed for managed
Kubernetes (EKS / AKS / GKE), Cozy-in-Cozy tenant clusters, or any other
cluster where control-plane nodes aren't visible: the webhook simply
schedules elsewhere.
- **Permissive `tolerations`** (`{operator: Exists}`) so a control-plane node
with `NoSchedule` taints accepts the pod when the soft affinity is
satisfiable.
- **Soft `podAntiAffinity`** on `kubernetes.io/hostname` so replicas
best-effort spread across nodes.
- **`PodDisruptionBudget`** with `maxUnavailable: 1`. At `replicas: 2+` it
caps disruption to one pod; at `replicas: 1` it's a useful no-op.
- **Service `spec.trafficDistribution: PreferClose`** so the apiserver
prefers a webhook endpoint on its own node when one exists, and
transparently falls over to a remote endpoint otherwise. Requires
Kubernetes ≥ 1.31; older clusters silently fall back to default
cluster-wide distribution (still safe, just no locality preference).

This shape works as-is on every Kubernetes distribution Cozystack supports.
You shouldn't need to override anything in normal operation.

## Increasing replicas

If you want more than two replicas — for instance, to keep one webhook pod
co-located with each apiserver on a five-node control plane — override the
`replicas` value via the `cozystack.cozystack-engine` Package, the same way
you'd override any other component (see
[Components]({{% ref "/docs/next/operations/configuration/components" %}})):

```yaml
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
name: cozystack.cozystack-engine
namespace: cozy-system
spec:
variant: default
components:
lineage-controller-webhook:
values:
lineageControllerWebhook:
replicas: 5
```

## `localK8sAPIEndpoint.enabled` is deprecated

The chart still exposes `localK8sAPIEndpoint.enabled`, which when set to
`true` injects `KUBERNETES_SERVICE_HOST=status.hostIP` and
`KUBERNETES_SERVICE_PORT=6443` so the webhook talks to the apiserver on its
own node. It was originally added to avoid latency on the
webhook-to-apiserver path. It's now defaulted to `false` and slated for
removal once the latency motivation is addressed in the webhook itself.

{{% alert title="Important" color="warning" %}}
Do not enable `localK8sAPIEndpoint.enabled` with the default chart values.
The injected `status.hostIP` is only valid when the pod runs on a node that
hosts a kube-apiserver, and the chart's soft control-plane affinity does not
guarantee that. With the flag enabled and the pod scheduled off a
control-plane node, the controller crash-loops dialing a non-apiserver IP —
and combined with `failurePolicy: Fail` that means a tenant CREATE/UPDATE
outage.
{{% /alert %}}

## Verifying the deployment

```bash
kubectl -n cozy-system get deploy lineage-controller-webhook
kubectl -n cozy-system get pods -l app=lineage-controller-webhook
kubectl -n cozy-system get svc lineage-controller-webhook -o yaml | grep trafficDistribution
```

A quick end-to-end check, exercising the webhook through the apiserver:

```bash
kubectl create ns lineage-webhook-test
kubectl -n lineage-webhook-test create service clusterip probe \
--clusterip=None --dry-run=server
Comment on lines +103 to +104
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

For kubectl (current stable docs), does kubectl create service clusterip NAMErequire--tcp/port arguments? Please cite the command reference and examples.

💡 Result:

Yes, kubectl create service clusterip NAME requires the --tcp or port arguments. The official Kubernetes documentation shows the syntax as kubectl create service clusterip NAME [--tcp=<port>:<targetPort>] [--dry-run=server|client|none], where --tcp is listed as a required flag for specifying port mappings. All provided examples include it, such as kubectl create service clusterip my-cs --tcp=5678:8080 or for headless kubectl create service clusterip my-cs --clusterip="None", but the latter still typically pairs with ports for functionality. Without ports, the service would have no ports defined, rendering it ineffective.

Citations:


Add --tcp=80:80 to kubectl service creation command for functionality.

The kubectl create service clusterip command requires port specification via the --tcp flag. Without it, the service is created with no ports defined, making the verification step unreliable. The official Kubernetes documentation lists --tcp as necessary for port mappings.

Suggested edit
-kubectl -n lineage-webhook-test create service clusterip probe \
-  --clusterip=None --dry-run=server
+kubectl -n lineage-webhook-test create service clusterip probe \
+  --tcp=80:80 --clusterip=None --dry-run=server
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
kubectl -n lineage-webhook-test create service clusterip probe \
--clusterip=None --dry-run=server
kubectl -n lineage-webhook-test create service clusterip probe \
--tcp=80:80 --clusterip=None --dry-run=server
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@content/en/docs/next/operations/configuration/lineage-controller-webhook.md`
around lines 197 - 198, The kubectl service creation command shown ("kubectl -n
lineage-webhook-test create service clusterip probe --clusterip=None
--dry-run=server") is missing a port mapping; add the --tcp=80:80 flag so the
service is created with a port (e.g., update that command to include
--tcp=80:80) to ensure the verification step is reliable and matches Kubernetes
requirements.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this step is to create a service, whose creation is gated by the lineage webhook. If the creation fails due to a webhook error, the user is immediately alerted. If it works fine, the most basic function of the webhook is proven to work. The service does not need to point to anything or route real traffic anywhere.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lllamnyp, that's a good clarification — the --dry-run=server flag is precisely the key detail I missed. Since the request reaches the apiserver (and therefore the webhook) but is never persisted, a portless headless service is entirely valid as a probe. The command is correct as written. Apologies for the noise.


🧠 Learnings used
Learnt from: lllamnyp
Repo: cozystack/website PR: 416
File: content/en/docs/operations/oidc/self-signed-certificates.md:102-110
Timestamp: 2026-02-12T15:52:39.278Z
Learning: When documenting kubelogin usage with kubectl config set-credentials, omit the --exec-interactive-mode option entirely. Do not include this flag in examples or explanations. If encountered in a doc, remove it and note that interactive mode is not supported. Apply this guideline to all Markdown docs under content to ensure consistent, accurate guidance.

kubectl delete ns lineage-webhook-test
```

The dry-run CREATE goes through the mutating admission webhook; if the
webhook isn't reachable, it fails with `failed calling webhook
"lineage.cozystack.io"`.