Replace fluentd with fluent-bit in the operator

[hjiawei]

Description

New feature (Calico Enterprise): the LogCollector controller now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, completing the log-collector migration on the operator side.

• Resource identity migration: namespace tigera-fluentd → calico-system; DaemonSet/ServiceAccount fluentd-node → calico-fluent-bit; TLS secret → calico-fluent-bit-tls; metrics port 9081 → 2020 (fluent-bit's built-in HTTP server).
• Configuration is rendered in fluent-bit's YAML schema into per-OS ConfigMaps (calico-fluent-bit-conf/-windows), subPath-mounted on Linux and directory-mounted on Windows, started with -c. The render loads the Go plugins via plugins_file, defines parsers inline, applies the record-transform Lua filter, and inlines user-provided fluent-bit YAML filter lists. A rendered-config hash annotation rolls the pods on config-only changes.
• Tail inputs use the producing components' real log paths, with SQLite offset DBs and filesystem buffering under /var/log/calico/calico-fluent-bit; a pos-migrator init container (Linux and Windows) seeds offsets from the legacy fluentd .pos files and pre-creates the tailed directories. Windows tails the same log types the fluentd Windows variant shipped.
• The linseed output matches only Linseed-bound tags, authenticates with mTLS + the pod's ServiceAccount token, and retries without limit against bounded filesystem storage. S3/Splunk/Syslog outputs mirror fluentd's per-type fan-out (standard AWS credential env vars, endpoint scheme honored, syslog ships the whole record as JSON via a per-output Lua processor with TLS properly enabled).
• NonClusterHost renders the :9880 HTTP input with client-certificate verification; the input Service is cleaned up when the resource is removed.
• eks-log-forwarder runs the fluent-bit image with a rendered in_eks → linseed pipeline and health probes (no startup init container; the input plugin resolves its resume point from Linseed).
• Probes hit :2020/api/v1/health; the ServiceMonitor scrapes plain HTTP (fluent-bit's monitoring server has no TLS) with access restricted by the component NetworkPolicy, and legacy fluentd monitors are deleted.
• The LogCollector controller no longer creates/owns the calico-system namespace (deleting the LogCollector must not garbage-collect it); the deprecated fluentdDaemonSet override is honored as an alias of the new calicoFluentBitDaemonSet field (with container-name translation); legacy tigera-fluentd resources — the namespace last — are cleaned up idempotently.

Testing: render, controller, and monitor unit suites updated/extended (ConfigMap-content assertions replacing the env-var assertions); the rendered configuration was validated against the real fluent-bit binary; the full migration was validated end-to-end on a test cluster — all log types flowing to Linseed/Elasticsearch, fluentd resources fully removed, tail-offset handover without re-shipping, NonClusterHost ingestion with client-certificate enforcement, and EKS/Windows render shapes verified.

Release Note

The LogCollector now deploys fluent-bit (calico-fluent-bit in calico-system) in place of fluentd, with operator-rendered configuration and automatic migration of fluentd tail positions.

Breaking changes and behavior changes in this migration:
- The tigera-fluentd namespace is removed; all log-collector resources are renamed and move to calico-system (DaemonSet/container calico-fluent-bit, TLS secret calico-fluent-bit-tls). Tooling referencing the old names (e.g. kubectl logs -c fluentd) must be updated.
- The LogCollector fluentdDaemonSet override field is deprecated in favor of calicoFluentBitDaemonSet; existing overrides (including legacy container names) keep applying for one release.
- User flow/DNS log filters: the fluentd-filters ConfigMap is no longer read. Filters must be recreated under the new fluent-bit-filters name as fluent-bit YAML filter lists; unparseable content raises a TigeraStatus warning while log shipping continues.
- Log-collector metrics are now fluent-bit's native Prometheus metrics (fluentbit_* metric names, plain HTTP on port 2020 at /api/v2/metrics/prometheus, guarded by NetworkPolicy). Dashboards and alerts keyed on fluentd_* metric names or the 9081 mTLS endpoint must be updated.
- S3 archive object keys change from fluentd's flat layout (<bucketPath>/flows20260101_<n>.gz) to directory-style keys (<bucketPath>/flows/20260101_<uuid>.gz), with non-cluster-host flows archived under their own non_cluster_flows/ directory. Downstream tooling anchored to the old flat patterns needs a one-time update.
- Log buffering moves from fluentd's in-memory buffers to fluent-bit filesystem storage under /var/log/calico/calico-fluent-bit/ on each node: buffered-but-unsent chunks now survive pod restarts, and host disk usage grows accordingly (capped per output by storage.total_limit_size).
- Syslog forwarding no longer applies fluentd's 1024-byte packet-size default. When spec.additionalStores.syslog.packetSize is unset, the fluent-bit syslog output uses its own default for the RFC5424 format the operator renders (2048 bytes), reducing truncation of longer log lines; set packetSize to cap the message size explicitly (messages above the cap are truncated).

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.

[Copilot]

Pull request overview

This PR completes the operator-side migration of the LogCollector from fluentd to fluent-bit (Calico Enterprise), including updated resource identities (namespace/workload names), rendered configuration shape, and updated policy/monitoring expectations across the rendering and controllers.

Changes:

• Replace fluentd-specific rendering/controller wiring with fluent-bit equivalents (names, ports, RBAC, ServiceMonitor behavior).
• Update expected Calico policies/selectors to reflect fluent-bit running in calico-system and using port 2020.
• Extend the LogCollector CRD/API to add calicoFluentBitDaemonSet and treat the deprecated fluentdDaemonSet as an alias during migration.

Reviewed changes

Copilot reviewed 47 out of 48 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/render/tiers/tiers_test.go	Updates tier rendering tests to stop expecting a separate LogCollector namespace.
pkg/render/testutils/expected_policies/node_local_dns_ipv6.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_ipv4.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/node_local_dns_dual.json	Removes legacy tigera-fluentd namespace from node-local-dns selector expectations.
pkg/render/testutils/expected_policies/linseed.json	Updates Linseed ingress policy sources from fluentd labels/namespace to fluent-bit in calico-system.
pkg/render/testutils/expected_policies/linseed_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/linseed_ocp_dpi_enabled.json	Same as above for OpenShift + DPI-enabled scenario.
pkg/render/testutils/expected_policies/linseed_dpi_enabled.json	Same as above for DPI-enabled scenario.
pkg/render/testutils/expected_policies/guardian.json	Updates Guardian policy expectations to allow fluent-bit instead of fluentd.
pkg/render/testutils/expected_policies/guardian_ocp.json	Same as above for OpenShift policy expectations.
pkg/render/testutils/expected_policies/fluentbit_unmanaged.json	Updates fluent-bit “allow metrics” policy expectations (name/namespace/selector/port).
pkg/render/testutils/expected_policies/fluentbit_unmanaged_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/fluentbit_managed.json	Updates managed-cluster policy expectations for fluent-bit metrics access.
pkg/render/testutils/expected_policies/es-gateway.json	Updates ES gateway ingress policy sources from fluentd to fluent-bit.
pkg/render/testutils/expected_policies/es-gateway_ocp.json	Same as above for OpenShift.
pkg/render/testutils/expected_policies/dns.json	Removes legacy tigera-fluentd namespace from DNS policy selector expectations.
pkg/render/testutils/expected_policies/dns_ocp.json	Same as above for OpenShift DNS policy expectations.
pkg/render/nonclusterhost/nonclusterhost.go	Adds Linseed RBAC permissions for DNS logs in non-cluster-host flow.
pkg/render/nonclusterhost/nonclusterhost_test.go	Updates RBAC test expectations to include dnslogs.
pkg/render/monitor/monitor.go	Renames fluentd monitoring constants to fluent-bit and switches ServiceMonitor to plain HTTP fluent-bit metrics endpoint.
pkg/render/monitor/monitor_test.go	Updates monitor rendering tests for renamed ServiceMonitor and the removal of TLS/scheme relabeling.
pkg/render/manager.go	Updates manager-rendered NetworkPolicy destination service name and cluster-wide namespace list (includes a problematic entry noted in comments).
pkg/render/manager_test.go	Updates manager tests to expect fluent-bit service names.
pkg/render/logstorage/linseed/linseed.go	Updates Linseed policy source entity rule reference to fluent-bit.
pkg/render/logstorage/esgateway/esgateway.go	Updates ES gateway policy source entity rule reference to fluent-bit.
pkg/render/intrusion_detection.go	Updates comments to reflect fluent-bit consuming IDS logs.
pkg/render/guardian.go	Updates Guardian policy source entity rule reference to fluent-bit.
pkg/render/fluentd.go	Removes the fluentd renderer implementation.
pkg/render/fluentd_test.go	Removes fluentd rendering unit tests.
pkg/render/common/elasticsearch/service.go	Renames isFluentd flag to isFluentBit and updates related commentary for managed-cluster Linseed routing.
pkg/render/applicationlayer/applicationlayer.go	Updates comment about WAF logs being consumed by fluent-bit.
pkg/imports/crds/operator/operator.tigera.io_logcollectors.yaml	Adds calicoFluentBitDaemonSet and updates fluentdDaemonSet docs/enums for the migration (some descriptions still say “Fluentd”, noted in comments).
pkg/controller/tiers/tiers_controller.go	Stops treating LogCollector as its own namespace when computing tier namespace lists.
pkg/controller/monitor/prometheus.go	Watches fluent-bit ServiceMonitor name instead of fluentd.
pkg/controller/monitor/monitor_controller.go	Switches monitored TLS secret from fluentd to fluent-bit.
pkg/controller/monitor/monitor_controller_test.go	Updates controller tests to reference fluent-bit ServiceMonitor and secrets.
pkg/controller/logstorage/secrets/secret_controller.go	Updates upstream cert collection references from fluentd TLS secret to fluent-bit TLS secret.
pkg/controller/logstorage/secrets/secret_controller_test.go	Updates tests to create/check fluent-bit TLS secret scenarios.
pkg/controller/logcollector/logcollector_controller.go	Rewires controller from fluentd to fluent-bit rendering (secrets, filters, services, namespace ownership behavior).
pkg/controller/logcollector/logcollector_controller_test.go	Updates controller tests for fluent-bit DaemonSet naming, config-driven outputs, and non-creation of calico-system.
pkg/components/enterprise.go	Renames enterprise component definitions from fluentd to fluent-bit (linux/windows).
hack/gen-versions/enterprise.go.tpl	Updates version generation template keys/structs for fluent-bit components.
config/enterprise_versions.yml	Replaces fluentd component entries with fluent-bit equivalents.
api/v1/zz_generated.deepcopy.go	Adds deepcopy support for CalicoFluentBitDaemonSet on LogCollectorSpec.
api/v1/logcollector_types.go	Adds calicoFluentBitDaemonSet field and documents fluentdDaemonSet as deprecated alias.
api/v1/fluentd_daemonset_types.go	Updates override container/initContainer enums to fluent-bit naming (backward-compat concerns noted in comments).

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 49 out of 50 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• api/v1/zz_generated.deepcopy.go: Generated file

Comments suppressed due to low confidence (2)

pkg/controller/monitor/monitor_controller.go:156

• Monitor no longer scrapes fluent-bit over mTLS (serviceMonitorFluentBit is plain HTTP), so the controller shouldn't watch or require the fluent-bit TLS secret. Keeping it here creates an unnecessary dependency on the LogCollector controller and can degrade Monitor reconciliation when LogCollector isn't installed (secret not found).

	for _, secret := range []string{
		certificatemanagement.CASecretName,
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		monitor.PrometheusServerTLSSecretName,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
		render.EKSLogForwarderTLSSecretName,
	} {

pkg/controller/monitor/monitor_controller.go:345

• Since fluent-bit metrics scraping is now plain HTTP, Monitor's trusted bundle no longer needs the fluent-bit certificate. Including it makes reconciliation fail with NotFound if LogCollector hasn't created the secret yet (or isn't installed).

	for _, certificateName := range []string{
		esmetrics.ElasticsearchMetricsServerTLSSecret,
		render.FluentBitTLSSecretName,
		render.NodePrometheusTLSServerSecret,
		render.CalicoAPIServerTLSSecretName,
		kubecontrollers.KubeControllerPrometheusTLSSecret,
	} {

[pasanw]

Might be worth touching base with @caseydavenport on the steps taken in this PR to remove direct ES connections - just to make sure you are aligned with what he had in mind

[hjiawei]

Will do — the ES retirement steps here follow the EV-6164 design (ES output dropped with fluentd, Linseed as sole structured-log sink). Casey, the specific surface in this PR: fluentd's ES env plumbing and the ElasticsearchEksLogForwarderUserSecret/cluster-config watches are removed from the LogCollector controller; nothing else touches ES.

[Brian-McM]

I know this wasn't there before, but you might want to add a daemonset watch to trigger re reconciliation if the daemonset was removed. This is missing in a lot of places currently.

[hjiawei]

Done. Added watches for the workloads this controller renders — the calico-fluent-bit and calico-fluent-bit-windows DaemonSets and the eks-log-forwarder Deployment (all in calico-system) — so deleting or editing any of them out-of-band now enqueues a reconcile that restores it. Reconcile already loads the singleton LogCollector and ignores the request key, so a workload-triggered event drives a full re-render. Wrapped it in a small utils.AddDaemonsetWatch helper mirroring the existing AddDeploymentWatch, since this gap exists in other controllers too.

[pasanw]

Are these still required? Right now (IIRC) they are permitting the fluent-bit SA to exec into pods into calico-system pods

[hjiawei]

One correction: the RoleBinding subject is the packetcapture-api ServiceAccount (render.PacketCaptureServiceAccountName), not the fluent-bit SA — the role grants packetcapture-api exec-into/list on pods in calico-system. It's still required: the PacketCapture API retrieves capture files by exec-ing into the per-node fluent-bit DaemonSet pod, which mounts the node's /var/log/calico hostPath where captures are written. Carried over unchanged from the fluentd era (only the namespace moved to calico-system), and rendered only when the PacketCapture API is installed. I clarified the misleading comment in rbac.go. Removing it would be a separate change needing validation against packetcapture-api's retrieval path, so I kept parity here.

[pasanw]

This RBAC role seemingly only existed in this controller because it was part of the fluentd namespace. But in the new scheme, PacketCapture API will exec into calico-node in the calico-system namespace.

As discussed, we should probably move this to a better home. It can be a follow-up PR

[hjiawei]

Done in this PR rather than deferring. Moved the pod/exec Role + RoleBinding to the packet-capture component (pkg/render/packet_capture_api.go), which owns the PacketCapture API ServiceAccount and renders only when the PacketCaptureAPI CR exists — the same condition that previously gated it. They keep their names (packetcapture-api-role / -binding) and namespace (calico-system) so the existing objects are adopted in place on upgrade, and I dropped the now-unused PacketCapture field, controller fetch, constants and rbac.go helpers from the log collector.

For context on why it lived here before: in the fluentd era the grant had to be in the fluentd-owned tigera-fluentd namespace and targeted the fluentd DaemonSet pods (the per-node pod mounting /var/log/calico, where captures are written). Since the API now execs into calico-node instead, the coupling to the log collector is gone. One note: tigera-fluentd was a dedicated single-workload namespace, so a namespaced exec grant there was effectively fluentd-only; in calico-system the same namespaced Role covers all pods in that namespace. Tightening that further (exec can't be scoped by resourceName) would be a separate change.

[pasanw]

Approved with some comments to consider before merging

[pasanw]

Regarding release notes, I'd suggest the following:

- The tigera-fluentd namespace is removed; all log-collector resources are renamed and move to calico-system (DaemonSet/container calico-fluent-bit, TLS secret calico-fluent-bit-tls). Tooling referencing the old names (e.g. kubectl logs -c fluentd) must be updated. 
- The LogCollector fluentdDaemonSet override field is deprecated in favor of calicoFluentBitDaemonSet; existing overrides (including legacy container names) keep applying for one release.
- User flow/DNS log filters: the fluentd-filters ConfigMap is no longer read. Filters must be recreated under the new fluent-bit-filters name as fluent-bit YAML filter lists; unparseable content raises a TigeraStatus warning while log shipping continues.
- Log-collector metrics are now fluent-bit's native Prometheus metrics (fluentbit_* metric names, plain HTTP on port 2020 at /api/v2/metrics/prometheus, guarded by NetworkPolicy). Dashboards and alerts keyed on fluentd_* metric names or the 9081 mTLS endpoint must be updated.
- S3 archive object keys change from fluentd's flat layout (<bucketPath>/flows20260101_<n>.gz) to directory-style keys (<bucketPath>/flows/20260101_<uuid>.gz), with non-cluster-host flows archived under their own non_cluster_flows/ directory. Downstream tooling anchored to the old flat patterns needs a one-time update.
- Log buffering moves from fluentd's in-memory buffers to fluent-bit filesystem storage under /var/log/calico/calico-fluent-bit/ on each node: buffered-but-unsent chunks now survive pod restarts, and host disk usage grows accordingly (capped per output by storage.total_limit_size).
- Non-cluster hosts can now forward DNS logs in addition to flow and policy-activity logs.

• Moved "The LogCollector fluentdDaemonSet override field is deprecated..." onto its own line
• Removed the line "The direct Elasticsearch log output is removed...", these are internal changes

[hjiawei]

Adopted your wording, with two changes:

• Dropped the "Non-cluster hosts can now forward DNS logs…" bullet — NCH DNS forwarding will be delivered and tested under its own epic, so it shouldn't ship as a release note here.
• Added a syslog packet-size bullet (see the thread on the syslog output): the 1024-byte fluentd default is gone; when packetSize is unset the fluent-bit syslog output uses its RFC5424 default (2048 bytes).

I'll fold the finalized block into the PR description's release-note section in the next update.

[pasanw]

Claude is flagging that we have lost the defaulting of syslog packet size - we used to explicitly set it to 1024. If this is true we should probably set it and reflect that in our golden tests

[hjiawei]

It's true the operator no longer defaults it to 1024 — but that's intentional, not a regression. fluentd's 1024-byte packet size truncated our own log lines, which is exactly the failure mode we want to avoid. The operator sets syslog_maxsize only when the user provides packetSize; when it's unset, fluent-bit's out_syslog applies its own default for the RFC5424 format we render (2048 bytes), which truncates less. So restoring an explicit 1024 would reintroduce the truncation.

I've corrected the stale Default: 1024 on the CRD field to describe the real behavior, and added a release note documenting the change. No golden update is needed — the goldens never set packetSize, so they already show no syslog_maxsize line (matching "fluent-bit default applies").