Skip to content

production/ksonnet/loki: rollout-operator incompatible with versions newer than v0.28.1 #20281

@nlowe

Description

@nlowe

Describe the bug
The rollout-operator added support for ZoneAwarePodDisruptionBudgets, a new custom resource. Loki's jsonnet manifests do not use the jsonnet manifests packaged for the rollout-operator and instead use a custom version. This custom version does not include RBAC permissions for the new ZPDB features.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy loki using the jsonnet manifests following the documentation
  2. Attempt to upgrade the rollout-operator image to v0.33.0 (the latest as of filing this issue)
  3. Observe the rollout-operator never goes ready:
E1228 23:50:57.509753       1 reflector.go:205] "Failed to watch" err="failed to list rollout-operator.grafana.com/v1, Resource=zoneawarepoddisruptionbudgets: zoneawarepoddisruptionbudgets.rollout-operator.grafana.com is forbidden: User \"system:serviceaccount:loki:rollout-operator\" cannot list resource \"zoneawarepoddisruptionbudgets\" in API group \"rollout-operator.grafana.com\" in the namespace \"loki\"" logger="UnhandledError" reflector="k8s.io/client-go/dynamic/dynamicinformer/informer.go:108" type="rollout-operator.grafana.com/v1, Resource=zoneawarepoddisruptionbudgets

Expected behavior
The rollout operator should be able to be upgraded to the latest version.

This can be worked around with

// ...
(import 'github.com/grafana/loki/production/ksonnet/loki/loki.libsonnet') +
{
  local role = $.rbac.v1.role,
  local policyRule = $.rbac.v1.policyRule,
  rollout_operator_role+:
    role.withRulesMixin([
      // https://github.com/grafana/rollout-operator/blob/6401f0ade9131590ca6352cacce5ac0fc59228e5/operations/rollout-operator/rollout-operator.libsonnet#L144-L148
      policyRule.withApiGroups('rollout-operator.grafana.com') +
      policyRule.withResources(['zoneawarepoddisruptionbudgets']) +
      policyRule.withVerbs(['get', 'list', 'watch']),
    ]),
} +
// ...

which allows the operator to start correctly on newer versions.

Environment:

  • Infrastructure: Kubernetes (v1.34.2+rke2r1) via RKE2 on bare metal
  • Deployment tool: jsonnet

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions