feat: Allow making logs less verbose than default

[tomasaschan]

We were alerted to some log patterns that were costing literally thousands of dollars per month in our setup, and one of them was from this line¹.

I tried using the spec.override.logLevels feature to dial down the logging a bit, but it turns out CRD-level validation require the log level specified there to be in the range 0-10 inclusive, which makes it impossible to turn off log lines like the one linked above, for two reasons:

1. It is not V-leveled; this PR addresses that for this specific logging statement, but it's likely that there is value in addressing this elswehere too in the project. Since we're not seeing enormous log volumes from other logging statements, though, I've considered that out of scope for this PR. To avoid changing the current behavior here, I've leveled this at 0, but I'm happy to bump that to turn it off by default for everyone.

2. The default log level is 0, which the CRD validation then does not allow turning off. This is addressed by relaxing the validation requirements and allow negative log level values.

klog itself doesn't have any issues with negative log levels (the log level is stored internally as an int32) so specifying -v=-1 on the command line works just fine, but since the RootSync CRD doesn't allow config like the following, we're stuck:

spec:
  override:
    logLevels:
      - containerName: reconciler
        logLevel: -1

To convince yourself that this is valid, you can run the following small program:

package main

import (
	"flag"
	"fmt"

	"k8s.io/klog/v2"
)

func main() {
	klog.InitFlags(nil)
	flag.Parse()

	fmt.Println("klog.V(0).Enabled():", klog.V(0).Enabled())

	klog.V(0).Infof("Leveled can be turned off!")
    klog.Infof("Unlevelled always logs... :(")
}

With this in klog-repro.go at the root of this repository, you'll see this:

$ go run ./klog-repro.go
klog.V(0).Enabled(): true
I0305 11:01:09.046724  159525 klog-repro.go:16] Leveled can be turned off!
I0305 11:01:09.046789  159525 klog-repro.go:17] Unlevelled always logs... :(

$ go run ./klog-repro.go -v=-1
klog.V(0).Enabled(): false
I0305 11:01:11.403201  159753 klog-repro.go:17] Unlevelled always logs... :(

Note: I get what looks like completely unrelated test errors from make test, so I'm going to rely on CI to find if I need to amend anything else somewhere.

---

1. Well, actually from this one since we're running a pretty old version of Config Sync... but even if we upgraded none of this would change, so I might as well point to the current source to avoid the whole "try upgrading first" discussion.

[google-oss-prow]

Hi @tomasaschan. Thanks for your PR.

I'm waiting for a GoogleContainerTools member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances logging control within Config Sync by allowing users to reduce log verbosity beyond the default settings. It addresses a critical issue of excessive logging costs by enabling the complete suppression of specific log messages and relaxing CRD validation to support negative log levels, which klog natively supports for disabling logs.

Highlights

• Relaxed Log Level Validation: Modified the ContainerLogLevelOverride CRD validation to permit negative logLevel values, extending the allowed range from 0-10 to -10-10. This enables users to effectively disable certain log statements.
• V-Leveled Logging for Remediator Watch Events: Converted a klog.Infof statement in filteredwatcher.go to klog.V(0).Infof. This change makes the log message V-leveled, allowing it to be controlled and potentially disabled via logLevel configuration, addressing a source of high logging costs.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• pkg/api/configsync/v1beta1/resource_override.go
  • Updated the logLevel field's kubebuilder:validation:Minimum annotation from 0 to -10.
  • Modified the comment for LogLevel to reflect the new allowed range of -10 to 10.
• pkg/remediator/watch/filteredwatcher.go
  • Changed klog.Infof to klog.V(0).Infof for the "Remediator handling watch event" log message.

Activity

• The author noted encountering unrelated test errors locally and is relying on CI to validate changes and identify any necessary amendments.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces two changes to allow for less verbose logging. It modifies a high-frequency log statement in filteredwatcher.go to be V-leveled, and updates the ContainerLogLevelOverride in the v1beta1 API to permit negative log levels. This allows disabling V-leveled logs with a level of 0. My feedback includes a suggestion to apply the log level validation change to the v1alpha1 API version as well to maintain consistency.

[tomasaschan]

/assign @cheftako

[google-oss-prow]

@tomasaschan: GitHub didn't allow me to assign the following users: cheftako.

Note that only GoogleContainerTools members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

/assign @cheftako

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiffanny29631]

This yields an update in doc of logLevel field https://docs.cloud.google.com/kubernetes-engine/config-sync/docs/reference/rootsync-reposync-fields. Worth noting that setting a logLevel to a high negative value (e.g., -10) may suppress almost all informational logs, making it difficult to monitor the container's health through standard output.

FYI run make generate to refresh the CRDs.

[tomasaschan]

Thanks! Did so (but with make configsync-crds); also, note that the way the Makefile sets up the installation of controller-gen it doesn't lock the version, and also gets confused if the user has set GOBIN (since it only sets GOPATH when installing), so it took me a few tries to make it run the version you had used previously.

[tiffanny29631]

Good catch! We could resolve this together with the Makefile improvement, focusing on the GOBIN consistency and version pinning ,@Camila-B the driver.

For now we could update the contribution doc to remind contributors to unset the var before building.

[tiffanny29631]

/ok-to-test

[cheftako]

/assign @tiffanny29631

[tiffanny29631]

@tomasaschan: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
kpt-config-sync-presubmit 98f2ba5 link true /test kpt-config-sync-presubmit
Details

Need to run make generate to update the CRDs.

[tomasaschan]

I'm flailing here.

For some reason I can't get the local codegen to generate the same code as what this checks for; I assume there's some version of some tool that's assumed by the build system but not specified in the repo, and I get a different one than whatever the build machine is using.

Could someone help me run the appropriate codegen for me and either push to this branch or just subsume this PR into one that builds cleanly?

[Camila-B]

/retest

[tiffanny29631]

/retest-required

[Camila-B]

Seems like TestOverrideRootSyncLogLevel is failing

[tomasaschan]

/retest-required

[tomasaschan]

/retest

[tiffanny29631]

/lgtm

FYI this change will be included with Config Sync 1.24, target date 2026-05-07

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tiffanny29631

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [tiffanny29631]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tomasaschan]

Thanks both for the help to get this to a mergeable state! We're looking forward to 1.24! :)