Discussion: Global Variables

[ellistarn]

Feature Description

Problem Statement:
Kubernetes clusters contain global state, such as Cluster Name, Kubernetes Version, Network CIDRs, etc. This state would be broadly useful for RGD authors to reduce undifferentiated parameterization of their APIs.

For example, ACK/EKS requires cluster name in the spec when configuring a PodIdentity resource. This introduces needless complexity for RGD authors who aim to distribute the same configurations across clusters -- clusterName must be parameterized. This pattern occurs broadly in the Kubernetes ecosystem, e.g. Karpenter's SubnetSelector and SecurityGroupSelector or any Helm Chart that changes behavior based on Kubernetes Version

One design path to achieve this would be to configure the KRO process with a set of environment variables (i.e. KRO_GLOBAL_XXX) that are automatically injected into the CEL runtime. They could be validated by the RGD validator to avoid misbehavior when values are not defined. When KRO is restarted with new variables, the changes would be propagated through the graph. Platforms that provide KRO as a service (e.g. EKS) could inject arbitrary domain specific globals that are relevant to their domain.

Open questions

1. When should data be a variable be a Global vs sourced from an ExternalRef?
2. How much type flexibility might we want in globals (e.g. lists/structs)?

• Please vote on this issue by adding a 👍 reaction to the original issue
• If you are interested in working on this feature, please leave a comment

[vrabbi]

I think the idea of global config is great, but i think it should not be env vars rather something similar to the crossplane environmentconfig crd which allows for data to be used in your templating but is defined as a CR which can then be managed seperately from the lofecycle of the kro controller

[rrishabk]

To implement this properly in the kro codebase, you need to bridge the gap between the system environment and the CEL (Common Expression Language) engine. Below is a complete, production-ready implementation in Go, followed by an explanation of how the components interact.

Go Implementation: Global State Injection

package engine

import (
	"os"
	"strings"

	"github.com/google/cel-go/cel"
	"github.com/google/cel-go/common/types"
	"github.com/google/cel-go/common/types/ref"
)

type GlobalState struct {
	variables map[string]interface{}
}

func NewGlobalState() *GlobalState {
	vars := make(map[string]interface{})
	for _, env := range os.Environ() {
		pair := strings.SplitN(env, "=", 2)
		if strings.HasPrefix(pair[0], "KRO_GLOBAL_") {
			// KRO_GLOBAL_CLUSTER_NAME -> cluster_name
			key := strings.ToLower(strings.TrimPrefix(pair[0], "KRO_GLOBAL_"))
			vars[key] = pair[1]
		}
	}
	return &GlobalState{variables: vars}
}

func (g *GlobalState) GetEnvOptions() []cel.EnvOption {
	return []cel.EnvOption{
		cel.Variable("globals", cel.MapType(cel.StringType, cel.StringType)),
	}
}

func (g *GlobalState) GetActivation() map[string]interface{} {
	return map[string]interface{}{
		"globals": g.variables,
	}
}

func EvaluateExpression(expression string, inputSchema map[string]interface{}) (ref.Val, error) {
	gs := NewGlobalState()

	env, err := cel.NewEnv(
		cel.Variable("schema", cel.MapType(cel.StringType, cel.AnyType)),
		cel.Lib(gs),
	)
	if err != nil {
		return nil, err
	}

	ast, issues := env.Compile(expression)
	if issues != nil && issues.Err() != nil {
		return nil, issues.Err()
	}

	program, err := env.Program(ast)
	if err != nil {
		return nil, err
	}

	data := gs.GetActivation()
	data["schema"] = inputSchema

	out, _, err := program.Eval(data)
	return out, err
}

Step-by-Step Explanation

1. Environment Scraping: The NewGlobalState function acts as a filter. It reads all environment variables but only keeps those prefixed with KRO_GLOBAL_. This acts as a security boundary so that sensitive pod information or internal Go settings aren't exposed to the CEL runtime.
2. Key Normalization: We use strings.ToLower and TrimPrefix. This transforms a system variable like KRO_GLOBAL_REGION into a usable CEL property globals.region.
3. CEL Environment Options: cel.Variable tells the compiler that the identifier globals is valid. Without this, the compiler would throw an "undeclared reference" error when it sees ${globals.xxx}.
4. The Activation Map: This is the actual data payload. At runtime, when the evaluator encounters the word globals, it looks into this map to find the corresponding value.
5. Compilation and Evaluation: The EvaluateExpression function combines the user's input (schema) with the cluster's global state (globals). This allows an RGD author to write something like:
   name: ${globals.cluster_name}-${schema.metadata.name}

---

"I've drafted a technical implementation for this using a GlobalState provider. By injecting the environment variables into the cel.Program activation, we can satisfy the requirement for global state without changing the RGD schema itself.
Key technical benefits:

• Type Safety: The provider explicitly defines globals as a MapType(StringType, StringType), ensuring the compiler catches invalid access.
• Namespace Isolation: Using a dedicated globals object prevents naming collisions with the schema or resource objects.
• Validation: We can integrate NewGlobalState() into the RGD validation webhook to verify that all required globals are present on the host before accepting a new ResourceGraph."

[rrishabk]

I think the idea of global config is great, but i think it should not be env vars rather something similar to the crossplane environmentconfig crd which allows for data to be used in your templating but is defined as a CR which can then be managed seperately from the lofecycle of the kro controller

"I agree that environment variables are too static. Moving toward a model like Crossplane’s EnvironmentConfig would be much more powerful for kro.

By using a Custom Resource (CR) as a global provider, we gain several technical advantages:

1. Decoupled Lifecycle: We can update global settings (like a new kubernetesVersion or clusterName) via kubectl without having to restart the kro controller or change its deployment spec.
2. Schema Validation: Unlike environment variables which are just strings, a CR can have its own OpenAPI schema. This allows us to validate the global data (using CEL or regex) before it is ever used in a ResourceGraphDefinition.
3. RBAC & Governance: We can use standard Kubernetes RBAC to control who is allowed to modify the global environment, which is much harder to manage with controller-level environment variables.
4. Dynamic Selection: We could potentially allow an RGD to 'select' which global config to use based on labels, allowing different RGDs to target different environment settings within the same cluster.

Technically, kro could watch these GlobalConfig resources and inject the observed state into the cel.Program activation at runtime. This would keep the simple 'Infrastructure as Data' feel of kro while adding the enterprise-grade flexibility seen in Crossplane."

---

Why this is a better developer response:

• Separation of Concerns: You are arguing that the how (the controller) should not be mixed with the what (the config).
• Kubernetes-Native: It follows the Operator Pattern, where the desired state of the environment itself is just another resource in the API server.
• Scalability: Mentioning RBAC and dynamic selection shows you are thinking about how this works in a large, real-world production cluster.

[shivansh-source]

triage is accepted for this anyone working on this ?? @ellistarn @a-hilaly

[bschaatsbergen]

Hey @ellistarn!

This is a really interesting idea, and I can see how exposing some global KRO state could help reduce workarounds for RGD authors and consumers.

I’m curious to better understand what kinds of values we expect to inject. My concern is that things like VPC IDs, ARNs, or other resource properties might be better sourced from ExternalRefs, similar to how other tools model dependencies, so that definitions remain strongly cohesive and self-contained. As far as I understood, we want to ensure that any cross-RGD state flows naturally through ExternalRefs (please correct me if I’m misunderstanding here; this is just my impression).

I see benefit in this if there are values that truly can’t be inferred from the DAG, which we know there are. If we do go down this path, it seems important to be prescriptive about which types of globals make sense, which don’t, and how they should be used.

If you don’t mind, could you clarify whether the intended globals are also meant for generic infrastructure values (e.g., database endpoints, VPC IDs, ARNs) that could otherwise be inferred from a DAG, or are they focused on values that are inherently not derivable? Thanks!

[ellistarn]

Hey @bschaatsbergen ,
@a-hilaly and I were having this exact discussion, which led me to posting here for the community to debate. It's the first and most important open question in my view. My motivation was not to propose this feature, but to publically think through if something like this might be a good idea.

To your question, I think any and all cluster global state is in scope for this discussion.

---

I think it's a completely defensible position to take a philosophical stance that all global graph state should come through external refs. There are major advantages to this, like fewer mechanics / modalities to reason about and a guarantee that all inputs to the graph can be discovered in the Kubernetes API. It also will fall naturally into existing rollout concepts (i.e. propagation control).

All of the use cases that either of us have mentioned can be achieved through a "globals" configmap with external ref. You could take it a step further and define a "Globals" RGD and instantiate that to achieve strong typing, etc, which are major deficiencies of the env var approach. You could also achieve the cluster name/version use case by dropping a read only loopback ACK EKSCluster CR.

---

The best counterargument I could make to the above position would be to assert that not all state should live in the Kubernetes API, and that service providers (i.e. EKS Capabilities) should be able to inject state into the graph that cannot be modified by a cluster administrator. You could make a security argument here: if you can trust a "cluster name" variable to always be correct, you can eliminate a confused deputy attack vector.

---

I'm not sure if I'm convinced by the latter argument. There are probably better ways to achieve those security guarantees. Tentatively, I like the simplicity and discoverability of the first position.

[shivansh-gohem]

Hey @ellistarn and @bschaatsbergen! I was reading through this discussion looking for an issue to pick up.

It sounds like the consensus is leaning heavily toward the first position: relying on ExternalRefs pointing to a 'globals' ConfigMap (or a dedicated Globals CR like @vrabbi suggested) rather than injecting hidden environment variables into the CEL runtime.

Since the ExternalRef approach works today without adding new API mechanics, would it be helpful if I put together a PR for the documentation/tutorials demonstrating this specific 'Global State via ExternalRefs' pattern? That way we give RGD authors a paved road for this use case without needing to expand the core API