feat: Virtual cluster lifecycle state model

[SamBarker]

Summary

Introduces proposal 016 defining a lifecycle state model for virtual clusters. Virtual clusters are modelled as entities in the Domain-Driven Design sense — each has a persistent identity (its name) that carries through state changes. Reload is a transition the entity passes through, not a replacement of it.

The model defines five states:

• initializing — cluster being set up; not yet serving traffic
• serving — proxy has completed setup and is serving traffic
• draining — no new connections or requests accepted; existing in-flight requests given the opportunity to complete
• failed — configuration not viable; all partially-acquired resources released
• stopped — terminal; cluster permanently removed from configuration

Key Design Decisions

• Virtual clusters are entities — each cluster's identifier is its name; the state machine tracks the entity, not a configuration instance. Renaming a cluster is a destructive operation.
• serving not accepting — the state name reflects that the proxy is actively serving traffic, not merely open to new connections
• Runtime health is out of scope — serving is not split into healthy/degraded; runtime health is orthogonal to lifecycle and better addressed by readiness probes and metrics. Circuit-breaking is noted as a manifestation of the same concern.
• Draining takes the client's view of in-flight — a request is in-flight from the moment the client sends it until it receives a response. New requests on existing connections are rejected during drain to ensure the cluster quiesces.
• Graceful drain is best-effort — the drain timeout is the hard backstop; aligns with Kafka ecosystem norms (brokers drain before stopping)
• partialInitialisationPolicy: serve-none | serve-others — governs proxy behaviour when cluster initialisation fails, whether on startup or reload. Default is serve-none (matches current behaviour).
• stopped is terminal — reload routes through draining → initializing, never through stopped
• Observability is public API — management endpoint and metrics (current state, time in state, transition count) are committed to in the proposal

Relationship to Other Proposals

This provides a foundation for 012 - Hot Reload, which can define reload transitions on this state model. Note: the Hot Reload proposal's "Cluster modification: remove + add" framing describes internal runtime mechanics; from the lifecycle model's perspective the named entity persists through modification.

Test plan

• Review state definitions for clarity and completeness
• Verify state transition diagram consistency with transition descriptions
• Review rejected alternatives for completeness
• Consider whether future enhancements section adequately captures known follow-up work

[tombentley]

Thanks @SamBarker I think this is useful work, but have doubts that we really need to have "healthy" as a state.

[tombentley]

It's not immediately apparent to me how this would work in practice. Suppose I have a producer pipelining Produce requests. How do we intend to stop that flow in such a way that the producer doesn't end up with an unacknowledged request?

You mention later on about timeout, but:

• the broker doesn't know what the client's timeout it configured to, so it doesn't know how long it needs to wait
• if the client timesout request 1 while thinking that 2, 3 and 4 are in flight then it will send request 5

[SamBarker]

The exact mechanics of how the proxy tracks in-flight requests per connection are an implementation concern rather than a design one. The design's position is that draining is best-effort: clients are given a time window to complete naturally, with the drain timeout as the hard backstop after which connections are closed regardless. Kafka clients are required to handle connection loss, so forced closure after timeout is protocol-compliant — the goal is to avoid unnecessary disruption, not to guarantee zero errors. I've clarified this in the proposal.

[tombentley]

Sorry @SamBarker, but I think this does need to be explained.

In the reloading proposal it's been observed that idempotent (but not transactional) producers get their idempotency from the scope of a connection. So trying to minimise duplicate records is clearly a concern for some users, and I think it's only fair that you at least sketch how you think this will work, so everyone can see what the trade-offs are.

clients are given a time window to complete naturally

This is simply not the case, at least as far as I understand what's proposed.

For a real Kafka broker, the clean shutdown process involves the broker telling the controller that it needs to stop, the controller selects replacement leaders for all the partitions that that broker is currently leading, and only once the broker is not leading any partitions does it shut down. In that way producers avoid having any requests in-flight to the broker that's shutting down.

The proxy lacks that mechanism, which means that we are delivering a weaker idempotency guarantee under disconnections than Kafka does. That's an existing problem. But it will be made worse by building a dynamic restart feature which more frequent disconnections.

To me it looks like this is the proposal where we should be figuring out a solution and everyone can assure themselves that it will work well enough.

[SamBarker]

Sorry, I've got local changes that were meant to go out in sync with comment.

Also I thought you were pushing for the opposite solution, let the protocol handle it all and skip the drain.

I'll push the changes shortly and come back to this.

[SamBarker]

The idempotent producer concern is real and the proposal now acknowledges it explicitly. However, this is an existing proxy limitation that predates this proposal — any proxy restart today drops all connections with the same effect. A reload mechanism building on this lifecycle model actually improves the situation by confining disruption to a single virtual cluster rather than the whole process.

Solving the fundamental problem — replicating broker-style partition leader migration to avoid session disruption entirely — is a substantial piece of work well beyond the scope of a lifecycle model, with a multitude of possible solutions that the proxy cannot currently express. The proposal notes the limitation and directs users to consider how analogous situations (broker crash, network split) are handled without the proxy, since clients must already be prepared for those. Users requiring exactly-once guarantees across reconnections should use transactional producers.

[SamBarker]

The draining section has been updated to clarify in-flight semantics and correct an earlier misstatement about idempotent producers.

On the specific concern about idempotent producers: a proxy-initiated drain is no different from any other connection loss event from the client's perspective. Idempotent producers retain their Producer ID and sequence numbers in client memory across reconnections — a connection drop does not start a new session. When the client reconnects after a drain, it retries unacknowledged requests with the same (PID, epoch, sequence) tuple, and the broker deduplicates them using its existing producer state. A new PID is only assigned on producer process restart, not on reconnection. The drain timeout is the hard backstop after which connections close, but this is protocol-compliant — Kafka clients are required to handle connection loss, and the idempotent producer's deduplication mechanism is designed to survive exactly this scenario.

References:

• KIP-360 documents how the broker retains producer state and how idempotent producers can safely bump their epoch when broker state is lost — the key point being that the PID is retained client-side across reconnections.
• The Idempotent Producer design doc confirms that deduplication is based on (PID, epoch, sequence) per partition, not per connection: "The broker maintains in memory the sequence numbers it receives for each topic partition from every PID."
• KIP-854 covers broker-side producer state expiry — the state is retained well beyond any reasonable drain timeout.

The lifecycle model actually improves the posture for idempotent producers compared to the status quo. Today, any configuration change requires a full process restart, dropping every connection across every virtual cluster. With per-cluster lifecycle and reload, only the affected cluster's connections drain — the blast radius shrinks from "all clusters" to "one cluster" and planned restarts become less frequent.

I think the proposal goes far enough on draining. The lifecycle model defines that draining exists as a state and its high-level semantics (reject new connections, reject new requests, allow in-flight to complete, timeout as backstop). The detailed mechanics — specific error codes for rejected requests, interaction with pipelining, TCP-level behaviour — are implementation concerns that don't change the lifecycle state model and are better resolved in the reload proposal where draining becomes a frequent operation rather than a rare shutdown event.

[SamBarker]

@tombentley can we resolve this thread?

[tombentley]

You've defined how proxy started works wrt the VM state machines. It's not really clear to me whether we need a proposal about the rest, unless you think that that state is also exposed via metrics/mgmt etc.

[SamBarker]

Yes — proxy-level state would also be exposed via metrics and management endpoints, which is the argument for capturing it as future work. The per-cluster metrics defined in this proposal are one layer; a proxy-level lifecycle would add an aggregate layer above them covering port binding, process startup/shutdown sequencing, and overall proxy health. Defining it now felt premature, but it's a natural next step once the per-cluster model is established.

[SamBarker]

The future enhancements section flags proxy-level lifecycle as a problem space that exists, not as work we're committed to. Whether proxy-level state needs its own formal model, observable via metrics/management endpoints, or is simply emergent from the per-VC lifecycles plus standard process management is an open question — and one that might not need answering.

Every proxy has some form of process-level lifecycle (HAProxy expresses it through old/new process coexistence during reload, nginx through worker generations, httpd through child worker replacement), but none of them formalise it as a state machine — it's just how startup and shutdown work. Kroxylicious may end up the same way: the proxy starts VCs, manages their lifecycles, and exits. If that's sufficient, no proposal is needed.

I've reworded the section to make it clearer that these are genuine open questions without obvious answers (port binding significance, health aggregation across deployment models) rather than planned future work.

[SamBarker]

@tombentley similar question here, are we happy?

[tombentley]

Thanks @SamBarker, I think this is looking pretty good. I had a question is about how the configuration API for the termination behaviour.

[tombentley]

I'm on-board with the broad concept, but:

• I don't find this name to be very intuitive
• Is there an equivalent thing which defines how the proxy handles failures during reload? I don't think we'd really need a separate config for that, (or did you have a use case in mind?)

This really seems to be about specifying the user's tolerance for virtual clusters entering/remaining in the failed state. Currently the state machine diagram shows a retry loop, but the way you're describing it here is that loop is, in actual fact, the kubernetes pod crash loop. And therefore that control loop does not exist when the proxy is not running in kube.

Perhaps we should borrow some ideas from how Kubernetes allows pods to fail, but apply them directly in the proxy (whether or not it happens to be running in Kube). Specifically, some number of retries, a retry delay ± a backoff. And implicitly we terminate when we run out of retries for any VC. (You can configure the number of reties as Integer.MAX_VALUE if you don't want to terminate). We can add some other policy than terminate later on if it's needed.

[SamBarker]

I can see where your coming from. I had hoped that initialisation would be read as VC initialisation rather than proxy startup.

How about restructuring as:

proxy:
  onVirtualClusterFailure:
    serve: none          # default
    # serve: remaining   # keep serving healthy clusters

onVirtualClusterFailure avoids ambiguity with target clusters. The values describe the proxy-level consequence rather than the individual cluster's fate.

I think retry is orthogonal to this policy. serve answers "what about the rest of the proxy?" while retry answers "what about the failed cluster?" They compose independently. By making onVirtualClusterFailure a structured block rather than a flat string, new keys can be added alongside serve in future proposals without breaking existing configs. For example:

proxy:
  onVirtualClusterFailure:
    serve: remaining
    # future extensions — new keys, existing configs unaffected:
    # retry:
    #   maxRetries: 3
    #   backoff: 5s
    # escalate:
    #   url: http://example.slack.com/
    #   messageFormat: "%s failed with %s"
    #   args: [VIRTUAL_CLUSTER, EXCEPTION_MESSAGE]

This proposal defines serve with its two values. The rejected alternative on automatic retry still applies — we're not committing to implementing retry here, but the config structure leaves room for it without a breaking change.

[tombentley]

The proxy cannot know when/whether the client receives a response that's been forwarded to it (at least not without inferring that fact from the receipt of another request, but that defeats the point).

In any case, I just don't see that this can really be completely solved by the proxy in its current shape. To solve it properly I think we'd need to "virtualise the brokers". That would allow the proxy to leverage Kafka's own technique for shutting down brokers cleanly: Tell the client that this broker is no longer the leader or coordinator for anything, so that all the clients update their metadata to find the new leader/coordinator. Doing that would require a proper cluster-of-proxies, which is not something we have today.

So I think what you're describing here is not perfect, but it's as good are we're likely to get for now, and we shouldn't block this proposal while we figure out a more perfect solution.

[SamBarker]

Agreed on both points. "As best it can" now hedges the in-flight definition — the proxy observes its own reads and writes, not what the client has actually sent or received.

And yes, virtualising the brokers would let the proxy leverage Kafka's own clean shutdown mechanism (leader migration via metadata updates). That would be a really compelling capability if we can figure out how to make it work. For now, best-effort draining with a timeout backstop is as good as we can get without it.

[hrishabhg]

The entity model is well-defined, and the state machine is clean and practical. The separation of runtime health from lifecycle state is the right call. One minor suggestion left on drainTimeout scoping, but nothing blocking.

cc: @Uzziee

[hrishabhg]

Since virtual clusters are modelled as independent entities with their own lifecycles, should drainTimeout be configured at the virtual cluster level rather than the root proxy: level? Different VCs may proxy to Kafka clusters with very different timeout characteristics (e.g. max.poll.interval.ms, producer acks=all latency), so a single global drain timeout could be too aggressive for some and too lenient for others.

[hrishabhg]

Should draining be opt-out? Kafka clients are designed to handle broker disconnections — they retry, reconnect, and rediscover leaders automatically. A proxy restart without switchover looks like a broker disconnect to clients. Draining primarily adds value during switchover scenarios where in-flight requests must complete on the old path before traffic moves to a new one. For simpler cases like reloading authz rules or config changes that don't affect connectivity, the drain phase adds unnecessary latency. Consider supporting drainTimeout: 0s (or an explicit drain: false) to skip draining entirely and transition directly to stopped. This also aligns with making drainTimeout per-VC — some VCs may never need draining.

cc: @Uzziee

[SamBarker]

Good suggestion on per-VC drain timeouts — the proposal now does exactly that. Different clusters can have different workload profiles, and per-cluster config also provides the right foundation for reload where only the affected cluster drains.

On making draining opt-out: draining is best-effort rather than a guarantee. It operates within the Netty shutdown window, which is the hard backstop — if a drain exceeds it, Netty force-closes everything. So the cost of having draining enabled is bounded: at worst it uses time that Netty was going to wait anyway. Making it opt-out adds configuration surface without a clear benefit, since disabling it just means connections close immediately rather than getting a chance to complete gracefully.

The proposal also now acknowledges that some operations (notably long-polling consumers) will routinely exceed any reasonable drain timeout. Synthesising early responses to unblock these is a possible future enhancement, but would require its own proposal as it introduces a new signal to the filters.

[hrishabhg]

Thanks for accepting the per-VC drainTimeout.

On the other point, my reading of the updated design is that drainTimeout must be ≤ network.proxy.shutdownTimeout (default 15s), since Netty's graceful shutdown is the hard outer boundary. If so, the default drainTimeout of 60s exceeds that bound which gives a misleading impression that drain can run longer than Netty allows. Implementation should have warning log for this and default should be set as 15s.

The proposal should clarify the relationship between drainTimeout and shutdownTimeout. As I understand it, drainTimeout is the lower bound (how long we attempt to let in-flight requests complete) and shutdownTimeout is the upper bound (Netty's hard stop). This also means drainTimeout: 0s is a natural opt-out — skip straight to Netty's shutdown, no backpressure or drain phase needed. This framing would help implementors understand that precise in-flight tracking isn't necessary — drain is best-effort within the window, and Netty force-closes at the upper bound regardless.

[hrishabhg]

Thanks for accepting the per-VC drainTimeout.

On the other point, my reading of the updated design is that drainTimeout must be ≤ network.proxy.shutdownTimeout (default 15s), since Netty's graceful shutdown is the hard outer boundary. If so, the default drainTimeout of 60s exceeds that bound which gives a misleading impression that drain can run longer than Netty allows. Implementation should have warning log for this and default should be set as 15s.

The proposal should clarify the relationship between drainTimeout and shutdownTimeout. As I understand it, drainTimeout is the lower bound (how long we attempt to let in-flight requests complete) and shutdownTimeout is the upper bound (Netty's hard stop). This also means drainTimeout: 0s is a natural opt-out — skip straight to Netty's shutdown, no backpressure or drain phase needed. This framing would help implementors understand that precise in-flight tracking isn't necessary — drain is best-effort within the window, and Netty force-closes at the upper bound regardless.

[tombentley]

Many thanks @SamBarker