Skip to content

[fix][fn] Allow retainKeyOrdering on Go functions - #26421

Merged
david-streamlio merged 3 commits into
apache:masterfrom
david-streamlio:fix-fn-cli-runtime-markers
Aug 26, 2026
Merged

[fix][fn] Allow retainKeyOrdering on Go functions#26421
david-streamlio merged 3 commits into
apache:masterfrom
david-streamlio:fix-fn-cli-runtime-markers

Conversation

@david-streamlio

Copy link
Copy Markdown
Contributor

Master Issue: #26404
Follow-up to #26414, which closed #26405.

Motivation

#26414 taught the Go runtime to honour retainOrdering and retainKeyOrdering: resolveSubscriptionType in pulsar-function-go/pf/instance.go selects Failover for the first and KeyShared for the second, matching python_instance.py.

Half of it cannot be reached. doGolangChecks still refuses retainKeyOrdering outright:

if (functionConfig.getRetainKeyOrdering() != null && functionConfig.getRetainKeyOrdering()) {
    throw new IllegalArgumentException("Retain Key Orderering not yet supported in Go function");
}

validateNonJavaFunction has a single caller, the worker REST API (FunctionsImpl), so cluster submission is exactly what the guard blocks. LocalRunner never calls it, which is why the KeyShared branch is reachable under localrun and nowhere else. As master stands, pulsar-admin functions create --go ... --retain-key-ordering fails at creation, and the code #26414 added for it is dead on the cluster path.

retainOrdering is unaffected — it was never gated — so that half of #26414 works today.

This is the same shape of gap as the Python dead letter queue in #26400, with one difference: convert carries retainKeyOrdering into FunctionDetails unconditionally (L297), so this guard is the only thing in the way.

#26405 was auto-closed when #26414 merged. Its body only described the runtime gap, so the validation half was never in its scope; happy to reopen it instead of tracking this here if that is preferred.

Modifications

Drop the guard from doGolangChecks.

The combinations that would genuinely be contradictory are already rejected in doCommonChecks, for every runtime:

  • retainKeyOrdering with EFFECTIVELY_ONCE"When effectively once processing guarantee is specified, retain Key ordering cannot be set"
  • retainKeyOrdering together with retainOrdering"Only one of retain ordering or retain key ordering can be set"

doGolangChecks refuses EFFECTIVELY_ONCE for Go before either is reached, so nothing needs repeating for Go specifically. The maxMessageRetries guard stays — the Go runtime does not honour retryDetails yet (#26406) — and a test pins it so this change cannot be widened by accident.

The Go client needs no extra configuration for KeyShared: ConsumerOptions leaves KeySharedPolicy nil, toProtoKeySharedMeta(nil) returns nil, and the broker then applies its default auto-split hash range.

Correct the CLI runtime markers. The @Option descriptions in CmdFunctions carry a marker that the docs sync parses into the Support column of the published pulsar-admin CLI reference, and that functions create --help prints verbatim. --retain-ordering and --retain-key-ordering were both #Java, which has been wrong for Python since long before #26414python_instance.py has always selected Failover and KeyShared for them. Both now read #Java, Python, Go.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Five tests in FunctionConfigUtilsTest: Go accepts retainKeyOrdering and converts to a KEY_SHARED subscription; accepts retainOrdering and converts to FAILOVER; rejects both ordering modes together; rejects retainKeyOrdering with EFFECTIVELY_ONCE; and still refuses message retries.
  • FunctionConfigUtilsTest: 42/42 pass.
  • Confirmed the tests are not vacuous: restoring the guard fails testGoFunctionAcceptsRetainKeyOrdering and nothing else.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints

This is relaxing only. A configuration that was accepted before is still accepted; one input that was rejected — retainKeyOrdering on a Go function — is now accepted, and behaves as the runtime already implements. Java and Python functions are untouched.

Documentation

  • doc-required
  • doc-not-needed

The FunctionConfig table in docs/functions-cli.md documents retainOrdering and retainKeyOrdering with no runtime qualification. With this change that is accurate for the first time — all three runtimes honour both — so the table needs no edit. The #Java, Python, Go markers above keep the generated CLI reference in step.

### Motivation

apache#26414 taught the Go runtime to honour retainKeyOrdering: resolveSubscriptionType
in pulsar-function-go/pf/instance.go selects a KeyShared subscription for it, the
same way python_instance.py does.

Nothing can reach that code on the cluster path. doGolangChecks still refuses
retainKeyOrdering outright ("Retain Key Orderering not yet supported in Go
function"), and validateNonJavaFunction is called by the worker REST API
(FunctionsImpl), so cluster submission is exactly what the guard blocks.
LocalRunner never calls it, which is why the KeyShared branch is reachable under
localrun and nowhere else.

Unlike the Python dead letter case, convert() carries retainKeyOrdering into
FunctionDetails unconditionally, so this guard is the only thing in the way.

### Modifications

Drop the guard from doGolangChecks.

The combinations that would genuinely be contradictory are already rejected in
doCommonChecks, for every runtime: retainKeyOrdering with EFFECTIVELY_ONCE
("retain Key ordering cannot be set"), and retainKeyOrdering together with
retainOrdering ("Only one of retain ordering or retain key ordering can be set").
doGolangChecks refuses EFFECTIVELY_ONCE for Go before either is reached, so
nothing needs repeating here.

The Go client needs no extra configuration for KeyShared: ConsumerOptions leaves
KeySharedPolicy nil, toProtoKeySharedMeta(nil) returns nil, and the broker then
applies its default auto-split hash range.

The maxMessageRetries guard stays: the Go runtime does not honour retryDetails
yet, and a test pins that so this change cannot be widened by accident.

### Verifying this change

Five tests in FunctionConfigUtilsTest: Go accepts retainKeyOrdering and converts
to a KEY_SHARED subscription; accepts retainOrdering and converts to FAILOVER;
rejects both ordering modes together; rejects retainKeyOrdering with
EFFECTIVELY_ONCE; and still refuses message retries.

Confirmed the tests are not vacuous: restoring the guard fails
testGoFunctionAcceptsRetainKeyOrdering.
…in-key-ordering

The @option descriptions in CmdFunctions carry a runtime marker that the docs
sync parses into the "Support" column of the published pulsar-admin CLI
reference, and that `functions create --help` prints verbatim.

Both flags were marked #Java. That has been wrong for Python since long before
apache#26414: python_instance.py selects Failover for retainOrdering and KeyShared for
retainKeyOrdering. apache#26414 added the same to the Go runtime, and the preceding
commit makes retainKeyOrdering reachable for a Go function submitted to a
cluster, so both now read #Java, Python, Go.
@david-streamlio david-streamlio added area/function doc-not-needed Your PR changes do not impact docs labels Aug 26, 2026

@freeznet freeznet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation and conversion paths are consistent with the Java and Python function behavior. I left one non-blocking simplification suggestion inline.

The active constraints are already stated in doCommonChecks, which rejects
retainKeyOrdering with EFFECTIVELY_ONCE and retainKeyOrdering together with
retainOrdering, and both are covered by tests. Keeping doGolangChecks limited to
active validation avoids a second description of the rules, and drops references
to specific Go and Python implementation files that would go stale.
@david-streamlio
david-streamlio merged commit ecf6dae into apache:master Aug 26, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/function doc-not-needed Your PR changes do not impact docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Go Functions] retainOrdering and retainKeyOrdering are ignored, so key ordering is silently not preserved

2 participants