Configure the confluent:// transport from app settings via kafkaesq - #761
Open
wbarnha wants to merge 2 commits into
Open
Configure the confluent:// transport from app settings via kafkaesq#761wbarnha wants to merge 2 commits into
wbarnha wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #761 +/- ##
==========================================
+ Coverage 96.06% 96.07% +0.01%
==========================================
Files 103 104 +1
Lines 11072 11153 +81
Branches 1191 1196 +5
==========================================
+ Hits 10636 10715 +79
Misses 345 345
- Partials 91 93 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Faust deals with three spellings of the same Kafka client configuration:
aiokafka's constructor kwargs (session_timeout_ms), librdkafka's dotted keys
(session.timeout.ms), and app settings (broker_session_timeout, in seconds).
Moving a config between them is manual work today.
Add kafkaesq as an optional dependency -- pip install
"faust-streaming[kafkaesq]" -- and a ClientConfig class that holds one
configuration and hands it out in whichever spelling is wanted:
config = ClientConfig.from_confluent(client_properties)
consumer = AIOKafkaConsumer('topic', **config.as_aiokafka())
settings = config.as_app_settings()
app = faust.App(settings.pop('id'), **settings)
and the other way round, from_app/from_app_settings turn an app's settings
into config for a plain client -- from_app taking the brokers from the app's
transport, so the client connects where the app does.
Values are clamped to the ranges librdkafka enforces on the way out, since
Faust's settings are not bounded by them: producer_request_timeout defaults
to 20 minutes, above librdkafka's 15-minute ceiling for request.timeout.ms,
which would fail client construction outright.
Nothing in Faust depends on this: the transports configure themselves from
app settings exactly as before, and neither needs kafkaesq installed. Without
it the class raises ImproperlyConfigured with the install hint.
Authentication is out of scope -- Faust configures it with a
broker_credentials object built at runtime, which no config file can
describe, so security keys are reported as unmapped when converting to app
settings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G
wbarnha
force-pushed
the
claude/faust-kafkaesq-optional-ub0s7p
branch
from
August 6, 2026 22:10
8df31e2 to
2c7cc1e
Compare
faust.transport.drivers.aiokafka has a reference page; its confluent counterpart never did, and was silenced in conf.py's apicheck_ignore_modules so `make apicheck` would not report the gap. Give it a page, drop the silencer, and list both drivers together under Transports. The docs build imports every documented module for autodoc and the confluent driver imports confluent_kafka at module level (unlike the optional stores, which guard theirs), so the docs environment now installs the ckafka extra. The broker setting's transport list gains a reference link per driver. That list lives in the setting's docstring, which docs/includes/settingref.txt is generated from, so the client-config-conversion section added there earlier moves into the docstring too -- the generated file now matches what `make configref` produces for that section, instead of drifting from it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Switching a Faust app between the kafka:// (aiokafka) and confluent://
(librdkafka) transports meant losing most of its configuration: the
confluent driver translated seven settings by hand and ignored
broker_credentials entirely, so an app with SASL simply connected as
plaintext.
Add kafkaesq as an optional dependency -- pip install
"faust-streaming[kafkaesq]" -- and put it to work in both directions in the
new faust.transport.kafkaesq module:
which the confluent driver now uses for its consumers and producer. With
the extra installed the full settings table is translated; without it the
driver falls back to the mapping it always had, so the extra is genuinely
optional.
librdkafka config into faust.App arguments.
broker_credentials to and from librdkafka security keys, which kafkaesq
leaves alone because Faust configures authentication with a runtime
object.
Three things the driver got wrong, found by feeding the generated configs to
real confluent_kafka clients (a new test does this, since a mocked-out
client cannot reject a key):
fetch response rather than one partition; aiokafka's equivalent is
max.partition.fetch.bytes.
refuses outright. It now gets a group of its own, so it still cannot
disturb the worker group.
15-minute ceiling for request.timeout.ms, which fails client construction.
Values are clamped into librdkafka's ranges, with a warning.
The consumer no longer sends request.timeout.ms, a producer-only property
librdkafka warns about and ignores, and the deprecated default.topic.config
wrapper is gone -- auto.offset.reset is a top-level key now, and follows
consumer_auto_offset_reset instead of always being "earliest".
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G