Skip to content

Configure the confluent:// transport from app settings via kafkaesq - #761

Open
wbarnha wants to merge 2 commits into
masterfrom
claude/faust-kafkaesq-optional-ub0s7p
Open

Configure the confluent:// transport from app settings via kafkaesq#761
wbarnha wants to merge 2 commits into
masterfrom
claude/faust-kafkaesq-optional-ub0s7p

Conversation

@wbarnha

@wbarnha wbarnha commented Aug 6, 2026

Copy link
Copy Markdown
Member

Switching a Faust app between the kafka:// (aiokafka) and confluent://
(librdkafka) transports meant losing most of its configuration: the
confluent driver translated seven settings by hand and ignored
broker_credentials entirely, so an app with SASL simply connected as
plaintext.

Add kafkaesq as an optional dependency -- pip install
"faust-streaming[kafkaesq]" -- and put it to work in both directions in the
new faust.transport.kafkaesq module:

  • app_settings_to_confluent derives a librdkafka config from app settings,
    which the confluent driver now uses for its consumers and producer. With
    the extra installed the full settings table is translated; without it the
    driver falls back to the mapping it always had, so the extra is genuinely
    optional.
  • confluent_to_app_settings goes the other way, turning an existing
    librdkafka config into faust.App arguments.
  • credentials_to_confluent / credentials_from_confluent convert
    broker_credentials to and from librdkafka security keys, which kafkaesq
    leaves alone because Faust configures authentication with a runtime
    object.

Three things the driver got wrong, found by feeding the generated configs to
real confluent_kafka clients (a new test does this, since a mocked-out
client cannot reject a key):

  • consumer_max_fetch_size was sent as fetch.max.bytes, which bounds the whole
    fetch response rather than one partition; aiokafka's equivalent is
    max.partition.fetch.bytes.
  • A client-only app built a consumer with no group.id, which librdkafka
    refuses outright. It now gets a group of its own, so it still cannot
    disturb the worker group.
  • producer_request_timeout defaults to 20 minutes, above librdkafka's
    15-minute ceiling for request.timeout.ms, which fails client construction.
    Values are clamped into librdkafka's ranges, with a warning.

The consumer no longer sends request.timeout.ms, a producer-only property
librdkafka warns about and ignores, and the deprecated default.topic.config
wrapper is gone -- auto.offset.reset is a top-level key now, and follows
consumer_auto_offset_reset instead of always being "earliest".

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G

@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.07%. Comparing base (4af976b) to head (d80875d).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
faust/transport/kafkaesq.py 95.83% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #761      +/-   ##
==========================================
+ Coverage   96.06%   96.07%   +0.01%     
==========================================
  Files         103      104       +1     
  Lines       11072    11153      +81     
  Branches     1191     1196       +5     
==========================================
+ Hits        10636    10715      +79     
  Misses        345      345              
- Partials       91       93       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Faust deals with three spellings of the same Kafka client configuration:
aiokafka's constructor kwargs (session_timeout_ms), librdkafka's dotted keys
(session.timeout.ms), and app settings (broker_session_timeout, in seconds).
Moving a config between them is manual work today.

Add kafkaesq as an optional dependency -- pip install
"faust-streaming[kafkaesq]" -- and a ClientConfig class that holds one
configuration and hands it out in whichever spelling is wanted:

    config = ClientConfig.from_confluent(client_properties)
    consumer = AIOKafkaConsumer('topic', **config.as_aiokafka())

    settings = config.as_app_settings()
    app = faust.App(settings.pop('id'), **settings)

and the other way round, from_app/from_app_settings turn an app's settings
into config for a plain client -- from_app taking the brokers from the app's
transport, so the client connects where the app does.

Values are clamped to the ranges librdkafka enforces on the way out, since
Faust's settings are not bounded by them: producer_request_timeout defaults
to 20 minutes, above librdkafka's 15-minute ceiling for request.timeout.ms,
which would fail client construction outright.

Nothing in Faust depends on this: the transports configure themselves from
app settings exactly as before, and neither needs kafkaesq installed. Without
it the class raises ImproperlyConfigured with the install hint.

Authentication is out of scope -- Faust configures it with a
broker_credentials object built at runtime, which no config file can
describe, so security keys are reported as unmapped when converting to app
settings.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G
@wbarnha
wbarnha force-pushed the claude/faust-kafkaesq-optional-ub0s7p branch from 8df31e2 to 2c7cc1e Compare August 6, 2026 22:10
faust.transport.drivers.aiokafka has a reference page; its confluent
counterpart never did, and was silenced in conf.py's apicheck_ignore_modules
so `make apicheck` would not report the gap. Give it a page, drop the
silencer, and list both drivers together under Transports.

The docs build imports every documented module for autodoc and the confluent
driver imports confluent_kafka at module level (unlike the optional stores,
which guard theirs), so the docs environment now installs the ckafka extra.

The broker setting's transport list gains a reference link per driver. That
list lives in the setting's docstring, which docs/includes/settingref.txt is
generated from, so the client-config-conversion section added there earlier
moves into the docstring too -- the generated file now matches what `make
configref` produces for that section, instead of drifting from it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BYhhm6H1FQskUmDqjrft7G
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants