Skip to content

Fix the remaining latent bugs found by the type checker - #760

Open
wbarnha wants to merge 6 commits into
masterfrom
claude/faust-latent-bug-fixes
Open

Fix the remaining latent bugs found by the type checker#760
wbarnha wants to merge 6 commits into
masterfrom
claude/faust-latent-bug-fixes

Conversation

@wbarnha

@wbarnha wbarnha commented Aug 6, 2026

Copy link
Copy Markdown
Member

Six bugs across five modules, all surfaced by the type checker in #758 and left marked XXX there because fixing them changes runtime behaviour. One commit per bug, each with a regression test.

#758 has since been squash-merged, so this branch has been rebased onto master and now contains only the six fixes. Independent of #759, which fixes the two aiokafka crashes — they touch disjoint files and can merge in either order.

What was broken

Module Bug
auth.py SSLCredentials() raised TypeError — could not build a context at all
__init__.py faust.version_info.major was 'v'; every field was wrong
tables/base.py on_window_close never received the aggregated window data
tables/recovery.py changelog events applied to an unrelated table; aborted-tx fixup silently skipped
cli/base.py require_app = False commands crashed — faust completion unusable without -A
transport/drivers/confluent.py key_partition dead on arrival; livelock detector raised every tick

SSLCredentials could not be constructed

purpose defaulted to None and went straight into ssl.create_default_context(purpose=...), which starts if not isinstance(purpose, _ASN1Object): raise TypeError(purpose). Any call that did not pass an explicit context raised — which defeats the cafile/capath/cadata parameters entirely. Now defaults to ssl.Purpose.SERVER_AUTH, the same default create_default_context() itself uses and the right one for a client verifying a broker; the tests assert the resulting context has check_hostname=True and verify_mode=CERT_REQUIRED.

faust.version_info held strings in its int fields

The regex groups (prefix, version, suffix) were splatted positionally into VersionInfo(major, minor, micro, ...), giving VersionInfo(major=None, minor='0.11.5', micro=''). Now parsed properly, with any non-numeric tail (dev1+g1234, rc1) going to releaselevel. An unparsable version degrades to VersionInfo(0, 0, 0, ...) instead of raising, so import faust can no longer fail on it.

This changes a public value. Code reading faust.version_info.minor as the version string must use faust.__version__, which is unchanged. Nothing in the repo consumes it and no docs reference it.

on_window_close never got its window data

_del_old_keys read _partition_timestamp_keys with the whole (start, end) tuple as the second key element; the map is keyed on range_end alone, written that way by _maybe_set_key_ttl and read that way by _maybe_del_key_ttl. The lookup could never hit, so triggered_windows was always [None, ...] and on_window_close only ever saw the raw per-key value.

User-visible: applications with an on_window_close handler will start receiving the aggregated data the API always promised.

Recovery applied events to whichever table came last

_slurp_changelogs binds table/offsets/bufsize per TP. The else: branch for an untracked TP only logged a warning and fell through, so the event was applied using the previous iteration's bindings — or raised UnboundLocalError if it was the first event. Now skipped. Worth noting a bare continue would be wrong: _maybe_signal_recovery_end() and the standby bookkeeping at the bottom of the loop must still run every iteration, so only the application block is skipped.

Separately, detect_aborted_tx compared await consumer.position(tp) >= highwater unguarded. position is Optional[int] and does return None, raising TypeError — swallowed by the caller's except Exception, which then skipped the aborted-transaction fixup for every remaining partition.

require_app = False was unusable

_app_from_str returns None for such a command invoked without -A, then AppCommand.__init__ did key_serializer or self.app.conf.key_serializer unconditionally. self.app is now a property over an Optional[AppT], with on_stop and blocking_timeout no longer assuming an app. Behaviour with an app present is unchanged, and require_app = True still raises the same UsageError.

confluent driver

key_partition reached for list_topics on ProducerThread.producer — the Faust producer — rather than ._producer, the confluent handle. And Consumer.verify_event_path delegated to a method no thread class defines, so the livelock detector raised every tick; it now has the same documented no-op the base Consumer.verify_event_path already is. A real livelock implementation is separate work.

Verification

mypy -p faust clean, scripts/check clean, suites 2207 → 2234 passed, 4 skipped.

Every new test was checked against the pre-fix code by overlaying the test files onto a pristine tree built from git archive. All fail there except the two noted below.

Two honest caveats:

  • The confluent fixes are unverified by execution. tests/unit/transport/drivers/test_confluent.py opens with pytest.importorskip("confluent_kafka"), and confluent-kafka is the optional faust[ckafka] extra that the CI test environment does not install. The tests are written and will run where the extra is present; here they are skipped. Those two fixes rest on reading the class definitions.
  • tests/unit/cli/test_base.py::test_init__serializers_without_app passes against pre-fix code too — both serializers it passes are truthy, so the old code short-circuits before dereferencing the None app. It documents the case but is not a guard; test_init__no_app_when_not_required is the one that actually fails pre-fix.

No existing test was weakened or deleted. The only removed test lines are an unused import, three mock_ranges calls replaced with explicit (start, end) tuples (the mock returned bare floats, which is not what _window_ranges yields), and two mock-setup lines that pointed at the very attribute the confluent fix corrects.

🤖 Generated with Claude Code

claude added 6 commits August 6, 2026 20:06
`SSLCredentials.__init__` defaulted `purpose` to None and passed it straight
into `ssl.create_default_context(purpose=...)`, which begins

    if not isinstance(purpose, _ASN1Object):
        raise TypeError(purpose)

So `SSLCredentials()` and `SSLCredentials(cafile=...)` -- any call that does not
supply an explicit `context` -- raised `TypeError: None`.  The class could only
ever be constructed by handing it a context built elsewhere, which defeats the
cafile/capath/cadata parameters entirely.

Default to `ssl.Purpose.SERVER_AUTH`: the same default `create_default_context()`
itself applies, and the correct one for a client verifying a broker.  It implies
`check_hostname=True` and `verify_mode=CERT_REQUIRED`, which the tests assert.
An explicitly passed `purpose` is still forwarded unchanged.

Found by the type checker in #758 and marked `XXX` there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
`VersionInfo` declares `major`, `minor` and `micro` as ints, but the module
splatted the regex groups `(prefix, version, suffix)` into them positionally:

    VersionInfo(major=None, minor='0.11.5', micro='')

So `faust.version_info.major` was the `'v'` prefix or None, `.minor` was the
whole version string, `.micro` was the suffix, and `.releaselevel` was always
None -- every field wrong except by accident.

Parse the dotted numbers properly instead, putting any non-numeric tail
(`dev1+g1234`, `rc1`, a local segment) into `releaselevel`.  Missing components
pad with zero, and an unparsable version degrades to `VersionInfo(0, 0, 0, ...)`
rather than raising, so `import faust` can never fail on the version string --
the old `RuntimeError('THIS IS A BROKEN RELEASE!')` branch is gone with it.

This changes a public value, which is why #758 left it marked `XXX` rather than
fixing it: code reading `faust.version_info.minor` as the version *string* must
switch to `faust.__version__`, which is unchanged.  Nothing in the repo consumes
it and no docs reference it.

`_parse_version` is injected into the lazy module's `__dict__` so it is
reachable for testing; it is private and stays out of `__all__` and `dir()`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
`Collection._del_old_keys` read `_partition_timestamp_keys` with

    self._partition_timestamp_keys.get((partition, window_range))

where `window_range` is the whole `(start, end)` tuple.  The map is keyed
`(partition, range_end)` -- an `(int, float)` pair -- written that way by
`_maybe_set_key_ttl` and read that way by `_maybe_del_key_ttl`.

So the lookup could never hit.  `triggered_windows` was always `[None, ...]`,
`window_data` stayed empty, and `on_window_close` was only ever handed the raw
per-key value instead of the aggregated window data it exists to receive.

Use `(partition, window_range[1])`, matching the writer.

This is user-visible: applications with an `on_window_close` handler will start
receiving the aggregated data the API always promised.  That is why #758 marked
it `XXX` instead of fixing it.

Two existing tests relied on `mock_ranges` returning bare floats, which is not
what `_window_ranges` yields; they now pass real `(start, end)` tuples.  Their
assertions are unchanged and the ranges still match nothing in
`_partition_timestamp_keys`, so they continue to cover the untriggered path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
Two bugs in `Recovery`, both found by the type checker in #758.

`_slurp_changelogs` classifies each event's TP as active or standby and binds
`table`, `offsets` and `bufsize` accordingly.  The `else:` branch for a TP that
is neither only logged `"recovery unknown topic"` and fell through -- so the
event was applied using the *previous* iteration's bindings: written into an
unrelated table's buffer and offset map, and passed to that table's
`on_changelog_event`.  On the first event of the loop there is nothing bound
yet, so it raised `UnboundLocalError` instead.

Skip applying an event for an untracked TP.  Note a bare `continue` would be
wrong: the statements at the bottom of the loop body -- `_maybe_signal_recovery_end()`
and the standby-ready bookkeeping -- must keep running on every iteration, or
recovery-end signalling loses a trigger.  Only the event-application block is
skipped.

`detect_aborted_tx` compared `await self.app.consumer.position(tp) >= highwater`
unguarded.  `ConsumerT.position` is `Optional[int]` and does return None when a
partition has no position yet, so that raised TypeError -- swallowed by the
caller's `except Exception`, which then silently skipped the aborted-transaction
fixup for every remaining partition in the loop.  Skip a TP with no position.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
`_app_from_str` returns None for a `require_app = False` command invoked
without `-A` -- that is the documented escape hatch, and `faust completion` is
exactly such a command.  `_finalize_app` handed that None straight back, and
`AppCommand.__init__` then did

    self.key_serializer = key_serializer or self.app.conf.key_serializer

unconditionally, so the command died with
`AttributeError: 'NoneType' object has no attribute 'conf'`.  The escape hatch
was unusable: `faust completion` could not run without the `-A` it is written
not to need.

Make `AppCommand` tolerate having no app.  `self.app` becomes a property over
an `Optional[AppT]`, the serializer defaults fall back to None when there is no
app, and `on_stop` and `blocking_timeout` no longer assume one.  Behaviour with
an app present is unchanged, and a command with `require_app = True` still gets
the same `UsageError` from `_app_from_str` as before.

Found by the type checker in #758 and marked `XXX` there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
`Producer.key_partition` called `self._producer_thread.producer.list_topics()`.
`ProducerThread.producer` is the *Faust* Producer; the confluent_kafka handle is
`ProducerThread._producer`.  Faust producers have no `list_topics`, so the
method raised AttributeError and was dead on arrival.  Read the confluent
producer instead.

`Consumer.verify_event_path` delegated to `self._thread.verify_event_path(...)`,
but neither `ConsumerThread` nor `ConfluentConsumerThread` defines it, so the
commit-livelock detector (`_commit_livelock_detector` ->
`verify_all_partitions_active`) raised AttributeError on every tick.  Add the
no-op to `ConfluentConsumerThread`, matching the documented no-op stub the base
`faust.transport.consumer.Consumer.verify_event_path` already is.  This makes
livelock detection inert for this driver rather than raising -- a real
implementation is separate work.

Both were found by the type checker in #758 and marked `XXX` there.

Note these tests do not run here or in CI: tests/unit/transport/drivers/test_confluent.py
starts with `pytest.importorskip("confluent_kafka")`, and confluent-kafka is the
optional `faust[ckafka]` extra, which the CI test environment does not install.
The fixes are verified by reading the class definitions, not by an executed test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012N2yysiNbzzVcvYhtrpsM7
@wbarnha
wbarnha force-pushed the claude/faust-latent-bug-fixes branch from 14601cb to f3b2097 Compare August 6, 2026 20:08
@wbarnha
wbarnha changed the base branch from claude/faust-mypy-compat-xyb41h to master August 6, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants