Qualify the receiver before believing a delivery number (#348 post-mortem; corrects two #349 measurements) - #351
Merged
Conversation
A Jaguar1 session left the chip in the ACT power state with its RF front end live, indefinitely, long after the process that brought it up had exited. Nothing else in the tree does that: Jaguar2 resets at init, Jaguar3 has run a full card-disable at teardown for a long time. The cost was a transmitter that quietly lost its dense constellations. Across back-to-back sessions on an 8812AU at ch6/20 MHz, constant TX config, the chip thermal meter climbed 43 -> 53 and MCS7 (64-QAM 5/6) delivery fell 83% -> 71% while the MCS1 control held at 98% and RSSI never moved — no power is lost, only signal quality. Nothing logs an error, and the signature is indistinguishable from a link too weak to carry the rate. Idling did not help, because a chip held in ACT keeps dissipating whether or not it is transmitting: 120 s of idle bought 4 thermal units where a 12 s VBUS cycle bought 9. That is why only pulling power appeared to recover it. Stop() and the destructor now run HalModule::rtw_hal_deinit(): halt the MAC engines (REG_CR, REG_RCR) then the die's card-disable power sequence, through the same HalPwrSeqCmdParsing and the same three-way chip dispatch InitPowerOn already uses. The sequence tables have been sitting in hal/Hal88*PwrSeq.c since the port began with nothing calling them. Both call sites are best-effort; a chip already off the bus makes the writes fail, which is fine at teardown. Measured effect, same repro: 60 s of idle with no power cycle went from thermal 48 / MCS7 73.0% to thermal 40 / MCS7 85.8%. An ordinary gap between sessions now restores the part completely without touching the hardware. Deliberately not included. There is no in-flight recovery API: the mechanism is heat, recovery needs the radio to stop transmitting for around a minute, and a call that restores the rate by killing the link for a minute is not a recovery for an airborne link. An init-side power-off was also rejected — it would run immediately before power-on and buys no cooling time, while adding risk to a bring-up path that works. Jaguar2 keeps its current teardown: its thermal meter did not move across the same run and the link available was too marginal at MCS7 to resolve a small effect, so there is no measurement to justify changing it. Validated on hardware: 8812AU, 8814AU and 8821AU all re-init cleanly across three consecutive sessions after the power-down; teardown_gen_sanity 10/10; tx_teardown_asan 6/6 including max-duty kills, which is the case that matters now that teardown writes registers while URBs may be in flight. The 8814AU barely heats at this duty (34 -> 37) and shows no degradation to fix. Also flips a documented wart: a beacon loaded into the MAC reserved page now stops when the session ends instead of airing until re-enumeration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… refuted it The previous commit asserted that warm-session TX degradation is caused by chip temperature. That was correlation across sessions presented as causation, and two experiments added here argue against it: thermal_offtime_sweep.sh degrades the part, then powers it down for 5/15/30/60/ 120 s before an otherwise identical probe. The reset is bit-identical in every arm; only cooling time varies — which is the one way to separate "the die cooled" from "the state was reset", since every recovery observed does both at once. Temperature fell cleanly with off-time (47 -> 40) while delivery scattered 63-83% with no relation to off-time or temperature: the best result came from the shortest off-time and the hottest chip. thermal_causation_probe.sh measures delivery inside a single uninterrupted session — one bring-up, no re-init, no state change, so only temperature can move. The thermal meter stayed pinned for 240 s of max-duty TX while MCS7 delivery still drifted ~10% and the control held flat. probe_repeatability.sh is the measurement that should have come first: ten identical back-to-back probes, nothing changed between them, give MCS7/20 sd 1.8 and a 5.7-point spread (MCS1 control 98.0 +/- 0.2). A single probe is worth about +/-3 points, so effects under ~4 points are not findings — and several plausible-looking decay curves in this investigation sat inside that band. Run it before believing a delivery difference. What survives: the fix itself. A 60 s idle recovers MCS7 from 73.0% to 85.8% only once the teardown power-down exists — ~13 points, about 7 sd, comfortably real — where before it idling recovered nothing. And the change is right on its own terms regardless of mechanism: leaving a radio powered and transmitting-capable after its owning process has exited is wrong, and is what every other generation already avoids. What does not: any claim about why. Heat is a real correlate of delivery within one probe sequence (thermal 39 -> 45 costing ~5 points, control flat, RSSI unchanged) and demonstrably not the driver in the two experiments above. The mechanism is open, and the docs now say so instead of asserting it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…invisible ones Investigating #348 needs to answer "what state did the last session leave this chip in?", and until now nothing could: every entry point reconfigures the chip on the way in. claim_interface_then_reset re-enumerates it, Init/InitWrite re-run the power sequence, reload the BB/RF/MAC tables and re-calibrate. The act of looking destroyed the evidence. examples/chipstate opens, claims, does NOT reset, constructs the device (chip identity resolves from SYS_CFG2 at construction, no bring-up needed), dumps and exits. It also forces tuning.teardown_power_down=0 for itself — otherwise the device destructor would card-disable the chip on the way out and the tool would destroy the state it just read, one look and gone. Verified: two consecutive dumps of the same chip are byte-identical. --init runs a normal bring-up first, for a healthy reference on a freshly power-cycled adapter. Two register groups join the canary set because they are exactly the state that could explain the bug and exactly the state nothing could see: - TX IQ correction (0xcc4/0xcc8/0xccc/0xcd4 and the 0xec* mirrors), written by IQK via FillTxIqc. Nothing in array_mp_8812a_phy_reg rewrites them, so unlike everything else in the dump they are NOT re-established by a bring-up — whatever the last calibration left is what transmits. - RF 0x58, the LOK the IQK loads from RF 0x08[19:10]. Not in the RF table, not in IQK's own backup/restore. Both damage TX EVM when wrong, which kills the dense constellations while the robust rates ride through — the exact signature in #348. The TX IQC registers live on BB page C1 (0x82c[31]=1), so a plain read returns whichever bank happens to be selected; the first version of this dump read them as zero for that reason. They are now read deliberately with the bank snapshotted and restored, and reported under their own BBC1 kind so a diff cannot silently compare across banks. On a live 8812AU that turns 0x000000 into path A X=0x1FC/Y=0x04, path B X=0x203/Y=0x0F — converged calibration values, not the 0x200/0x0 the failure path writes. tuning.teardown_power_down (DEVOURER_TEARDOWN_POWER_DOWN=0|1, default on) gates the card-disable added earlier. A powered-down chip answers every register read with the CARDEMU fill, so post-mortem inspection needs it off — and it is the A/B lever for measuring what that power-down is actually worth, which is still unestablished. PowerOff() now reads back 0x0005 and REG_CR after the sequence and logs them rather than trusting the return value, so the teardown path is provably not inert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three experiments, run in this order, each answering the one the previous made unavoidable. severe_form_hunt.sh adds exactly one stressor per arm on top of an identical session loop — band switch, bandwidth switch, ch13/14 spur, power extremes, failed 5 GHz tunes — so a collapse names its cause instead of just happening. Measured over a devourer->devourer link it reported the #348 severe form at last: the band arm took MCS7 from 32.2% to 0.0% with the MCS1 control at 97.5%. That HIT did not survive contact with a second receiver. The vendor A/B, judged by an AR9271 that shares no silicon or code with either side, showed the same "dead" transmitter delivering MCS7 at 62.5% against the vendor driver's 69.8% — minutes after the hunt scored it at zero. band_stressor_sniffer.sh therefore re-runs the winning stressor with the sniffer as judge, so the verdict is about the transmitter alone. Eight band-switching sessions moved the sniffer's MCS7 count from 3721 to 3915. No degradation. The transmitter is not what breaks. ground_rx_degradation.sh isolates the other end by recovering the receiver in steps while leaving the transmitter alone: fresh ground, aged ground, RX process restart, RX power cycle, and finally a TX power cycle as the control. On the 8812AU->8822BU pair MCS7 sits at ~0.1-0.4% at every step while MCS1 holds at 95-98%, and none of the recoveries move it — including power-cycling both ends. The conclusion this forces is uncomfortable and is the point: delivery is a property of a link, and every measurement in #348 so far — including the original report — inferred a transmitter defect from a single devourer<->devourer link. Judged independently the transmitter is healthy under every stressor tried. What is actually broken lives on the receive side of that link, and remains unidentified; ambient co-channel traffic on ch6 also rose from ~19 to ~58 frames/3s across the session, which hurts 64-QAM far more than it hurts BPSK, so an environmental explanation is not yet excluded either. No mechanism is claimed here. These scripts exist so the next attempt starts from measurements that can distinguish the two ends. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…driver too
The band-switch arm's 32-point collapse was not a transmitter defect and not a
devourer defect. It was the ground station's receiver dying mid-run.
tests/rx_vendor_ab.sh holds the transmitter constant and changes only the
RECEIVER's driver on the same dongle. On the 8822BU that had been the ground
station for nearly every measurement in this investigation:
MCS7/20 devourer 34/5862 = 0.6% vendor 94/5845 = 1.6%
MCS1/20 devourer 5834/5936 = 98.3% vendor 5152/5922 = 87.0%
Neither driver decodes 64-QAM on it; both decode BPSK fine. At the same time the
same transmitter is decoded at MCS7 by an 8814AU ground at 82.9% and by an AR9271
sniffer at 62%, and matches the vendor driver on the transmit side. So the
adapter's receive path is degraded, not our code — the one conclusion that had
never been tested because the vendor driver had only ever been run as a
TRANSMITTER control.
That single dying adapter is behind a long trail of wrong answers in this issue:
a "warm-session TX degradation", a thermal correlation, and finally a
band-switch stressor that looked like a clean root cause. All of them were
measured through it.
So severe_form_hunt.sh gains the two things that would have caught it:
- both ends are recovered before every arm, and the ground process restarted
with them. Cycling only the DUT let receive-side decay accumulate across the
whole run and surface as a per-arm delta.
- a drift canary: re-measure the first arm's cold baseline at the very end,
with both ends recovered exactly as at the start. Nothing in the run should
move it. If it moved by more than the noise floor the reference shifted
underneath every arm, the run exits non-zero, and the report says so instead
of presenting differences between two different rigs as stressor effects.
On the run that produced the "hit" the arms' own cold baselines had slid
62 -> 32 -> 1.2 -> 1.8 -> 1.9 -> 0.7.
Also fixes rx_vendor_ab.sh's transmit helper to sleep-then-SIGTERM rather than
letting `timeout` kill txdemo: the final tx.stats carries the frames-sent
denominator and is only emitted on a clean signal, so a slow bring-up silently
turned a measured cell into a divide-by-zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retracting "the 8822BU's receive path is degraded". It is not. The TP-Link Archer T3U has INTERNAL antennas; the 8814AU it was compared against has two external ones. That is several dB of receive gain, and it puts the T3U's modulation cliff in the middle of the rates under test. Same transmitter, same channel, minutes apart: T3U (8822BU, internal) MCS3 97.8 MCS4 98.3 MCS5 79.1 MCS6 49.0 MCS7 2.9 RTL8814AU (external x2) MCS3 80.6 MCS4 80.2 MCS5 80.8 MCS6 80.6 MCS7 80.4 A textbook waterfall rolling off through the 64-QAM rates, against a receiver that is flat with margin to spare. Both adapters are healthy. The vendor driver failing identically on the T3U was never evidence of damage — a link-budget shortfall predicts exactly that, and so does hardware failure. Raising TX power does not rescue it either: at RSSI ~67 the front end is already near-field saturated, so no offset from -5 to +10 dB moved MCS7 while MCS1 held at 98%. This is the mechanism behind the whole investigation. A receiver perched on its cliff swings between fine and zero on a few dB of ambient while the robust rates stay pinned — indistinguishable from a transmitter that degrades and recovers, which is precisely the shape of the original report. Earlier in the same session, with lower ambient, that same T3U delivered MCS7 at 68%. The cliff never moved; the operating point did. tests/ground_station_qualify.sh sweeps the ladder on a candidate ground station and refuses the pairing (exit 1) when the test rate is off the flat part, with the reason spelled out. Self-tested against the known-marginal T3U: NOT QUALIFIED for MCS7/20, 2.8% against 98.1% best on its own ladder. Recorded in CLAUDE.md next to the noise-floor rule, because both are about the same thing: knowing whether a measurement can support the claim before making the claim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The numbers in this doc were taken through a TP-Link Archer T3U, whose internal
antennas put its modulation cliff in the middle of the rates under test. A
Comfast CF-924AC V2 — the SAME RTL8822BU silicon, same code path, but with two
external antennas — is flat at 94-96% across MCS1..MCS7 where the T3U rolls off
to 2.9%. Re-measured against it, two of this doc's claims were wrong:
"2 spatial streams above MCS2 read zero on this bench, two dongles in near
field give the streams too little spatial decorrelation"
-> wrong. VHT2SS MCS0 95.3%, MCS4 90.8%, MCS7 15.5%, all modulation-verified.
That was the receiver's cliff, not spatial decorrelation.
"VHT1SS_MCS8 (256-QAM) delivered 295/4675"
-> understated 12x. 77.1% on a qualified receiver, 37/37 frames decoded as
vht1ss_mcs8.
So the 256-QAM result is considerably stronger than published, and the 2SS
ceiling is MCS7 rather than MCS2.
What did NOT change is the 40 MHz gap: rxdemo with DEVOURER_BW=40 still delivers
zero on a qualified receiver, including the most robust 40 MHz rate. That
survives the better instrument, so it is a real receive-path gap rather than a
margin artifact — the claim is now properly evidenced instead of merely repeated.
The remaining gap to the 2x2 headline is 256-QAM on two streams (VHT2SS_MCS8,
0.6% where 2SS MCS7 reaches 15.5%) — a ladder rolling off, i.e. link margin,
not a property of the mode.
Doc now opens with the receiver-qualification requirement and the two-adapter
comparison that motivates it, because every figure in it depends on that.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… post-mortem The teardown power-down's only quantitative justification — a 60 s idle recovering MCS7 delivery from 73.0% to 85.8% — was measured through the ground station that turns out to have been the source of every wrong answer in this investigation. It cannot support the claim, so the claim is gone from the code comment, the subtree CLAUDE.md and the doc. The change itself stands on architecture alone, which was always the stronger argument: Jaguar1 was the only generation leaving the chip in ACT with its RF front end live indefinitely after the owning process exited, and an autonomously-airing beacon kept transmitting. docs/warm-tx-degradation.md is rewritten as what it actually is — a post-mortem of four successive wrong root causes (latched chip state, thermal, band switching, degraded adapter) that all had one origin: a receiver whose modulation cliff sat inside the rates under test. Two RTL8822BU adapters, identical silicon and code, differing only in antennas: Archer T3U (internal) MCS5 79.1 MCS6 49.0 MCS7 2.9 CF-924AC V2 (external x2) MCS5 95.7 MCS6 94.4 MCS7 95.9 The root CLAUDE.md thermal bullet is narrowed to the part that survived: the correlation is real, it is a trap, and a power cycle changes chip state and temperature together so it can never separate them. Whether the original field report describes something real is left explicitly open. Nothing measured here is evidence either way, because it was all taken through an instrument that could not support the conclusion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #348 (as unreproducible — see below). Corrects two measurements published
in #349.
What #348 turned out to be
The reported bug — an RTL8812AU whose 64-QAM rates die across sessions while
robust rates stay perfect, recovering only on a power cycle — was a
measurement artifact. The ground station was a TP-Link Archer T3U, an
RTL8822BU with internal antennas, whose modulation cliff sits inside the
rates under test.
Two RTL8822BU adapters. Identical silicon, identical code path, same
transmitter, same channel, minutes apart. Only the antennas differ:
A receiver on its cliff swings between fine and zero on a few dB of ambient
while the rates below stay pinned — indistinguishable from a transmitter that
degrades and recovers. In one session that same T3U gave MCS7 68%, later 0.6%;
the cliff never moved, the ambient did.
Four successive root causes were wrong, all from this one instrument:
latched chip state (refuted by reading the registers — the BB swing is rewritten
every init), thermal (refuted by an off-time sweep varying only cooling time,
and by a within-session probe where the meter is pinned while delivery drifts),
a band-switching TX defect (a real 32-point collapse over the devourer link —
but an AR9271 sniffer showed the same stressor moving decoded MCS7 from 3721 to
3915, i.e. not at all), and a degraded adapter (the vendor driver failed
identically, which a link-budget shortfall predicts just as well).
Corrections to #349
Both were measured through the T3U and are wrong:
chip" → wrong. On a qualified receiver: VHT2SS MCS0 95.3%, MCS4
90.8%, MCS7 15.5%, all modulation-verified. That was the receiver's cliff.
77.1%, 37/37 frames decoded as
vht1ss_mcs8.So the 256-QAM-on-2.4 GHz result is materially stronger than published and the
2SS ceiling is MCS7, not MCS2.
What did not change:
DEVOURER_BW=40still delivers zero on a qualifiedreceiver, including the most robust 40 MHz rate. That gap survives the better
instrument, so it is real and now properly evidenced rather than merely repeated.
The guard, so this cannot recur
tests/ground_station_qualify.shsweeps the ladder on a candidate groundstation and refuses the pairing (exit 1) when the test rate is off the flat
part. Self-tested against the T3U: NOT QUALIFIED for MCS7/20, 2.8% vs 98.1%
best on its own ladder.
Plus the harnesses each wrong answer needed to kill it:
probe_repeatability.shrx_vendor_ab.shband_stressor_sniffer.shground_rx_degradation.shsevere_form_hunt.shthermal_offtime_sweep.shthermal_causation_probe.shCode changes
examples/chipstate— reads a chip's registers with no USB reset and nobring-up, the only way to inspect what a session left behind. It forces its
own
teardown_power_down=0or it destroys the state it just read (verified:two consecutive dumps byte-identical). Canary set extended with the TX IQ
correction (
0xcc4/0xcc8/0xccc/0xcd4+0xec*, on BB page C1 — a plain readreturns the wrong bank) and RF
0x58LOK; neither is rewritten by a bring-upand neither was visible before.
Stop()/destructor runHalModule::rtw_hal_deinit()(halt MAC engines, then the die's card-disablesequence, using tables that sat unused in
hal/Hal88*PwrSeq.csince the portbegan). Jaguar1 was the only generation leaving the chip in ACT with RF live
after process exit. No performance claim attached — the delivery recovery
an earlier revision cited came from the unqualified T3U. It also makes a
stray hardware beacon stop at session end (a documented wart).
tuning.teardown_power_down(DEVOURER_TEARDOWN_POWER_DOWN=0|1) — apowered-down chip reads back the CARDEMU fill, so post-mortems need it off.
PowerOff()reads back0x0005/REG_CRinstead of trusting a return code.Behaviour change
A beacon in the MAC reserved page now stops when a Jaguar1 session ends instead
of airing until re-enumeration. Matches Jaguar3 and the vendor driver.
Testing
ctest48/48.teardown_gen_sanity.sh10/10.tx_teardown_asan.sh6/6 underASan including max-duty kills (teardown now writes registers with URBs in
flight). 8812AU/8814AU/8821AU each re-init cleanly across consecutive sessions
after the power-down.
On closing #348
Whether the original field report describes something real is unknown and
deliberately left open in the doc. Nothing measured is evidence either way — it
was all taken through an instrument that could not support the conclusion. A
re-report is worth taking seriously from a rig where the ground station passes
ground_station_qualify.sh, a second receiver corroborates, and the noise flooris established first.
🤖 Generated with Claude Code