TX degrades across warm sessions: high-order constellations die, only a VBUS power cycle recovers (a drone cannot power-cycle in flight)

[josephnef]

Summary

Repeated devourer sessions on one adapter progressively degrade high-order
constellation TX while leaving the robust rates untouched. Only a VBUS
power cycle recovers it — libusb_reset, teardown/re-init, and idling do
not.

This is a real deployment bug, not just a bench annoyance: an airborne link
cannot power-cycle its adapter. A drone that restarts its TX process, or runs
a long session, silently loses its top rates with nothing in any log to say so.

Why it is dangerous

It is not a wedge, and that is what makes it easy to misdiagnose. Throughout
the degradation the adapter:

• keeps enumerating and accepting frames;
• reports every frame submitted, zero failures;
• keeps delivering the robust rates at full ratio;
• reports strong RSSI and clean EVM on the frames that do decode (because
  only the low-order frames decode, and those are clean).

The signature is indistinguishable from "the link is too weak to carry a denser
constellation". In the investigation that surfaced this, it produced a
completely coherent but wrong conclusion — a peaked delivery-vs-power curve,
the same ceiling on an empty channel (ch14), and the same ceiling across three
different receivers — all of it an artifact of accumulated chip state.

Reproduction

tests/warm_tx_degradation_repro.sh (added alongside this report). It holds one
ground receiver up for the whole run so the reference never moves, then
alternates {probe} and {N warm devourer sessions} on the DUT, and finishes with
two controls.

sudo REGRESS_VBUS_MAP="0bda:8812=3-2.3.4,3;2357:012d=10,2" \
     tests/warm_tx_degradation_repro.sh

Two independent runs, RTL8812AU TX -> RTL8822BU ground, ch6, 20 MHz, constant
TX config:

stage	MCS7 (64-QAM 5/6)	MCS1 control
cold baseline	63.3% / 62.6%	97.7%
after 3 warm sessions	58.1% / 58.1%	97.2%
after 6 warm sessions	56.3% / 56.6%	97.3%
after 9 warm sessions	57.4% / 55.2%	97.1%
after 12 warm sessions	56.3% / 55.1%	97.2%
after 60 s idle, no power cycle	57.6% / 59.6%	97.0%
after VBUS cold cycle	64.0% / 61.9%	97.1%

The control rate is flat at ~97% the whole way, so this is not ambient drift,
and the recovery is what confirms it: only removing power restores the rate.

The severe form (observed, not yet isolated)

The table above is the mild, reliably reproducible form: ~7 points over 12
sessions. During the original investigation the same adapter reached a far worse
state — MCS5, MCS6 and MCS7 all at exactly 0% delivery while MCS0–MCS4 kept
working normally at ~88%. A VBUS cycle restored MCS7 to 58% at the same channel,
power and inter-frame gap.

That state followed roughly a hundred sessions that also included band switches
(2.4 -> 5 GHz), bandwidth switches (20/40 MHz), flat TX-power index overrides,
and large power offsets in both directions. Plain repetition at fixed
config does not reproduce it, and neither does max-power/max-duty repetition
(WARM_PWR=63 WARM_GAP=0, which the script supports) — that gives the same ~7
point curve. Isolating the aggravating factor is the main open question.

Investigation leads

• The selectivity is the clue. The datapath, USB, descriptors and MAC are
  demonstrably fine — the low rates ride the exact same path at full delivery.
  What degrades is analog transmit quality, i.e. EVM. That points at
  calibration state (IQK / LCK / DPK) or power/thermal tracking, not DMA.
• Not simple thermal. 60 s idle does not recover the plain form. The
  max-power variant recovers partially on idle (59.6% vs 55.1%), suggesting a
  thermal component stacked on top of a persistent state component.
• Prior art, likely the same root cause. The RTL8814AU "post-use
  degradation" wedge (chip stops delivering after a few devourer runs; only
  VBUS recovers; libusb_reset and idle do not) looks like the severe end of
  this same axis. A proposed deinit-before-init fix was reverted as unvalidated
  because the warm gate never fired — devourer issues libusb_reset before
  claim, so the chip never presents as "already on". Worth re-examining with
  the constellation-selective probe here, which is a far sharper signal than
  "wedged / not wedged".
• Candidate fix direction. If it is calibration state, an explicit
  de-init (vendor card_disable_flow) or a forced re-calibration on bring-up
  should recover it without removing power. That is the outcome that matters
  for the field: a software-only recovery path, since the drone cannot
  power-cycle.

Impact

• Long-running or restart-prone field links silently lose their top rates.
• Any bench measurement of a rate/modulation ceiling is invalid unless it
  cold-cycles per cell — it otherwise measures accumulated chip state. This is
  now documented in CLAUDE.md and enforced by default in
  tests/nitroqam_waterfall.sh.

Chips

Characterized on the RTL8812AU (Jaguar1). The 8814AU shows the probable severe
variant. Unknown on Jaguar2/Jaguar3/Kestrel — the repro script is
chip-agnostic (DUT_VID/DUT_PID) and running it per generation is the first
concrete task.

[josephnef]

Root cause found: chip temperature, not latched state

Instrumented the repro with the chip thermal meter, RSSI and EVM per probe before changing any code. One run settled it — RTL8812AU, ch6/20 MHz, constant TX config:

stage	MCS7 (64-QAM)	MCS1 control	thermal	RSSI
cold baseline	83.0%	98.0%	43	69
after 24 max-duty sessions	71.3%	98.0%	53	69
120 s idle, chip still powered	70.8%	98.3%	49	69
after VBUS cold cycle	79.2%	97.7%	44	69

RSSI never moves — no power is lost, only signal quality, and the thermal meter tracks the loss one-for-one.

And the reason only VBUS appeared to recover it: a Jaguar1 session left the chip in ACT with the RF front end live indefinitely, so idling never cooled it. 120 s of idle bought 4 thermal units; a 12 s power cycle bought 9. Jaguar2 resets at init and Jaguar3 card-disables at teardown — Jaguar1 did neither.

Also retires the 'stale BB TX-swing index' theory from the leads list: 0xc1c/0xe1c are rewritten unconditionally at every init by phy_SetBBSwingByBand_8812A, which then re-seeds the power-tracking baseline. Not stale.

Fix in #350

Stop() + destructor run a card-disable power sequence. The chip now cools whenever no session owns it:

stage	before	after
60 s idle, no power cycle	thermal 48, MCS7 73.0%	thermal 40, MCS7 85.8%

Why this issue stays open

1. The severe form is unreproduced. Worst observed here is thermal 53 costing ~12 points. The originally reported state — MCS5/6/7 all at 0% while MCS0–4 held ~88% — did not reappear even at max power and max duty. Same mechanism further along, or something else? Unresolved.
2. SIGKILL still leaves the chip powered and heating — neither Stop() nor the destructor runs. Not fixable in-library; an init-side power-off buys no cooling because it runs immediately before power-on.
3. No in-flight recovery is possible for a thermal mechanism. Measured: recovery needs ~60 s of not transmitting. A call that restores the rate by killing the link for a minute is not a recovery for an airborne link. The lever for a long flight is thermal-aware operation — DEVOURER_THERMAL_POLL_MS, watch the rising delta, back off SetTxPowerOffsetQdb or duty. Deliberately not shipped as an API.
4. Jaguar2 audit inconclusive — its thermal meter did not move (33–34) and the available link was too marginal at MCS7 to resolve a small effect. It remains the one generation leaving the chip powered at exit.

Per-chip: 8812AU is where the effect lives (thermal 43→53). The 8814AU barely heats at the same duty (34→37) and shows no degradation — the effect scales with how hot the part gets.

[josephnef]

Correction: retracting the thermal root cause

My previous comment claimed the root cause is chip temperature. That was correlation across sessions presented as causation. Retracting it.

Every recovery that works removes power, and removing power resets chip state and lets the die cool. No power-cycle experiment separates those, so none of the earlier evidence was proof. Two new harnesses try to separate them, and both argue against cooling:

tests/thermal_offtime_sweep.sh — degrade the part, then power it down for 5/15/30/60/120 s before an otherwise identical probe. The reset is bit-identical in every arm; only cooling time varies.

off(s)	MCS7	thermal
5	83.1%	47
15	63.2%	45
30	64.9%	43
60	68.7%	41
120	72.4%	40

Temperature falls cleanly with off-time; delivery does not follow it at all. The best result comes from the shortest off-time and the hottest chip.

tests/thermal_causation_probe.sh — delivery inside a single uninterrupted session (one bring-up, no re-init, no state change, so only temperature can move). Thermal stayed pinned through 240 s of continuous max-duty TX while MCS7 drifted down ~10% and the MCS1 control held flat.

And a measurement-validity problem

tests/probe_repeatability.sh — ten identical back-to-back probes, nothing changed between them:

	mean	sd	range
MCS7/20	67.3%	1.8	65.0 – 70.7
MCS1/20 control	98.0%	0.2	97.5 – 98.3

A single probe is worth ±3 points. The ~7-point 'decay curve' I originally reported in this issue is only marginally outside that band, so the mild form is much weaker evidence than I presented it as. Anything under ~4 points needs repetition.

What still stands

• Jaguar1 left the chip in ACT with RF live indefinitely after process exit — real, and fixed in Jaguar1: power the chip down at teardown (mechanism open; thermal explanation retracted) #350.
• The fix has a large measured effect: a 60 s idle recovers MCS7 from 73.0% to 85.8% only once the teardown power-down exists (~7 sd). Before it, idling recovered nothing.
• The severe form (MCS5/6/7 at 0% while MCS0–4 held ~88%) remains unreproduced.

What this issue now needs

The mechanism. Candidates not yet separated: latched state cleared by card-disable, something in the analog/RF domain that a re-enumeration does not touch, or a thermal effect with a time constant the on-die meter does not represent. A proper attack needs temperature varied independently of power state — external cooling or heating with the session live — which the bench cannot currently do.

[josephnef]

The transmitter is not what breaks — reframing this issue

Continued investigation. The headline: every measurement in this issue, including the original report, inferred a transmitter defect from a single devourer↔devourer link. Judged by an independent receiver the transmitter is healthy under every stressor tried.

Reproduced the severe form — then lost it

tests/severe_form_hunt.sh adds exactly one stressor per arm on top of an identical session loop, 3 probes per measurement (noise is ±3):

arm	cold	after	delta	MCS1 ctrl
baseline	62.0	51.5	−10.5	98.2	mild
band (2.4↔5 GHz)	32.2	0.0	−32.2	97.5	HIT
bandwidth	1.2	0.7	−0.5	94.0	—
spur (ch13/14)	1.8	1.4	−0.4	97.4	—
power extremes	1.9	7.1	+5.2	98.0	—
failed 5 GHz tune	0.7	0.0	−0.7	97.8	—

That is the severe form: 64-QAM at zero, robust rate untouched. It did not survive a second receiver.

The transmitter, measured alone

Minutes later, the same "dead" transmitter against an AR9271 (different vendor, no shared silicon/firmware/driver) and the vendor 88XXau driver on the same dongle:

rate	devourer	vendor
MCS4	63.8%	56.0%
MCS7	62.5%	69.8%

So tests/band_stressor_sniffer.sh re-ran the winning stressor with the sniffer as judge — the transmitter alone. Eight band-switching sessions: MCS7 decoded 3721 → 3915. No degradation.

The receive side

tests/ground_rx_degradation.sh recovers the receiver in steps, leaving the TX alone: fresh ground / aged ground / RX process restart / RX VBUS / TX VBUS (control). On 8812AU→8822BU, MCS7 sits at 0.1–0.4% at every step while MCS1 holds 95–98%. Nothing recovers it — not restarting the RX process, not power-cycling either adapter.

Where that leaves this issue

• The transmitter is exonerated for every stressor tried, by two independent judges.
• What is broken lives on the receive side of the devourer↔devourer link, and is not identified.
• Not excluded: ambient. Co-channel traffic on ch6 rose from ~19 to ~58 frames/3s across the session, which hurts 64-QAM far more than BPSK. A ch14 control was inconclusive — that link is fully dead there (MCS1 = 0 too), not rate-selectively.

I am claiming no mechanism. Two were published in this issue and both were retracted; this round's contribution is tooling that can tell the two ends apart, plus the finding that the transmitter should stop being the suspect.

Also added

examples/chipstate reads a chip's registers with no USB reset and no bring-up — the only way to inspect what a session left behind, since every other entry point reconfigures the chip on the way in. It also had to force its own teardown power-down off, or it destroyed the state it had just read. It exposes two things nothing could see before: the TX IQ correction (0xcc4/0xcc8/0xccc/0xcd4 + 0xec*, on BB page C1 — a plain read returns the wrong bank) and RF 0x58 LOK, neither of which is rewritten by a bring-up. Those remain the best candidates if a real TX-side latch ever is demonstrated.

[josephnef]

Answer: no — and it is not a devourer bug

Asked whether the band arm's 32.2% → 0.0% collapse could be fixed. It cannot, because it is not ours. The ground station's receiver was dying, and it fails under the vendor kernel driver too.

tests/rx_vendor_ab.sh (new) holds the transmitter constant and changes only the receiver's driver on the same dongle — the one control never run, because the vendor driver had only ever been used as a transmit control:

rate	devourer RX	vendor RX (88x2bu_ohd)
MCS7/20	34/5862 = 0.6%	94/5845 = 1.6%
MCS1/20	5834/5936 = 98.3%	5152/5922 = 87.0%

Neither driver decodes 64-QAM on that 8822BU; both decode BPSK fine. Cross-checks at the same time, same transmitter:

• 8814AU as ground: MCS7 82.9%
• AR9271 sniffer: MCS7 62%
• transmitter vs vendor driver on the TX side: 62.5% vs 69.8%

So: transmitter healthy, two other receivers fine, this one receiver dead at 64-QAM under both drivers. The adapter's receive path is degraded.

What that means for this issue

That single adapter was the ground station for nearly every measurement here. It is behind the whole trail of wrong answers — the "warm-session TX degradation", the thermal correlation, and the band-switch "root cause". All were measured through a receiver that was progressively losing its dense constellations, which is exactly the signature the issue set out to explain.

#348 as written — a transmitter that degrades and only recovers on a power cycle — is not supported by any evidence that survives an independent receiver. I would not close it outright: the original field report may be a real and different thing. But it needs re-reporting from a rig whose ground station has been validated, because nothing measured so far can be trusted.

Shipped so the next attempt cannot repeat this

• rx_vendor_ab.sh — vendor A/B on the receive side.
• severe_form_hunt.sh now recovers both ends before every arm (cycling only the DUT let receive-side decay accumulate and surface as a per-arm delta), and ends with a drift canary: re-measure the first arm's cold baseline with both ends recovered as at the start. If it moved more than the noise floor the run exits non-zero and says the reference shifted. On the run that produced the "hit", the arms' own cold baselines had slid 62 → 32 → 1.2 → 1.8 → 1.9 → 0.7.
• ground_rx_degradation.sh, band_stressor_sniffer.sh, probe_repeatability.sh, examples/chipstate.

Branch: jaguar1-latched-state-investigation. No fix PR — there is no devourer defect demonstrated to fix.

[josephnef]

Retracting "the adapter is degraded" — it's the antennas, and that explains everything

The T3U (8822BU) has internal antennas; the 8814AU I compared it against has two external ones. That is several dB of receive gain, and it puts the T3U's modulation cliff in the middle of the rates under test. Same transmitter, same channel, minutes apart:

ground	MCS3	MCS4	MCS5	MCS6	MCS7
Archer T3U (internal)	97.8	98.3	79.1	49.0	2.9
RTL8814AU (external ×2)	80.6	80.2	80.8	80.6	80.4

A textbook waterfall rolling off through 64-QAM, against a receiver that is flat with margin to spare. Both adapters are healthy.

My "degraded" call was under-determined: the vendor driver failing identically on the T3U is predicted just as well by a link-budget shortfall as by hardware damage. And power doesn't rescue it either — at RSSI ~67 the front end is already near-field saturated, so no offset from −5 to +10 dB moved MCS7 while MCS1 held 98%. Per-chain telemetry also shows both RX chains alive (RSSI [68,70], SNR [30,28]), so it isn't a dead chain.

This is the mechanism behind the entire issue

A receiver perched on its cliff swings between fine and zero on a few dB of ambient, while the robust rates stay pinned. That is indistinguishable from a transmitter that degrades and recovers — which is exactly the shape of the original report. Earlier in this same session, with lower ambient, that same T3U delivered MCS7 at 68%. The cliff never moved; the operating point did.

Every wrong answer in this issue was measured through that receiver: the "warm-session TX degradation", the thermal correlation, the band-switch "root cause", and my "degraded adapter". One marginal ground station, four mis-attributions.

Ship

tests/ground_station_qualify.sh — sweeps the ladder on a candidate ground station and refuses the pairing (exit 1) when the test rate is off the flat part. Self-tested against the T3U: NOT QUALIFIED for MCS7/20, 2.8% vs 98.1% best on its own ladder. Documented in CLAUDE.md beside the noise-floor rule.

Recommendation for this issue

No devourer defect is demonstrated, and the evidence base is now known to be invalid. I'd suggest re-reporting from a rig where:

1. the ground station passes ground_station_qualify.sh at the rate of interest (prefer external-antenna adapters for high-MCS work),
2. a second independent receiver corroborates, and
3. probe_repeatability.sh establishes the noise floor first.

If the effect survives all three it is real and worth chasing. Nothing measured so far survives even the first.

[josephnef]

Closing as unreproducible — the reported behaviour was a measurement artifact, not a devourer defect. Full post-mortem merged in #351 (09819b6, docs/warm-tx-degradation.md).

Root cause of the reports: the ground station was a TP-Link Archer T3U — an RTL8822BU with internal antennas — whose modulation cliff sits inside the rates under test. Two RTL8822BU adapters, identical silicon and code path, same transmitter and channel, differing only in antennas:

ground station	MCS1	MCS4	MCS5	MCS6	MCS7
Archer T3U — internal	98.1	98.3	79.1	49.0	2.9
CF-924AC V2 — external ×2	95.8	95.8	95.7	94.4	95.9

A receiver on its cliff swings between fine and zero on a few dB of ambient while the rates below stay pinned — indistinguishable from a transmitter that degrades and recovers. Four successive root causes (latched chip state, thermal, band switching, degraded adapter) all came from that one instrument.

Guard against recurrence: tests/ground_station_qualify.sh refuses a pairing (exit 1) when the test rate is off the flat part of the receiver's ladder, plus probe_repeatability.sh (noise floor), rx_vendor_ab.sh (vendor driver on the receive side) and the rest of the harnesses in #351.

Whether the original field report describes something real is left open. Nothing measured here is evidence either way — it was all taken through an instrument that could not support the conclusion. Reopen with a re-report from a rig where the ground station passes ground_station_qualify.sh at the rate of interest, a second independent receiver corroborates, and the noise floor has been established first.

One genuine bug did survive the better instrument: the 40 MHz RX gap, filed separately.