v2.0 Weightslab - #298
Merged
Merged
Conversation
…files, heading levels
- PRs now formatted as [#N](url) title — date with author GitHub profile links
- Contributors section links to github.com/{login} (from PR authors)
- Commits capped at 25 most recent non-merge commits
- Title: ## **Weightslab** (no version, ## level)
- Sections demoted to ### level
- Removed separator between title and LinkedIn/Graybx links
- Dev release routes PRs from --base dev, main from --base main
- Doc build gated on main-branch check (not just tag pattern)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, bb_B], batch_idx=[batch_idA, B, ..etc]
* Add to detection usecase dump history and custom signal labelling * upgrade documentation with new functions and examples * add and fix utests * refactor the logger and add instances history and queries functions, with a user wl.write_history function * add df writing for user during exp * fix code quality issues
* Guarantee no model-internal interactions unless asked; light=True by default
New `light` kwarg on `ModelInterface.__init__`, default `True`. When
light=True the wrap skips `init_attributes` (the shallow `vars(model)`
iteration that created class-level property forwarders), the
architecture-change hook, and the `CheckpointManager` auto-load block
(which would otherwise call `load_state_dict(strict=True)` on a
discovered checkpoint). Also forces `compute_dependencies=False`.
Result: zero traversal or mutation of the wrapped model. Retains only
what's needed for consistency and metrics — device placement, ledger
registration, `guard_*_context.model = self` binding, `hp_config` read,
and the `get_age` / `tracking_mode` / `set_tracking_mode` methods
inherited from `NetworkWithOps`.
Opt out with `light=False` to enable model surgery, attribute
forwarding, and checkpoint auto-load.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Ultralytics harmonization: example ladder + WL integration helper
The integration ladder for blending Ultralytics YOLO idioms with WeightsLab,
plus the helper module that supports the destination interface.
Files (all in examples/PyTorch/ws-detection/src/):
ul10_wl00_main.py pure Ultralytics anchor (zero WL).
ul08_wl02_main.py + read-only WL data inspection during vanilla
`model.train()`. No signal capture, no edits.
ul07_wl03_main.py + per-sample signal capture (loss + IoU) via UL
callbacks. Still no edits, no manual loop.
ul06_wl04_main.py reserved for the edits rung (watch_or_edit on model
/ optimizer / hparams). Stub.
ideal_main.py destination — imperative, top-level `watch_or_edit`
calls only. No session, no context manager. Atexit
handles the silent join so `wl.keep_serving()` is
no longer needed.
wl_ultralytics.py the helper module: `attach(model)` installs UL
callbacks (deferred to `on_train_start`), and a
dispatch around `wl.watch_or_edit` routes YOLO
instances through `attach` so callers write
`model = wl.watch_or_edit(model)` symmetrically with
the other registrations. Atexit on first `attach`
keeps the studio backend alive after training ends.
Built on top of the `light=True` default from `light-mode-default`: the
model wrap guarantees no model-internal interactions unless
`light=False` is explicitly passed.
Local-only branch — not for pushing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Trajectory checkpoint: ul05_wl05 + ul06_wl04, wl_ultralytics consolidated
Consolidation:
* wl_ultralytics.py absorbs utils/criterions.py + utils/data.py contents.
Becomes the single canonical home for per-sample loss / IoU / detection
metrics, YOLODatasetWL + collate, load_config, attach, and the
wl.watch_or_edit YOLO dispatch. Env defaults also live here so they
run before weightslab is imported by this module.
* utils/criterions.py and utils/data.py become thin re-export shims —
main.py keeps working unchanged.
ideal_main.py reaches its minimal destination shape: 17 LOC, three
wl.watch_or_edit calls + YOLO + serve + train.
Two new convergence rungs:
* ul05_wl05_main.py — one step from main.py toward ideal: keeps the
DetectionTrainer subclass shell, but deletes the manual train() and
do_validate() overrides. UL's natural training drives; WL listens via
callbacks installed in __init__.
* ul06_wl04_main.py — one step from ideal_main toward verbose
(sketch, not verified): no subclass, wl.watch_or_edit(model) dispatch
unfolded into explicit add_callback() lines, edits-rung opt-in
visible (light=False on the model wrap), env defaults inline.
ul07_wl03_main.py and ul08_wl02_main.py updated to import dataset
helpers from wl_ultralytics directly.
Local-only branch — not for pushing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* WL light-mode fixes + minimal YOLO integration that runs end-to-end
Three bugs fixed in WL that blocked the YOLO integration trajectory:
1. backend/ledgers.py — Proxy.get(ref, default) now returns the plain
default value when `ref` is not in the underlying mapping. Previously
it wrapped the default in a _ValueProxy unconditionally, which broke
UL's `YAML.save(args)` ("cannot represent ValueProxy") when callers
used `cfg.get("epochs", 100)` on a hparams-wrapped dict.
2. backend/model_interface.py — ModelInterface.__init__ device check
now normalizes via `th.device(d).type == 'cuda'`. The previous
exact-string `device == 'cuda'` check silently dropped
`torch.device('cuda:0')` / `torch.device('cuda')` to CPU, which
moved the wrapped model off the GPU and caused later device
mismatches.
3. backend/model_interface.py — `_apply` and `train` overrides on
ModelInterface so `.to / .half / .float / .cuda / .cpu / .train /
.eval` reach `self.model`. `self.model` is intentionally kept in
`__dict__` (custom __getattr__ relies on it), so nn.Module's
default submodule recursion misses it; the overrides propagate
explicitly.
wl_ultralytics.attach() is now minimal:
* Dataset wrap, optimizer wrap.
* Model wrap with `forced_model_wrapping=True` (avoids stale Proxy
re-use from prior runs) — only for ledger handle + age counter.
* pause_controller.resume + @wl.eval_fn.
Per-sample loss / IoU / detection-metric emission is removed for now —
UL's DetectionTrainer doesn't expose `trainer.preds`, so capturing
per-batch predictions needs a forward hook on the underlying model.
Deferred follow-up.
End-to-end smoke test against TrespassColor (epoch 1, batch 4, imgsz
1024): training + validation both complete; validator reports
P=0.839 R=0.747 mAP50=0.815 mAP50-95=0.394 (reasonable for a
pretrained yolo11s warmup).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ul05_wl05: first step from main.py toward ideal — verified end-to-end
Make ul05_wl05_main.py functional. Same overall shape as before (thin
DetectionTrainer subclass, no manual loop, callbacks installed in
__init__) but aligned with the working integration pattern from
wl_ultralytics.attach:
* `forced_model_wrapping=True` on the model wrap (avoids stale Proxy
from prior runs hosting weights on the wrong device).
* Per-sample loss/IoU emission dropped — UL's DetectionTrainer doesn't
expose `trainer.preds`; capturing per-batch outputs needs a forward
hook on the underlying model. Deferred follow-up.
* Drop `workers=0` — with workers=0 the main-process dataloader
iteration sees our `loader.dataset.__class__ = YOLODatasetWL` swap,
but UL's default collate expects the original YOLODataset's dict
output (not YOLODatasetWL's tuple). UL's default workers fork the
dataset before on_train_start runs, so the swap is invisible to
them — sidesteps the collate mismatch.
Smoke-tested against TrespassColor (epoch 1 of 1000 partial run):
training + validation complete; P=0.787 R=0.671 mAP50=0.727
mAP50-95=0.444 for a yolo11s warmup.
Trajectory status:
main.py (manual loop, verbose)
-> ul05_wl05 (subclass shell + callbacks, UL natural training) ✅
-> ul06_wl04 (no subclass, explicit callbacks, edits opt-in) sketch
-> ideal_main (clean, dispatch hides callbacks) ✅
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ul05_wl05: capture per-sample signals via forward hook + preprocess patch
UL's DetectionTrainer doesn't store `trainer.preds` or `trainer.batch` —
the batch is a local variable in `_do_train`'s loop body, and predictions
flow through DetectionModel.forward without being stashed on the trainer.
Hook two surfaces to recover what we need:
* Forward hook on the raw DetectionModel: DetectionModel.forward(x)
routes by input type. With a batch dict it returns (loss, loss_items);
with an image tensor (recursive call inside .loss()) it returns raw
preds. Both fire per training step; we keep only the prediction call.
* Patch `trainer.preprocess_batch` and `validator.preprocess` to stash
the device-prepared batch into shared state. UL keeps batch local in
the loop, so the preprocess wrap is our only handle from callbacks.
`on_train_batch_end` / `on_val_batch_end` then call the per-sample
`PerSampleDetectionLoss` (bbxs / clsf / dfl) and `PerSampleIoU` channels
with the captured (preds, batch). PerSampleDetectionLoss is built against
the raw DetectionModel (v8DetectionLoss does `model.model[-1]` which
needs the raw class, not our wrapper).
Smoke test against TrespassColor: 237 train batches + full validation
both complete with no callback errors (P=0.787 R=0.671 mAP50=0.727
mAP50-95=0.444 from UL's aggregated metrics). Per-sample WL signals
flow through 6 channels per split (train/val × bbxs/clsf/dfl) plus
miou/{split} on every batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ul05_wl05: revert false per-sample signal capture
The previous commit (2c66ea2) claimed per-sample signal emission worked
end-to-end. It does NOT — though the script ran without an unhandled
exception (we silently dropped errors in a try/except), the actual
WL `add_scalars` call was never reached:
* forward hook captured preds correctly via the monkey-patched
`underlying.forward` (the register_forward_hook didn't work because
DetectionModel.loss() calls self.forward(image) directly, bypassing
__call__ and forward hooks).
* preprocess_batch / validator.preprocess wraps captured batch.
* BUT — PerSampleDetectionLoss.forward, called with our captured
(preds, batch), crashes inside UL's v8DetectionLoss internals on
`batch["cls"].view(-1, 1)` with "Type must be a sub-type of ndarray
type". The PerSample* classes were designed for main.py's manual
loop and don't slot onto UL 8.4.51's training-mode dict pred output
once the per-sample slicing has happened.
So nothing was actually logged to WL. Reverting to a clean shell that
runs end-to-end with no per-sample emission and no false promises.
Plumbing for capture (forward-patch + preprocess-patch) was removed
too — it'd need to come back when we have a working per-sample
computation surface. That'll likely require either:
(a) decoding preds ourselves consistently with main.py's flow, or
(b) hooking AFTER UL's criterion runs to read its per-anchor loss
tensor and reduce per-sample (still inside the hook idiom).
Per-sample work is deferred; the trajectory file is honest again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ul05_wl05: per-sample cls + batch box/cls/dfl + aggregated val via hooks
Implement option-1 MVP from the integration-hook-don't-recompute memo.
No UL method is overridden; signal capture rides on:
* forward_hook(criterion.bce) → per-anchor cls tensor of
shape (bs, num_anchors, nc).
Reduced via .sum(dim=(1,2))
gives per-sample cls.
* trainer.loss_items → batch-level (box, cls, dfl)
scalars already aggregated
by UL; just read and emit.
* validator.metrics.results_dict → aggregated val scalars
(precision/recall/mAP50/
mAP50-95/fitness) read at
on_val_end.
`_Sink(nn.Module)` is a passthrough that lets pre-computed values flow
through `wl.watch_or_edit(_Sink(), flag="loss"|"metric", ...)`'s normal
signal-logging pipeline. We never recompute anything UL already did.
Criterion is created eagerly in on_train_start (normally lazy on first
.loss() call) so the bce hook is in place before any batch fires.
Channels registered:
train/cls_per_sample (per_sample=True), train/box, train/cls, train/dfl,
val/precision, val/recall, val/mAP50, val/mAP50-95, val/fitness.
Verified end-to-end:
* Channels register without error.
* On each batch end, per-sample cls vals are real and distinct per
sample (e.g. [235.8, 183.7, 209.2, 223.8]).
* loss_items match UL's epoch progress line.
* UL's natural training and validation complete.
Known infrastructure issue (NOT in our code): WL's DATAFRAME_M Proxy
target is never set ("Failed to apply data: Proxy target not set" at
session start), and `save_signals` → `DATAFRAME_M.enqueue_batch`
silently no-ops. So values reach `LoggerQueue.add_scalars` in memory
but never persist to h5. Investigation in a follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add weightslab.integrations.ultralytics SDK module
New `WLAwareTrainer` + `WLAwareDataset` two-name surface replaces the
hand-wired bridge under `examples/PyTorch/ws-detection/src/`. The trainer
routes both train and val loaders through `wl.watch_or_edit(flag='data',
loader_name=...)` so each split gets a disjoint uid range from the
global counter — fixes the silent uid-namespace collision where val's
positional uids overwrote train rows in the shared ledger.
Per-sample signals: train cls/box/dfl + live NMS overlay (conf=1e-4 so
early-training overlays are non-empty); val IoU + AP@0.5 + post-NMS
overlay.
WL core changes motivated by this integration:
* src.py: drop `get_active_sample_mask` + per-sample masking in
wrappered_fwd. Per-sample discard is enforced at the data sampler
now (deny-aware sampler excludes discarded samples from batches),
so the post-hoc signal mask was a redundant second filter that hid
true loss values from per-sample logging.
* src.py (post-merge): import `get_active_origin` directly to fix
`NameError: name 'global_monitoring' is not defined` introduced by
the dev merge.
* backend/model_interface.py: getattr guard on
`_checkpoint_auto_every_steps` — light-mode wrapping (used by
WLAwareTrainer via forced_model_wrapping=True) binds the method to
a model that never ran __init__.
* components/checkpoint_manager.py: restore `model.criterion` in a
finally block after architecture save — `install_per_sample_signals`
hooks crit.bce + crit.get_assigned_targets_and_loss, and the save
path was leaving them orphaned.
Example cleanup:
* `main.py` now uses `WLAwareTrainer` directly (~80 LOC, no bridge).
* `main_explicit.py` preserved as the pre-SDK hand-wired version for
reference.
* Removed throwaway ladder: ul0{5,6,7,8,10}_*_main.py, ideal_main.py,
wl_ultralytics.py.
* `config.yaml`: load_model=false + dump_model_architecture=false to
preserve model identity across restarts (no model surgery here).
UI package markers: restore `ui/__init__.py`, `ui/docker/__init__.py`,
`ui/envoy/__init__.py` — dropped by a prior merge, causing
ModuleNotFoundError on `weightslab ui docker se`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Simplify integrations.ultralytics: tap, don't recompute
signals.py: 162 → 104 LOC; _utils.py: 145 → 24 LOC.
What's still shipped:
* train/cls_per_sample — forward hook on `crit.bce`, sum(1,2) per image.
Pure reduction over the (bs, na, nc) tensor UL already computes.
* train preds overlay — forward hook on `Detect` head captures the raw
training-mode preds dict (UL training skips decoding). We then call
UL's own `Detect._inference` + `non_max_suppression` for decode/NMS.
UL's code path, no rewrite.
* val preds overlay — wrap `validator.update_metrics`; `preds` is
already NMS'd by UL. Pure tap.
* Aggregate curves (train box/cls/dfl, val P/R/mAP/fitness) — taps in
`WLAwareTrainer` callbacks over `loss_items` and `results_dict`.
What was dropped vs. the prior bridge:
* `_utils.preds_for_overlay`, `_scatter_per_image` — inlined where needed.
* `_utils._mini_ap`, `per_sample_iou_post_nms`, `per_sample_map50_post_nms`
— were reimplementing IoU + 11-point AP that UL's validator already
computes per image in `_process_batch`. To recover per-sample IoU/AP,
tap that method instead.
* `bbox_loss.forward` override that reimplemented UL's box+dfl math.
Per-sample box/dfl can be recovered via pre-hook on `bbox_loss` +
forward hook on `dfl_loss` + UL's own `bbox_iou` — no math rewrite.
* `Sink` class — replaced with `torch.nn.Identity` (same behavior).
* `enable_train_overlay` / `nms_conf_thres` / `nms_iou_thres` knobs on
`WLAwareTrainer` — no longer needed; train overlay is always on and
uses tiny NMS conf so early-training overlays aren't empty.
Cleanup also reverts unrelated changes that snuck into the prior commit:
* `backend/model_interface.py`, `components/checkpoint_manager.py` —
drop the light-mode hardening (out of scope for this PR).
* `data/dataframe_manager.py` — was a docstring-only change tied to a
deleted helper; not worth pushing on its own.
* `examples/.../config.yaml` — out of scope (local dev config).
* `examples/.../client_discard_test.py` — local test artifact, untrack.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Restore per-sample box/dfl + val IoU; drop light-mode kwarg
signals.py: add the four per-sample signals dropped in the prior cleanup
back via taps over UL — no math reimplementation:
* train/box_per_sample — `bbox_loss` pre-hook captures fg_mask +
target_scores; a tap on `ultralytics.utils.loss.bbox_iou` captures
the per-fg-anchor IoU UL computes inside its own forward (one
reference assignment, no extra call). Form `(1-iou)*weight` and
scatter per image.
* train/dfl_per_sample — `bbox_loss.dfl_loss` forward hook captures
the per-fg-anchor DFL tensor; multiply by the same weight UL uses
and scatter per image.
* val/iou_per_sample — wrap `validator._process_batch` (UL calls it
per image inside `update_metrics`); use UL's own `box_iou` once to
derive a per-image mean-of-max-IoU-per-GT scalar.
Per-image mAP@0.5 is NOT restored — UL doesn't expose per-image AP and
recovering it would require reimplementing the precision/recall curve,
which is exactly what we just removed.
model_interface.py: revert the `light=True` kwarg added in 1e17662 —
the integration uses `forced_model_wrapping=True`, not `light=True`,
so the light-mode work is unrelated to this branch and the prior commit
was carrying it by accident.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Revert integration-side WL fixes — keep branch focused on SDK only
ledgers.py, model_interface.py, checkpoint_manager.py had lingering
non-SDK fixes from earlier exploratory commits on this branch:
* Proxy.get returns plain default when key missing (UL YAML serialize)
* ModelInterface device normalization for 'cuda:N' / torch.device
* ModelInterface._apply + train overrides to propagate to self.model
* CheckpointManager th.load map_location, param-fingerprint log,
captured missing/unexpected keys from load_state_dict
All real fixes, but not part of the integrations.ultralytics SDK push —
restoring to origin/dev so this branch only carries the SDK changes.
Saved locally as patch for re-application in a dedicated PR:
~/wl_integration_side_fixes_20260608.patch
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Revert config.yaml — it's local dev config, not for this PR
Saved local dirty version to /tmp/cfg_dirty.yaml so it can be restored
on disk after the commit (without going back into git).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Drop main_explicit + ui/__init__ markers from branch (kept locally)
Two batches of files were carrying over on this branch but are
orthogonal to the detection SDK push:
* examples/PyTorch/ws-detection/src/main_explicit.py — the old
pre-SDK explicit bridge, preserved as reference. Not needed for
the SDK PR itself; the SDK example lives in main.py.
* ui/__init__.py, ui/docker/__init__.py, ui/envoy/__init__.py —
empty package markers that fix a separate ModuleNotFoundError on
`ui docker se` (their absence breaks UI launch). Belongs in its
own PR, not this one.
Both saved as a patch for re-application in a dedicated PR:
~/wl_branch_extras_20260608.patch
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Rename: explicit version takes back main.py; SDK version → main_ul_native.py
* main.py — now holds the explicit hand-wired bridge (pre-SDK), which is
the version dev currently ships under this name. Reclaims its original
identity in the example tree.
* main_ul_native.py — the WLAwareTrainer-based example that uses UL's
native YOLO().train() entry point with `trainer=WLAwareTrainer`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* src.py: revert to dev + minimal import fix
Restore origin/dev's src.py verbatim, then add a single import:
+from weightslab.components import global_monitoring
so the pre-existing `global_monitoring.get_active_origin()` call in
`wrappered_fwd` (line 485 in dev) resolves at runtime. Without this
the path NameErrors the first time UL signals fire without an explicit
origin kwarg — which is what tripped us up on the dev merge.
Reverts the per-sample-mask removal and `get_active_sample_mask`
deletion from this branch — those are an architectural cleanup
("discard at the sampler, not at the loss") that belongs in its own
PR, not bundled with the SDK introduction.
`get_active_group_mask` stays untouched: group-level discard still
goes through wrappered_fwd in dev, and reorganizing that path is the
same separate architectural conversation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals.py Shape B + load-bearing fixes for SDK to actually run
signals.py: refactored into composable primitives + declarative records.
* Capture primitives: fwd_hook, pre_hook, fn_tap, per_call_buffer —
each returns a zero-arg getter for the captured value, closures bind
it into a signal's reduce/preds.
* Signal dataclass: (name, flag, reduce, preds=None). Reduce is a
function from batch → (B,) tensor or None.
* Orchestrators install_train_pipeline / install_val_pipeline are
~15 LOC each; they own the sync-point wrap and ship loop.
* Default packs default_train_signals(model) / default_val_signals(validator)
return list[Signal]; back-compat wrappers install_per_sample_signals /
install_per_sample_val_signals delegate to the pipeline.
* Adding signal N+1 is now appending one Signal(...) record + a small
closure. No more 80-LOC imperative paragraph edits.
* _overlay_dict: move tensor to CPU BEFORE the scale-divide — fixes a
cross-device RuntimeError when training on cuda.
trainer.py: pass compute_dependencies=False on the model wrap call
(architecture-change is a future-release feature in dev and raises if
left default).
backend/model_interface.py: three load-bearing fixes for forced model
wrapping — without these the SDK doesn't even run a single step.
* _apply override: propagate .to / .half / .cuda / .cpu to self.model
(which lives in self.__dict__, not self._modules, so nn.Module's
default recursion misses it).
* train override: same reason, for train/eval mode propagation.
* Device normalization accepts 'cuda:N' / torch.device — previous
exact 'cuda' string match silently dropped non-'cuda' to CPU.
main_ul_native.py: amp=False workaround. With AMP on, UL's autocast
scope doesn't see through ModelInterface to apply FP16 casts on weights;
input ends up FP16 while weights stay FP32. Disabling AMP is a clean
workaround until we work out how to surface autocast through the wrap.
client_discard_test.py: rewrote the discard scenario test to verify
per-sample SIGNAL semantics in addition to last_seen. After discard:
- last_seen must NOT advance for discarded samples (sampler skip), AND
- per-sample signal values (train box/cls/dfl, val iou) must NOT
change (WL never overwrites discarded rows behind the sampler).
Sorts top-N by train/box_per_sample for train_loader and by
val/iou_per_sample for val_loader; discards both, waits ≥300 post-
discard steps, then asserts both invariants on both splits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Untrack client_discard_test.py — local-only repro
The discard scenario test is useful for local verification but not part
of the SDK PR. Keeping it untracked on disk; users who want it can pull
from this commit's parent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Revert main.py to dev — SDK example is its own file, don't replace the old one
main.py stays untouched on dev; the SDK demo lives in main_ul_native.py.
Local backup of the explicit-bridge version kept at:
~/main_explicit_backup_20260608.py
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: wrap _ship_round in torch.no_grad — capture is observational
The train overlay path runs UL's `Detect._inference` + `non_max_suppression`
on captured raw_preds DURING training, while autograd is still recording.
The intermediate tensors that decode+NMS allocate were getting added to the
backward graph and kept alive until backward, compounding across steps and
causing OOM at any non-trivial batch size on an 8GB GPU.
Wrap the whole `_ship_round` (reduce + preds + channel ship) in
`torch.no_grad()` — signal capture and overlay are pure observation, no
gradients needed. Lets the SDK fit at batch 16 instead of being OOM-bound
to batch 4.
Verified end-to-end with the discard-scenario client:
* 15/15 train victims frozen on last_seen AND per-sample signals
(cls/box/dfl) over 2 full post-discard epochs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* dataset: extract _to_six_col helper
Both `fast_get_label` and `get_items(include_labels=True)` were assembling
the same `[x1, y1, x2, y2, class_id, confidence=1.0]` row from different
xyxy sources (PIL-fallback letterbox math vs UL's pipeline output). Pull
the assembly out as `_to_six_col(xyxy, cls)` — saves the duplicated empty
check + concatenate + dtype dance and makes the "target = bboxes + cls +
confidence=1" semantic explicit. ~10 LOC less; the rest (UL-pipeline-decode
vs letterbox-recompute) is the actual difference between the two paths.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: wrap DFLoss in __call__ proxy — forward hook never fires on it
DFLoss in ultralytics.utils.loss overrides __call__ directly instead of
going through nn.Module.__call__ → forward(). That means our
`fwd_hook(bl.dfl_loss)` never receives the call: forward hooks only fire
when Module.__call__ dispatches into them, but the override bypasses that
path entirely.
Wrap the bound dfl_loss instance in a thin `_DFLossTap` that:
* `__getattr__` proxies to the original (so UL's
`self.dfl_loss.reg_max` reads still work);
* `__call__` invokes the inner instance and captures its output for
the per-sample DFL signal.
Without this, `train/dfl_per_sample` was silently never written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: symmetric primitives + val pipeline order fix
Two fixes:
1. Add `method_call_tap(obj, attr)` primitive — captures the return value
of a callable attribute whose `__call__` doesn't route through
`nn.Module.__call__` (so forward hooks never fire). UL's `DFLoss` is
exactly this case: it inherits from `nn.Module` but overrides
`__call__` directly, so the bce-style `fwd_hook(bl.dfl_loss)` never
fires and `train/dfl_per_sample` was silently never written.
With this, `default_train_signals` reads symmetrically — each
capture uses the right primitive for the kind of callable it wraps:
get_bce = fwd_hook(crit.bce) # plain nn.Module
get_iou = fn_tap(ul_loss, "bbox_iou") # plain function
get_dfl = method_call_tap(bl, "dfl_loss") # __call__-override module
get_bl_args = pre_hook(bl)
Replaces the inlined `_DFLossTap` class from the previous commit.
2. Val pipeline ran `_ship_round` BEFORE the original `update_metrics`,
but `_orig(preds, batch)` is exactly what fires `_process_batch` per
image — i.e. what fills our IoU buffer. We were shipping an empty
buffer every cycle and `val/iou_per_sample` had zero entries in the
ledger. Swap the order: original first, ship second.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: method_call_tap — bypass nn.Module._modules guard
`bl.dfl_loss` is a registered submodule on BboxLoss, so
`nn.Module.__setattr__` rejects swapping it for a non-Module proxy.
Pop the child out of `_modules` first, then write the proxy into
`__dict__` directly. Attribute lookup still resolves to our `_Tap`
via Module.__getattr__ falling back to __dict__.
Was crashing the trainer at `on_train_start` with `TypeError: cannot
assign '_Tap' as child module 'dfl_loss'`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: cap overlay to top-10 per image by confidence
Train NMS uses conf_thres=1e-4 so early-training overlays aren't all
empty; UL's `non_max_suppression` then caps at `max_det=300` per image.
That floods the studio with hundreds of low-confidence boxes that drown
out the few high-confidence ones.
Post-filter in `_overlay_dict`: sort by conf column, keep top-10. Single
constant `_OVERLAY_TOPK` controls it. Applies to both train and val
overlays since both go through `_overlay_dict`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: collapse val signals + add WL_PROFILE / WL_CORRECTNESS gates
* Merge val/preds_per_sample (constant-zero carrier) into val/iou_per_sample
by attaching the overlay to the iou signal. Studio histogram now shows
a real distribution for val; one fewer ghost channel.
* WL_PROFILE=1 — log avg signal-ship wall time every 50 steps so we can
see overlay/NMS cost vs total step time.
* WL_CORRECTNESS=1 — every 50 steps log sum|w| of trainer.model and
trainer.ema, plus ||ema - model||_2. Sanity-check that:
- model_sum_abs changes (optimizer is stepping)
- ||ema - model||_2 grows then plateaus (EMA actually updating off
the wrapped model)
Both opt-in via env var; zero cost when off.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals/trainer: WL_PROFILE / WL_CORRECTNESS cadence to 50
10-step cadence was the debug-session setting; 50 is calmer if the env
var is left on accidentally. Same data, less noise.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals/trainer: lean cleanup for review
Three focused cleanups:
* Drop WL_PROFILE / WL_CORRECTNESS instrumentation. They were useful
for the verification pass (model_sum_abs changing → optimizer steps;
EMA diverging → EMA updates; ~3% ship overhead) but they bloat the
file and shouldn't ship in the SDK. Re-add ad-hoc when needed.
* Tighter defensive checks in `default_train_signals`:
- Drop the "if detect_head is None" guard in `overlay_p` — instead
only attach the overlay to the cls Signal when a Detect head was
found at install. One less branch per training step.
- Drop the broad try/except in `overlay_p`. Real bugs in
`_inference` + NMS should surface, not be silently swallowed.
- cls_r: skip the redundant `bce is None` check — `crit.bce` is
invoked every training step's loss, so by the time `_ship_round`
runs the BCE hook has fired.
- box_r/dfl_r: drop the inner `iou is None` / `dfl is None`
checks — the `_fg_state() is None` guard already covers the
"bbox_loss was skipped because fg_mask was empty" case, which
is the only path where the iou/dfl taps can fail to fire.
- val overlay_p: drop the `_wl_preds is None` check; the val
pipeline stashes preds unconditionally before calling reducers.
* Rename `_OVERLAY_TOPK = 10` → `OVERLAY_MAX_DETS = 50` (module-public
constant). Both train and val overlays share the cap. 50 is a
visual-readability bound; UL's NMS otherwise produces up to 300 boxes
per image, which floods the studio. Exposing as `OVERLAY_MAX_DETS`
so users can override at module level rather than digging into
private internals.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: expose overlay NMS thresholds as module constants
Train overlay's NMS used hardcoded `conf_thres=1e-4` and `iou_thres=0.45`.
Promote both to module-level constants alongside `OVERLAY_MAX_DETS`, with
a docstring explaining why we override UL's `model.args.{conf,iou}`
(those are unset during training).
OVERLAY_CONF_THRES = 1e-4 # tiny so early-epoch overlays aren't empty
OVERLAY_IOU_THRES = 0.45
OVERLAY_MAX_DETS = 50
Users can rebind at module level (`signals.OVERLAY_CONF_THRES = 0.05`)
without subclassing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: overlay NMS thresholds inherit from `model.args`
`overlay_p` was hardcoding `conf_thres=1e-4, iou_thres=0.45`. Now it
reads `model.args.conf` / `model.args.iou` first — i.e. whatever the
user passed to `YOLO().train(conf=..., iou=...)`. The two module-level
constants are renamed to `OVERLAY_CONF_FALLBACK` / `OVERLAY_IOU_FALLBACK`
and kick in only when `model.args.{conf,iou}` is None (which is the
case during training because UL only auto-populates those for predict).
`OVERLAY_MAX_DETS` stays a module constant — it's a studio-readability
display cap (50 boxes per image), not a model hyperparameter.
`_overlay_nms_thresholds(model)` is the single lookup point.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: revert NMS .cpu() workaround
Was added to dodge missing-CUDA-backend in some torchvision wheels.
Cost was ~2x train step time. Proper fix is matched torch+torchvision
install (`pip install torch torchvision --index-url
https://download.pytorch.org/whl/cuXXX`), not a permanent CPU bounce
for everyone.
Keeping the shape-guard in `box_r` / `dfl_r`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* signals: factor staleness check into `_fresh` + root-cause docstring
Same shape-guard logic as before, with the cause from Guillaume's
investigation named: validator.py runs `model.loss` on the EMA model,
which dispatches through the same module-level `ul_loss.bbox_iou` tap
and refreshes our IoU cache — but it's the EMA's bbox_loss that ran,
not the train instance's, so the train pre-hook stays stale.
When the next train step has zero foreground, bbox_loss is skipped
entirely (UL guards with `if fg_mask.sum()`), so the train pre-hook
also stays stale — and we end up with shape-mismatched stale args
vs fresh EMA-val IoU. `_fresh` enforces "this round's value pairs
1:1 with this round's fg weights"; mismatch ⇒ skip.
Co-Authored-By: Guillaume PELLUET <guillaume@graybx.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix discarded data among data loader
* Add new configs in config files for signals
* Remove learning rate customization from UI and related stuff related to the modelling part
* Integration: WLAwareTrainer empty-val guard + LoggerQueue auto-register
- validate(): return ({}, 0.0) when val loader is fully discarded, instead
of letting UL crash on np.concatenate([]) inside metrics.process. ({}, 0.0)
unpacks cleanly into UL's `{**self.metrics, ...}` while None does not.
- _on_train_start: register LoggerQueue from trainer.save_dir (no user code
needed; uses UL's `project/name` -> save_dir mapping).
- main_ul_native: pass project/name from cfg so UL's save_dir lands the WL
logger in the configured root_log_dir; drop the manual LoggerQueue call.
- README.md: setup matrix (UL/Python/OS), required train kwargs,
shipped signals, and discard behavior including the all-val guard.
* Add scenario tests: discard, tag-discard, discard-all-val
gRPC-driven smoke tests for the WLAwareTrainer integration:
- client_discard_test.py: top-N by signal value -> discard ->
verify last_seen + per-sample signals freeze on victims while
non-victims keep advancing.
- client_tag_discard_test.py: set `tag:probe` -> tag-query ->
discard the query result (the agent path) -> verify freeze.
- client_discard_all_val_test.py: discard every active val sample
and watch 2 val cycles for the empty-val guard in
WLAwareTrainer.validate() -- PASS = no `np.concatenate([])`
or `'NoneType' object is not a mapping` crash.
- run_all_tests.sh: sequential orchestrator.
All three pass against the current branch.
* Drop scenario tests from branch; log when empty-val guard fires
- Remove the client_*_test.py + run_all_tests.sh added in a94945d:
they were useful for local validation but are too coupled to
the dev environment (paths, ports, dataset shape) to live in
the branch. Kept locally as untracked.
- WLAwareTrainer.validate(): print a single visible marker when
the empty-val guard returns ({}, 0.0). Without this, the case
is silent in the log -- harder to notice in production runs.
* fix empty bbx converted to mask
---------
Co-authored-by: Alexandru Rotaru <rotarualexandruandrei94@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Guillaume PELLUET <guillaume@graybx.com>
UL's DetectionTrainer.build_dataset hardcodes `rect=mode=='val'`, so val gets per-batch rect_shape padding (e.g. 224x352) while train pads to square (320x320). Same normalised bbox lands in different absolute positions across splits. Smallest surface: flip `dataset.rect = False` on the val dataset after self.build_dataset(...). UL's __getitem__ skips the rect branch and the default LetterBox(new_shape=(imgsz, imgsz)) applies, matching train.
Previous fix hardcoded `dataset.rect = False`. Move it next to the other ledger-driven loader settings so users can re-enable rect mode per-loader via `cfg.data.val_loader.rect: true` without code changes. Default False keeps val/train geometry aligned.
Two cases, stdlib-only, runs in ~1s on cached val cache: * default cfg -> dataset.rect=False, img.shape == (3, imgsz, imgsz) square, matching train geometry. * opt-in cfg.data.val_loader.rect=true -> dataset.rect=True, img.shape != (imgsz, imgsz) (per-batch rect_shape). Both PASS against the current SDK.
This reverts commit 3459058.
…istory The full-history path now reads signal_logger.get_signal_history_downsampled (downsampling moved into DuckDB); these two tests still mocked the old get_signal_history method, so the mock was never hit and the response came back empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_decode_outliers (and production's find_outliers) expect a list of
{"sample_id", "value"} dicts; the fixture wrote [name, value] pairs, which
_decode_outliers silently drops as non-dict items, so outlier_count never
got set on the returned entries even though the outliers column was
populated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 13:16 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 13:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 13:27 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 13:45 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 14:21 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 14:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 14:33 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
marked this pull request as ready for review
August 21, 2026 14:35
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 14:36 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
guillaume-byte
temporarily deployed
to
github-pages
August 21, 2026 14:58 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v2 — Major Feature Release & Stability Upgrade
New Features
Fixes & Improvements
Developer Experience
/clearand/compact, stable loop behavior.Experimental & Advanced