Skip to content

Capture Windows UIA evidence with native actions#45

Merged
abrichr merged 4 commits into
mainfrom
agent/windows-uia-structural-authoring-20260725
Jul 25, 2026
Merged

Capture Windows UIA evidence with native actions#45
abrichr merged 4 commits into
mainfrom
agent/windows-uia-structural-authoring-20260725

Conversation

@abrichr

@abrichr abrichr commented Jul 25, 2026

Copy link
Copy Markdown
Member

What changed

  • Adds the versioned public openadapt.capture.structural-observation/v1 schema and injectable read-only observer protocol.
  • Records available Windows UIA evidence at action time: AutomationId, role/control type, name, bounds, supported patterns, process/window identity, ancestry, and candidate cardinality/context.
  • Persists the observation beside the existing ActionEvent in recording.db and exposes it through raw events and processed CaptureSession.actions().
  • Keeps old recordings compatible through the existing additive SQLite migration and treats unavailable or malformed optional evidence as absent.
  • Keeps non-Windows and headless imports safe; no new recorder, actuation path, or input injection was added.
  • Keeps RDP/Citrix black-box: UIA describes the local Windows tree and is not presented as crossing the remote pixel boundary.

Independent review fixes

Review against pywinauto 0.6.9 found that the first draft's fakes were more permissive than the real API. The updated branch now:

  • deterministically quantizes cross-platform float coordinates to the integer POINT ABI before Desktop.from_point;
  • resolves keyboard targets through IUIA().get_focused_element() and UIAElementInfo, rather than the nonexistent Desktop.get_active();
  • enumerates candidates with filters that pywinauto 0.6.9 actually supports, then compares AutomationId exactly instead of passing its unsupported auto_id filter to descendants();
  • opens, uses, and closes UIA on the input-delivery thread through a thread-local, balanced COM lifecycle;
  • clears the service-owned pywinauto singleton before COM teardown so it is never reused across apartments;
  • keeps UIA optional: an unavailable provider omits structural evidence without aborting the canonical pixel/input recording.

The tests now mirror those exact API shapes and verify lifecycle start/event/stop all execute on the same delivery thread.

Why

openadapt-capture is the canonical native demonstration recorder. Native Windows authoring needs live structural evidence beside pixels and input timing so openadapt-flow can normalize and compile stronger structural bindings without rebuilding capture in the engine.

Dependency and package boundary

pywinauto>=0.6.9 is a Windows-only conditional dependency and is loaded lazily only on the Windows input-delivery thread. PyPI identifies 0.6.9 as BSD 3-clause. It is not vendored or bundled into these source archives.

The built wheel and sdist both passed scripts/verify_distribution.py, include the MIT license plus the new schema/observer modules, and contain no OpenIMIS, AGPL, real-EMR, or private-corpus payload matches.

Validation

  • ruff check openadapt_capture tests/test_structural_observation.py tests/test_input_observer.py
  • pytest -q tests/ --ignore=tests/test_browser_bridge.py -m 'not slow' → 333 passed, 8 deselected
  • Focused persistence/schema/processing/input lifecycle suite → 151 passed
  • Exact-head Windows CI smoke against a live synthetic WinForms UI: real pywinauto point and focused-element resolution, duplicate candidate cardinality, and a second COM/UIA lifecycle; no mouse or keyboard injection
  • Wheel/sdist build and release-boundary verifier
  • Built-wheel import/headless smoke on macOS

Artifact SHA-256:

  • wheel: eff076c5bcf4c214e57eca0e572bd3a4cab88b07141514859306097ed23f5ec0
  • sdist: a289ad94c92f981cac0fd2d33371efb023ad17e1da85f96788350c4c8b66ec61

Follow-up boundary

This draft intentionally stops at the capture-side contract. Flow normalization/compiler changes belong in a separate PR.

@abrichr
abrichr marked this pull request as ready for review July 25, 2026 04:32
@abrichr
abrichr merged commit ba08f0b into main Jul 25, 2026
12 checks passed
@abrichr
abrichr deleted the agent/windows-uia-structural-authoring-20260725 branch July 25, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant