Skip to content

feat(android): add display selection to AndroidAgent#296

Merged
programminx-askui merged 2 commits into
mainfrom
feat/android-display-selection
Jul 23, 2026
Merged

feat(android): add display selection to AndroidAgent#296
programminx-askui merged 2 commits into
mainfrom
feat/android-display-selection

Conversation

@programminx-askui

@programminx-askui programminx-askui commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

What & why

On multi-display Android hardware with non-standard display port numbering (e.g. some automotive head units), AndroidAgent could only auto-select the first detected display, and the auto-detected logical display id could be wrong. Detection infers the logical id as port + 1 (with port 0 → 0), which matches typical devices where ports are small and sequential (0, 1, 2 → 0, 2, 3) but not hardware that reports large/non-sequential ports. This adds explicit, caller-controlled display selection so such hardware can bypass detection.

Changes

  • display parameter on AndroidAgent / PpadbAgentOs: AndroidDisplay | list[AndroidDisplay] | int | str | None.
    • An AndroidDisplay (or a list) becomes the authoritative display list, bypassing SurfaceFlinger auto-detection. Every selector — including the model's select_display_by_unique_id tool — then resolves against the supplied ids.
    • int selects by index, str by name (both via detection).
  • Optional ids on AndroidDisplay: display_id (logical id → input -d) and unique_display_id (physical id → screencap -d) may now be None, which omits the -d flag so that command targets adb's default display. This removes the need for a custom no-flag subclass and subsumes SingleAndroidDisplay. A new model-facing __str__ prevents None ids from being shown to the model as selectable values.
  • display_allow_switching (default True): when False, the runtime display/device selection tools are dropped so a pinned display cannot be changed mid-run.
  • Docs: new "Selecting a display" section in docs/02_using_agents.md.
  • Tests: tests/unit/tools/android/test_display_selection.py (15 cases).

Possible follow-up (not in this PR)

The port + 1 inference in get_connected_displays() is correct for typical devices but breaks when a device reports non-sequential ports. A robustness improvement would be to read the logical displayId directly from dumpsys display (mViewports, keyed by uniqueId), which is correct regardless of port numbering. Passing an explicit AndroidDisplay already sidesteps this, so it is optional.

🤖 Generated with Claude Code

programminx-askui and others added 2 commits July 23, 2026 19:33
Add a `display` parameter to AndroidAgent/PpadbAgentOs accepting an
AndroidDisplay, a list of them, an index, or a name. Explicit
AndroidDisplay(s) become the authoritative display list, bypassing the
SurfaceFlinger auto-detection so every path - initial selection,
set_display_by_*, and the model's select_display_by_unique_id tool -
resolves against caller-supplied ids.

AndroidDisplay ids are now optional: a None display_id/unique_display_id
omits the `-d` flag so input/screencap target adb's default display. This
replaces the need for a no-flag subclass and subsumes SingleAndroidDisplay.
A model-facing __str__ keeps None ids from being shown as selectable.

Add `display_allow_switching` (default True); when False the runtime
display/device selection tools are removed so a pinned display cannot be
changed mid-run.

Docs: new "Selecting a display" section in 02_using_agents.md.
Tests: tests/unit/tools/android/test_display_selection.py (15 cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@programminx-askui
programminx-askui merged commit 053503a into main Jul 23, 2026
1 check passed
@programminx-askui
programminx-askui deleted the feat/android-display-selection branch July 23, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant