feat(android): add display selection to AndroidAgent#296
Merged
Conversation
Add a `display` parameter to AndroidAgent/PpadbAgentOs accepting an AndroidDisplay, a list of them, an index, or a name. Explicit AndroidDisplay(s) become the authoritative display list, bypassing the SurfaceFlinger auto-detection so every path - initial selection, set_display_by_*, and the model's select_display_by_unique_id tool - resolves against caller-supplied ids. AndroidDisplay ids are now optional: a None display_id/unique_display_id omits the `-d` flag so input/screencap target adb's default display. This replaces the need for a no-flag subclass and subsumes SingleAndroidDisplay. A model-facing __str__ keeps None ids from being shown as selectable. Add `display_allow_switching` (default True); when False the runtime display/device selection tools are removed so a pinned display cannot be changed mid-run. Docs: new "Selecting a display" section in 02_using_agents.md. Tests: tests/unit/tools/android/test_display_selection.py (15 cases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
On multi-display Android hardware with non-standard display port numbering (e.g. some automotive head units),
AndroidAgentcould only auto-select the first detected display, and the auto-detected logical display id could be wrong. Detection infers the logical id asport + 1(withport 0 → 0), which matches typical devices where ports are small and sequential (0, 1, 2 → 0, 2, 3) but not hardware that reports large/non-sequential ports. This adds explicit, caller-controlled display selection so such hardware can bypass detection.Changes
displayparameter onAndroidAgent/PpadbAgentOs:AndroidDisplay | list[AndroidDisplay] | int | str | None.AndroidDisplay(or a list) becomes the authoritative display list, bypassing SurfaceFlinger auto-detection. Every selector — including the model'sselect_display_by_unique_idtool — then resolves against the supplied ids.intselects by index,strby name (both via detection).AndroidDisplay:display_id(logical id →input -d) andunique_display_id(physical id →screencap -d) may now beNone, which omits the-dflag so that command targets adb's default display. This removes the need for a custom no-flag subclass and subsumesSingleAndroidDisplay. A new model-facing__str__preventsNoneids from being shown to the model as selectable values.display_allow_switching(defaultTrue): whenFalse, the runtime display/device selection tools are dropped so a pinneddisplaycannot be changed mid-run.docs/02_using_agents.md.tests/unit/tools/android/test_display_selection.py(15 cases).Possible follow-up (not in this PR)
The
port + 1inference inget_connected_displays()is correct for typical devices but breaks when a device reports non-sequential ports. A robustness improvement would be to read the logicaldisplayIddirectly fromdumpsys display(mViewports, keyed byuniqueId), which is correct regardless of port numbering. Passing an explicitAndroidDisplayalready sidesteps this, so it is optional.🤖 Generated with Claude Code