`snapshot --raw` returns XCUITest runner's own UI tree instead of target app on physical iOS devices

[EarthXP]

Title

snapshot --raw returns XCUITest runner's own UI tree instead of target app on physical iOS devices

Description

Summary

On physical iOS devices, agent-device snapshot --raw returns the UI tree of the XCUITest runner app (AgentDeviceRunner) instead of the target app under test. The default snapshot (without --raw) works correctly because it uses a different underlying API.

Environment

• Device: iPhone 13 Pro Max, iOS 18.6.2 (physical device)
• agent-device: latest (installed via npm), 0.7.1
• Xcode: 16.x

Steps to reproduce

1. Connect a physical iOS device
2. Launch any app (e.g. Outlook): agent-device open com.microsoft.Office.Outlook
3. Take a default snapshot: agent-device snapshot -i → returns correct target app UI tree
4. Take a raw snapshot: agent-device snapshot --raw --json → returns the runner app's UI tree (AgentDeviceRunner) instead of Outlook

On a simulator, both commands return the correct target app tree.

Root cause

snapshotFast (default path) and snapshotRaw (--raw path) use fundamentally different XCUITest APIs to traverse the element tree:

snapshotFast — uses static snapshot() API (works cross-process)

// RunnerTests.swift L1330-1332
let rootSnapshot: XCUIElementSnapshot
rootSnapshot = try queryRoot.snapshot()
// Then traverses via snapshot.children — all in-memory, no IPC

XCUIElementSnapshot.snapshot() invokes the _XCT_snapshotForElement:attributes:parameters:reply: XPC protocol, which is Apple's designed mechanism for cross-process element tree retrieval. It returns an atomic, in-memory tree of the target app's UI.

snapshotRaw — uses live XCUIElement queries (fails cross-process on physical devices)

// RunnerTests.swift L1470
let children = element.children(matching: .any).allElementsBoundByIndex
// Each access triggers cross-process IPC via XCUIElementQuery

XCUIElement.children(matching:).allElementsBoundByIndex resolves elements through XCUIElementQuery. On simulators, both processes share the same Mac-hosted accessibility server, so cross-process queries work. On physical devices, the runner and target app are fully separate processes communicating through the device's testmanagerd, and the query silently falls back to the runner's own UI tree.

Why snapshotRaw uses live queries

This is not an oversight. snapshotRaw uses live XCUIElement queries to access element.isHittable — a property that XCUIElementSnapshot does not expose:

// RunnerTests.swift L1760-1762
private func snapshotHittable(_ snapshot: XCUIElementSnapshot) -> Bool {
    // XCUIElementSnapshot does not expose isHittable; use enabled as a lightweight proxy.
    return snapshot.isEnabled
}

The shouldInclude filtering logic (L1705-1728) heavily relies on isHittable for compact and interactive-only modes. This was a deliberate trade-off: live queries for accurate isHittable vs static snapshots for cross-process reliability.

Impact

• snapshot --raw is unusable on physical iOS devices — it returns the runner's tree, not the target app's
• snapshot --raw --json is often used to get detailed element attributes (value, hittable state) that the default snapshot omits
• The current docs say --raw is "for troubleshooting only" but do not mention this physical device limitation

External validation

This is not specific to agent-device. The same cross-process limitation is well-known across the iOS automation ecosystem:

1. Appium WebDriverAgent deliberately avoids live element queries for tree retrieval. Its page source, XML, and XPath operations all use the static snapshotWithError: API. The one method that uses descendantsMatchingType + allElementsBoundByIndex (fb_stableInstanceWithUid) explicitly marks results as fb_isResolvedNatively = @NO.

2. Appium XCUITest driver docs explicitly warn: "Sometimes, even if visually it looks like UI elements belong to the same application, they are referenced by absolutely different apps" and "There are known cases where application interface/behavior might differ in simulators and real devices."

3. Detox issue wix/Detox#3463 proposes injecting unique accessibility identifiers as a workaround for unreliable cross-process element matching — confirming XCUITest's native queries are insufficient. Detox has never fully supported physical device testing partly due to these constraints.

4. Appelium SnapshotQuery was created specifically because live cross-process queries are "less reliable and flakier" — snapshot-based queries are ~1000x faster (60μs vs 50-100ms per element) and eliminate race conditions.

5. The original Facebook WDA was explicitly "A WebDriver server for iOS that runs inside the Simulator" — physical device support was added later by the Appium fork with significant architectural adjustments.

Proposed fix

Replace live XCUIElement tree traversal in snapshotRaw with the same snapshot() API used by snapshotFast, and compute isHittable from snapshot properties instead of querying it live:

private func computeHittable(snapshot: XCUIElementSnapshot, viewport: CGRect, siblings: [XCUIElementSnapshot]) -> Bool {
    guard snapshot.isEnabled else { return false }
    let frame = snapshot.frame
    guard frame.width > 0 && frame.height > 0 else { return false }
    guard viewport.intersects(frame) else { return false }
    // Check if any later sibling fully occludes this element
    for sibling in siblings {
        if sibling.frame.contains(frame) { return false }
    }
    return true
}

This approach:

• Uses snapshot() for reliable cross-process tree on both simulators and physical devices
• Computes isHittable from frame geometry, enabled state, viewport intersection, and sibling occlusion — eliminating the need for live XCUIElement access
• Aligns with how WDA, Detox, and SnapshotQuery handle this problem
• Maintains the accuracy goals of --raw mode without the cross-process limitation

[EarthXP]

I'm not sure if the root cause analyzed above, I thought snapshot raw is basic function of agent-device, if it has this kind of limitation it should be documented somewhere. I have tested this in android physical device, snapshot raw works as expected.

[thymikee]

It’s a bug, would you like to send a PR fix?

[EarthXP]

the proposal: migrate snapshotRaw to the snapshot() API and compute isHittable from snapshot geometry instead of querying it live.

private func computeHittable(
_ snapshot: XCUIElementSnapshot,
viewport: CGRect,
laterSiblings: [XCUIElementSnapshot]
) -> Bool {
guard snapshot.isEnabled else { return false }
let frame = snapshot.frame
guard frame.width > 0 && frame.height > 0 else { return false }
let center = CGPoint(x: frame.midX, y: frame.midY)
guard viewport.contains(center) else { return false }
for sibling in laterSiblings {
if sibling.frame.contains(center) { return false }
}
return true
}

This checks enabled state → non-zero frame → center within viewport → not occluded by higher z-order siblings — matching the core logic of Apple's isHittable.

Known edge cases where the computed value may differ from live isHittable:

• Container with isUserInteractionEnabled = false but accessibility isEnabled = true (rare)
• App overrides point(inside:with:) for custom hit regions (rare)
• Cousin-level occlusion from a different subtree (moderate — could be addressed by checking all nodes with higher tree index instead of siblings only)

None of these affect the current isEnabled-only proxy in snapshotFast either. The geometry-based computation is strictly more accurate than the existing fallback while fully resolving the physical device cross-process issue.

A few questions before I put together a PR:

1. Is this accuracy trade-off acceptable? Or does --raw need to guarantee byte-identical isHittable values?
2. Should the same computeHittable also replace the isEnabled proxy in snapshotFast?

[thymikee]

Please check #148

[EarthXP]

Great, one more thing you may check if you want to fix in your PR: - Cousin-level occlusion from a different subtree (could be addressed by checking all nodes with higher tree index instead of siblings only)