Skip to content

docs: capture macOS UI test permission prompt guidance #91

Description

@gaelic-ghost

Summary

Add guidance to the Socket Apple/Xcode UI-testing skills for avoiding repeated macOS permission prompts during UI, real-app, and helper-backed validation runs.

This came up while validating Sirious external-helper text insertion. The helper itself was installed and reachable over XPC, but the real-app TextEdit scenario still failed because the Xcode-hosted test process was the process asking for Accessibility trust during setup/verification:

TextEdit real-app scenario cannot run because macOS still reports the active Xcode test host as untrusted for Accessibility after Sirious requested the system prompt.
Current host: com.galewilliams.Sirious at .../Build/Products/Debug/Sirious.app.

That failure mode is easy to reproduce in local UI/real-app tests: every new build location, runner, host app, helper, or generated test process can become a distinct TCC identity. If the test harness itself performs Accessibility, file, Apple Events, microphone, or other protected operations, users can get repeated prompts even when the product architecture is trying to keep those permissions scoped to the app or helper under test.

Proposed Skill Guidance

For Apple UI testing and real-app validation skills, add guidance like:

  1. Prefer assigning protected actions to the real process under test.

    • If a helper, app extension, or installed app owns the product permission, make the test call that process through the product API instead of having the Xcode test host perform the protected action directly.
    • Example: for helper-owned Accessibility insertion, validate via the helper's XPC command and avoid using the test runner's own Accessibility access for setup or assertion unless that is the behavior being tested.
  2. Use explicit test plans for prompt-heavy scenarios.

    • Put prompt-heavy checks behind a dedicated .xctestplan with environment variables such as SIRIOUS_RUN_EXTERNAL_HELPER_REAL_APP_SCENARIOS=1.
    • Do not rely on shell environment variables before xcodebuild; they may not reach the test runner. Use test-plan environmentVariableEntries for reliable propagation.
    • Keep ordinary local/unit test plans free of prompt-heavy gates.
  3. Keep generated artifacts out of prompt-prone locations.

    • Prefer repo-local ignored Build/, DerivedData/, .build/, or /tmp paths for build/test artifacts.
    • Avoid test code that writes to Desktop, Documents, Downloads, home-directory roots, or app-selected user folders unless file-access behavior is the thing under test.
    • Add ignore rules for recurring generated files like Build/ or *.profraw when Xcode writes them in the repo.
  4. Treat TCC identity as part of the test matrix.

    • Record which process is expected to own each permission: app, helper, Xcode, xcodebuild, XCTRunner, generated test runner, or installed app.
    • Include the active bundle identifier and bundle path in failure diagnostics so the user can approve the correct System Settings entry.
    • Avoid changing build/install locations between runs when TCC stability matters.
  5. Separate setup, action, and assertion ownership.

    • If setup needs to focus a real app, prefer product-owned setup APIs, helper commands, or deterministic app launch/open flows before falling back to Accessibility from the test host.
    • If assertion requires protected access, consider whether the product/helper can return observable state or whether the scenario should clearly document that the test host itself needs permission.

Acceptance Criteria

  • Apple/Xcode UI testing skill docs include a section for macOS permission prompt minimization.
  • The guidance explicitly covers TCC identity drift across Xcode build products, test runners, helpers, and installed app locations.
  • The guidance recommends dedicated .xctestplan environment gates for prompt-heavy scenarios.
  • The guidance warns that shell environment variables passed to xcodebuild are not reliable proof that the test runner received the gate; use test-plan environment entries instead.
  • The guidance recommends repo-local ignored artifact directories and avoiding protected user folders unless file access is under test.

Evidence From Sirious

  • External LaunchAgent helper reachable after install: scripts/manage-external-automation-helper.sh install --check-xpc returned SiriousAutomationHelper is available.
  • Dedicated helper test plan was required for reliable gate propagation: SiriousExternalHelperRealAppScenarios.xctestplan with SIRIOUS_RUN_EXTERNAL_HELPER_REAL_APP_SCENARIOS=1.
  • Shell-level SIRIOUS_RUN_EXTERNAL_HELPER_REAL_APP_SCENARIOS=1 xcodebuild ... was not enough evidence that the Swift Testing process received the gate.
  • The helper-backed TextEdit scenario then reached the real gate and failed on test-host Accessibility trust, proving the remaining prompt owner was the Xcode-hosted app/test process rather than the helper XPC reachability path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions