tests: add UI test infrastructure for VM Wizard#7631
tests: add UI test infrastructure for VM Wizard#7631sdrabb wants to merge 14 commits intoutmapp:mainfrom
Conversation
|
Hi please confirm that you have read https://github.com/utmapp/UTM/blob/main/CONTRIBUTING.md and have followed the guidelines. Thanks! |
Add a UTMUITests target (com.apple.product-type.bundle.ui-testing) that depends on the macOS app target and wire it into the macOS scheme so that tests can be run both from Xcode and via: xcodebuild test -scheme macOS -destination 'platform=macOS' Only the minimum required project.pbxproj entries are added: PBXBuildFile, PBXContainerItemProxy, PBXFileReference, PBXFrameworks/Resources/SourcesBuildPhase, PBXGroup, PBXNativeTarget, PBXTargetDependency, XCBuildConfiguration (Debug + Release), and XCConfigurationList. No unrelated target settings are touched. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a test-utm job that runs UTMUITests against the macOS scheme
on every push/PR. The job depends on build-sysroot (same as
build-utm) so it can share the cached sysroot and run in parallel
with the other build jobs.
xcodebuild test -scheme macOS -destination 'platform=macOS' \
CODE_SIGNING_ALLOWED=NO
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
sdrabb
force-pushed
the
feature/ui-testing
branch
from
f685795 to
466f481
Compare
sdrabb
changed the title
|
Any chance to run this on the self hosted runner that we have? |
Comment out iOS and visionOS configurations for ARM64 and x86_64 architectures.
Replace CODE_SIGNING_ALLOWED=NO with ad-hoc signing so that com.apple.security.cs.disable-library-validation (from macOS-unsigned.entitlements) and com.apple.security.get-task-allow are honoured by the kernel. Without this the QEMU dylibs fail to load and the test runner hangs before establishing connection. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Gatekeeper quarantines build products on self-hosted macOS runners, blocking UTM.app from launching during the test run. Strip the quarantine xattr from the workspace before invoking xcodebuild test. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Split test into build-for-testing + test-without-building so we can strip the quarantine xattr from DerivedData (where UTMUITests-Runner is created) between the two steps. Using -derivedDataPath keeps all build products inside GITHUB_WORKSPACE for a clean xattr sweep. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
xattr -dr on the full DerivedData tree hits read-only files in SourcePackages/checkouts (git-managed, Permission denied) and never reaches UTMUITests-Runner. The runner binary lives in DerivedData/Build/Products so target that subtree instead. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Ad-hoc signing produces TeamIdentifierPrefix="-" which makes the app group "-.com.utmapp.UTM" — invalid. The sandbox SIGKILLs the process before the test runner can connect. Disable the sandbox for the test build since it is not needed to validate UI behaviour. ENABLE_HARDENED_RUNTIME is kept so QEMU dylibs can still load via cs.disable-library-validation. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove the build-for-testing / test-without-building split and the DerivedData path override. Use xcodebuild test directly with xattr to clear quarantine from the workspace before launch. Quarantine on XCTRunner itself must be removed once on the runner: sudo xattr -dr com.apple.quarantine /Applications/Xcode_26.0.app Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove quarantine attribute before running tests.
|
I configure the one I have as self hosted for a run |
|
@copilot please review this PR for any errors. |
osy
left a comment
osy
left a comment
There was a problem hiding this comment.
I looked through all the changes and it's fine and is very similar to the code I have generated with my own AI agent. The reason why I didn't commit it is because the bulk of the work (99%) is actually defining the tests and implementing them. You can probably use AI to implement most of the tests (assuming you properly review it, etc) but the real work is defining all the test cases. That takes a lot of work in understanding the use cases and where the complexity lies and is not something that you can tell the AI to do without a lot of context and guidance (especially since by default these AI agents are not too good at debugging UI issues). What I want to see (in human written form) is a test plan that covers everything that is important across all the different platforms (not just macOS but it is okay to start with just macOS).
|
Ok then I'll do a proposal, thanks for the feedback |
sdrabb
force-pushed
the
feature/ui-testing
branch
from
e538438 to
b03a98c
Compare
3 participants
Summary
UTMUITeststarget wired into the macOS scheme so tests run viaXcode and
xcodebuild test -scheme macOS -destination 'platform=macOS'VMWizardViewTestwith a window close button smoke testTESTING.mdguide covering how to run and extend the test suitetest-utmCI job tobuild.ymlthat runs on every push/PR,parallel to
build-utm, sharing the same sysroot cache#2565
Testing
Tested on: Mac mini M4, macOS 26.3 (Tahoe beta)
xcodebuild test -scheme macOS -destination 'platform=macOS'passes locallyNotes
Code is AI-assisted. All commits carry
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>.I have read the AI Contribution Guidelines and can attest that I have followed each item to the best of my ability.