Add MAUI iOS Inner Loop measurements for CI#5187
Draft
davidnguyen-tech wants to merge 96 commits intodotnet:mainfrom
Draft
Add MAUI iOS Inner Loop measurements for CI#5187davidnguyen-tech wants to merge 96 commits intodotnet:mainfrom
davidnguyen-tech wants to merge 96 commits intodotnet:mainfrom
Conversation
- Create iOSInnerLoopParser.cs: binlog parser for iOS inner loop build timings, extracting iOS-specific tasks (AOTCompile, Codesign, MTouch, etc.) and targets (_AOTCompile, _CodesignAppBundle, _CreateAppBundle, etc.) plus shared tasks (Csc, XamlC, LinkAssembliesNoShrink) - Wire into Startup.cs: add iOSInnerLoop to MetricType enum and map it to iOSInnerLoopParser in the parser switch expression - Fix Reporter.cs: guard against null/empty PERFLAB_BUILDTIMESTAMP to prevent ArgumentNullException on DateTime.Parse(null) when the env var is unset (falls back to DateTime.Now) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- const.py: Add IOSINNERLOOP constant and SCENARIO_NAMES mapping - ioshelper.py: New module with iOSHelper class for simulator and physical device management (boot, install, launch, terminate, uninstall, find bundle) - runner.py: Add iosinnerloop subparser, attribute assignment, and full execution branch (first build+deploy+launch, incremental loop with source toggling, binlog parsing, report aggregation, and Helix upload) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pre.py: Install maui-ios workload, create MAUI template (no-restore for Helix), strip non-iOS TFMs with flexible regex, inject MSBuild properties (AllowMissingPrunePackageData, UseSharedCompilation), copy merged NuGet.config for Helix-side restore, create modified source files for incremental edit loop, check Xcode compatibility. test.py: Thin entrypoint that builds TestTraits and invokes Runner. post.py: Uninstall app from simulator, shut down dotnet build server, clean directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create the Helix machine setup script for MAUI iOS inner loop measurements. This script runs on the macOS Helix machine before test.py and handles: 1. DOTNET_ROOT/PATH configuration from the correlation payload SDK 2. Xcode selection — auto-detects highest versioned Xcode_*.app, matching the pattern used by maui_scenarios_ios.proj PreparePayloadWorkItem 3. iOS simulator runtime validation via xcrun simctl 4. Simulator device boot with graceful already-booted handling 5. maui-ios workload install using rollback file from pre.py, with --ignore-failed-sources for dead NuGet feeds 6. NuGet package restore with --ignore-failed-sources /p:NuGetAudit=false 7. Spotlight indexing disabled via mdutil to prevent file-lock errors Follows the same structure as the Android inner loop setup_helix.py: context dict pattern, step-by-step functions, structured logging to HELIX_WORKITEM_UPLOAD_ROOT for post-mortem debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define the Helix .proj file for iOS inner loop measurements, modeled after the Android inner loop .proj and existing maui_scenarios_ios.proj patterns. Key design decisions: - Build on Helix machine (not build agent) because deploy requires a connected device/simulator. PreparePayloadWorkItem only creates the template and modified source files via pre.py. - Workload packs stripped from correlation payload (RemoveDotnetFromCorrelation Staging) and reinstalled on Helix machine by setup_helix.py. - Environment variables set via shell 'export' in PreCommands (not in Python) because os.environ changes don't persist across process boundaries. - No XHarness — iOS inner loop uses xcrun simctl directly. - Simulator-only for now; physical device support (ios-arm64, code signing) is structured as a future TODO pending runner.py device support. - 01:30 timeout to accommodate iOS build + workload install + NuGet restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- sdk-perf-jobs.yml: Add Mono Debug job entry for maui_scenarios_ios_innerloop on osx-x64-ios-arm64 (Mac.iPhone.17.Perf queue) - run-performance-job.yml: Add maui_scenarios_ios_innerloop to the in() check so --runtime-flavor is forwarded to run_performance_job.py - run_performance_job.py: Add maui_scenarios_ios_innerloop to get_run_configurations() (CodegenType, RuntimeType, BuildConfig) and to the binlog copy block for PreparePayloadWorkItems artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ioshelper.py: Add detect_connected_device() with auto-detection via xcrun devicectl (JSON + fallback text parsing), uninstall_app_physical, terminate_app_physical, close_physical_device, and cleanup() dispatch - runner.py: Add --device-type arg (simulator/device) to iosinnerloop subparser, auto-infer from RuntimeIdentifier, auto-detect device UDID, branch setup/install/startup/cleanup for physical vs simulator - setup_helix.py: Detect device type from IOS_RID env var, skip simulator boot for physical device, add detect_physical_device() for Helix - post.py: Handle physical device uninstall via devicectl with UDID auto-detection fallback - maui_scenarios_ios_innerloop.proj: Add physical device HelixWorkItem (conditioned on iOSRid=ios-arm64), pass IOS_RID to Pre/PostCommands, add --device-type arg to both simulator and device workitems Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e sort Fix 1 (Major): Replace non-existent 'devicectl terminate --bundle-id' with '--terminate-existing' flag on launch command. Make terminate_app_physical() a no-op with documentation explaining why. Fix 2 (Medium): Write devicectl JSON output to temp file instead of /dev/stdout, which mixes human-readable table and JSON. Applied in both ioshelper.py and setup_helix.py with proper temp file cleanup. Fix 3 (Medium): Add standard UUID pattern (8-4-4-4-12) to UDID regex in _detect_device_fallback() for CoreDevice UUID format compatibility. Fix 4 (Medium): Normalize MAUI template to always use Pages/ subdirectory in pre.py. If template puts MainPage files at root, move them to Pages/. Add explanatory comment in .proj documenting the coupling. Fix 5 (Minor): Use tuple-of-ints version sort for Xcode selection instead of string comparison (fixes 16.10 < 16.2 ordering bug). Fix 6 (Minor): Make simulator boot failure fatal with sys.exit(1). Add dynamic fallback to latest available iPhone simulator before failing. Fix 7 (Nit): Add missing trailing newline to runner.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace deprecated tempfile.mktemp() with tempfile.mkstemp() in both ioshelper.py and setup_helix.py to avoid TOCTOU race condition. - Fix unreachable fallback in detect_connected_device(): when devicectl exits non-zero (e.g., older Xcode without --json-output), call _detect_device_fallback() instead of returning None immediately. - Guard against missing JSON report in runner.py IOSINNERLOOP branch: Startup.cs only writes reports when PERFLAB_INLAB=1, so local runs would crash with FileNotFoundError. Now degrades gracefully with empty counters and a warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Temporarily disable all other scenario jobs to speed up CI iteration while validating the new MAUI iOS Inner Loop scenario. This change should be reverted before merging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Capture dotnet build output instead of crashing on CalledProcessError - Create traces/ directory before first build - Fix setup_helix.py to write output.log (matches .proj expectation) - Improve error handling for build failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet build stdout/stderr wasn't appearing in Helix console logs, making it impossible to diagnose build failures. Explicitly capture and print build output through Python logging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2943141 hit the 90-minute timeout. iOS first build with AOT compilation can take 30+ minutes, plus 3 incremental iterations. Increasing to 2.5 hours to allow full completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helix machines have Xcode 26.2 but the iOS SDK requires 26.3. The minor version difference shouldn't affect build correctness, so bypass the check with ValidateXcodeVersion=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mac.iPhone.17.Perf queue uses Intel x64 machines which need iossimulator-x64, not iossimulator-arm64. Add architecture detection in setup_helix.py and update default RID in .proj. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The traces upload directory already exists from the first build, causing copytree() to fail on subsequent iterations. Clear it before each parsetraces() call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The -v:n flag was added to debug build errors but produces excessive file copy logs. Default verbosity shows errors/warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add /p:MtouchLink=None to disable managed linker for Debug inner loop builds, avoiding MT0180 errors on machines without Xcode 26.3 - Add minimum Xcode version check in setup_helix.py for fast failure with clear diagnostics when machine has Xcode < 26.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove temporary ${{ if false }}: wrappers that disabled all jobs
except iOS inner loop during iterative CI debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…loop - Separate install from simulator/device setup in ioshelper.py - Capture install time for first and incremental deploys in runner.py - Add "Install Time" counter to both perf reports - Add CoreCLR Debug job entry in pipeline YAML - Add device (ios-arm64) job entries for both Mono and CoreCLR - Wire iOSRid env var through to MSBuild for device builds - [TEMP] Disable non-iOS-inner-loop jobs for CI validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the dynamically-resolved manifest references SDK packs not yet propagated to NuGet feeds, fall back to installing without the rollback file. This avoids CI being blocked by transient feed propagation delays. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the rollback file references SDK packs not yet propagated to NuGet feeds, retry without the rollback file. Matches the fallback pattern already added to pre.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The simulator HelixWorkItem was unconditionally included, even when iOSRid=ios-arm64. This caused the simulator to receive device RID in _MSBuildArgs, producing ARM64 binaries that can't install on a simulator. Add Condition to exclude it from device jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t.py Consolidate 12 duplicated simulator/device methods in ioshelper.py into a unified API (setup_device, install_app, measure_cold_startup, cleanup) that dispatches internally based on is_physical_device. Removes all if-is_physical dispatch branches from runner.py. Extract merge_build_deploy_and_startup and _make_counter to module-level helpers. Inline the incremental iteration loop (was a nested function with 10 parameters). Simplify post.py to reuse ioshelper instead of duplicating device detection. Extract inject_csproj_properties in pre.py. Net reduction: -232 lines across 4 files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ling Re-apply run_env_vars (including iOSRid) right before perf_send_to_helix() so the MSBuild .proj ItemGroup conditions can correctly exclude the simulator work item from device jobs. Add '|| exit $?' to setup_helix.py PreCommands so that when setup_helix exits non-zero (e.g., Xcode too old), the Helix shell stops instead of continuing to run test.py which would fail with a less clear error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Env var inheritance through msbuild.sh/tools.sh is unreliable for iOSRid. Add ios_rid field to PerfSendToHelixArgs and pass it as /p:iOSRid=<value> on the MSBuild command line so it reaches .proj evaluation deterministically. Also set it via set_environment_variables as belt-and-suspenders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mono_InnerLoop → Mono_InnerLoop_Simulator CoreCLR_InnerLoop → CoreCLR_InnerLoop_Simulator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace simctl (simulator) and devicectl (device) install/launch commands with mlaunch to match the real Visual Studio F5 developer experience: - Simulator: --launchsim combines install + launch (install_app returns 0) - Device: --installdev for install, --launchdev for launch - Device cleanup: --uninstalldevbundleid replaces devicectl uninstall - Simulator cleanup: unchanged (simctl terminate + uninstall) - Added _resolve_mlaunch() to find mlaunch from iOS SDK packs Device detection (devicectl) and simulator management (simctl boot/ terminate/uninstall) remain unchanged. The install_app/measure_cold_startup API is preserved so runner.py requires no changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of making install_app() a no-op for simulator, use mlaunch --installsim to get a separate install measurement. measure_cold_startup() still uses --launchsim for launch timing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…stall After installing the maui-ios workload, read _RecommendedXcodeVersion from the SDK's Versions.props and switch to the matching Xcode_*.app if the currently active Xcode doesn't match. This handles the case where Helix agents have a newer Xcode than the SDK requires. Falls back gracefully to the already-selected Xcode if no matching installation is found. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… artifact discovery
Build 2967827 made big strides on Mac.iPhone.13.Perf — build succeeded
end-to-end on simulator with the cross-arch RID fix, and 'mlaunch
--installsim' actually installed the .app — but two distinct failures
still blocked all 4 jobs:
Simulator (mono + coreclr): cold-startup measurement timed out after
120s with 'bundle never appeared in launchctl list'. iOS simulator
'launchctl list' only enumerates daemons/agents, NOT UI apps managed
by SpringBoard, so the bundle id was never going to show up there.
Fix: poll 'simctl spawn <UDID> ps -A' for the app's executable name
instead. Also raise timeout to 180s (cold sim launch on macOS 26
with dyld_sim setup can be slow), and on timeout dump mlaunch's
stdout/stderr plus 'simctl listapps' and 'simctl spawn ps -A' so
future timeouts have actionable diagnostics in the Helix log.
Device (mono + coreclr): mlaunch --installdev internally invokes
xcrun devicectl, which rejects the build with MIInstallerErrorDomain
error 13 'No code signature found'. The .csproj sets
EnableCodeSigning=false expecting ioshelper.sign_app_for_device() to
copy embedded.mobileprovision into the .app and run the Helix-provided
'sign' tool, but on Mac.iPhone.13.Perf neither artifact was on PATH or
in CWD.
Fix: setup_helix.py now searches /etc/helix-prep, /Users/helix-runner,
/Users/Shared/Helix, /var/helix, /usr/local/{bin,share} for both
files. If found, embedded.mobileprovision is copied into
HELIX_WORKITEM_ROOT (where ioshelper.sign_app_for_device looks) and
'sign' is symlinked into the work item venv bin (already on PATH per
.proj PreCommands, so shutil.which('sign') will resolve it).
Logs warnings (not errors) when artifacts are missing, so we can
see exactly what's missing on a given queue.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+330
to
+334
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: |
Comment on lines
+49
to
+56
| matches = sorted(glob.glob(pattern)) | ||
| if not matches: | ||
| raise FileNotFoundError( | ||
| f"mlaunch not found. Searched: {pattern}\n" | ||
| f"Ensure the iOS SDK workload is installed (dotnet workload install ios)." | ||
| ) | ||
| # Use the last match (highest version when sorted lexicographically) | ||
| mlaunch = matches[-1] |
Comment on lines
+623
to
+626
| rid_patterns = ['ios-arm64'] if is_physical else ['iossimulator-*', 'ios-arm64'] | ||
| for rid_pattern in rid_patterns: | ||
| pattern = os.path.join(build_output_dir, 'bin', configuration, 'net*', rid_pattern, f'{app_name}.app') | ||
| matches = glob.glob(pattern) |
Comment on lines
+1231
to
+1235
| # Toggle source files (odd → modified, even → original) | ||
| for dest, original, modified in edit_pairs: | ||
| content = modified if iteration % 2 == 1 else original | ||
| with open(dest, 'w') as f: | ||
| f.write(content) |
Comment on lines
+101
to
+110
| # Use the Python helper that matches Xcode to the iOS SDK version. | ||
| # It checks rollback_maui.json, then SDK packs, then falls back to highest. | ||
| # stderr has diagnostics (suppressed here); stdout has the path. | ||
| XCODE_PATH=$(python3 "$SCENARIO_DIR/select_xcode.py" \ | ||
| --scenario-dir "$SCENARIO_DIR" \ | ||
| ${DOTNET_ROOT:+--dotnet-root "$DOTNET_ROOT"} 2>/dev/null) \ | ||
| || XCODE_PATH="" | ||
| if [[ -z "$XCODE_PATH" || ! -d "$XCODE_PATH" ]]; then | ||
| log "WARNING: select_xcode.py failed; falling back to /Applications/Xcode.app" | ||
| XCODE_PATH="/Applications/Xcode.app" |
- Replace mlaunch --launchsim with xcrun simctl launch for the simulator cold-startup measurement. Earlier iterations hit a state-loss bug where the simulator was Booted → Shutdown during/after mlaunch --launchsim with no diagnostic output, producing 180s timeouts. simctl launch is what mlaunch invokes internally for the actual launch step, so the measurement is equivalent without managing simulator state. - Verify simulator is Booted via _assert_simulator_booted before the launch and verify the app is installed via simctl get_app_container before starting the timer, so install/registration failures are not misclassified as launch failures. - After launch, parse the PID from simctl launch output and confirm it is alive 2s later via simctl spawn ps -p <pid>, so a crashed launch is reported as failure rather than success. - find_and_stage_signing_artifacts now returns a bool. When the device signing infrastructure (cert + provisioning profile + sign tool) is missing on this Helix queue, setup_helix.py writes a SKIPPED.flag sentinel and exits 0; test.py honours the sentinel and exits 0 with a clear log message. This treats the missing infra as a documented skip rather than failing the build, so simulator passes are visible while the queue is provisioned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
iOS Simulator's userland does not include /bin/ps, so 'simctl spawn <UDID> ps' fails with ENOENT regardless of whether the launched app is alive. Build 2967895 hit this: simctl launch returned PID 2380 successfully and the app was visible in 'simctl listapps', but the 2-second stabilization check failed with 'No such file or directory' and we mis-reported this as a crash. iOS Simulator apps run as real macOS processes (sandboxed but in the host process table — the PID returned by 'simctl launch' IS the host PID), so use host 'ps -p <pid>' to confirm the process survived past the launch. Diagnostic dump similarly drops 'simctl spawn ps' in favour of host 'ps -A' so the dump itself doesn't error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+709
to
+726
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - win-x86-viper | ||
| #- ubuntu-x64-1804 reenable under new machine on new ubuntu once lttng/events are available | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: sdk_scenarios | ||
| projectFileName: sdk_scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} |
Comment on lines
+254
to
+263
| "machineName": "Davids-MacBook-Pro" | ||
| }, | ||
| "build": { | ||
| "repo": "dotnet/performance", | ||
| "branch": "nguyendav/maui-ios-inner-loop", | ||
| "architecture": null, | ||
| "locale": null, | ||
| "gitHash": "c3d1ab62f641767c94d78f9013f1cade2200f66c", | ||
| "buildName": "local-20260414123803", | ||
| "timeStamp": "2026-04-14T12:38:03+02:00", |
Comment on lines
+253
to
+262
| "machineName": "Davids-MacBook-Pro" | ||
| }, | ||
| "build": { | ||
| "repo": "dotnet/performance", | ||
| "branch": "nguyendav/maui-ios-inner-loop", | ||
| "architecture": null, | ||
| "locale": null, | ||
| "gitHash": "c3d1ab62f641767c94d78f9013f1cade2200f66c", | ||
| "buildName": "local-20260414123803", | ||
| "timeStamp": "2026-04-14T12:38:03+02:00", |
| /// <summary> | ||
| /// Parses iOS inner loop (build+deploy) target and task durations from a binary log file. | ||
| /// </summary> | ||
| public class iOSInnerLoopParser : IParser |
Comment on lines
+376
to
+394
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-android-arm64-pixel | ||
| - win-x64-android-arm64-galaxy | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: maui_scenarios_android | ||
| projectFileName: maui_scenarios_android.proj | ||
| channels: | ||
| - main | ||
| runtimeFlavor: mono | ||
| codeGenType: Default | ||
| buildConfig: Release | ||
| additionalJobIdentifier: Mono | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} |
… gap Build 2967912 collected all 4 startup measurements successfully (cold startups of 290-513 ms across iterations) but exited 1 because scripts/upload.py raised FileNotFoundError on /Users/helix-runner/certs/LabCert1.pfx — the perf-lab certificate is not provisioned on Mac.iPhone.13.Perf and managed-identity auth is also unavailable from that queue. The measurement results are already written to disk before upload, so upload failure does not invalidate the run; treating it as fatal turns a green scenario red for an unrelated infra gap. Inside the iOS inner loop branch only, log a warning and continue when upload throws or returns non-zero so the work item passes when measurement succeeded. Other scenarios that share runner.py are unaffected — this code path is gated by the iosinnerloop test type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- runner.py: revert the iOS inner-loop upload to fail-fast on non-zero upload_code. Result-publishing failures should be loud — silently passing a work item that didn't publish results hides the regression in the dashboard pipeline. - ioshelper.py: add a module docstring with a tooling matrix (device vs simulator, install vs launch) and an explicit Note explaining why simulator launch uses 'xcrun simctl launch' rather than 'mlaunch --launchsim' (the latter caused silent simulator shutdowns on Apple Silicon Helix queues with no diagnostic output). Also documents the device code-signing prerequisite so future readers don't repeat the investigation. - setup_helix.py: add a 'Device path & infrastructure prerequisites' section to the module docstring describing the SKIPPED.flag mechanism, why it exists (machine-image gap on Mac.iPhone.13.Perf), and how it self-heals once the queue is provisioned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
111
to
+115
| private static Build ParseBuildInfo(IEnvironment environment) | ||
| { | ||
| var buildTimestampStr = environment.GetEnvironmentVariable("PERFLAB_BUILDTIMESTAMP"); | ||
| var buildTimestamp = !string.IsNullOrEmpty(buildTimestampStr) ? DateTime.Parse(buildTimestampStr, CultureInfo.InvariantCulture) : DateTime.Now; | ||
|
|
Comment on lines
+87
to
+97
| dotnet_root = os.environ.get('DOTNET_ROOT', os.path.expanduser('~/.dotnet')) | ||
| pattern = os.path.join(dotnet_root, 'packs', 'Microsoft.iOS.Sdk.*', '*', 'tools', 'bin', 'mlaunch') | ||
| matches = sorted(glob.glob(pattern)) | ||
| if not matches: | ||
| raise FileNotFoundError( | ||
| f"mlaunch not found. Searched: {pattern}\n" | ||
| f"Ensure the iOS SDK workload is installed (dotnet workload install ios)." | ||
| ) | ||
| # Use the last match (highest version when sorted lexicographically) | ||
| mlaunch = matches[-1] | ||
| getLogger().info("Resolved mlaunch: %s", mlaunch) |
| if self.framework: | ||
| base_cmd.extend(['-f', self.framework]) | ||
| if self.msbuildargs: | ||
| base_cmd.extend(shlex.split(self.msbuildargs.replace(';', ' '))) |
Comment on lines
327
to
+352
| - ${{ if parameters.runPrivateJobs }}: | ||
|
|
||
| # Scenario benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - ubuntu-x64-viper | ||
| - win-arm64-ampere | ||
| - ubuntu-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - ubuntu-x64-viper | ||
| - win-arm64-ampere | ||
| - ubuntu-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
|
|
||
| # Affinitized Scenario benchmarks (Initially just PDN) | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - win-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios_affinitized.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| additionalJobIdentifier: 'Affinity_85' | ||
| affinity: '85' # (01010101) Enables alternating process threads to take hyperthreading into account | ||
| runEnvVars: | ||
| - DOTNET_GCgen0size=410000 # ~4MB | ||
| - DOTNET_GCHeapCount=4 | ||
| - DOTNET_GCTotalPhysicalMemory=400000000 # 16GB | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml |
Comment on lines
+2
to
+105
| "tests": [ | ||
| { | ||
| "categories": [ | ||
| "Startup" | ||
| ], | ||
| "name": "MAUI iOS Inner Loop - Mono Default - Incremental Build and Deploy", | ||
| "additionalData": {}, | ||
| "counters": [ | ||
| { | ||
| "name": "Build Time", | ||
| "topCounter": true, | ||
| "defaultCounter": true, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 22.534105, | ||
| 22.960042, | ||
| 24.691066, | ||
| 22.006338, | ||
| 23.735795, | ||
| 21.999691, | ||
| 24.129217, | ||
| 23.561018, | ||
| 21.718979, | ||
| 22.470737 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "Csc Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.6258360000000001, | ||
| 0.9932960000000001, | ||
| 0.27010700000000004, | ||
| 0.476783, | ||
| 0.187942, | ||
| 0.296952, | ||
| 0.286233, | ||
| 0.20488399999999998, | ||
| 0.212752, | ||
| 0.244082 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "XamlC Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.021858, | ||
| 1.035089, | ||
| 0.029793, | ||
| 1.290103, | ||
| 1.342634, | ||
| 0.024404, | ||
| 0.999846, | ||
| 0.025325, | ||
| 0.024375, | ||
| 0.022213999999999998 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "AOTCompile Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.03290100000000001, | ||
| 0.032951999999999995, | ||
| 0.033388, | ||
| 0.033648000000000004, | ||
| 0.034679, | ||
| 0.032544, | ||
| 0.035817, | ||
| 0.032668, | ||
| 0.032465, | ||
| 0.033374 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "Codesign Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.250228, | ||
| 0.260359, | ||
| 0.253315, | ||
| 0.249833, | ||
| 0.25204, | ||
| 0.252579, | ||
| 0.241097, | ||
| 0.26532799999999995, | ||
| 0.262065, | ||
| 0.260322 | ||
| ] | ||
| }, | ||
| { |
Comment on lines
+1
to
+71
| { | ||
| "tests": [ | ||
| { | ||
| "categories": [ | ||
| "Startup" | ||
| ], | ||
| "name": "MAUI iOS Inner Loop - CoreCLR Default - First Build and Deploy", | ||
| "additionalData": {}, | ||
| "counters": [ | ||
| { | ||
| "name": "Build Time", | ||
| "topCounter": true, | ||
| "defaultCounter": true, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 72.81466999999999 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "Csc Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 1.3180619999999998 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "XamlC Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.029151 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "Codesign Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.833853 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "LinkNativeCode Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 1.122443, | ||
| 0.209675 | ||
| ] | ||
| }, | ||
| { | ||
| "name": "DetectSigningIdentity Task Time", | ||
| "topCounter": true, | ||
| "defaultCounter": false, | ||
| "higherIsBetter": false, | ||
| "metricName": "s", | ||
| "results": [ | ||
| 0.41167000000000004 | ||
| ] | ||
| }, | ||
| { |
Comment on lines
+1
to
+10
| { | ||
| "tests": [ | ||
| { | ||
| "categories": [ | ||
| "Startup" | ||
| ], | ||
| "name": "MAUI iOS Inner Loop - CoreCLR Default - Incremental Build and Deploy", | ||
| "additionalData": {}, | ||
| "counters": [ | ||
| { |
The 6 JSON files under src/scenarios/mauiiosinnerloop/results/ are local measurement output produced by run-local.sh (which copies traces/*.binlog and *-versions.json from a local run for inspection). They were accidentally committed in earlier work-in-progress commits and have no place in the PR — production results are uploaded to the perf-lab via upload.py. Add a .gitignore entry inside the scenario directory so this can't happen again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously fix_coresimulator_permissions ran 'sudo chown -R user:staff /Library/Developer/CoreSimulator', which on Apple Silicon Helix machines descends into the iOS runtime image at .../Volumes/iOS_NN/... — an Apple-mounted read-only volume containing hundreds of thousands of files we have no permission to chown. Each one produced an 'Operation not permitted' line that was tee'd into the work-item console log, amounting to ~727k log lines and >200 MB per work item. Both the device and simulator paths called the function, so device runs (where the chown is only needed because actool spawns AssetCatalogSimulatorAgent during 'dotnet build' for ios-arm64) were just as bloated. Replace the plain 'chown -R' on the system-wide path with a 'find ... -prune -o -exec chown' that excludes the Apple-managed read-only subtrees (Volumes, Profiles, Cryptex, Images). Per-user ~/Library/... paths still use plain 'chown -R' because they don't contain those mount points. Verified the find -prune syntax on macOS against a synthetic tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The PYTHONPATH change ($PYTHONPATH -> ${PYTHONPATH:-}) is a useful
robustness fix for set -u callers, but it is unrelated to the iOS
inner loop work and affects every dotnet-performance scenario. Keep
the PR focused — the change can land separately on its own merits.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| private static Build ParseBuildInfo(IEnvironment environment) | ||
| { | ||
| var buildTimestampStr = environment.GetEnvironmentVariable("PERFLAB_BUILDTIMESTAMP"); | ||
| var buildTimestamp = !string.IsNullOrEmpty(buildTimestampStr) ? DateTime.Parse(buildTimestampStr, CultureInfo.InvariantCulture) : DateTime.Now; |
| startup.parsetraces(self.traits) | ||
|
|
||
| elif self.testtype == const.IOSINNERLOOP: | ||
| import hashlib |
Comment on lines
+95
to
+99
| # Use the last match (highest version when sorted lexicographically) | ||
| mlaunch = matches[-1] | ||
| getLogger().info("Resolved mlaunch: %s", mlaunch) | ||
| iOSHelper._mlaunch_path = mlaunch | ||
| return mlaunch |
Comment on lines
+26
to
+32
| helper.setup_device(bundle_id, None, device_udid, is_physical=True) | ||
| helper.cleanup() | ||
| else: | ||
| logger.warning("No device UDID available — skipping uninstall") | ||
| else: | ||
| helper.setup_device(bundle_id, None, 'booted', is_physical=False) | ||
| helper.cleanup() |
Comment on lines
327
to
+333
| - ${{ if parameters.runPrivateJobs }}: | ||
|
|
||
| # Scenario benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - ubuntu-x64-viper | ||
| - win-arm64-ampere | ||
| - ubuntu-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml |
Previously, when find_and_stage_signing_artifacts() failed to locate embedded.mobileprovision and the 'sign' tool, setup_helix.py wrote a SKIPPED.flag sentinel and test.py treated it as a pass (sys.exit 0). The intent was 'don't fail the build for a known queue provisioning gap', but the result is exactly the failure mode we don't want: a green build that hides the fact the scenario didn't actually run. Hard-fail instead. When signing artifacts are missing, log a loud 'WORK ITEM FAILED — DEVICE INFRA UNAVAILABLE' banner with the exact reason (which artifact is missing, which roots were searched, what needs provisioning) and sys.exit(1). The work item turns red, the console log makes the gap obvious to humans, and ESI can ticket the queue provisioning work without anyone wondering why the dashboard shows green. - setup_helix.py: replace write_skip_sentinel + return 0 path with a banner + sys.exit(1); drop the SKIP_SENTINEL constant and the write_skip_sentinel helper; update the module docstring. - test.py: drop _SKIP_SENTINEL and _check_skip_sentinel(); the scenario runner now starts immediately under __main__. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2968516's Mono Device job failed with: Unable to boot device due to insufficient system resources. maxUserProcs: 1333, runningUserProcs: 1221, enforcedProcBuffer: 183 Earlier in the same build the CoreCLR Device job logged 5 leftover 'PerfTest-iPhone-*' simulators in the 'Booted' state from prior workitem runs. Each booted simulator forks ~150-200 system daemons, so a handful of leaked simulators is enough to exhaust the per-user macOS process rlimit before this workitem's own simctl boot can succeed (NSPOSIX 67). Leaks happen when a previous workitem's post.py crashed before reaching delete_simulator(), or when the workitem was killed mid-run by a Helix timeout. Existing post.py cleanup is fine for the happy path; the new sweep is a defensive safety net at the start of create_and_boot_simulator that shuts down + deletes any existing PerfTest-iPhone-* simulators. Conservative scope: only touches devices whose name starts with the well-known 'PerfTest-iPhone-' prefix (set by _unique_simulator_name), so unrelated workitems sharing the queue are never disturbed. Best- effort — failures are logged but don't abort setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+114
to
+115
| var buildTimestamp = !string.IsNullOrEmpty(buildTimestampStr) ? DateTime.Parse(buildTimestampStr, CultureInfo.InvariantCulture) : DateTime.Now; | ||
|
|
Comment on lines
+87
to
+97
| dotnet_root = os.environ.get('DOTNET_ROOT', os.path.expanduser('~/.dotnet')) | ||
| pattern = os.path.join(dotnet_root, 'packs', 'Microsoft.iOS.Sdk.*', '*', 'tools', 'bin', 'mlaunch') | ||
| matches = sorted(glob.glob(pattern)) | ||
| if not matches: | ||
| raise FileNotFoundError( | ||
| f"mlaunch not found. Searched: {pattern}\n" | ||
| f"Ensure the iOS SDK workload is installed (dotnet workload install ios)." | ||
| ) | ||
| # Use the last match (highest version when sorted lexicographically) | ||
| mlaunch = matches[-1] | ||
| getLogger().info("Resolved mlaunch: %s", mlaunch) |
Comment on lines
329
to
+333
| # Scenario benchmarks | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - ubuntu-x64-viper | ||
| - win-arm64-ampere | ||
| - ubuntu-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} | ||
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml |
Comment on lines
+149
to
159
| - ${{ if and(containsValue(parameters.buildMachines, 'osx-x64-ios-arm64'), not(eq(parameters.isPublic, true))) }}: # iPhone ARM64 13 only used in private builds currently | ||
| - template: ${{ parameters.jobTemplate }} | ||
| parameters: | ||
| osGroup: osx | ||
| archType: x64 | ||
| osVersion: 15 | ||
| pool: | ||
| vmImage: 'macos-15' | ||
| queue: Mac.iPhone.17.Perf | ||
| machinePool: iPhone17 | ||
| queue: Mac.iPhone.13.Perf | ||
| machinePool: iPhone13 | ||
| ${{ insert }}: ${{ parameters.jobParameters }} |
Comment on lines
+16
to
+18
| Default is iossimulator-x64 because the Mac.iPhone.17.Perf Helix | ||
| queue runs on Intel x64 machines (e.g. DNCENGMAC045). --> | ||
| <iOSRid Condition="'$(iOSRid)' == ''">iossimulator-x64</iOSRid> |
… simulator fallback Audit triggered by user concern: 'are we falling back to simulator in CI if the devices aren't available? I do NOT want that'. Confirmed end-to-end: no path silently downgrades a device job to a simulator. runner.py:1093 already raises an Exception when no UDID is detected, and IOS_RID/RuntimeIdentifier are never flipped from ios-arm64 to iossimulator-* anywhere. But the device branch of setup_helix.main() was sloppy: when detect_physical_device() returned None, it logged a WARNING and let setup continue. The work item then wasted ~5 minutes on dotnet build before runner.py finally raised the cryptic 'Physical device mode requires a device UDID' exception during install. Replace the soft warn-and-continue with a hard 'WORK ITEM FAILED — NO PHYSICAL DEVICE' banner + sys.exit(1) at the same point we fail for missing signing infra. Same shape, same loud telemetry, same intent: queue provisioning gaps must show up as red builds with an obvious reason in the console log, never as silent skips or accidental simulator measurements masquerading as device numbers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…vice path The device-job branch was running find_and_stage_signing_artifacts() first, then detect_physical_device() second. On Mac.iPhone.13.Perf the signing artifacts are missing, so the work item bails before logging anything about what physical hardware is (or isn't) attached. Result: when a human asks 'what iPhone is on that queue?' the only answer in the logs is 'we never checked'. Reorder so physical-device detection runs first (its own loud failure banner) and signing-artifact discovery runs after (its own loud failure banner). Both gates check independent infrastructure — a disconnected iPhone vs missing keychain materials — and surface independently. Now the device log always shows what hardware 'xcrun devicectl list devices' returned, regardless of signing state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+89
to
+96
| matches = sorted(glob.glob(pattern)) | ||
| if not matches: | ||
| raise FileNotFoundError( | ||
| f"mlaunch not found. Searched: {pattern}\n" | ||
| f"Ensure the iOS SDK workload is installed (dotnet workload install ios)." | ||
| ) | ||
| # Use the last match (highest version when sorted lexicographically) | ||
| mlaunch = matches[-1] |
Comment on lines
+612
to
+621
| collect_cmd = ['sudo', 'log', 'collect', '--device', | ||
| '--start', start_ts, '--output', logarchive] | ||
| RunCommand(collect_cmd, verbose=True).run() | ||
|
|
||
| # Parse SpringBoard watchdog events for this bundle ID | ||
| show_cmd = ['log', 'show', | ||
| '--predicate', '(process == "SpringBoard") && (category == "Watchdog")', | ||
| '--info', '--style', 'ndjson', logarchive] | ||
| show = RunCommand(show_cmd, verbose=True) | ||
| show.run() |
Comment on lines
+5
to
+6
| <!-- No XHarness needed — iOS inner loop uses xcrun simctl directly for | ||
| simulator deploy and does not use the XHarness test infrastructure. --> |
Comment on lines
+330
to
+348
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - ubuntu-x64-viper | ||
| - win-arm64-ampere | ||
| - ubuntu-arm64-ampere | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: scenarios | ||
| projectFileName: scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} |
Comment on lines
+709
to
+726
| - ${{ if false }}: # [TEMP] Disabled for iOS inner loop CI validation | ||
| - template: /eng/pipelines/templates/build-machine-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml | ||
| buildMachines: | ||
| - win-x64-viper | ||
| - win-x86-viper | ||
| #- ubuntu-x64-1804 reenable under new machine on new ubuntu once lttng/events are available | ||
| isPublic: false | ||
| jobParameters: | ||
| runKind: sdk_scenarios | ||
| projectFileName: sdk_scenarios.proj | ||
| channels: | ||
| - main | ||
| - 9.0 | ||
| - 8.0 | ||
| ${{ each parameter in parameters.jobParameters }}: | ||
| ${{ parameter.key }}: ${{ parameter.value }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.
What's included:
Measurements:
Targets:
Based on: