AB#12334: Prototype duration-balanced host build tests on Helix

[mmitche]

Live validation runs

• First opt-in run: build 1563210 - exposed an incorrect Linux NUnit filter (Name = BuildTest matched zero tests); Windows validation continues in that run.
• Corrected opt-in run at c41b6aa: build 1563289
• Metadata-corrected opt-in run at 112d251: build 1563300. The first run showed 1,504 Windows tests packed into 41 fallback work items; submission then failed because the project did not preserve generated Command metadata.
• Payload-partitioned opt-in run at 911c6b3: build 1563332. Build 1563300 successfully discovered 167 Linux tests in 5 work items, then exposed a 13.57 GiB monolithic payload and the Helix SDK in-memory ZIP limit; payloads are now capped at 1 GiB and the duplicated NuGet cache is omitted.
• Device-trimmed payload run at 39d7209: build 1563387. Build 1563332 showed that the prepared SDK included a device-only Android 29 x86_64 system image larger than 1 GiB; emulator images, skins, and sources are now excluded from host-only payloads.
• Measured payload composition: build 1563446 showed 7.75 GiB Linux / 8.69 GiB Windows. SDK docs/extras, Java.Interop source, unrelated test assemblies, and device-only content are now removed.
• Host-only iteration run at c6114de: build 1563474, using skipTests=true only to accelerate prototype validation; the final parity run will retain all existing jobs.

This run preserves the existing Windows/Linux jobs in the same build for inventory and outcome parity comparison. Results will be recorded here after both the baseline jobs and Helix work items complete.

Azure DevOps tracking - Parent: DNCENG Epic 11483 - Initiative: DNCENG Epic 12333 - Improve dotnet/android P80 PR build time - Implementation task: AB#12334 - Investigate moving dotnet/android tests to Helix ## Summary Prototype moving host-only Xamarin.Android.Build.Tests fan-out from fixed Azure Pipelines shards to fine-grained Helix work items. This is additive and guarded: the existing Windows and Linux jobs remain unchanged, and the new stage is omitted unless enableHostBuildTestsHelixPrototype is enabled. ## Duration-balanced work items - Default target: 15 estimated wall-clock minutes per work item. - Configure with hostBuildTestsHelixTargetMinutes. - Test discovery uses the repository's existing dotnet-test-slicer NUnit support. - Timing data uses the slicer's existing balance.xml format. - Work items use deterministic best-fit-decreasing duration packing. - A test whose duration exceeds the target remains in an isolated work item. - Missing tests use the average of matching historical timings. - With no matching history, generation explicitly reports count-fallback and uses hostBuildTestsHelixFallbackTestDurationSeconds rather than claiming count slicing is duration-balanced. - hostBuildTestsHelixDurationParallelism (default 2.5) translates summed per-test elapsed time to expected work-item wall time while preserving the current NUnit worker behavior. ## Architecture and data flow For each host OS, Azure Pipelines performs setup once and stages a Helix correlation payload containing: - the prepared repository-local .NET SDK and Android workload packs; - Android SDK and JDK; - NuGet and Gradle caches; - test assemblies and required repository test/build inputs; - the existing apkdiff tool. Small per-work-item payloads contain only NUnit runsettings, a native Windows/Linux command script, and generation metadata. Work items run on host queues that do not require an Android device: - Windows.Amd64.VS2026.Open - Ubuntu.2404.Amd64.Open Windows currently covers the full host test assembly. Linux initially covers the existing long, unsliced BuildTest group. Each work item runs in its own Helix sandbox and directs temporary/test output to work-item-specific paths. Failures preserve the dotnet test exit code. Uploaded diagnostics include TRX, console output, NUnit/MSBuild test attachments and binlogs, generation metadata, and crash dumps when produced. After completion, successful TRX timings are converted back to balance.xml and published as OS-specific artifacts. A subsequent run can consume them by setting hostBuildTestsHelixTimingBuildId to the prior build ID. ## Opt in Queue the public pipeline with: - enableHostBuildTestsHelixPrototype=true - hostBuildTestsHelixTargetMinutes=15 - optionally hostBuildTestsHelixTimingBuildId=<prior build ID> The per-work-item timeout defaults to 00:45:00 and is configurable through hostBuildTestsHelixWorkItemTimeout. Example: powershell az pipelines run ` --organization https://dev.azure.com/dnceng-public ` --project public ` --id <dotnet-android-public-definition-id> ` --parameters enableHostBuildTestsHelixPrototype=true hostBuildTestsHelixTargetMinutes=15 ## Validation - PowerShell generator tests pass, covering deterministic packing, explicit no-history fallback, partial history, oversized tests, zero-duration TRX entries, escaping, generated payloads, and Windows/Linux commands. - Both changed YAML files parse successfully. - The Helix MSBuild project parses, restores, and evaluates successfully with the repository SDK and synthetic work items. - Synthetic end-to-end payload preparation succeeds. - Historical-duration simulation produced: - Windows: 1,910 timed tests into 52 work items, estimated 10.6-15 minutes. - Linux BuildTest: 239 timed tests into 8 work items, estimated 8.8-15 minutes. - git diff --check passes. ## Known limitations / first experiment - No real Helix submission has been run yet; the first opt-in pipeline run should validate queue image prerequisites, correlation payload upload/download size, and actual setup amortization. - The 2.5 timing parallelism factor is an initial calibration and should be adjusted from observed Helix wall times. - To control correlation payload size, the source copy excludes src/Mono.Android; tests consume that product through the staged workload packs. The first run should confirm no host test requires additional source-only inputs. - Existing Azure Pipelines jobs intentionally remain enabled during comparison, so the first run duplicates coverage. - Linux initially moves only BuildTest; the other Linux smoke groups remain on their existing path.