Add ASP.NET Core perf-build pipeline#5243
Conversation
Stand up an Azure DevOps perf-build pipeline hosted in dotnet/performance that builds every commit on dotnet/aspnetcore main from source, packs per-RID Microsoft.AspNetCore.App.Runtime archives, and uploads them to the Build Cache Service for dotnet/crank bisection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Azure DevOps perf-build pipeline in dotnet/performance to build dotnet/aspnetcore main commits from source, pack per-RID Microsoft.AspNetCore.App.Runtime archives, and upload/register them in BCS using the existing shared templates (extended to support a caller-supplied SHA).
Changes:
- Introduces
eng/pipelines/aspnetcore-perf-build.yml+eng/pipelines/aspnetcore-perf-build-jobs.ymlto build/pack/publish ASP.NET Core runtime-pack artifacts for the locked set of 5 config keys. - Adds
shaparameter plumbing through the shared register/upload dispatcher + leaf templates so non-self(resource) builds can write BCS blobs under the triggering repo commit. - Adds
eng/pipelines/tools/pack-bcs-archives.ps1to produce the BCS archive layout from runtime-pack nupkgs.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/pipelines/aspnetcore-perf-build.yml | New root pipeline with aspnetcore repo-resource trigger and Register/Build/Upload stages. |
| eng/pipelines/aspnetcore-perf-build-jobs.yml | New authored build jobs to build aspnetcore from source (Windows multi-arch + Linux x64/arm64) and publish artifacts. |
| eng/pipelines/tools/pack-bcs-archives.ps1 | New packaging script to turn runtime-pack nupkgs into BCS archive layout and validate contents. |
| eng/pipelines/upload-build-artifacts-jobs.yml | Adds sha parameter and threads it through aspnetcore upload branches. |
| eng/pipelines/templates/upload-build-artifacts-job.yml | Adds sha parameter and uses it in the BCS blob path for uploads. |
| eng/pipelines/register-build-jobs.yml | Adds sha parameter and forwards it into per-buildType register jobs. |
| eng/pipelines/templates/register-build-job.yml | Adds sha parameter and uses it in the BCS blob path for buildInfo.json. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add an early, unconditional RegisterBuild job that tags the run with aspnetcore-sha:<40-char sha> (the aspnetcore commit from the repo-resource version macro). The build record's sourceVersion is the performance commit, so this tag is the dotnet-performance-infra indexer's sole aspnetcore-sha signal. Additive; the BCS {sha} override and buildInfo.json are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…version)
The dotnet-performance-infra indexer derives the aspnetcore commit directly from the build's resources.repositories.aspnetcore.version via the runs API, so the dedicated TagBuild job is redundant. The BCS {sha} path override and buildInfo.json are unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Convert job-level variables: blocks in aspnetcore-perf-build-jobs.yml from the invalid list form (- key: value) to the mapping form so AzDO schema validation passes and _AspNetCoreRoot/_PackScript/_ShippingDir/_StagingRoot expand (all 3 build jobs). - Clarify pack-bcs-archives.ps1 .DESCRIPTION (explicit pwsh step, not an afterBuild hook; reference aspnetcore-perf-build-jobs.yml) and add a local-run .EXAMPLE that passes -ShippingDir/-StagingRoot explicitly. - Clarify the ATOMICITY comment in aspnetcore-perf-build.yml to note the always()-guarded log-publish steps are intentionally continueOnError:true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Thread `sha: ${{ parameters.sha }}` into the 15 runtime upload branches in
upload-build-artifacts-jobs.yml, matching the per-branch `repoName` threading
already present on all branches. Behavior-neutral: sha defaults to
$(Build.SourceVersion) (the leaf template default), so runtime perf-build
uploads resolve to the same BCS path as before; only callers that pass an
explicit sha (e.g. the aspnetcore pipeline's resource version) change the
path. register-build-jobs.yml already forwards sha to every buildType via its
${{ each }} loop, so no change was needed there.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the single multi-arch Windows_build job with three independent, boolean-gated jobs (Windows_x64_build / Windows_x86_build / Windows_arm64_build), mirroring the Linux job structure. Each job now: - is gated solely by its own aspnetcore_*_windows boolean (x64 is no longer an unconditional base build that runs whenever any Windows arch is requested), and - builds standalone with -all + -nativeToolsOnMachine; x86/arm64 keep -noBuildNative so native runtime bits restore from packages. Repoint the three Windows aspnetcore_*_windows upload branches' dependencyJobName to the new per-arch job names. This gives exact per-config gating for the forward-compat indexer (queue only arm64 -> only Windows_arm64_build runs) and runs the three Windows arches in parallel instead of sequentially. Standalone Windows-arch native restore is validated by the first real queued run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After U1 (4f41751) every upload branch threads sha: ${{ parameters.sha }}, so the parameter doc claiming runtime branches omit sha and inherit the leaf default is no longer accurate. Reword to say all branches pass sha explicitly and it resolves to the $(Build.SourceVersion) default when no caller overrides it. Comment-only; no behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ults Two follow-up Copilot comments on the latest push (c43515f): 1. aspnetcore-perf-build.yml: the Build stage was the only ungated stage. It checks out the internal-only aspnetcore mirror on the internal DncEng pool, so a manual queue in a public project would fail confusingly while the already-gated RegisterBuild/UploadArtifacts stages are compile-time removed. Add a runtime condition: matching the sibling internal + (ResourceTrigger | Manual) gate. A runtime condition (not ${{ if }}) keeps the stage present-but-skipped in public, avoiding a zero-stage (invalid) pipeline. Behavior-neutral in the steady-state internal trigger path. 2. pack-bcs-archives.ps1: $ShippingDir/$StagingRoot defaults called Join-Path on BUILD_SOURCESDIRECTORY/BUILD_ARTIFACTSTAGINGDIRECTORY, which throws during parameter binding when those env vars are unset (local runs). Fall back to the current directory so the documented local-override workflow works off-agent. Unchanged on the agent where the BUILD_* vars are always set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The pack script is aspnetcore-specific: it is hardcoded to find
Microsoft.AspNetCore.App.Runtime.{rid} nupkgs and emit the lowercase
microsoft.aspnetcore.app.runtime.{rid}/Release layout. It shares the
eng/pipelines/tools/ directory with the dotnet/runtime perf-build pipeline, so an
-aspnetcore suffix makes its scope explicit and avoids a future collision with a
runtime-specific pack script. Matches the aspnetcore-* naming of its sibling
pipeline files. Updated the 5 _PackScript references and the header comment in
aspnetcore-perf-build-jobs.yml and the 2 .EXAMPLE self-references in the script.
No contract depends on the filename (only the pipeline references it).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…netcore
Copilot review: the pack script selected the runtime-pack nupkg with
`Sort-Object Name | Select-Object -First 1`, which silently picks the
lexicographically-smallest match if multiple versions are present in the Shipping
directory -- packing an unintended version into the BCS archive.
A clean from-source build produces exactly one non-symbols
Microsoft.AspNetCore.App.Runtime.{rid} pack per RID, so collect all non-symbols
matches and assert exactly one: throw a clear error listing the offenders when
more than one is found, instead of guessing. Matches the script's existing
assert-the-contract style (single archive root entry). Renamed the local to
$nupkgMatches to avoid shadowing PowerShell's automatic $Matches variable under
Set-StrictMode. Verified parse + 0/1/2-match behavior off-agent (symbols pack
correctly excluded).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| $pattern = "Microsoft.AspNetCore.App.Runtime.$rid.*.nupkg" | ||
| $matches = @(Get-ChildItem -Path $ShippingDir -Filter $pattern -File -ErrorAction SilentlyContinue | | ||
| Where-Object { $_.Name -notmatch '\.symbols\.nupkg$' } | | ||
| Sort-Object Name) | ||
| if ($matches.Count -eq 0) { | ||
| throw "Could not find runtime pack nupkg matching '$pattern' under '$ShippingDir'." | ||
| } | ||
| if ($matches.Count -gt 1) { | ||
| # A clean from-source build produces exactly one non-symbols runtime pack | ||
| # per RID. More than one means a stale/duplicate version is lingering in | ||
| # the Shipping dir; pick-first would silently pack the lexicographically | ||
| # smallest (likely wrong) version, so fail loudly instead. | ||
| $names = ($matches | ForEach-Object { $_.Name }) -join ', ' | ||
| throw "Expected exactly one runtime pack nupkg matching '$pattern' under '$ShippingDir', found $($matches.Count): $names." | ||
| } | ||
| $nupkg = $matches[0] | ||
| Write-Host "Found nupkg: $($nupkg.FullName)" |
There was a problem hiding this comment.
Good catch — fixed in c96b8cd. Renamed the local $matches to $nupkgMatches (all 6 references) so it no longer shadows PowerShell's automatic $Matches variable, which matters once Set-StrictMode is in play or regex matching is added later. Pure rename, no behavior change; script still parses and the exactly-one-nupkg assertion is unaffected.
| @@ -0,0 +1,206 @@ | |||
| <# | |||
There was a problem hiding this comment.
Why does this file even exist? It seems to just unpack the nupkg, then re-pack it. It seems like it would be a lot simpler to just save off the nupkg to the archive.
There was a problem hiding this comment.
The main thing is that this transforms the nupkg to match what we already have from dotnet/runtime. Both storing the nupkg or these archives work, each with their own pros and cons. Storing nupkgs leaves us with a simpler pipeline here and having the whole artifact but causes a mismatch between this and dotnet/runtime packs in what is stored and how crank can use each (may be able to share more if both are tar/zip) (still a WIP). While storing the archives makes this pipeline more complex but should allow for more reuse on the crank side as the stored packages with match dotnet/runtime build format.
There was a problem hiding this comment.
Good call — that extract/repack round-trip was pointless, so I cut it. We're now storing the runtime-pack nupkg verbatim in BCS instead of transforming it into an archive. The script (pack-bcs-archives-aspnetcore.ps1 → renamed stage-bcs-nupkg-aspnetcore.ps1) is now just "find the one Microsoft.AspNetCore.App.Runtime.{rid} nupkg and copy it to the fixed BCS artifact name." It still earns its keep for two reasons: it asserts exactly one non-symbols nupkg exists (a stale/dup version should fail loudly, not get silently picked), and it renames the version-stamped filename to the predictable BuildArtifacts_{os}_{arch}_Release_aspnetcore.nupkg that crank resolves from a SHA. crank does the managed/native filtering at consume time, so there's nothing to strip here. The matching crank-side change is in dotnet/crank#878.
PowerShell's $Matches is an automatic variable populated by the -match operator; assigning to the case-insensitive local $matches shadows it and is risky under Set-StrictMode if regex matching is later added. Rename the local to $nupkgMatches and update all references. No behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the extract/validate/repack pack-bcs-archives-aspnetcore.ps1 with a thin
stage-bcs-nupkg-aspnetcore.ps1 that copies the single runtime-pack nupkg to the
fixed BCS artifact name. The BCS now stores the verbatim nupkg (crank filters at
consume time) rather than a transformed archive, so there is no produce-time
layout transform to maintain and crank can re-derive whatever it needs.
- New stage-bcs-nupkg-aspnetcore.ps1 (find the one runtime-pack nupkg, copy to
BuildArtifacts_{os}_{arch}_Release_aspnetcore.nupkg); old pack script removed
- Build jobs drop -Format and invoke the staging script
- Upload dispatcher aspnetcore files -> .nupkg
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What this does
Stands up a new Azure DevOps perf-build pipeline hosted in dotnet/performance that builds every commit on dotnet/aspnetcore
mainfrom source, stages each per-RIDMicrosoft.AspNetCore.App.Runtimeruntime-pack nupkg, and uploads it to the Build Cache Service (BCS @pvscmdupload) so dotnet/crank can resolve ASP.NET Core runtime binaries by commit SHA for performance-regression bisection.This moves work that currently lives in dotnet/aspnetcore (a
perf-build.ymlto be retired there) into dotnet/performance.Why move it here
Team merge control. Hosting the pipeline in dotnet/performance gives the perf team full ownership of the build/upload contract instead of it living in aspnetcore.
What's in the change
New
eng/pipelines/aspnetcore-perf-build.yml— root pipeline: 5 per-config boolean params, theaspnetcorerepo-resource trigger, and the 3 stages (RegisterBuild / Build / UploadArtifacts).eng/pipelines/aspnetcore-perf-build-jobs.yml— authored build jobs, one self-sufficient job per config (Windows x64; Windows x86; Windows arm64; Linux x64; Linux arm64).eng/pipelines/tools/stage-bcs-nupkg-aspnetcore.ps1— the canonical "find the one runtime-pack nupkg → copy it to the fixed BCS artifact nameBuildArtifacts_{os}_{arch}_Release_aspnetcore.nupkg" recipe. The BCS stores the runtime-pack nupkg verbatim (no extract/repack): a nupkg is a zip on every OS, so crank extracts it with the same code path and gets the nupkg's ownruntimes/{rid}/...layout. Storing the raw nupkg keeps the full shipped artifact — crank filters managed/native at consume time — instead of a lossy archive projection that can't be re-derived for historical bisection. (Supersedes the earlier extract/validate/repackpack-bcs-archives-aspnetcore.ps1moved from aspnetcore.)Edited (shared runtime templates — back-compat preserved)
register-build-jobs.yml,templates/register-build-job.yml,upload-build-artifacts-jobs.yml,templates/upload-build-artifacts-job.yml— added ashaparameter (default$(Build.SourceVersion)), threaded through to the BCS blob path. The 5aspnetcore_*upload branches (added in Generalize BCS upload + register-build templates with repoName parameter #5241) now threadshatoo, and their artifact file lists are.nupkg.Key design decisions / things to scrutinize
Can't reuse aspnetcore's
default-build.ymlcross-repo. That template has hard@selfreferences, and@selfalways resolves to the root pipeline repo (= performance), which doesn't carry aspnetcore'seng/commoncontract. So the build jobs are rebuilt from source here as plain- job:blocks that invoke aspnetcore's owneng/build.cmd/eng/build.sh.One job per arch (Windows split). Unlike aspnetcore's
ci-public.yml, which builds Windows x64→x86→arm64 sequentially in one job (x86/arm64 reuse the x64 step's managed build + on-machine native toolchain), each config here is its own self-sufficient job gated by its boolean — uniform with the Linux jobs. This makes per-config gating exact (e.g. queue onlyaspnetcore_arm64_windows→ onlyWindows_arm64_buildruns; x64 isn't built just to host arm64) and runs the three Windows arches in parallel instead of back-to-back. The tradeoff: each Windows arch carries its own-all(full managed build) and-nativeToolsOnMachine(the toolchain the shared x64 step used to provide); x86/arm64 keep-noBuildNative(faithful to ci-public's per-arch args — native runtime bits restore from packages rather than cross-building on the x64 host). Scrutinize: standalone Windows-arch builds are not locally provable; the first real queued run validates that x86/arm64 restore native correctly without the preceding x64 step.SHA inversion (load-bearing). Because the pipeline is hosted in performance,
Build.SourceVersionis the performance commit, not the aspnetcore one. The correct BCS{sha}is the triggering aspnetcore commit =$(resources.repositories.aspnetcore.version). The root pipeline passes that assha:to the register/upload templates. The shared templates defaultshato$(Build.SourceVersion)so the existing dotnet/runtime perf-build is unaffected.Build↔commit correlation (Option B). The build record's sourceVersion is performance's commit, so the dotnet-performance-infra indexer does not rely on sourceVersion. Instead it derives the aspnetcore commit directly from the build's
resources.repositories.aspnetcore.versionvia the runs API — intrinsic to every build that carries the aspnetcore repo resource, so no pipeline-side run tagging is needed. This avoids the silent-empty-latestBuildsfailure mode a forgotten or hand-edited build tag could otherwise introduce.Repo-resource CI trigger for per-commit firing. The aspnetcore AzDO mirror
internal/dotnet-aspnetcoreis atype: gitrepository resource withtriggeronmain(batch: false→ eager, per-commit). A push to aspnetcore main fires this pipeline (Build.Reason == ResourceTrigger); jobs check out that resource at the triggering commit. (Note: stage gating usesResourceTrigger, notIndividualCI— another consequence of the host inversion vs. the aspnetcore reference.) All three stages carry the internal + (ResourceTrigger|Manual) gate: RegisterBuild/UploadArtifacts via${{ if }}(compile-time removed in public), Build via a runtimecondition:so it skips rather than fails in a public/manual misqueue (a fully${{ if }}-gated pipeline would expand to zero stages, which AzDO rejects).Build from PUBLIC feeds. No internal runtime download, no
enable-internal-runtimes/get-delegation-sasstep, nodotnetbuilds-internal-readconnection. Proof: aspnetcore's ownci-public.ymlbuilds every public PR with_InternalRuntimeDownloadArgsempty. Only the BCS upload uses the proven.NET Performanceconnection.Per-config booleans for forward-compat indexing. 5 boolean params (default true) gate their build jobs and feed the register/upload
buildTypearrays, so the dotnet-performance-infraMissingBuildsTriggerPerConfiguration indexer can count exactly these keys. In v1's eager mode all 5 are always built; the booleans let a future Function queue a subset.No 1ES. Per-commit perf artifacts go to a private BCS cache (unsigned, not redistributed), so we don't extend
1ES.Official— matching dotnet/runtime's perf-build.Atomicity.
continueOnError: falseeverywhere; any failure sinks the build to Failed and the indexer skips that SHA. UploadArtifactsdependsOn: [Build, RegisterBuild],condition: succeeded().The 5 configs / RIDs (locked)
aspnetcore_x64_linuxLinux_x64_buildaspnetcore_arm64_linuxLinux_arm64_buildaspnetcore_x64_windowsWindows_x64_buildaspnetcore_arm64_windowsWindows_arm64_buildaspnetcore_x86_windowsWindows_x86_buildBCS layout:
builds/aspnetcore/buildArtifacts/{sha}/{configKey}/{buildInfo.json,nupkg}.Validation status
The full pipeline can't be run locally. All 7 YAML files parse; job-name ↔
dependencyJobName, artifact names, and the staging-script output filename were cross-checked for consistency. Real validation = a manual queued run after merge (pool image availability, aspnetcore from-source build on the dnceng internal pool, end-to-end BCS upload). The aspnetcore-hosted version will be retired once this is validated.