feat(platform): selectable AKS modes, pinned service images, Entra-only data plane, and cluster-side CI/CD onboarding - #36
Draft
yuchen-wang99 wants to merge 19 commits into
Draft
Conversation
AKS Automatic remains the default cluster topology. The validated Base SKU with Node Autoprovisioning becomes an explicit alternative in infra/aks-base.bicep, sharing the VNet, NAT gateway, control-plane identity, OIDC issuer, storage profile, and output contract with the Automatic template. Both templates now publish the kubelet identity object ID and the managed Istio revision they pin, so the kubelet AcrPull grant is never silently skipped and the sidecar revision is never hardcoded downstream. Also imports the Microsoft-tenant data-plane posture: Gremlin and Service Bus disable local authentication, Cosmos data-plane roles are granted in every subscription, system Cosmos secrets and the per-partition record container are provisioned, and the AKS system pool uses region-aware availability zones. Refs: ADR-031, ADR-032, ADR-033 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
The ingress profile becomes the only Flux owner of the Gateway, so an HTTPS profile can no longer be reverted to HTTP by a competing reconciler. The osdu namespace no longer hardcodes a managed Istio revision; it is substituted from spi-ingress-config so Flux and the CLI converge on the revision the cluster actually runs. Also brings in the OSDU service and init chart changes: creator seeding during Entitlements initialization, ordered replaceable init hooks, per-service Application Insights role names, Workload Identity data-plane settings, and the image-lock keys used by the GHCR service baseline. Refs: ADR-030, ADR-032, ADR-034, ADR-035 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Adds the deployment-mode surface: spi up --aks-mode automatic|base, the spi-aks-mode resource-group tag, validation of an existing cluster's real SKU and node-provisioning profile, and rejection of in-place mode changes. Brings across the SPI CLI capabilities: spi onboard with cluster rehome and role-scope preservation, the no-data test identity, per-identity creator resolution and Entitlements seeding, spi reconcile --suspend/--resume, the public GHCR image baseline pinned by digest, and the complete Application Insights mode with persistence and disabled-agent fallback. Hardens deployment: the kubelet identity and Istio revision are required outputs, the kubeconfig pins the deployment tenant, cluster-admin verification uses raw ARM instead of Graph-dependent lookups, and registry reads retry transient failures. spi info reads the Flux extension ConfigMap from the namespace the extension actually owns. Refs: ADR-024, ADR-025, ADR-028, ADR-030, ADR-031, ADR-032, ADR-035 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Imports the SPI Stack decision records as ADR-024 through ADR-030 without altering their content, renumbering only to avoid collisions with the upstream sequence. Adds five new decisions for the reconciled architecture instead of editing accepted records in place: selectable AKS deployment modes (ADR-031), runtime-resolved managed Istio revision (ADR-032), the Entra-first data-plane boundary (ADR-033), single Gateway ownership (ADR-034), and per-identity authorization with explicit membership seeding (ADR-035). The index records which earlier decisions they supersede or extend. Also updates the README, architecture and design docs for the two deployment modes, and extends CI to build both AKS templates and to what-if both Application Insights modes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Resolving the AKS mode or the Application Insights mode against an existing environment can legitimately fail, for example when a rerun requests a mode the environment cannot change to. Those failures printed a Python traceback instead of the message they carry. Both resolutions now surface as a normal CLI error with exit code 2, matching how invalid image and creator options are already reported. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
The Automatic SKU validates the system pool against its recommended zone set and rejects cluster creation with AKSAutomaticSKUFeatureValidationError when a region is given a reduced list. Automatic now requests all three zones. The region-aware zone map stays in the Base template, where it is still needed: eastus2 has no zonal capacity in zone 2 for the Base system VM size. Also removes the Microsoft-tenant deploy journal. It is a point-in-time engineering log carrying environment-specific identifiers, and its durable decisions are already recorded in the ADRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Three defects found while reviewing the reconciliation: Flux substitutes an undefined variable with an empty string, so an environment created before the revision was published would apply an empty istio.io/rev on resume and silently lose sidecar injection. spi reconcile --resume now backfills ISTIO_REVISION in spi-ingress-config from the live cluster and fails if the bootstrap ConfigMap is absent. spi down validated the AKS mode before deleting anything, so a stale tag or a tag/cluster mismatch could block the very command an operator needs to recover. Teardown now warns and proceeds. Service-principal deployments resolved an empty deployer object ID, which silently skipped the Key Vault Secrets Officer grant and failed the later bootstrap secret writes. The object ID is now looked up by appId, degrading to the previous behavior when Graph is unavailable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Availability zones are restricted per subscription, not just per region, so a size published in three zones can be usable in fewer. Hardcoding zones failed both ways: the full set was rejected with AvailabilityZoneNotSupported where a zone is restricted, and a reduced set was rejected by the Automatic SKU, which validates the system pool against the region's recommended zones. The CLI now resolves the usable zones for the system pool size in the target region and passes them, along with the size itself, to whichever AKS template is selected. The region-specific zone map is gone from the Base template. Automatic fails early with an actionable message when a region cannot offer every zone, naming the restricted zones and pointing at --aks-mode base. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Node autoprovisioning resolves the node subnet as the cluster's control-plane identity. Bicep declares that grant, but the identity is created in the same deployment, and principalType ServicePrincipal tells ARM to skip the principal existence check rather than wait for it. When Entra replication has not caught up, ARM reports the assignment as created and it is never committed: the deployment succeeds, no write appears in the activity log, and the assignment does not exist. The failure is invisible until Karpenter provisions the first workload pool, roughly forty minutes after spi up exits, and then every AKSNodeClass reports SubnetsReady=False with a 403 on virtualNetworks/subnets/read. The CLI now verifies the assignment through raw ARM after cluster creation and reapplies it when missing, matching how the deployer cluster-admin grant is already verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Upstream reorganized ingress and profiles while this branch was in flight: the minimal profile arrived, routes split into middleware and osdu trees, the gateway TLS overlay became spi-gateway-tls, and TLS certificates moved into platform. Adopt the upstream structure rather than carrying a parallel one. The profiles own the base Gateway and the ingress profiles own the TLS overlay, so the local single-owner variant and its ADR are dropped; whether the two owners drift across reconcile intervals is a question for upstream, with evidence, rather than a unilateral restructure here. Profile.FULL was removed upstream in favor of minimal, so the alias that mapped it onto core goes with it and the Flux profile path is taken straight from the selected profile. Keeps the local additions on top: the namespaces Kustomization still substitutes ISTIO_REVISION in both profiles, and the ADRs renumber to 026-036 to leave room for upstream ADR-024 and ADR-025. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
The community registry garbage-collects master SHA tags, keeping only the few most recent builds. The pinned df3e786 tag no longer resolves, so the schema-load Job fails with ErrImagePull and spi-osdu-schema-load plus the dependent spi-osdu-reference Kustomizations never become Ready. Repin to 146f2f6, the newest surviving tag, which is also the commit the schema-service image is built from so loader and service agree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Two upstream-readiness defects found while preparing the port back to
Azure/osdu-spi-stack.
1. DEFAULT_GHCR_ORG defaulted to a personal fork organization, so an adopter
running 'spi up' with no flags would silently pull all 13 OSDU service
images from an org they have no relationship with. Default to 'Azure' and
describe the organization as selectable via --image-org. The GitHub API
path uses the value verbatim, and the GHCR path already lowercases it, so
'Azure' yields /repos/Azure/<svc> and ghcr.io/azure/<svc>. Adopters whose
images are published elsewhere pass --image-org, or --image-source
community for the OSDU GitLab fallback.
2. The default region had been reverted to eastus2, undoing the upstream
change that moved it to westus3 and documented the reason ('eastus2 and
centralus have shown API Server VNet Integration capacity constraints').
This was an unintended rebase clobber and it actively breaks the AKS
Automatic mode added here, because eastus2 restricts availability zone 2
for Standard_D4lds_v5 while AKS Automatic requires all zones. Restore
westus3 in the CLI, Config and infra/main.bicep, including the upstream
help text.
The two affected tests hardcoded a region in their cluster fixtures and now
derive it from Config, so they no longer depend on the default's value.
201/201 tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
…e-safe Four defects this branch introduced against azure/main, all found while preparing the upstream port. The schema-load Job's image was repinned, but a Job's spec.template is immutable. Flux's server-side apply fails with an immutable-field error, the stale Job survives, spi-osdu-schema-load reports Ready=False and the dependent reference-services Kustomization stays blocked. Observed for real: after the repin commit a reconcile left the Job on the previous tag until it was deleted by hand. Add kustomize.toolkit.fluxcd.io/force so Flux recreates it; safe for a one-shot loader because already-registered schemas return 409 and are skipped. The same commit raised the Job's activeDeadlineSeconds to 3600s but left the Kustomization timeout at 35m, and left its comment still claiming 1800s. Upstream had deliberately coupled the two. Raise the timeout to 65m and correct the comment so a legitimately slow load is not reported as failed while the Job is still running. The osdu Namespace now takes istio.io/rev from a substituted variable, but ensure_istio_revision_published() was only called under --resume. Plain reconcile also fetches the new commit and annotates spi-namespaces, and Flux renders an undefined variable as an EMPTY string rather than failing, so an environment created before the key existed would be relabelled istio.io/rev: " and silently lose sidecar injection for every OSDU service. Publish the revision on both paths, and give the manifest a defensive default so an empty label is impossible. Suspend is unaffected: it applies no manifests and returns earlier. Three ADR cross-references were left stale by renumbering on import: 027 cited a nonexistent ADR-037, 028 was labelled ADR-031 while linking 033, and 033 attributed Base+NAP to ADR-026, which is the integration-tests decision. Adds three reconcile regression tests. Verified the plain-reconcile test fails when the fix is reverted. 204 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
…M-token OID Four defects in code this branch adds, found while preparing the upstream port. Deployer OID resolution had regressed against upstream. azure/main resolves the principal via SPI_DEPLOYER_OID, then the ARM access token, and only then Microsoft Graph, because Conditional Access token protection can refuse a Graph token (AADSTS530084) while ARM access is fine. This branch kept that logic in _grant_deployer_cluster_admin but added a second resolver that went straight to Graph: users hard-failed, and service principals degraded to an empty object ID that silently skipped the Key Vault Secrets Officer grant, surfacing only during Phase 6 secret writes after the cluster already existed. Try the ARM token first in both paths, keeping Graph as the fallback. Onboarding fetched cluster credentials with check=False and then ran every kubectl without pinning a context, so a failed fetch would apply Flux RBAC and the entitlements seeding Job to whichever cluster the operator was last pointed at. Make the fetch fatal, give the run its own named context, and route every kubectl through a helper that supplies it. The seed Job's client-side wait was 240s while the Job's own deadline was 600s and the in-Job group discovery retried for 300s, so the cleanup in inally destroyed Jobs that were still going to succeed; a cold cluster can spend minutes just provisioning a node. Express the three budgets as ordered constants and wait for the Job deadline plus scheduling margin. Secret reconciliation checked only AZURE_CLIENT_ID while writing three secrets, and detected a rehome only when the AZURE_CLIENT_ID variable was present. A repo missing the tenant or subscription secret, or missing the variable that records which cluster it belongs to, kept stale or absent values indefinitely. Check all three and treat an unknown link as a reason to rewrite. Adds 10 regression tests. Each fix was mutation-tested: reverting it makes the corresponding test fail. Validated live against spi-stack-auto2, including deleting AZURE_TENANT_ID and confirming a re-onboard repairs it. 214 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
…esolution Two capabilities in this branch had no decision record. ADR-037 covers 'spi onboard', the cluster-side half of service-fork CI/CD onboarding: it establishes the managed identity, federated credentials, Azure RBAC, Flux read access, entitlements membership and repository secrets that the deploy lane needs, as one idempotent command that reconciles rather than assumes and re-homes a fork to a rebuilt cluster in a single step. ADR-038 covers resolving the system pool's availability zones from the target subscription. The cluster template already exposes an availabilityZones parameter documented as an override; the decision here is that the CLI supplies it by reading the compute SKU catalogue and subtracting subscription restrictions, because zone availability is per subscription rather than per region and the Automatic SKU rejects a reduced set. Both are appended after the existing 026-036 block, which keeps the record chronological and gapless (001-038) and leaves every existing number untouched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR significantly extends the SPI Stack CLI + GitOps footprint by adding selectable AKS deployment modes (Automatic vs Base+NAP), switching the default service-image baseline to GHCR with digest pinning, tightening the Entra-first data-plane posture (local auth off where supported), and adding cluster/bootstrap improvements to support CI/CD onboarding and safer reconciliation.
Changes:
- Add AKS mode selection and persistence (
automaticdefault,baseas an immutable per-environment choice) with a newinfra/aks-base.bicepentrypoint. - Introduce a first-class service image supply chain: resolve images from GHCR (tag or git ref) and pin immutable OCI digests into
osdu-image-lock, consumed by HelmReleases. - Expand bootstrap and runtime wiring: managed Istio revision publishing/substitution, creator identity projection/seeding, Application Insights opt-in with a safe “disabled” fallback, and improved CLI robustness on Windows (
resolve_command).
Reviewed changes
Copilot reviewed 87 out of 87 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_reconcile.py | Adds tests for Flux suspend/resume behavior and image selector resolution semantics. |
| tests/test_istio_revision.py | Adds tests for publishing/backfilling the managed Istio revision into Flux substitution inputs. |
| tests/test_images.py | Expands image tests to cover GHCR tag/ref resolution and digest pinning, plus lock rendering metadata. |
| tests/test_deployer_oid.py | Adds regression tests ensuring deployer principal resolution prefers ARM token over Graph. |
| tests/test_creator_identity.py | Adds tests for JWT claim projection, creator seeding inputs, and init chart ordering/behavior. |
| tests/test_config_validation.py | Adds validation coverage for new config defaults (profiles, App Insights, GHCR image baseline). |
| tests/test_cluster_network_grant.py | Adds tests for repairing/ensuring the cluster identity VNet role assignment (Base+NAP dependency). |
| tests/test_bicep_compile.py | Improves Azure CLI discovery for bicep compilation tests (Windows-friendly). |
| tests/test_azure_infra.py | Adds tests ensuring idempotent reruns read kubelet identity + Istio revision outputs. |
| tests/test_availability_zones.py | Adds tests for subscription-aware system-pool availability zone resolution. |
| tests/test_aks_mode.py | Adds tests for persistent AKS mode resolution, tagging, template selection, and immutability rules. |
| src/spi/templates.py | Adds App Insights “disabled” fallback config, updates Istio identity projection behavior, and injects creator IDs into init values. |
| src/spi/shell.py | Adds resolve_command() and applies it to subprocess-based kubectl/az calls for Windows PATHEXT compatibility. |
| src/spi/secrets.py | Updates direct kubectl subprocess calls to use resolve_command(). |
| src/spi/ingress.py | Publishes ISTIO_REVISION into spi-ingress-config and resolves az/kubectl executables via resolve_command(). |
| src/spi/info.py | Reads Flux extension metadata configmap from flux-system. |
| src/spi/images.py | Adds GHCR + GitHub API resolution paths, digest pinning, and image-source selection model. |
| src/spi/identity.py | Introduces shared JWT claim decoding and projected-identity helpers. |
| src/spi/guard.py | Makes SPI cluster fingerprinting/Flux namespace detection namespace-agnostic and uses resolve_command(). |
| src/spi/deploy.py | Reworks bootstrap sequencing, adds Flux “extension-only then activate” flow, image-lock resolution updates, and config-driven restarts on ConfigMap changes. |
| src/spi/config.py | Adds config for AKS mode, image-source selection, App Insights, creator seeding, and updates defaults/validation. |
| src/spi/checks.py | Updates kubelogin description to reflect Azure RBAC for Kubernetes usage beyond AKS Automatic. |
| src/spi/bootstrap.py | Improves Istio revision detection and makes namespace creation idempotent via apply. |
| software/stacks/osdu/services/workflow.yaml | Adds digest-pinned image support for workflow service. |
| software/stacks/osdu/services/storage.yaml | Adds digest-pinned image support and enables MSI flag for token-based data-plane paths. |
| software/stacks/osdu/services/search.yaml | Adds digest-pinned image support for search service. |
| software/stacks/osdu/services/schema.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services/partition.yaml | Adds digest-pinned image support and tweaks Redis cache TTL to reduce stale empty partition-list risk. |
| software/stacks/osdu/services/legal.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services/indexer.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services/indexer-queue.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services/file.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services/entitlements.yaml | Adds digest-pinned image support and enables MSI flag. |
| software/stacks/osdu/services-reference/unit.yaml | Adds digest-pinned image support for reference service. |
| software/stacks/osdu/services-reference/crs-conversion.yaml | Adds digest-pinned image support for reference service. |
| software/stacks/osdu/services-reference/crs-catalog.yaml | Adds digest-pinned image support for reference service. |
| software/stacks/osdu/schema-load/job.yaml | Forces Flux recreation of immutable Job and extends deadline; updates schema-load image pin rationale. |
| software/stacks/osdu/profiles/minimal/stack.yaml | Adds substituteFrom wiring so ISTIO_REVISION substitution is available in minimal profile too. |
| software/stacks/osdu/profiles/core/stack.yaml | Adds substituteFrom wiring and increases schema-load Kustomization timeout to match Job deadline. |
| software/stacks/osdu/init/release.yaml | Switches HelmRelease reconcile strategy to Revision and updates init-job behavior notes. |
| software/components/namespaces/namespaces.yaml | Substitutes Istio revision from spi-ingress-config to prevent drift/hardcoding. |
| software/charts/osdu-spi-service/values.yaml | Adds image.digest value to support digest pinning. |
| software/charts/osdu-spi-service/templates/deployment.yaml | Renders repository@digest when available and sets per-service App Insights role name. |
| software/charts/osdu-spi-service/Chart.yaml | Updates chart description and bumps chart version. |
| software/charts/osdu-spi-init/values.yaml | Adds creatorUserIds value used for Entitlements membership seeding. |
| software/charts/osdu-spi-init/templates/scripts.yaml | Extends entitlements-init script to seed creator identities via Entitlements API. |
| software/charts/osdu-spi-init/templates/partition-init.yaml | Converts init job into replaceable ordered Helm hooks. |
| software/charts/osdu-spi-init/templates/entitlements-init.yaml | Converts init job into replaceable ordered Helm hooks and injects creator IDs JSON. |
| software/charts/osdu-spi-init/Chart.yaml | Bumps init chart version. |
| README.md | Updates top-level README to document selectable AKS modes, image selection, and creator seeding controls. |
| infra/modules/rbac.bicep | Clarifies deployer principal docs, adds kubelet AcrPull context, and documents least-privilege Service Bus roles. |
| infra/modules/partition.bicep | Hard-disables Service Bus local auth, keeps SB KV secret as DISABLED placeholder, and clarifies Cosmos SQL local-auth boundary + RBAC role assignment. |
| infra/modules/cosmos-gremlin.bicep | Disables Gremlin local auth and requires principalId for Gremlin RBAC. |
| infra/main.bicep | Reorders and clarifies App Insights resources/outputs and removes Gremlin key secret path (local auth off). |
| infra/flux.bicep | Adds extension-only activation mode, gitops namespace configurability, and optional private-repo auth reference. |
| infra/aks.bicep | Updates comments and outputs kubelet object ID + pinned Istio revision. |
| infra/aks-base.bicep | Adds the Base SKU + NAP cluster template with BYO VNet + managed Istio and matching outputs. |
| docs/design/workload-identity.md | Updates workload identity documentation, including Service Bus posture and indexer-queue compatibility notes. |
| docs/design/secret-lifecycle.md | Updates secret lifecycle to reflect local-auth-disabled Service Bus and DISABLED placeholder KV secret. |
| docs/design/README.md | Minor documentation wording updates. |
| docs/design/gateway-ingress.md | Updates ingress documentation for selectable AKS modes and single gateway ownership model. |
| docs/design/flux-reconciliation.md | Updates Flux layer description around gateway ownership placement. |
| docs/design/deployment-lifecycle.md | Updates lifecycle documentation for AKS mode selection, bootstrap ordering, and image baseline changes. |
| docs/design/bicep-architecture.md | Updates Bicep staging explanation for two AKS entrypoints and shared downstream stages. |
| docs/decisions/README.md | Updates ADR index and status notes for newly added ADRs. |
| docs/decisions/038-subscription-resolved-availability-zones.md | Adds ADR-038 documenting subscription-aware zone resolution. |
| docs/decisions/037-cluster-side-onboarding.md | Adds ADR-037 documenting cluster-side onboarding model. |
| docs/decisions/036-per-identity-authorization.md | Adds ADR-036 documenting per-identity auth projection + membership seeding. |
| docs/decisions/035-entra-first-data-plane.md | Adds ADR-035 documenting Entra-first data-plane boundary and exceptions. |
| docs/decisions/034-runtime-resolved-istio-revision.md | Adds ADR-034 documenting runtime publication/substitution of Istio revision. |
| docs/decisions/033-selectable-aks-deployment-modes.md | Adds ADR-033 documenting persistent AKS mode selection and immutability. |
| docs/decisions/032-spi-ghcr-service-images.md | Adds ADR-032 documenting GHCR baseline + digest pinning supply chain. |
| docs/decisions/031-record-ingestion-data-plane.md | Adds ADR-031 documenting record-ingestion enablement requirements. |
| docs/decisions/030-application-insights-telemetry.md | Adds ADR-030 documenting optional App Insights wiring and disabled fallback. |
| docs/decisions/029-disable-local-auth-data-services.md | Adds ADR-029 documenting local-auth disabling posture and service requirements. |
| docs/decisions/028-aks-base-node-autoprovisioning.md | Adds ADR-028 (superseded) documenting Base+NAP adoption rationale. |
| docs/decisions/027-deploy-lane-invariants.md | Adds ADR-027 documenting deploy-lane CI invariants and fixes. |
| docs/decisions/026-adme-aligned-integration-tests.md | Adds ADR-026 documenting ADME-aligned integration testing approach. |
| docs/architecture.md | Updates architecture doc for selectable AKS modes and staging details. |
| .github/workflows/validate.yml | Expands bicep build validation and adds what-if coverage for App Insights off/on. |
| .github/workflows/codeql.yml | Avoids CodeQL SARIF/database upload failures on private forks lacking Advanced Security. |
| .github/copilot-instructions.md | Updates repo agent context and documents new key decisions and updated ADR counts. |
Comments suppressed due to low confidence (1)
docs/design/workload-identity.md:67
- Same issue as above: this section references
Azure Service Bus Data Owner, but the stack actually assignsAzure Service Bus Data SenderandAzure Service Bus Data Receiverto cover publish/consume without granting entity management.
Service Bus local authentication is disabled in SPI Stack. Services must use the shared UAMI via Workload Identity and the `Azure Service Bus Data Owner` role assignment.
Review follow-ups on the upstream pull request. CodeQL flagged incomplete URL substring sanitization in the creator-identity helper: the token issuer was checked with `in`, which matches anywhere in the string. Verified by execution that lookalike issuers such as https://evil.example.com/sts.windows.net/tenant/ and https://sts.windows.net.attacker.io/tenant/ were accepted and returned the attacker-supplied identity. Exposure is bounded, because the claims come from a token the local Azure CLI already obtained and the cluster-side Istio RequestAuthentication independently pins the issuer, but this helper decides which principal is seeded as a creator, so parse the URL and compare the host exactly. Adds regression tests covering userinfo and case variations. The extension-only Flux pass waited for the osdu-flux namespace, which ensure_namespaces() already created earlier in the same phase, so the wait always passed instantly and proved nothing. Wait for flux-system, the namespace the extension itself creates, so GitOps activation cannot race controller installation. Confirmed against a live cluster that the Flux controllers run in flux-system while osdu-flux holds only the Flux objects. kubectl_json() appends -o json itself, so the guard's call site passed the flag twice. Two design docs claimed the stack grants Azure Service Bus Data Owner, but rbac.bicep assigns the least-privilege Data Sender and Data Receiver pair; the docs now match what is deployed. One bullet in the gateway design doc contained a literal escape sequence that merged two list items. 216 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
The onboarding changes introduced a few lines ruff formats differently, and the new regression tests indexed re.search() results directly, which the type checker rejects because the result is Match | None. Assert the match first, then read the group, which also produces a clearer failure if the embedded seed script is ever restructured. No behaviour change. 216 tests pass; ruff format, ruff check and ty all clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ae6b083b-4c0b-4a2c-baa9-2c93f0b5eb42
Collaborator
|
Holding on this for now to implement in a smaller feature by feature scenario. |
This was referenced Jul 28, 2026
danielscholl
marked this pull request as draft
August 20, 2026 19:05
This was referenced Aug 24, 2026
This was referenced Aug 24, 2026
Collaborator
|
Copilot resolve the merge conflicts in this pull request |
Resolve conflicts: adopt main's Entra-only data plane (ADR-027), run_process launcher, spi-cluster-config Istio revision, and deployer-principal threading; keep the branch's AKS mode selection, GHCR image supply chain, and onboarding. Renumber colliding ADRs 026-028 to 040-042 and drop the branch ADRs superseded by ADR-027. Co-authored-by: danielscholl <26447046+danielscholl@users.noreply.github.com>
Co-authored-by: danielscholl <26447046+danielscholl@users.noreply.github.com>
Contributor
Merged Where main and the branch disagreed, main's trunk decision won:
Branch features kept intact: selectable AKS modes, GHCR image supply chain with digest pinning, ADR numbering collided on 026-029; the branch's files were renumbered to 040 (ADME-aligned integration tests), 041 (deploy-lane invariants), and 042 (AKS Base + NAP), references were repointed, and Verified: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the SPI Stack with a second cluster topology, a first-class service image supply chain, an Entra-only data plane, and the cluster-side onboarding needed to drive CI/CD from service forks.
Validated end to end on live clusters in two Azure tenants, including a policy-constrained corporate tenant.
What this adds
Selectable AKS deployment modes
--aks-modechooses between two validated topologies, and the choice persists per environment:automaticbaseBase mode exists because Automatic constrains the mutating webhooks cert-manager and CloudNativePG rely on. Both modes share one GitOps stack and one CLI.
infra/aks-base.bicepis added alongside the existing template. (ADR-033, ADR-028)Service images as a pinned supply chain
Service images are resolved once from a tag selector and then pinned by immutable OCI digest into a cluster-side image lock that the Flux manifests substitute from.
--image-tagselects a coordinated release,--image-refresolves a Git ref across every service repository for feature validation, and--image-source communityretains the OSDU GitLab registry. (ADR-032)Entra-only data plane
Local (key and SAS) authentication is disabled on the Cosmos and Service Bus accounts, and services reach them through Workload Identity with data-plane role assignments instead. This removes the key material from the deployment entirely — and it is what allows the stack to deploy in tenants whose policy denies creating data services with local auth enabled. (ADR-029, ADR-035, ADR-021)
Per-identity authorization
The Istio identity filter projects only the caller's identity; authorization is decided by explicit entitlements membership rather than inferred from the token. Deploying operators and CI identities are seeded deliberately rather than implicitly trusted. (ADR-036)
Cluster-side CI/CD onboarding
spi onboardis a new command that establishes everything a service fork needs to deploy into the cluster: managed identity, federated credentials matching the exact OIDC subjects, Azure RBAC, Flux read access for the deploy lane's pre-flight, entitlements membership for the acceptance suite, and the repository secrets and link variables that name it all.It reconciles rather than assumes, so re-running repairs partial state, and pointing it at a different cluster re-homes a fork in one step.
--dry-runprints the full plan without changing anything. (ADR-037, and ADR-026/ADR-027 for the lane it serves)Deployment robustness
Validation
End-to-end deployment, two tenants
Corporate tenant, policy-constrained. Deployed on AKS Automatic 1.36 with the full Azure PaaS estate. The Entra-only data plane is what makes this succeed: the tenant's policy denies creating Cosmos accounts with local auth enabled. Both bootstrap jobs completed, all Flux Kustomizations reconciled, and the external smoke suite passed against the public gateway — including the full ingestion path, where a record written through the storage API was found in the search index, exercising storage → Service Bus → indexer → search.
Secondary tenant. Deployed again on AKS Automatic 1.36 from a clean subscription: 24/24 Kustomizations Ready, 16/16 service pods, both bootstrap jobs completing unattended, schema load registering all 1,386 schemas. The external smoke suite passed here too, and the runtime-resolved Istio revision was confirmed flowing from the cluster template through the CLI into the namespace label, with sidecars injected.
CI/CD onboarding and deploy lane
A service fork was onboarded to a freshly created cluster with
spi onboardand then exercised through its full pipeline —yuchen-osdu/storagerun 30301239194:Also verified: OIDC federation via the smoke workflow, break-glass rollback moving the service back to its previous digest with the pod healthy afterwards, and re-homing a fork that had previously been onboarded to a now-deleted cluster.
Automated tests
214 tests pass, covering deployment-mode selection, availability-zone resolution, image resolution and locking, Istio revision handling, creator identity projection, onboarding, and reconcile behaviour.
Notes for reviewers
automaticremains the default, so existing behaviour is unchanged unless--aks-mode baseis selected.