support background steps

[lokesh755]

Summary

Adds runner support for background steps - enabling concurrent step execution within a single GitHub Actions job. This introduces four new workflow YAML keywords: background: true, wait, wait-all, and cancel.

Motivation

Today, all steps in a job run sequentially. Common patterns like "start a dev server, run tests, stop the server" or "upload artifacts while tests continue" require workarounds (& backgrounding, service containers). Background steps provide first-class support for concurrent step execution with explicit synchronization primitives.

What's Changed

New step types (SDK layer)

• WaitStep, WaitAllStep, CancelStep — new pipeline step types with JSON deserialization support
• ActionStep.Background — boolean property indicating a step should run asynchronously

Core execution engine (StepsRunner.cs)

• Background steps launch via Task.Run and execute concurrently with subsequent foreground steps
• wait: <id> blocks until specific background step(s) complete
• wait-all: true blocks until all prior background steps complete
• cancel: <id> sends SIGTERM with a 7.5s grace period
• Concurrency limited to 10 concurrent background steps via SemaphoreSlim
• Implicit wait-all injected before post-job hooks if any background steps haven't been explicitly waited on (visible in UI as a timeline entry)

Thread safety (StepsContext.cs)

• Added lock around SetOutput, SetConclusion, SetOutcome — background steps write to the shared steps context from thread pool threads

Related to

https://github.com/github/actions-runtime/issues/5420
https://github.com/github/actions-runtime/issues/5421

[Copilot]

Pull request overview

Adds first-class “background step” support to the runner by allowing action steps to run concurrently, introducing control steps (wait / wait-all / cancel), and surfacing related metadata to the Results service contracts.

Changes:

• Introduces new pipeline step types (Wait, WaitAll, Cancel) and runner step implementations to coordinate background execution.
• Extends StepsRunner to start background actions concurrently, provide wait/cancel semantics, and attempt to isolate per-step GitHub context.
• Adds Results/RunService contract fields and L0 tests to validate concurrency and metadata propagation.

Show a summary per file

File	Description
src/Test/L0/Worker/BackgroundStepsL0.cs	Adds L0 coverage for concurrent background steps, waits, cancels, and steps-context thread-safety.
src/Sdk/WebApi/WebApi/ResultsHttpClient.cs	Adds mapping of timeline record variables into workflow step payloads (but also includes debug file logging).
src/Sdk/WebApi/WebApi/Contracts.cs	Extends Results service Step contract with background/control-step metadata DTOs.
src/Sdk/RSWebApi/Contracts/StepResult.cs	Extends RunService step result contract with background/control-step metadata fields.
src/Sdk/DTPipelines/Pipelines/WaitStep.cs	Adds pipeline model for a “wait” step.
src/Sdk/DTPipelines/Pipelines/WaitAllStep.cs	Adds pipeline model for a “wait-all” step.
src/Sdk/DTPipelines/Pipelines/CancelStep.cs	Adds pipeline model for a “cancel” step.
src/Sdk/DTPipelines/Pipelines/StepConverter.cs	Enables JSON deserialization for new step types.
src/Sdk/DTPipelines/Pipelines/Step.cs	Registers new known step types and extends the StepType enum.
src/Sdk/DTPipelines/Pipelines/ActionStep.cs	Adds Background flag and ensures it’s cloned.
src/Runner.Worker/WaitStepRunner.cs	Adds runner step type for “wait” control step.
src/Runner.Worker/WaitAllStepRunner.cs	Adds runner step type for “wait-all” control step.
src/Runner.Worker/CancelStepRunner.cs	Adds runner step type for “cancel” control step.
src/Runner.Worker/StepsRunner.cs	Implements background execution, wait/wait-all/cancel handling, slot limiting, and GitHubContext isolation.
src/Runner.Worker/StepsContext.cs	Adds locking around step output/outcome/conclusion mutations.
src/Runner.Worker/JobExtension.cs	Wires new pipeline step types into job initialization and sets timeline variables for results metadata.
src/Runner.Worker/ExecutionContext.cs	Adds timeline-record variable setter and maps those variables into StepResult; adjusts template evaluator feature gating.
src/Runner.Worker/BackgroundStepContext.cs	Adds per-background-step tracking (task, CTS, result, external id).

Copilot's findings

Comments suppressed due to low confidence (4)

src/Sdk/WebApi/WebApi/ResultsHttpClient.cs:568

• File.AppendAllText("/tmp/bg-steps-debug.log", ...) is executed without a try/catch here. If the path is unavailable (e.g., Windows runner, locked filesystem, permissions), it will throw and can break step updates. Remove this statement or guard it behind safe, platform-appropriate diagnostics.

            System.IO.File.AppendAllText("/tmp/bg-steps-debug.log", $"[BG-DEBUG]   Result: name={step.Name}, isBackground={step.IsBackground}, stepType={step.StepType}\n");
            return step;

src/Sdk/WebApi/WebApi/ResultsHttpClient.cs:644

• Serializing and logging full StepsUpdateRequest JSON payloads to /tmp can expose tokens/PII (steps metadata can contain user-controlled values) and adds unbounded I/O in a hot path. Please remove this debug logging or route it through existing trace logging with appropriate redaction and opt-in controls.

                // DEBUG: Serialize and log the JSON payload
                try
                {
                    var json = Newtonsoft.Json.JsonConvert.SerializeObject(request, Newtonsoft.Json.Formatting.None);
                    System.IO.File.AppendAllText("/tmp/bg-steps-debug.log", $"[BG-DEBUG] JSON payload: {json}\n");
                }
                catch (Exception ex)
                {
                    System.IO.File.AppendAllText("/tmp/bg-steps-debug.log", $"[BG-DEBUG] Serialize error: {ex.Message}\n");
                }

src/Runner.Worker/StepsRunner.cs:605

• In HandleWaitAsync, the local completed is assigned but never used. With TreatWarningsAsErrors enabled in this repo, this will fail the build. Remove the unused assignment (or use it if intended).

                Trace.Info($"Waiting for {tasks.Count} background step(s)...");
                var cancelTask = Task.Delay(Timeout.Infinite, cancellationToken);
                var completed = await Task.WhenAny(Task.WhenAll(tasks), cancelTask);
                if (cancellationToken.IsCancellationRequested)
                {

src/Runner.Worker/JobExtension.cs:494

• This sets cancel_step_id on the cancel step timeline record to the logical step id, but later logic expects to publish the background step's external/timeline id. If StepsRunner can't resolve the id, this logical value will be reported upstream. Consider deferring this variable until you can resolve the external id (or always overwrite/clear it during execution).

                            cancelRunner.ExecutionContext.SetTimelineRecordVariable("step_type", "cancel");
                            if (!string.IsNullOrEmpty(cancelRunner.CancelStepId))
                            {
                                cancelRunner.ExecutionContext.SetTimelineRecordVariable("cancel_step_id", cancelRunner.CancelStepId);
                            }

• Files reviewed: 18/18 changed files
• Comments generated: 14

[TingluoHuang]

instead of using lock, should we change everything to be thread-safe, ex: using concurrecnybag/dictionary/queue, etc.
So we won't have problem for any future code that forgot to add lock

[lokesh755]

Good point, I looked into this. The main challenge is that List uses .AddRange() in a couple of places and there's no ConcurrentHashSet in .NET, so swapping types ends up touching ~20 call sites across 8 files. Feels too invasive for this PR, happy to do it as a follow-up.

[TingluoHuang]

we can front load the change in its own PR, and change all the consumers.
i am pretty sure we will forget calling lock on this in the future and cause bugs.

[TingluoHuang]

it's a little bit odd we store the data on the client.
It also mean, we will lost all the data if we ever need to recreate the client, ex: hitting network error force us recreate connection and client

[lokesh755]

this is very transient data. Also metadata in this is useful for rendering the UI.