Skip to content

Dynamic labels: runnerLabels variable accumulates labels across batch groups causing incorrect runner registration #5175

@Brend-Smits

Description

@Brend-Smits

Bug

When multiple jobs with different ghr-* dynamic labels are processed in the same scale-up lambda invocation, runners are registered with labels from all groups instead of only their own group's labels.

Root Cause

In scale-up.ts, the runnerLabels variable is declared with let at function scope (outside the group processing loop) and mutated inside the loop:

// Line 331 — declared once outside the loop
let runnerLabels = process.env.RUNNER_LABELS || '';

// Line 468 — mutated each iteration, ACCUMULATING labels from all groups
for (const [group, { ... }] of validMessages.entries()) {
  // ...
  runnerLabels = runnerLabels ? `${runnerLabels},${allDynamicLabels.join(',')}` : allDynamicLabels.join(',');
  // ...
  createRunners({ runnerLabels, ... }); // passes accumulated labels
}

Each group iteration appends its ghr-* labels to the same variable. By the time the Nth group is processed, runnerLabels contains labels from groups 1 through N.

Impact

  • Runners get registered with labels from unrelated jobs
  • GitHub's superset label matching assigns runners to wrong jobs (any runner in a batch matches any job in the batch)
  • The ghr-job-id uniqueness guarantee is broken because all job IDs end up on every runner
  • Instance type mismatches: a job requesting m7a.large gets assigned a runner that was actually launched as m7i.xlarge

Reproduction

  1. Enable enable_dynamic_labels = true
  2. Submit multiple jobs simultaneously with different dynamic labels:
    • Job A: ghr-ec2-instance-type:m7a.large, ghr-job-id:abc
    • Job B: ghr-ec2-instance-type:m7i.xlarge, ghr-job-id:def
    • Job C: ghr-ec2-instance-type:c7a.large, ghr-job-id:ghi
  3. All jobs land in the same SQS batch → single lambda invocation
  4. Runner for Job C is registered with labels from all three jobs

Observed Behavior (from GitHub Settings > Runners)

A single runner registered with labels from 3 different jobs:

  • ghr-ec2-instance-type:m7a.large
  • ghr-ec2-instance-type:m7i.large
  • ghr-job-id:27542571479-2-combo
  • ghr-job-id:27542571479-2-custom
  • ghr-job-id:27542571479-2-inst-0
  • ghr-team:platform

Fix

Scope the label variable per loop iteration:

const runnerLabels = process.env.RUNNER_LABELS || ''; // const, not let

for (const [group, { ... }] of validMessages.entries()) {
  let groupRunnerLabels = runnerLabels; // fresh copy each iteration
  // ...
  groupRunnerLabels = groupRunnerLabels ? `${groupRunnerLabels},${allDynamicLabels.join(',')}` : allDynamicLabels.join(',');
  // ...
  createRunners({ runnerLabels: groupRunnerLabels, ... });
}

PR incoming.

Metadata

Metadata

Assignees

No one assigned

    Labels

    dynamic-labelsIssue/PR related to the Dynamic Label Feature

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions