Bug
When multiple jobs with different ghr-* dynamic labels are processed in the same scale-up lambda invocation, runners are registered with labels from all groups instead of only their own group's labels.
Root Cause
In scale-up.ts, the runnerLabels variable is declared with let at function scope (outside the group processing loop) and mutated inside the loop:
// Line 331 — declared once outside the loop
let runnerLabels = process.env.RUNNER_LABELS || '';
// Line 468 — mutated each iteration, ACCUMULATING labels from all groups
for (const [group, { ... }] of validMessages.entries()) {
// ...
runnerLabels = runnerLabels ? `${runnerLabels},${allDynamicLabels.join(',')}` : allDynamicLabels.join(',');
// ...
createRunners({ runnerLabels, ... }); // passes accumulated labels
}
Each group iteration appends its ghr-* labels to the same variable. By the time the Nth group is processed, runnerLabels contains labels from groups 1 through N.
Impact
- Runners get registered with labels from unrelated jobs
- GitHub's superset label matching assigns runners to wrong jobs (any runner in a batch matches any job in the batch)
- The
ghr-job-id uniqueness guarantee is broken because all job IDs end up on every runner
- Instance type mismatches: a job requesting
m7a.large gets assigned a runner that was actually launched as m7i.xlarge
Reproduction
- Enable
enable_dynamic_labels = true
- Submit multiple jobs simultaneously with different dynamic labels:
- Job A:
ghr-ec2-instance-type:m7a.large, ghr-job-id:abc
- Job B:
ghr-ec2-instance-type:m7i.xlarge, ghr-job-id:def
- Job C:
ghr-ec2-instance-type:c7a.large, ghr-job-id:ghi
- All jobs land in the same SQS batch → single lambda invocation
- Runner for Job C is registered with labels from all three jobs
Observed Behavior (from GitHub Settings > Runners)
A single runner registered with labels from 3 different jobs:
ghr-ec2-instance-type:m7a.large
ghr-ec2-instance-type:m7i.large
ghr-job-id:27542571479-2-combo
ghr-job-id:27542571479-2-custom
ghr-job-id:27542571479-2-inst-0
ghr-team:platform
Fix
Scope the label variable per loop iteration:
const runnerLabels = process.env.RUNNER_LABELS || ''; // const, not let
for (const [group, { ... }] of validMessages.entries()) {
let groupRunnerLabels = runnerLabels; // fresh copy each iteration
// ...
groupRunnerLabels = groupRunnerLabels ? `${groupRunnerLabels},${allDynamicLabels.join(',')}` : allDynamicLabels.join(',');
// ...
createRunners({ runnerLabels: groupRunnerLabels, ... });
}
PR incoming.
Bug
When multiple jobs with different
ghr-*dynamic labels are processed in the same scale-up lambda invocation, runners are registered with labels from all groups instead of only their own group's labels.Root Cause
In
scale-up.ts, therunnerLabelsvariable is declared withletat function scope (outside the group processing loop) and mutated inside the loop:Each group iteration appends its
ghr-*labels to the same variable. By the time the Nth group is processed,runnerLabelscontains labels from groups 1 through N.Impact
ghr-job-iduniqueness guarantee is broken because all job IDs end up on every runnerm7a.largegets assigned a runner that was actually launched asm7i.xlargeReproduction
enable_dynamic_labels = trueghr-ec2-instance-type:m7a.large,ghr-job-id:abcghr-ec2-instance-type:m7i.xlarge,ghr-job-id:defghr-ec2-instance-type:c7a.large,ghr-job-id:ghiObserved Behavior (from GitHub Settings > Runners)
A single runner registered with labels from 3 different jobs:
ghr-ec2-instance-type:m7a.largeghr-ec2-instance-type:m7i.largeghr-job-id:27542571479-2-comboghr-job-id:27542571479-2-customghr-job-id:27542571479-2-inst-0ghr-team:platformFix
Scope the label variable per loop iteration:
PR incoming.