Adding a MaxConcurrentTasks flag to the self hosted worker #27
Open
Adding a MaxConcurrentTasks flag to the self hosted worker #27
MaxConcurrentTasks flag to the self hosted worker #27Conversation
ianhodge
changed the title
MaxConcurrentTasks flag to the self hosted worker
bnavetta
reviewed
Mar 3, 2026
internal/worker/worker.go
Outdated
| sendChan chan []byte | ||
| activeTasks map[string]context.CancelFunc | ||
| tasksMutex sync.Mutex | ||
| taskSemaphore chan struct{} // nil when unlimited; buffered channel used as counting semaphore |
Collaborator
There was a problem hiding this comment.
We should probably use https://pkg.go.dev/golang.org/x/sync/semaphore since it handles context cancellation for free.
That'd also let us use TryAcquire rather than Acquire, so that if a self-hosted worker is at capacity, the task can more quickly get routed to another worker or requeued server-side
| NoCleanup: CLI.NoCleanup, | ||
| Volumes: CLI.Volumes, | ||
| Env: envMap, | ||
| APIKey: CLI.APIKey, |
Collaborator
There was a problem hiding this comment.
We're probably gonna want a config file format pretty soon
bnavetta
reviewed
Mar 6, 2026
| if w.taskSemaphore != nil { | ||
| if !w.taskSemaphore.TryAcquire(1) { | ||
| log.Warnf(w.ctx, "At max concurrency (%d), rejecting task: taskID=%s", w.config.MaxConcurrentTasks, assignment.TaskID) | ||
| if err := w.sendTaskFailed(assignment.TaskID, "worker at maximum concurrency"); err != nil { |
Collaborator
There was a problem hiding this comment.
This will mark the task as failed and unretryable, right? I think we need a new message type that signals "I will not claim this task"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WISOTT
Default to 0 for existing behavior