feat: configurable web worker count via WEB_WORKERS env var#1145
feat: configurable web worker count via WEB_WORKERS env var#1145
Conversation
Local dev: WEB_WORKERS > 1 runs uvicorn with --workers (no --reload). Default single-worker mode keeps --reload for hot reloading. Production: explicit --workers for gunicorn (defaults to 4). Co-Authored-By: Claude <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe startup scripts for local and production Django environments are enhanced to support configurable web server workers via the WEB_WORKERS environment variable. Local development conditionally enables reload behavior based on worker count, while production dynamically configures Gunicorn workers. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Deploy Preview for antenna-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for antenna-ssec ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
compose/local/django/start (1)
14-15: Consider guarding against non-integerWEB_WORKERSfor a clearer error.If
WEB_WORKERSis set to a non-integer value (e.g."two","2.5"),[ "$WORKERS" -gt 1 ]will fail withinteger expression expectedandset -o errexitwill exit silently. A short guard produces a much friendlier message.🛡️ Proposed guard
WORKERS="${WEB_WORKERS:-1}" + if ! [[ "$WORKERS" =~ ^[0-9]+$ ]]; then + echo "Error: WEB_WORKERS must be a non-negative integer (got: '$WORKERS')" >&2 + exit 1 + fi if [ "$WORKERS" -gt 1 ]; then🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@compose/local/django/start` around lines 14 - 15, The current check using [ "$WORKERS" -gt 1 ] will fail if WEB_WORKERS contains non-integer text; modify the start-up script to validate WORKERS is an integer before the numeric comparison by testing WORKERS against a digits-only pattern (e.g. a regex like ^[0-9]+$ or a shell case) and if it doesn’t match print a clear error mentioning WEB_WORKERS and exit non‑zero; after the guard you can safely use the existing [ "$WORKERS" -gt 1 ] branch.compose/production/django/start (2)
10-10: Implicit default jump from 1 → 4 workers will affect all existing deployments.Previously, gunicorn ran without
--workers, which defaults to 1 worker. Any deployment that does not explicitly setWEB_WORKERSwill now start 4 workers automatically. This silently quadruples:
- Memory consumed per container instance.
- Database connections opened (each worker holds its own connection pool).
Ensure your DB connection limit and container memory limits can accommodate 4× the prior baseline before rolling this out.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@compose/production/django/start` at line 10, The change adds a hard default of 4 workers via the --workers flag which silently multiplies memory and DB connections for deployments that don't set WEB_WORKERS; update the start command so it does not unconditionally default to 4—either omit the --workers flag when WEB_WORKERS is unset or default WEB_WORKERS to 1 instead; locate the shell invocation using the --workers flag and the WEB_WORKERS expansion and change it to only emit --workers when WEB_WORKERS is set (or set the expansion to "${WEB_WORKERS:-1}" if you intend an explicit 1-worker default).
10-10:WEB_WORKERSaccepts any string — consider guarding against non-integer input.
${WEB_WORKERS:-4}passes the value verbatim to gunicorn. IfWEB_WORKERSis set to a non-numeric or zero/negative value, gunicorn will fail at startup with a cryptic error rather than a clear configuration message.🛡️ Optional: add a validation guard
+WEB_WORKERS="${WEB_WORKERS:-4}" +if ! [[ "${WEB_WORKERS}" =~ ^[1-9][0-9]*$ ]]; then + echo "ERROR: WEB_WORKERS must be a positive integer (got: '${WEB_WORKERS}')" >&2 + exit 1 +fi + exec newrelic-admin run-program /usr/local/bin/gunicorn config.asgi \ - --workers "${WEB_WORKERS:-4}" \ + --workers "${WEB_WORKERS}" \ --bind 0.0.0.0:5000 --chdir=/app -k uvicorn.workers.UvicornWorker🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@compose/production/django/start` at line 10, Guard against non-integer or non-positive WEB_WORKERS before passing it to gunicorn: validate the WEB_WORKERS environment variable in the start script (the place that builds the --workers "${WEB_WORKERS:-4}" argument), ensure it is a positive integer, and fall back to 4 (or exit with a clear error) if validation fails; update the start script to parse and coerce/validate WEB_WORKERS (e.g., regex/digit check or integer conversion) and only pass a validated positive integer to gunicorn's --workers flag.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@compose/local/django/start`:
- Around line 14-15: The current check using [ "$WORKERS" -gt 1 ] will fail if
WEB_WORKERS contains non-integer text; modify the start-up script to validate
WORKERS is an integer before the numeric comparison by testing WORKERS against a
digits-only pattern (e.g. a regex like ^[0-9]+$ or a shell case) and if it
doesn’t match print a clear error mentioning WEB_WORKERS and exit non‑zero;
after the guard you can safely use the existing [ "$WORKERS" -gt 1 ] branch.
In `@compose/production/django/start`:
- Line 10: The change adds a hard default of 4 workers via the --workers flag
which silently multiplies memory and DB connections for deployments that don't
set WEB_WORKERS; update the start command so it does not unconditionally default
to 4—either omit the --workers flag when WEB_WORKERS is unset or default
WEB_WORKERS to 1 instead; locate the shell invocation using the --workers flag
and the WEB_WORKERS expansion and change it to only emit --workers when
WEB_WORKERS is set (or set the expansion to "${WEB_WORKERS:-1}" if you intend an
explicit 1-worker default).
- Line 10: Guard against non-integer or non-positive WEB_WORKERS before passing
it to gunicorn: validate the WEB_WORKERS environment variable in the start
script (the place that builds the --workers "${WEB_WORKERS:-4}" argument),
ensure it is a positive integer, and fall back to 4 (or exit with a clear error)
if validation fails; update the start script to parse and coerce/validate
WEB_WORKERS (e.g., regex/digit check or integer conversion) and only pass a
validated positive integer to gunicorn's --workers flag.
There was a problem hiding this comment.
Pull request overview
This pull request adds support for configurable web worker count via the WEB_WORKERS environment variable to enable handling concurrent requests during local development and production deployments. The feature addresses a limitation where a single uvicorn worker couldn't handle concurrent requests from multiple DataLoader subprocesses during PSv2 integration testing.
Changes:
- Local development script conditionally enables multiple uvicorn workers when
WEB_WORKERS > 1, disabling hot-reload since it's incompatible with multi-worker mode - Production script explicitly passes
--workersflag to gunicorn with a default of 4 workers - Single-worker mode remains the default for local development, preserving hot-reload functionality
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| compose/production/django/start | Added --workers flag to gunicorn command with default value of 4 from WEB_WORKERS env var |
| compose/local/django/start | Added conditional logic to enable multiple uvicorn workers when WEB_WORKERS > 1, disabling --reload in multi-worker mode |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| exec python -Xfrozen_modules=off -m debugpy --listen 0.0.0.0:5678 -m uvicorn config.asgi:application --host 0.0.0.0 | ||
| else | ||
| exec uvicorn config.asgi:application --host 0.0.0.0 --reload --reload-include '*.html' | ||
| WORKERS="${WEB_WORKERS:-1}" |
There was a problem hiding this comment.
The numeric comparison on line 15 will fail with a cryptic error if WEB_WORKERS is set to a non-numeric value (e.g., WEB_WORKERS=abc). Consider adding input validation before the comparison to ensure WORKERS is a positive integer, or use a more defensive comparison that handles non-numeric values gracefully.
| WORKERS="${WEB_WORKERS:-1}" | |
| WORKERS="${WEB_WORKERS:-1}" | |
| # Ensure WORKERS is a positive integer; fall back to 1 if invalid | |
| if ! [[ "$WORKERS" =~ ^[0-9]+$ ]] || [ "$WORKERS" -lt 1 ]; then | |
| echo "Invalid WEB_WORKERS value '$WORKERS'. Falling back to 1 worker." >&2 | |
| WORKERS=1 | |
| fi |
| exec newrelic-admin run-program /usr/local/bin/gunicorn config.asgi \ | ||
| --workers "${WEB_WORKERS:-4}" \ |
There was a problem hiding this comment.
If WEB_WORKERS is set to a non-numeric value (e.g., WEB_WORKERS=abc) or zero/negative value, gunicorn will fail. Consider adding input validation to ensure WEB_WORKERS is a positive integer before using it, which would provide a clearer error message.
| exec newrelic-admin run-program /usr/local/bin/gunicorn config.asgi \ | |
| --workers "${WEB_WORKERS:-4}" \ | |
| # Validate WEB_WORKERS as a positive integer, defaulting to 4 if unset or invalid | |
| WEB_WORKERS_VALUE="${WEB_WORKERS:-4}" | |
| if ! [[ "${WEB_WORKERS_VALUE}" =~ ^[1-9][0-9]*$ ]]; then | |
| echo "Invalid WEB_WORKERS value '${WEB_WORKERS:-unset}'. Must be a positive integer. Falling back to 4." >&2 | |
| WEB_WORKERS_VALUE=4 | |
| fi | |
| exec newrelic-admin run-program /usr/local/bin/gunicorn config.asgi \ | |
| --workers "${WEB_WORKERS_VALUE}" \ |
| WORKERS="${WEB_WORKERS:-1}" | ||
| if [ "$WORKERS" -gt 1 ]; then | ||
| # --reload is incompatible with --workers, so skip it for multi-worker mode | ||
| exec uvicorn config.asgi:application --host 0.0.0.0 --workers "$WORKERS" |
There was a problem hiding this comment.
Consider adding documentation for the WEB_WORKERS environment variable. Based on the pattern in CLAUDE.md where DEBUGGER=1 is documented (lines 96-106), this new environment variable should also be documented to help developers understand how to use it for local development with multiple workers.
Merging worker config from #1142 and #1145After investigating production, here's what I found and a plan to consolidate. Discovery:
|
|
Closing in favor of #1142 which now uses the standard WEB_CONCURRENCY env var (gunicorn reads it natively). The USE_UVICORN=1 escape hatch is available for local dev if needed. |
1 participant
Summary
WEB_WORKERSenv var; values > 1 run uvicorn with--workers(disables--reloadsince they're incompatible)--workersto gunicorn (defaults to 4)Single-worker mode (default) is unchanged — keeps
--reloadfor hot reloading in dev.Context
During PSv2 integration testing, a single uvicorn worker couldn't handle concurrent requests from multiple DataLoader subprocesses. This makes it easy to run multiple workers locally with
WEB_WORKERS=2 docker compose up django.Test plan
docker compose up django— single worker with--reload(default, unchanged)WEB_WORKERS=2 docker compose up django— two workers, no--reloadDEBUGGER=1 docker compose up django— debugpy mode unchangedSummary by CodeRabbit