feat: Composite startup verbs for the flagsmith entrypoint#240
Conversation
Add serve, migrate, run-task-processor and migrate-and-serve verbs to the `flagsmith` CLI, mirroring the role sequencing Core API's run-docker.sh performs in shell but running in a single process so the Django app registry is initialised once instead of per step. The verbs stay generic: the databases to migrate or wait for, and any extra startup commands, are driven by settings the consuming app provides (FLAGSMITH_MIGRATE_DATABASES, FLAGSMITH_WAIT_FOR_MIGRATIONS_DATABASES, FLAGSMITH_STARTUP_COMMANDS). beep boop
Default --numthreads, --sleepintervalms, --graceperiodms and --queuepopsize from the TASK_PROCESSOR_* environment variables (falling back to the values run-docker.sh used), so the tuning survives the move to the `flagsmith` entrypoint. Adds environs to the task-processor extra, which now reads it. beep boop
There was a problem hiding this comment.
Code Review
This pull request introduces composite startup commands to the flagsmith entrypoint to handle startup sequencing in a single process, and adds environment variable configuration for the task processor. The review feedback highlights a critical bug in how the legacy TASK_PROCESSOR_SLEEP_INTERVAL (seconds) is parsed as milliseconds, which can cause validation failures or high CPU usage. Additionally, the feedback suggests splitting startup commands with arguments using shlex.split to prevent execution failures, updating the unit tests accordingly, and clarifying the README documentation.
| parser.add_argument( | ||
| "--sleepintervalms", | ||
| type=int, | ||
| help="Number of millis each worker waits before checking for new tasks", | ||
| default=2000, | ||
| default=env.int( | ||
| "TASK_PROCESSOR_SLEEP_INTERVAL_MS", | ||
| env.int("TASK_PROCESSOR_SLEEP_INTERVAL", 500), | ||
| ), | ||
| ) |
There was a problem hiding this comment.
The legacy environment variable TASK_PROCESSOR_SLEEP_INTERVAL is defined in seconds (e.g., 0.5 or 2), whereas --sleepintervalms expects milliseconds.
- If
TASK_PROCESSOR_SLEEP_INTERVALis set to a float like0.5,env.intwill raise aValidationErrorbecause"0.5"is not a valid integer. - If it is set to an integer like
2(representing 2 seconds),env.intwill return2, which means2milliseconds, causing the task processor to spin aggressively and consume 100% CPU.
We should parse TASK_PROCESSOR_SLEEP_INTERVAL as a float, multiply by 1000, and convert to an integer.
| parser.add_argument( | |
| "--sleepintervalms", | |
| type=int, | |
| help="Number of millis each worker waits before checking for new tasks", | |
| default=2000, | |
| default=env.int( | |
| "TASK_PROCESSOR_SLEEP_INTERVAL_MS", | |
| env.int("TASK_PROCESSOR_SLEEP_INTERVAL", 500), | |
| ), | |
| ) | |
| legacy_sleep_interval = env.float("TASK_PROCESSOR_SLEEP_INTERVAL", None) | |
| legacy_sleep_interval_ms = ( | |
| int(legacy_sleep_interval * 1000) | |
| if legacy_sleep_interval is not None | |
| else 500 | |
| ) | |
| parser.add_argument( | |
| "--sleepintervalms", | |
| type=int, | |
| help="Number of millis each worker waits before checking for new tasks", | |
| default=env.int( | |
| "TASK_PROCESSOR_SLEEP_INTERVAL_MS", | |
| legacy_sleep_interval_ms, | |
| ), | |
| ) |
There was a problem hiding this comment.
TASK_PROCESSOR_SLEEP_INTERVAL originally expected milliseconds.
| def migrate_and_serve(argv: list[str], *, prog: str) -> None: | ||
| """Migrate, run any configured startup commands, then start the API server.""" | ||
| _migrate() | ||
| startup_commands: list[str] = getattr(settings, "FLAGSMITH_STARTUP_COMMANDS", []) | ||
| for command in startup_commands: | ||
| django_execute_from_command_line(["flagsmith", command]) | ||
| django_execute_from_command_line(["flagsmith", "start", "api", *argv]) |
There was a problem hiding this comment.
If a startup command in FLAGSMITH_STARTUP_COMMANDS contains arguments (e.g., "bootstrap --force"), passing it directly to django_execute_from_command_line as a single string will fail because Django expects the subcommand and its arguments to be separate elements in the argv list. We should split the command string using shlex.split.
| def migrate_and_serve(argv: list[str], *, prog: str) -> None: | |
| """Migrate, run any configured startup commands, then start the API server.""" | |
| _migrate() | |
| startup_commands: list[str] = getattr(settings, "FLAGSMITH_STARTUP_COMMANDS", []) | |
| for command in startup_commands: | |
| django_execute_from_command_line(["flagsmith", command]) | |
| django_execute_from_command_line(["flagsmith", "start", "api", *argv]) | |
| def migrate_and_serve(argv: list[str], *, prog: str) -> None: | |
| """Migrate, run any configured startup commands, then start the API server.""" | |
| import shlex | |
| _migrate() | |
| startup_commands: list[str] = getattr(settings, "FLAGSMITH_STARTUP_COMMANDS", []) | |
| for command in startup_commands: | |
| django_execute_from_command_line(["flagsmith", *shlex.split(command)]) | |
| django_execute_from_command_line(["flagsmith", "start", "api", *argv]) |
| def test_add_arguments__legacy_sleep_interval__used_as_fallback( | ||
| monkeypatch: pytest.MonkeyPatch, | ||
| ) -> None: | ||
| # Given | ||
| monkeypatch.delenv("TASK_PROCESSOR_SLEEP_INTERVAL_MS", raising=False) | ||
| monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL", "750") | ||
|
|
||
| # When | ||
| args = _parse_defaults() | ||
|
|
||
| # Then | ||
| assert args.sleepintervalms == 750 | ||
|
|
||
|
|
||
| def test_add_arguments__sleep_interval_ms__takes_precedence_over_legacy( | ||
| monkeypatch: pytest.MonkeyPatch, | ||
| ) -> None: | ||
| # Given | ||
| monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL_MS", "300") | ||
| monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL", "750") | ||
|
|
||
| # When | ||
| args = _parse_defaults() | ||
|
|
||
| # Then | ||
| assert args.sleepintervalms == 300 |
There was a problem hiding this comment.
Since TASK_PROCESSOR_SLEEP_INTERVAL is defined in seconds, we should set it to "0.75" in the tests to assert 750 milliseconds.
def test_add_arguments__legacy_sleep_interval__used_as_fallback(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# Given
monkeypatch.delenv("TASK_PROCESSOR_SLEEP_INTERVAL_MS", raising=False)
monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL", "0.75")
# When
args = _parse_defaults()
# Then
assert args.sleepintervalms == 750
def test_add_arguments__sleep_interval_ms__takes_precedence_over_legacy(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# Given
monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL_MS", "300")
monkeypatch.setenv("TASK_PROCESSOR_SLEEP_INTERVAL", "0.75")
# When
args = _parse_defaults()
# Then
assert args.sleepintervalms == 300| | --- | --- | --- | | ||
| | `SKIP_WAIT_FOR_DB` | unset | When set, skip waiting for the database. | | ||
| | `TASK_PROCESSOR_NUM_THREADS` | `5` | Number of worker threads. | | ||
| | `TASK_PROCESSOR_SLEEP_INTERVAL_MS` | `500` | Millis each worker waits before checking for new tasks (falls back to `TASK_PROCESSOR_SLEEP_INTERVAL`). | |
There was a problem hiding this comment.
Clarify that TASK_PROCESSOR_SLEEP_INTERVAL is configured in seconds.
| | `TASK_PROCESSOR_SLEEP_INTERVAL_MS` | `500` | Millis each worker waits before checking for new tasks (falls back to `TASK_PROCESSOR_SLEEP_INTERVAL`). | | |
| | `TASK_PROCESSOR_SLEEP_INTERVAL_MS` | `500` | Millis each worker waits before checking for new tasks (falls back to `TASK_PROCESSOR_SLEEP_INTERVAL` in seconds). | |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #240 +/- ##
==========================================
+ Coverage 97.39% 97.55% +0.15%
==========================================
Files 104 107 +3
Lines 4613 4745 +132
==========================================
+ Hits 4493 4629 +136
+ Misses 120 116 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
FLAGSMITH_STARTUP_COMMANDS entries are command strings; passing an entry with arguments (e.g. "bootstrap --skip-fixtures") as a single argv element makes Django treat the whole string as an unknown subcommand. Shell-split each entry with shlex before dispatch. beep boop
Changes
Closes #239.
In this PR, we add the composite startup verbs to the
flagsmithCLI, fully compatible with Core API'srun-docker.sh.New
flagsmithverbs:servemigraterun-task-processormigrate-and-serveThey mirror
run-docker.sh's sequencing exactly, but run in a single process, so the Django app registry is initialised once instead of once permanage.pysubprocess. They stay generic — no Core-API specifics — driven by settings the consuming app provides:FLAGSMITH_MIGRATE_DATABASES(default["default"]) — databases themigratestep applies, in orderFLAGSMITH_WAIT_FOR_MIGRATIONS_DATABASES(default["default"]) — databasesrun-task-processorwaits onFLAGSMITH_STARTUP_COMMANDS(default[]) — commands run between migrate and serve inmigrate-and-serveSKIP_WAIT_FOR_DBis honoured in the verb layer. The server hand-off goes through the existingstartcommand via the management CLI path (notcall_command, which can't pass throughstart's gunicorn arguments alongside its subparsers).Separately, the task processor's worker tuning arguments (
--numthreads,--sleepintervalms,--graceperiodms,--queuepopsize) now default from theTASK_PROCESSOR_*environment variables, falling back to the valuesrun-docker.shused, so that tuning survives the move to the entrypoint.environsis added to thetask-processorextra, which now reads it.The README documents the new verbs and their settings and environment variables.
How did you test this code?
Unit tests cover each verb's exact call sequence, the settings-driven database lists, the bare-vs-targeted
migratesplit,SKIP_WAIT_FOR_DB, argument pass-through, dispatch routing, and the task processor env defaults, including theTASK_PROCESSOR_SLEEP_INTERVALfallback and precedence.