Skip to content

Conversation

@HardMax71
Copy link
Owner

@HardMax71 HardMax71 commented Jan 30, 2026


Summary by cubic

Replaces .env/pydantic-settings with TOML-based config loaded via tomllib, with per-worker overrides and a secrets.toml layer. Fixes #118 by simplifying config management and making settings consistent across dev, CI, and runtime.

  • Refactors

    • Rewrote Settings to a pydantic BaseModel that reads config.toml, merges secrets.toml, and optional override_path.
    • Removed .env/.env.test and dropped pydantic-settings/python-dotenv.
    • Added config.toml and config.test.toml plus per-worker overrides (e.g., config.k8s-worker.toml).
    • Updated workers to pass override configs; CI now copies config.test.toml to config.toml and copies secrets.example.toml to secrets.toml.
    • Docker/Docker Compose mount config.toml, secrets.toml, and per-worker TOMLs; deploy.sh references config.toml.
    • Cleaned test and load configs to use TOML-backed Settings; added secrets.example.toml.
    • Updated docs to a TOML-based Configuration Reference and revised getting started/deployment guides.
  • Migration

    • Stop using backend/.env; use backend/config.toml as the single source of truth.
    • Create backend/secrets.toml for SECRET_KEY and MONGODB_URL (see secrets.example.toml); CI can copy the example or populate it from secrets.
    • If you run workers directly, pass the override file (e.g., Settings(override_path="config.k8s-worker.toml")).

Written for commit bbf99d9. Summary will update on new commits.

Summary by CodeRabbit

  • Chores

    • Switched runtime configuration from env files to TOML (base + per-worker overrides) and added a separate secrets.toml (example committed, secrets ignored).
    • CI, Docker, docker-compose, entrypoints, and tests updated to load TOML configs; container runtime defaults consolidated into fixed config values.
  • Documentation

    • Updated docs and getting-started to describe TOML configuration, secrets handling, worker override files, and test configuration guidance.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link

coderabbitai bot commented Jan 30, 2026

📝 Walkthrough

Walkthrough

Replaces environment-variable (.env) configuration with layered TOML files (config.toml, secrets.toml, per-service overrides). Refactors Settings to load from TOML (BaseModel), updates Docker/compose/CI to copy/mount TOML files, and changes workers/tests to pass config/override paths.

Changes

Cohort / File(s) Summary
Core settings
backend/app/settings.py, backend/pyproject.toml, backend/app/main.py
Switch Settings from pydantic BaseSettings to BaseModel with TOML loading; remove pydantic-settings dependency; update docstrings and constructor signature.
Central & test config files
backend/config.toml, backend/config.test.toml, backend/secrets.example.toml
Add primary config.toml and test config; introduce a gitignored secrets.toml template (secrets.example.toml) for sensitive values.
Per-service overrides
backend/config.*.toml
backend/config.coordinator.toml, backend/config.dlq-processor.toml, backend/config.event-replay.toml, backend/config.k8s-worker.toml, backend/config.pod-monitor.toml, backend/config.result-processor.toml, backend/config.saga-orchestrator.toml
Add lightweight per-worker override TOML files (TRACING_SERVICE_NAME, KAFKA_CONSUMER_GROUP_ID) consumed via override_path.
Removed env files
backend/.env, backend/.env.test
Remove .env and .env.test; environment-driven defaults replaced by TOML files.
Docker & compose
backend/Dockerfile, backend/Dockerfile.base, docker-compose.yaml
Consume TOML files instead of env_file; add KUBECONFIG default in base image; hardcode Gunicorn params; mount config/*.toml and secrets.toml into services.
Workers & entrypoints
backend/workers/...
backend/workers/run_coordinator.py, run_event_replay.py, run_k8s_worker.py, run_pod_monitor.py, run_result_processor.py, run_saga_orchestrator.py, workers/dlq_processor.py
Initialize Settings with explicit config_path/override_path to load service-specific TOML overrides.
Tests & load tests
backend/tests/conftest.py, backend/tests/unit/services/pod_monitor/test_monitor.py, backend/tests/load/config.py
Tests updated to use config.test.toml; test/load config models migrated from BaseSettings to BaseModel and gained fields/methods (generate_plots, mode, output_dir, api()).
CI / scripts / deploy
.github/workflows/stack-tests.yml, .github/workflows/docs.yml, deploy.sh, backend/scripts/create_topics.py
CI/docs workflows and deploy script now copy/use TOML-based configs/secrets; add setup steps to create secrets.toml from example.
Docs & site
docs/*, mkdocs.yml
Replace Environment Variables docs with TOML Configuration Reference; update getting-started, deployment, and configuration reference to describe layered TOML + secrets workflow.
Repo hygiene
.gitignore, backend/secrets.example.toml
Add secrets.toml to .gitignore and include a secrets.example.toml template for local/dev usage.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant CI as CI
    participant Image as Docker Image
    participant Container as Container Runtime
    participant Settings as Settings loader
    participant Service as Worker/API

    CI->>Image: copy `backend/config.test.toml` & `backend/secrets.toml`
    Image->>Container: start container (mount config/*.toml & secrets.toml)
    Container->>Settings: Settings(config_path="config.toml", secrets_path="secrets.toml", override_path="config.service.toml")
    Settings-->>Container: return merged configuration
    Container->>Service: start service with resolved settings
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Poem

🐰 I hopped from .env to tidy TOML beds,
Files stacked in order, secrets tucked like threads.
Workers hum their tunes from per-service notes,
CI plants the keys, containers wear the coats.
A carrot-coded cheer — configuration spreads! 🥕

🚥 Pre-merge checks | ✅ 3 | ❌ 2
❌ Failed checks (1 warning, 1 inconclusive)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 76.92% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title 'fix: #118' is vague and does not clearly describe the main change; it only references an issue number without summarizing the actual implementation. Consider a more descriptive title such as 'fix: migrate configuration from .env/pydantic-settings to TOML-based config' to better communicate the scope of changes.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed The PR successfully replaces pydantic-settings with a TOML-based configuration system that ensures deterministic precedence by reading from config.toml, merging secrets.toml, and accepting optional per-worker overrides, directly addressing issue #118's requirement for file-specified values to take precedence over OS environment variables.
Out of Scope Changes check ✅ Passed All changes are in-scope: configuration migration from .env to TOML, Settings refactoring, worker updates, CI/Docker adjustments, and documentation updates all directly support the core objective of fixing pydantic-settings precedence issues.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/pydantic-settings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 31 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="backend/tests/load/config.py">

<violation number="1" location="backend/tests/load/config.py:10">
P2: Switching LoadConfig to BaseModel drops BaseSettings environment loading, so documented LOAD_* env overrides stop working (e.g., LOAD_BASE_URL). This is a functional regression for load test configuration.</violation>
</file>

<file name="backend/config.test.toml">

<violation number="1" location="backend/config.test.toml:6">
P1: Avoid committing secrets or passwords in config. Move SECRET_KEY and MongoDB credentials to environment variables or a secrets manager, and reference them here instead.</violation>
</file>

<file name="backend/config.toml">

<violation number="1" location="backend/config.toml:6">
P0: Hard-coded SECRET_KEY committed to the repo. Secrets should be provided via environment/secret management, not stored in versioned config.</violation>

<violation number="2" location="backend/config.toml:10">
P1: Plaintext MongoDB credentials are committed in config. Use environment/secret management and avoid root credentials in repo.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@backend/config.toml`:
- Line 10: The MONGODB_URL entry currently contains hardcoded credentials
("root:rootpassword"); remove the secret from the value and replace it with a
non-secret default or placeholder (e.g., a connection string that references
environment variables) and add a short comment in the config explaining this
file is for development defaults only and production must override via
environment variables; specifically update the MONGODB_URL setting to not embed
credentials and document how to supply real credentials (env var names) so
callers of MONGODB_URL (e.g., code that reads that config) will use secure,
overridable secrets in production.
- Line 6: Remove the hardcoded SECRET_KEY entry from the tracked config (the
SECRET_KEY setting in backend/config.toml) and migrate secret handling to a
secure source: add support for loading SECRET_KEY from an environment variable
(e.g., process.env.SECRET_KEY or the app's config loader) and update the config
loader to fall back to a gitignored config.local.toml for local overrides;
alternatively wire the loader to a secrets manager. Ensure you update README or
example config to show a placeholder SECRET_KEY and add config.local.toml to
.gitignore so secrets are never committed.
🧹 Nitpick comments (4)
backend/config.test.toml (1)

1-8: Acknowledge that SECRET_KEY is intentionally test-only.

Gitleaks flagged this as a potential secret exposure. While this is a test configuration file and the key is clearly intended for local/CI testing only, consider adding a comment to make this explicit and prevent future false positives:

 # Integr8sCode backend test configuration
 # Differences from config.toml: lower timeouts, faster bcrypt, relaxed rate limits
+# NOTE: SECRET_KEY below is for test/CI use only - never use in production
 
 PROJECT_NAME = "integr8scode"
 DATABASE_NAME = "integr8scode_db"
 SECRET_KEY = "uS5xBF-OKXHV-1vqU4ASLwyPcKpSdUTLqGHPYs3y-Yc"

This documents intent and helps security scanners (and reviewers) understand it's safe.

backend/Dockerfile (1)

29-42: Gunicorn parameters are hardcoded, ignoring WEB_* settings from config.toml.

The entrypoint script uses hardcoded values (--workers 4, --threads 4, --timeout 60, --backlog 2048) instead of reading from the TOML configuration. The WEB_CONCURRENCY, WEB_THREADS, WEB_TIMEOUT, and WEB_BACKLOG fields in config.toml and Settings are now effectively unused for the main backend service.

Consider either:

  1. Removing the WEB_* fields from Settings/config.toml if they're not needed
  2. Or using a wrapper script that reads the TOML and passes values to Gunicorn
backend/config.toml (1)

50-50: 100% tracing sampling rate may impact production performance.

TRACING_SAMPLING_RATE = 1.0 samples all requests, which can significantly impact performance and storage costs in production. The Settings class default is 0.1 (10%). Consider using a lower rate for production defaults.

backend/app/settings.py (1)

24-30: Consider adding user-friendly error handling for missing config files.

If the config file doesn't exist, FileNotFoundError propagates with a generic message. A clearer error message would help with debugging deployment issues.

♻️ Suggested improvement
 def __init__(self, config_path: str = "config.toml", override_path: str | None = None) -> None:
-    with open(config_path, "rb") as f:
-        data = tomllib.load(f)
+    try:
+        with open(config_path, "rb") as f:
+            data = tomllib.load(f)
+    except FileNotFoundError:
+        raise FileNotFoundError(f"Configuration file not found: {config_path}")
     if override_path:
-        with open(override_path, "rb") as f:
-            data |= tomllib.load(f)
+        try:
+            with open(override_path, "rb") as f:
+                data |= tomllib.load(f)
+        except FileNotFoundError:
+            raise FileNotFoundError(f"Override configuration file not found: {override_path}")
     super().__init__(**data)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@backend/app/settings.py`:
- Around line 36-44: The code opens config_path and override_path without
validating existence and only secrets_path uses Path(secrets_path).is_file();
update the block that builds data so all three paths are checked consistently:
verify Path(config_path).is_file() before opening and raise a clear, descriptive
exception if missing (including the path and context), use the existing Pattern
(Path(...).is_file()) for override_path before attempting to open it, and
preserve merging with tomllib.load and the final super().__init__(**data); also
consider catching tomllib errors to wrap them with a helpful message referencing
the failing path.

In `@docker-compose.yaml`:
- Around line 109-111: Add a secrets example and documentation because
docker-compose.yaml mounts ./backend/secrets.toml (used by services in the
compose file) but that file is gitignored and absent in fresh clones; create a
new backend/secrets.example.toml containing all required keys/placeholder values
and short comments, add backend/secrets.toml to .gitignore if not already, and
update README/CONTRIBUTING to document copying secrets.example.toml to
backend/secrets.toml (and any required env values) so docker-compose up won’t
fail on missing mounts.

Copy link

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 8 files (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="backend/secrets.example.toml">

<violation number="1" location="backend/secrets.example.toml:16">
P1: Do not commit real-looking secrets or credentials in the example secrets file. Keep placeholder values so teams are forced to supply their own secrets.</violation>

<violation number="2" location="backend/secrets.example.toml:17">
P1: Replace the example MongoDB URL with a placeholder (no real credentials) to avoid committing reusable passwords.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@backend/config.test.toml`:
- Around line 1-4: The comment in backend/config.test.toml incorrectly
references "secrets.test.toml" while CI actually uses the repository's secrets
file; either create and populate a dedicated secrets.test.toml or update the
comment to reference the real CI secrets filename (change "secrets.test.toml" to
"secrets.toml" or the correct secrets filename) so the comment matches the CI
workflow; edit the header comment in backend/config.test.toml accordingly.
🧹 Nitpick comments (1)
docs/reference/configuration.md (1)

36-38: Consider using named anchors instead of line numbers for snippets.

The snippets reference specific line ranges (e.g., backend/config.toml:1:9). If config.toml is modified, these line numbers may become stale, causing the documentation to show incorrect content. Consider using named anchors in the TOML file or documenting this fragility for maintainers.

@sonarqubecloud
Copy link

@HardMax71 HardMax71 merged commit 2ce06fc into main Jan 30, 2026
23 checks passed
@HardMax71 HardMax71 deleted the fix/pydantic-settings branch January 30, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: precedence of data read by pydantic-settings

2 participants