Skip to content

Wait for cluster DNS before starting nginx in the frontend - #4354

Open
imnasnainaec wants to merge 4 commits into
masterfrom
fix/frontend-dns-wait
Open

Wait for cluster DNS before starting nginx in the frontend#4354
imnasnainaec wants to merge 4 commits into
masterfrom
fix/frontend-dns-wait

Conversation

@imnasnainaec

@imnasnainaec imnasnainaec commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Split out of #4352, part 1 of 4 — see
#4352 (comment) for the split and how it was verified.

nginx resolves the hostnames in its proxy_pass directives when it loads its configuration, and exits if any of them
cannot be resolved:

[emerg] host not found in upstream "backend" in /etc/nginx/conf.d/default.conf:48

On a cold start the frontend container can be running before the cluster DNS is able to answer for the backend
service, which leaves the pod crash looping until the kubelet's restart backoff happens to line up with DNS being
ready.

  • Wait for the name in a docker-entrypoint.d script, and start nginx anyway after two minutes, so that a real
    misconfiguration is still reported rather than leaving a container that runs but never serves.
  • Add a readiness probe to the frontend, since with the wait in place a running container is no longer necessarily a
    serving one.
  • Note at the proxy_pass directives that a new cluster-internal upstream has to be added to the wait. All three of
    them currently point at backend, so the wait covers every one as it stands.

Scope

Three files, none of them touched by the other parts of #4352:

  • nginx/init/05-wait-for-backend-dns.sh (new) — Dockerfile already copies nginx/init/* into
    /docker-entrypoint.d/, so nothing else is needed to run it
  • nginx/templates/default.conf.template
  • deploy/helm/thecombine/charts/frontend/templates/deployment-frontend.yaml

Independent of the other three parts in both directions, textually and functionally: it shares no file with them and
no behavior, so it can merge at any point in the sequence. Verified conflict-free against #4353 and the database part.

QA and prod

Affected. Both changes are in the frontend image and the shared frontend chart, with no target conditionals, so they
ship to qa-kube.thecombine.app and thecombine.app as well as to the NUCs and desktop installs. On a server whose
cluster DNS is already up, the script's first getent succeeds and it exits immediately, and the two-minute cap keeps
it from ever hanging a deploy. The readiness probe changes when the frontend joins its Service endpoints on every
target.

Testing

Not yet exercised on hardware. The failure it fixes needs a cold cluster start to reproduce, so it wants a laptop or
NUC install rather than a QA deploy, where DNS is up before the frontend rolls.

🤖 Generated with Claude Code


This change is Reviewable

Summary by CodeRabbit

  • Bug Fixes
    • Improved frontend startup checks so deployments can better verify service readiness.
    • Added handling for delayed backend hostname resolution during nginx startup, reducing premature connection failures.
  • Documentation
    • Clarified which backend hostname is checked before nginx starts.

imnasnainaec and others added 2 commits August 21, 2026 08:16
nginx resolves the hostnames in its proxy_pass directives when it loads its
configuration and exits if any of them cannot be resolved:

  [emerg] host not found in upstream "backend" in
  /etc/nginx/conf.d/default.conf:48

On a cold start the frontend container can be running before the cluster DNS
can answer for the backend service, which leaves the pod crash looping until
the kubelet's restart backoff happens to line up with DNS being ready.

Wait for the name in a docker-entrypoint.d script, and start nginx anyway
after two minutes so that a real misconfiguration is still reported rather
than leaving a container that runs but never serves.

Add a readiness probe to the frontend, since with the wait in place a running
container is no longer necessarily a serving one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nginx exits if a proxy_pass host cannot be resolved when it loads its
configuration, and the wait only knows about "backend". Say so where the
upstreams are, so that a new cluster-internal one is added to the wait rather
than left to fail on a cold start.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 47677df9-821d-4a16-a1ba-105224065244

📥 Commits

Reviewing files that changed from the base of the PR and between b097bf0 and 5b8479e.

📒 Files selected for processing (3)
  • deploy/helm/thecombine/charts/frontend/templates/deployment-frontend.yaml
  • nginx/init/05-wait-for-backend-dns.sh
  • nginx/templates/default.conf.template

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The frontend deployment adds a TCP readiness probe on port 80. nginx adds a startup script that waits for backend DNS resolution, with polling and timeout handling. The nginx template documents the hostname covered by the script.

Changes

Readiness checks

Layer / File(s) Summary
nginx backend DNS startup check
nginx/init/05-wait-for-backend-dns.sh, nginx/templates/default.conf.template
nginx checks the configurable backend hostname, polls every two seconds for up to 120 seconds, and logs a timeout before continuing. The template documents this scope.
Frontend TCP readiness probe
deploy/helm/thecombine/charts/frontend/templates/deployment-frontend.yaml
The frontend deployment checks TCP port 80 after a five-second initial delay and at five-second intervals.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 5b847

The PR adds startup DNS waiting and frontend readiness gating, and no actionable merge-blocking risk remains. It is merge-ready after normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: waiting for cluster DNS before starting frontend nginx.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files. (2 skipped: 2 unsupported.)
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/frontend-dns-wait

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.96%. Comparing base (b097bf0) to head (5b8479e).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4354   +/-   ##
=======================================
  Coverage   75.96%   75.96%           
=======================================
  Files         305      305           
  Lines       11384    11384           
  Branches     1411     1411           
=======================================
  Hits         8648     8648           
  Misses       2332     2332           
  Partials      404      404           
Flag Coverage Δ
backend 87.22% <ø> (ø)
frontend 66.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@imnasnainaec

Copy link
Copy Markdown
Collaborator Author

Delivery note

Two of the three files here are image content rather than installer content:

  • nginx/init/05-wait-for-backend-dns.sh and nginx/templates/default.conf.template are baked into the frontend
    image (Dockerfile:63-66)
  • deployment-frontend.yaml is in the deploy/ tree, so it ships in combine-installer.run

So the DNS wait itself only takes effect once a release publishes a new frontend image. An installer built from master
before then carries the readiness probe alongside an image with no wait script, because
installer_release.yml takes the image tag from gh release view while actions/checkout gives it master's deploy/
tree.

That pairing is harmless: the probe is a tcpSocket on port 80, so an old frontend serves or crash loops exactly as it
does today and the fix is simply absent. It does matter for verification, though — reproducing the crash loop and
confirming the fix needs a frontend image built from this branch, not the released tag.

@imnasnainaec imnasnainaec added the 🟨Medium Medium-priority PR label Aug 21, 2026
@imnasnainaec imnasnainaec self-assigned this Aug 21, 2026
imnasnainaec and others added 2 commits August 21, 2026 10:39
The note at the proxy_pass directives repeated the script's own account of why
nginx exits on an unresolvable upstream. Keep only what is not written at the
destination: that a cluster-internal upstream has to be added to the wait.

Rewrap the script's header to the width of the rest of the repo.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The note at the proxy_pass directives told a maintainer to add cluster-internal
upstreams to the wait, which has one hardcoded host, no list to add to, and the
name in its own filename. State what it waits for instead: a host that is not
"backend" is then visibly not covered, and generalizing the script can wait for
a second upstream to actually exist.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@imnasnainaec
imnasnainaec marked this pull request as ready for review August 21, 2026 15:04
@imnasnainaec

This comment was marked as resolved.

@coderabbitai

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant