Skip to content

feat(bootstrap): use an init container instead of a separate Job for new instances - #11319

Open
leonardoce wants to merge 8 commits into
cloudnative-pg:mainfrom
leonardoce:remove-jobs-initcontainer
Open

feat(bootstrap): use an init container instead of a separate Job for new instances#11319
leonardoce wants to merge 8 commits into
cloudnative-pg:mainfrom
leonardoce:remove-jobs-initcontainer

Conversation

@leonardoce

@leonardoce leonardoce commented Aug 13, 2026 •
edited by mnencia
Loading

Copy link
Copy Markdown
Contributor

New primary and replica instances are now bootstrapped (initdb, recovery, pg_basebackup, join, and snapshot restore) directly on the instance Pod via an init container, instead of running a separate Kubernetes Job beforehand. This removes the extra Job resource previously created for every new instance, along with its own Pod, and the wait for that Job to complete before the actual instance Pod could start.

A stuck bootstrap surfaces as PhaseUnrecoverable on the Cluster status, and a replica resuming a stalled bootstrap reports PhaseCreatingReplica with an Event.

Closes: #11228

Original PR description

New primary and replica instances are now bootstrapped (initdb, recovery,
pg_basebackup, join, and snapshot restore) directly on the instance Pod via
an init container, instead of running a separate Kubernetes Job beforehand.
This removes the extra Job resource previously created for every new
instance, along with its own Pod, and the wait for that Job to complete
before the actual instance Pod could start.

Important

This is an alternative approach for #11231 that is less invasive on the codebase.
It requires #11315 to be merged too.

@leonardoce
leonardoce marked this pull request as ready for review August 13, 2026 08:04
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.28 release-1.29 release-1.30 labels Aug 13, 2026
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement 🪄 New feature or request labels Aug 13, 2026
@leonardoce leonardoce added do not backport This PR must not be backported - it will be in the next minor release and removed backport-requested ◀️ This pull request should be backported to all supported releases release-1.28 release-1.29 release-1.30 labels Aug 13, 2026
@leonardoce

leonardoce commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

/test bp=pr-1025 l=kind

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/31681054762

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Aug 13, 2026
@leonardoce
leonardoce force-pushed the remove-jobs-initcontainer branch from cb78fa2 to 4987ac8 Compare August 17, 2026 08:51
@leonardoce

leonardoce commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

/test bp=pr-1025 l=kind

@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/32012769354

@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Aug 17, 2026
@NiccoloFei
NiccoloFei force-pushed the remove-jobs-initcontainer branch from 4987ac8 to 0e06aee Compare August 21, 2026 12:47
@mnencia
mnencia force-pushed the remove-jobs-initcontainer branch 2 times, most recently from 632e3d9 to 51541be Compare August 25, 2026 17:35
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 25, 2026
@mnencia
mnencia force-pushed the remove-jobs-initcontainer branch from 51541be to 01825fc Compare August 25, 2026 17:37
@mnencia mnencia changed the title feat: bootstrap new instances with an init container instead of a separate Job feat(bootstrap): use an init container instead of a separate Job for new instances Aug 26, 2026
@mnencia

mnencia commented Aug 26, 2026

Copy link
Copy Markdown
Member

/test

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/32969475705

@mnencia

mnencia commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Full E2E run on EDB fork https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/33049566677

@mnencia
mnencia force-pushed the remove-jobs-initcontainer branch 2 times, most recently from 535f11c to fa08f68 Compare August 26, 2026 15:26
@mnencia

mnencia commented Aug 27, 2026

Copy link
Copy Markdown
Member

/test bp=pr-1025

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/33049147476

@mnencia mnencia mentioned this pull request Aug 27, 2026
Merged
leonardoce and others added 8 commits August 27, 2026 11:06
@leonardoce @mnencia
feat: bootstrap new instances with an init container instead of a sep…
005b2f9 
…arate Job

New primary and replica instances are now bootstrapped (initdb, recovery,
pg_basebackup, join, and snapshot restore) directly on the instance Pod via
an init container, instead of running a separate Kubernetes Job beforehand.
This removes the extra Job resource previously created for every new
instance, along with its own Pod, and the wait for that Job to complete
before the actual instance Pod could start.

Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
@mnencia
fix: surface a failing bootstrap init container as unrecoverable
405e68c 
A bootstrap init container has no Job-style backoff limit, so
kubelet retries a failing one forever with no signal; surface it on
first failure via Cluster status, leaving the Pod untouched so a
later success clears it again.

Assisted-by: Claude
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
refactor: extract reconcileFailedJobs for symmetry with the bootstrap…
8d8c813 
… check

Assisted-by: Claude
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
fix: surface a resumed replica bootstrap on the Cluster status
226d49c 
attachReplicaBootstrapInitContainer resumes a stalled replica's
bootstrap but never told the Cluster: no Event, no
PhaseCreatingReplica. Restore both, matching joinReplicaInstance and
the Job-based recreateReplicaBootstrapJob it replaced.

Assisted-by: Claude
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
fix: rename misnamed recovery var to bootstrap in getInvolvedSecretNames
ad82c7d 
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
fix: requeue quickly and log clearly on replica Pod AlreadyExists
bbe871c 
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
docs: drop phantom specs.resolveBarmanEndpointCA reference
066ee6f 
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
fix: drop dead BootstrapWorkContainerName exclusion in ComparePodSpecs
8a29ae2 
NewInstance stamps the stored PodSpec annotation before
AddBootstrapInitContainer runs, and the freshly-built target skips
that call too, so the container can never appear on either side. The
comment's premise, including a NewInstanceWithBootstrap citation that
doesn't exist, was wrong.

Assisted-by: Claude
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
@mnencia
mnencia force-pushed the remove-jobs-initcontainer branch from fa08f68 to 8a29ae2 Compare August 27, 2026 09:06
@mnencia

mnencia commented Aug 27, 2026

Copy link
Copy Markdown
Member

/test bp=pr-1025

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/33057043908

mnencia added a commit to cloudnative-pg/plugin-barman-cloud that referenced this pull request Aug 27, 2026
@armru @mnencia
feat: serve restore hooks from the instance sidecar (#1025)
7bb01a8 
CloudNativePG is moving the bootstrap of new instances from dedicated
Jobs into the instance pod itself (cloudnative-pg/cloudnative-pg#11319):
the restore that used to run in a recovery Job now happens in-process
inside the instance pod before PostgreSQL starts. The sidecar shipped in
that pod must therefore answer the same Restore RPC the operator sends
over the plugin sockets, so the instance mode now registers the
restore-job hooks and advertises the restore-job service capability.

A cluster that only bootstraps from an object store, without continued
archiving, previously received no sidecar at all in its instance pods;
under the new flow that leaves the bootstrap without a plugin socket,
both for the Restore RPC and for `wal-restore` during the recovery
replay. The injection condition is widened to match what the plugin
configuration already considers valid, so recovery-only clusters get the
sidecar too.

The sidecar is dropped once the instance's bootstrap completes
(cluster.Status.CurrentPrimary set), which triggers one deterministic
rollout to remove it, accepted rather than engineered around since it
uses the same switchover/restart machinery as any other pod-spec change.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mnencia mnencia mnencia approved these changes
@armru armru armru approved these changes
@NiccoloFei NiccoloFei Awaiting requested review from NiccoloFei NiccoloFei is a code owner
@litaocdl litaocdl Awaiting requested review from litaocdl litaocdl is a code owner
@jsilvela jsilvela Awaiting requested review from jsilvela

Assignees

No one assigned

Labels

do not backport This PR must not be backported - it will be in the next minor release enhancement 🪄 New feature or request lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Bootstrap PostgreSQL instances without a separate Job or init container

4 participants

@leonardoce @mnencia @armru @cnpg-bot