Skip to content

[DPE-10791] feat(config): own the patroni template and its render target (3/6) - #183

Merged
marceloneppel merged 14 commits into
16/edgefrom
skl-01-update-config-3-template
Aug 17, 2026
Merged

[DPE-10791] feat(config): own the patroni template and its render target (3/6)#183
marceloneppel merged 14 commits into
16/edgefrom
skl-01-update-config-3-template

Conversation

@marceloneppel

@marceloneppel marceloneppel commented Jul 11, 2026

Copy link
Copy Markdown
Member

Issue

Each charm ships its own templates/patroni.yml.j2, loaded from the current working directory. The library must own the template and load it as package data so render_patroni_yml_file works regardless of CWD.

Owning the render surface exposed a mismatch between what gets written and what reads it: the renderer wrote patroni.yaml on both substrates while K8s starts Patroni from patroni.yml, so on K8s the config subsystem drove a file Patroni never reads — a fresh pod comes up without its config, an upgraded one keeps running against a stale copy, and every later TLS, HBA or restore change silently never reaches the cluster.

Solution

  • Move both charms' patroni.yml.j2 into one lib-owned single_kernel_postgresql/templates/patroni.yml.j2 that renders either substrate from a single document. The values that genuinely vary — paths, addresses, the pgBackRest config flag — are bound once at the top; conditionals remain only where the substrates really differ: the Raft section, the watcher rule, the Kubernetes-only keys and the asymmetric authentication block.
  • Load it via importlib.resources instead of a CWD-relative open(); add "substrate" to the render context.
  • Put the config file name on the paths object as patroni_configpatroni.yaml on both substrates, matching shipped 16/edge so the on-disk filename is unchanged across a refresh — and read it from the renderer, the K8s Pebble layer and configure_patroni_on_unit, so what is written and what is started cannot drift apart again.
  • Emit the K8s data paths from the workload's major version rather than spelling 16 out. bin_dir already followed the version, so on a 17 workload the WAL, PostgreSQL-log and Patroni-log directories would have kept pointing at a 16 tree. Nothing renders differently on 16.
  • Key the authentication username on absence rather than falsiness, so collapsing the three role stanzas into one loop keeps the omission to the roles that mean it — K8s drops the key for replication and superuser, VM never does.

Rendered layout changes: on VM the output loses only blank lines, with every non-whitespace line identical and in the same order; on K8s the top-level keys are emitted in one sequence rather than that charm's own. The configuration itself is unchanged, and nothing downstream reads the file as text — the config hash is taken over the Juju config options, not the rendered output — so this carries no restart. The golden matrix in #188 compares the parsed documents against the preserved pre-merge charm templates, so any changed value, added key or reordered pg_hba rule fails the suite.

Two references the charms leave undefined are preserved rather than fixed, so this stays a move: conf_path on K8s and watcher_addr on VM both render empty, exactly as they do today. The watcher's health-check rule that watcher_addr gates is unreachable in any case — the user_databases_map loop emits a broader grant for the same user and database earlier in the list, and pg_hba takes the first match.

Third (code-only) PR in the update_config stack. The golden fixtures and the per-substrate render matrix live in the stack's test PR (#188), along with the minimal update to the existing render test carried here to keep the suite green.

Checklist

  • I have added or updated any relevant documentation.
  • I have cleaned any remaining cloud resources from my accounts.

@marceloneppel
marceloneppel force-pushed the skl-01-update-config-3-template branch from cba92af to fdf2a44 Compare July 11, 2026 16:12
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-2-params branch from a7ee622 to 786d9bd Compare July 20, 2026 13:22
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-3-template branch from fdf2a44 to ef76d9a Compare July 20, 2026 13:22
@marceloneppel marceloneppel changed the title feat(config): move patroni template into the lib as one substrate-conditional template feat(config): move patroni template into the lib as one substrate-conditional template (3/5) Jul 20, 2026
@marceloneppel marceloneppel changed the title feat(config): move patroni template into the lib as one substrate-conditional template (3/5) feat(config): move patroni template into the lib as one substrate-conditional template (3/6) Jul 20, 2026
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-2-params branch from 786d9bd to ddf2305 Compare July 20, 2026 19:04
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-3-template branch from ef76d9a to d994db8 Compare July 20, 2026 19:04
…tion

Internalizes PostgreSQL parameter calculation (worker-process auto/cap
rules, wal_compression, memory-limit conversion) into the library so
ConfigManager owns the full config-build flow instead of stopping at a
TODO. Resource discovery (cpu_cores, memory_bytes) moves onto the
workload rather than a substrate-specific manager: get_available_memory
already lives there, ConfigManager already holds self.workload, and
K8sManager doesn't exist yet when ConfigManager is constructed in
abstract_charm - so the workload is the only place both substrates can
share a single no-branch call site.

K8sWorkload gains required unit_name/namespace constructor args so its
lightkube Pod/Node lookups can resolve "my own pod" without depending on
a charm object reference, mirroring what the K8s charm does today via
self.unit.name/self.model.name. lightkube becomes a real (not merely
transitive) dependency via a new k8s pyproject extra, matching the
version pins the K8s test charm already uses.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-2-params branch from ddf2305 to f9a1111 Compare July 21, 2026 17:46
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-3-template branch from d994db8 to f5d55f2 Compare July 21, 2026 17:46
The parameter-building port placed lightkube node/pod resource lookups on K8sWorkload and injected unit_name/namespace into its constructor. Both duplicate identity the state already holds and put K8s-API code on the workload abstraction, which reviewers flagged.

Move the lookups onto K8sManager, which is already K8s-only-loaded and holds the state, sourcing pod/namespace from state.peer.unit_name and state.model_name. ConfigManager reaches the value through a new state.available_resources property backed by a per-substrate provider (the VM workload, or the K8s manager), so the call site stays substrate-agnostic and lightkube never enters the VM import graph.

Behavior is unchanged: node allocatable clamped by the container limit, 403 -> DeployedWithoutTrustError. The Node lookups drop the namespace argument, which lightkube ignores for cluster-scoped resources.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
…ditional template

The two charms each carried their own templates/patroni.yml.j2 and the config
manager loaded it with a CWD-relative open(), so the lib could not own the
render. Merge both copies into a single template the lib ships and loads as
package data, so the charms delete their copies in the adoption PR and the
template travels with the code that renders it rather than with each charm's
working directory.

The VM and K8s templates diverge in section ordering, whole blocks (raft,
bypass_api_service, pod_ip), path variables, and pg_hba rules, so the merged
template branches at the top level on a substrate context var with each
substrate's body kept verbatim. That guarantees byte-for-byte identical output
per substrate and avoids interleaving conditionals that could silently shift a
byte. render_patroni_yml_file now loads the template via importlib.resources and
passes the substrate the template branches on.

Golden tests render the merged template against byte copies of each charm's
original template across the full matrix of conditional dimensions (tls,
connectivity, ldap, restore/pitr, slots, peers, watcher, extra replication
endpoints, tags) and assert equality, so any divergent byte fails the suite.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
marceloneppel force-pushed the skl-01-update-config-3-template branch from f5d55f2 to 79f2a4f Compare July 23, 2026 17:34
Base automatically changed from skl-01-update-config-2-params to 16/edge July 24, 2026 19:55
The renderer wrote patroni.yaml on both substrates while the K8s Pebble
layer starts Patroni with patroni.yml, so on K8s the config subsystem
would drive a file Patroni never reads: a fresh pod misses its config,
an upgraded one keeps running against a stale one, and every later TLS,
HBA or restore change silently never reaches the cluster.

The two names are not interchangeable — each is what its substrate has
always used (snap: patroni.yaml, K8s charm: patroni.yml) — so the file
name belongs to the paths object, where render and launch both read it
from one place and cannot drift apart again.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The template gates the watcher's health-check rule on watcher_addr, but
only watcher — the ip:port Raft address — was ever supplied, so Jinja
resolved watcher_addr to undefined and the rule was dropped: a related
watcher joins Raft yet is refused by PostgreSQL when it probes.

The pristine VM charm has the same omission, so this is not a migration
regression; the port is where it becomes cheap to fix, since the address
is already in hand and only needs its port trimmed.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel marceloneppel changed the title feat(config): move patroni template into the lib as one substrate-conditional template (3/6) feat(config): own the patroni template, its render target, and its watcher context (3/6) Aug 10, 2026
The `'hostssl' if enable_tls else 'host'` ternary was spelled out at all 41
pg_hba rule sites, 40 characters of ceremony ahead of the part a reader is
actually there for, pushing every rule off the left margin.

Binding it once says the same thing in the name and lets the rules read as
pg_hba again.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
Each substrate spelled out replication, rewind and superuser as three
copies of the same stanza, so the TLS-guarded certificate lines existed six
times across the file and any change to them had to be made six times.

Driving the three roles from a list keeps the stanza in one place, and the
substrates keep their own copy because their certificate handling genuinely
differs: VM emits sslrootcert and always emits the client pair, K8s emits
neither unless TLS is on.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The seven charmed_* roles were seven rule lines per substrate that differed
only in the role name, so adding a role meant editing two places and the
grant itself was buried in fourteen near-identical lines.

Listing the roles once puts the set of roles where a reader looks for it and
leaves one rule to read.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The template was the two charm templates concatenated under a substrate
conditional, so roughly three quarters of it existed twice: the same 43
PostgreSQL parameters, the same restore command, the same pg_hba rules, each
needing every edit applied in two places. The two copies had already drifted
apart in ways nobody intended, down to a different indent for the same list.

Binding the values that genuinely vary per substrate once at the top collapses
that to a single document with conditionals only where the substrates really
differ - the raft section, the watcher rule, the Kubernetes-only keys and the
asymmetric authentication block.

Field order now follows one sequence rather than each charm's, which changes
the rendered layout; the parsed configuration is unchanged, verified across the
full render matrix.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
The rule this fed is unreachable. The user_databases_map loop emits
`postgres watcher 0.0.0.0/0` earlier in the same list whenever a watcher is
related - which is exactly when the watcher host is non-null - and pg_hba
takes the first matching record, so the narrower /32 rule below it never
authorises a connection the broader one has not already allowed.

The charm omits the same variable, so restoring the omission keeps this a
move rather than a behaviour change, and leaves the dead rule visible for
whoever decides what it was meant to do.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
K8s omits the username for the replication and superuser roles, which the
merged template expressed as a truthiness test. An empty username would then
be dropped on VM too, where the charm template always emitted the key.

Testing for none keeps the omission to the roles that mean it. The values are
non-empty constants today, so nothing rendered differently - this closes a gap
the golden matrix cannot see, since it pins both usernames at one value.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
bin_dir already followed the running major version while the WAL, PostgreSQL
log and Patroni log paths spelled 16 out, so a 17 workload would have kept
three directories pointing at a 16 tree.

The version is already in the render context, so this only reaches for what
was there. Nothing renders differently on 16, which is also why the golden
matrix cannot see the difference - it pins the version at 16.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel marceloneppel changed the title feat(config): own the patroni template, its render target, and its watcher context (3/6) feat(config): own the patroni template and its render target (3/6) Aug 12, 2026
@marceloneppel marceloneppel changed the title feat(config): own the patroni template and its render target (3/6) [DPE-10791] feat(config): own the patroni template and its render target (3/6) Aug 12, 2026
@marceloneppel
marceloneppel marked this pull request as ready for review August 12, 2026 20:13
@marceloneppel
marceloneppel requested a review from a team as a code owner August 12, 2026 20:13
@marceloneppel
marceloneppel requested review from carlcsaposs-canonical, dragomirp, juju-charm-bot and taurus-forever and removed request for a team August 12, 2026 20:13
Comment thread single_kernel_postgresql/workload/paths/k8s.py Outdated
…6/edge

Shipped 16/edge launches patroni.yaml on both substrates (K8s at
charm.py:2345, VM via the lib 16/edge already consumes). The .yml split
matched main, a different track, and would rename the config file on a
16/edge K8s refresh — risking a window where Patroni restarts before the
first render and finds no config. Unify patroni_config on patroni.yaml
for both substrates so the on-disk filename is unchanged across refresh.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
Both substrates returned the same value (patroni_conf / "patroni.yaml")
through identical overrides, differing only in docstring. Move it to one
concrete implementation on the base class and drop the per-substrate
overrides, so the file name lives in one place and cannot drift apart.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@marceloneppel
marceloneppel requested a review from akram09 August 17, 2026 12:00

@akram09 akram09 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems good for me!
One thing maybe regarding how we are getting the available resources, I am not sure if the best solution would be to pass through the state. Do you think maybe we can have the K8sManager as K8sClient and use the client in the Managers ?

@marceloneppel

Copy link
Copy Markdown
Member Author

Seems good for me! One thing maybe regarding how we are getting the available resources, I am not sure if the best solution would be to pass through the state. Do you think maybe we can have the K8sManager as K8sClient and use the client in the Managers ?

Yes, @akram09. The lightkube lookups moved onto K8sManager in ef52b09, but managers still reach them through state.available_resources, which is the part you're pointing at. Injecting the provider drops that passthrough along with the mutable resource_provider slot and its "not set" RuntimeError, and matches the constructor injection ConfigManager gains in #180, the next PR in this stack, for tls_manager/patroni_manager. One constraint: inject the existing ResourceProvider protocol rather than a K8s client type, since VM satisfies it through the workload and K8sManager isn't importable there. I'll do it as a follow-up so #183 stays as-is.

@dragomirp‚ this reverses the state route from #182, so say if you'd rather keep it.

@marceloneppel
marceloneppel merged commit 9173ca5 into 16/edge Aug 17, 2026
6 of 7 checks passed
@marceloneppel
marceloneppel deleted the skl-01-update-config-3-template branch August 17, 2026 19:11

@taurus-forever taurus-forever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tnx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants