Expose ZooKeeper auto-purge settings

[lfrancke]

Problem

ZooKeeper, by default, never deletes old snapshots or transaction logs. Cleanup is opt-in via two settings (available since ZK 3.4.0):

• autopurge.snapRetainCount (default 3, minimum 3) -> I suggest we keep the default of 3
• autopurge.purgeInterval (default 0 = disabled; positive integer = hours between purges) -> I suggest 24 (hours)

We currently don't set or expose them. Users can use configOverrides. We had customers whose PVC filled up with snapshots/logs multiple times, we should set a default.

But because the purge stuff depends on snapshots actually being created we should tweak other settings as well. Our default PVC size is 1GiB (which we could also increase to 2GiB).

• snapSizeLimitInKb defaults to 4GiB which doesn't fit on our default PVC. Should there be some humongous transactions it might never roll and "crash" on our default config. I suggest reducing this to 100MiB
• preAllocSize defaults to 64MiB, the docs say to change this when snapshots are written more frequently -> which they are. This ist he transaction log and just how much data is allocated initially, not a limit on its size I believe.

If I'm not totally wrong (the AI disagrees with me but I think I'm correct, I might be wrong though ;-) ) it means we have at most 5 x 100MiB for snapshots plus the transactions logs needed to replay all of those retained logs. A new log is created when a new snapshot is created. Which means (guessing) doubling the size. So the worst case would be:

3 x 100 MiB plus one current log (up to ~100MiB) + 3 old logs (up to ~100MiB) = ~700MiB.

I'm making a guess here at how large the logs are to be honest.

Either way I believe my suggested defaults are safer than what we have today.

Open decision

Do we want to expose any of this via the CRD? I initially leaned towards "yes" but now I'm leaning more towards "no".
Reason: Initially I thought the two autopurge settings are enough. Adding preAllocSize and snapSizeLimitInKb would mean adding two more or picking which to expose and which not...

Acceptance criteria

1. Enable auto-purge by default. Set sane defaults in zoo.cfg:

   • autopurge.snapRetainCount=3 (already the default, maybe good to hardcode it?)
   • autopurge.purgeInterval=24 (hours)
   • snapSizeLimitInKb ot 100 MiB (don't know the unit right now)
   • preAllocSize to 16 MiB (same, don't know the unit)

2. Update operator docs (docs/modules/zookeeper/pages/usage_guide/) with a short section on data-dir lifecycle.

3. Release note flagging the behavior change for upgrading users (their disk usage will start dropping after the first purge run, that's expected, not a bug).

Verification

• Deploy with defaults, write enough traffic to roll a few snapshots (zookeeper.snapCount defaults to 100 000 txns), confirm log.* and snapshot.* files older than the retention window are deleted within the purge interval.