Add automated backup and restore via dedicated CRDs

[discostur]

Add automated backup & restore via dedicated CRDs

Closes the long-standing request for operator-managed backups (#1795, #862). This
supersedes the gRPC-plugin approach of #1798 with a lighter, CRD-driven design that
wraps clickhouse-backup and mirrors the
existing ClickHouseKeeper controller-runtime pattern.

What it adds

Three new custom resources in the clickhouse.altinity.com/v1 group:

Kind	Short	Operator action
ClickHouseBackup	chb	one-off backup → Kubernetes Job
ClickHouseBackupSchedule	chbs	recurring backup → managed CronJob
ClickHouseRestore	chr	one-off restore → Kubernetes Job

Design

• No reinvented backup logic. clickhouse-backup runs as a sidecar (documented
  prerequisite, API_CREATE_INTEGRATION_TABLES=true); the operator-generated jobs trigger
  it remotely through the system.backup_actions integration table. The new controllers
  stay fully decoupled from CHI reconciliation.
• controller-runtime, mirroring pkg/controller/chk. Jobs/CronJobs are owned by the
  CR (automatic GC + status tracking via status.conditions).
• Schedule → native CronJob: schedule/suspend/concurrency/history come from the
  CronJob; remote retention is delegated to clickhouse-backup (BACKUPS_TO_KEEP_REMOTE).

Cluster awareness

• Backup defaults to one replica per shard (FirstPerShard) — correct and
  storage-efficient for Replicated* tables. AllReplicas is available for clusters with
  non-replicated/local tables.
• Restore applies schema on all replicas and data on the first replica of each
  shard, letting native replication synchronize the rest.

Restore safety (per mature DB-operator conventions, e.g. CloudNativePG)

• Preflight validation (target CHI Completed, topology reachable) surfaced via conditions.
• Overwrite guard: refuses a non-empty target unless overwrite: true.
• Restoring into a fresh, empty CHI is the recommended path; one-shot jobs
  (backoffLimit: 0, restartPolicy: Never).

Included

• API types + hand-written deepcopy (regenerable via dev/run_code_generator.sh).
• Controllers + minimal Job/CronJob builders + operator wiring (thread_backup.go).
• Hand-written CRD section + RBAC (incl. batch jobs/cronjobs); regenerated install
  bundles and Helm chart.
• docs/backup.md + docs/chb-examples/.
• Go unit tests (builders + controller reconcile) and a TestFlows e2e test
  (tests/e2e/test_backup_restore.py) doing a backup→restore round-trip with replica-sync
  verification.

Notes / follow-ups

• Host service names are resolved from cluster layout counts using the default naming
  scheme; explicit shard/replica lists / custom host names are a planned follow-up.
• For sharded clusters the sidecar remote path should include the {shard} macro
  (e.g. S3_PATH: backup/shard-{shard}).

🤖 Generated with Claude Code

[discostur]

Pushed an update extending the feature with several enhancements:

• Selective backups — spec.tables (clickhouse-backup --tables pattern) and spec.partitions.
• Incremental backups — spec.diffFromRemote → create_remote --diff-from-remote=<base>.
• Retention — spec.keepLastRemote keeps only the N most recent remote backups (best-effort prune via system.backup_list + delete remote).
• Verification — spec.verify runs a post-backup job that downloads the backup and checks its integrity (no cluster data touched); result surfaced as the Verified condition + a clickhouse_operator_backup_verifications_failed metric.
• Observability — Prometheus metrics on the operator's existing :9999/metrics endpoint (clickhouse_operator_backups_* / restores_*, a duration histogram, last-success timestamp), Kubernetes Events on backup/restore lifecycle, and duration in status + a duration printer column.
• Bootstrap-from-backup — annotate a fresh ClickHouseInstallation with clickhouse.altinity.com/recover-from-backup: <name> and the operator auto-creates a one-time ClickHouseRestore once the cluster is up.
• Compression & encryption — documented as clickhouse-backup sidecar settings (COMPRESSION_FORMAT, S3 S3_SSE/SSE_KMS_KEY_ID, etc.).

I also corrected the replicated-restore path to issue the schema ON CLUSTER (requires the sidecar's restore_schema_on_cluster) instead of per-replica.

[discostur]

Tested locally in dev k8s cluster ... happy to get some feedback from the maintainers ;)