fix: write delete tombstones on a separate ClickHouse client

[lesandie]

What broke

A delete phase died 78 minutes in, on the first batch it processed:

Code: 373. DB::Exception: Session … is locked by a concurrent client. (SESSION_IS_LOCKED)
  s3gc.py in do_use -> ch_client.insert(
  clickhouse_connect/.../orchestration.py -> yield QueryOp("DESCRIBE TABLE …")

The objects were already deleted from S3 when the tombstone write was rejected. The job exited non-zero having removed up to --deletebatchsize objects with no record, so a resumed run could not tell they were done.

Why

connect_to_ch() built a single clickhouse_connect client. The driver gives it an auto-generated session_id, and ClickHouse permits one query at a time per session. do_use() holds that session for the entire anti-join while it consumes query_row_block_stream, and insert() issues its own DESCRIBE TABLE before writing — a second concurrent query on the held session.

The bug is not new. It is present in every build back to the original single-client design; it simply had never fired.

Why it had never fired

Every earlier delete that actually reclaimed data ran with --order-by-objpath, which makes the server sort the whole result before streaming it, and ran against ClickHouse 25.x. The run that hit this was the first delete without global ordering — the documented default for Kubernetes Jobs — and against 24.8.

Which of the two previously masked it was not established. The fix doesn't depend on the answer, and depending on either to keep the session free was accidental rather than designed.

Changes

• connect_to_ch() builds a second client, used only for writes issued while a stream from the first is open. Tombstone writes move to it. The comment at the call site says why, because a future reader will otherwise read the second connection as redundant.
• USETOTAL is now settable from the Job template. --usetotal already existed on the CLI and, via env_prefix="S3GC", in the environment — but the renderer never emitted it, so the only delete available to a Kubernetes operator was unbounded. A bounded run exercises anti-join → deletion → tombstone write-back in minutes; one would have caught this defect for the price of minutes instead of a multi-hour run that died with objects already gone. Optional, and renders no variable at all when empty, since S3GC_USETOTAL is parsed as an integer. Existing env files render unchanged.
• CHANGELOG.md records both, with the evidence.
• TODO.md records a separate defect this investigation surfaced: no release has ever been published. container.yml triggers on tags: ['v*.*.*'], but the repository's tags are v0.5, v_0.1 and v_0.2 — none can match. Every deployment so far has pinned a hand-built dev-* digest with no org.opencontainers.image.revision, which is why identifying which commit was running here required comparing log-message formats between runs. Not fixed in this PR; it needs a vX.Y.Z tag, not a code change.

Tests

Both defects have regression tests that fail without the fix:

• test_tombstones_are_written_off_the_streaming_session — a fake client that rejects writes while its stream is open, modelling the one-query-per-session rule.
• test_connect_to_ch_builds_a_separate_writer_client — the two clients must be distinct, or the lock returns.
• Renderer tests for USETOTAL set, unset, invalid, and absent from the file (backward compatibility).

pytest -v -m "not dev_cluster"   65 passed
render.py example.env            OK, S3GC_USETOTAL absent
render.py (USETOTAL=5000)        OK, value: "5000"
kubeconform -strict              2 resources, Valid: 2

Suggested release

The delete path is unusable without this, so it warrants a published image — which per the TODO.md item above means the first one this repository has ever produced. v0.6.0 rather than v0.5.1, since USETOTAL adds deployment configuration surface.