diff --git a/pkg/storage/general.go b/pkg/storage/general.go index 107b58ed6..5445801db 100644 --- a/pkg/storage/general.go +++ b/pkg/storage/general.go @@ -121,9 +121,9 @@ func (bd *BackupDestination) RemoveBackupRemote(ctx context.Context, backup Back return retry.RunCtx(ctx, func(ctx context.Context) error { return bd.DeleteFile(ctx, path.Join(backup.BackupName, f.Name())) }) - } else { - return nil } + + return nil } return retry.RunCtx(ctx, func(ctx context.Context) error { return bd.DeleteFile(ctx, path.Join(backup.BackupName, f.Name())) diff --git a/test/testflows/clickhouse_backup/requirements/fips/QA_STP_ClickHouse_Backup_FIPS.md b/test/testflows/clickhouse_backup/requirements/fips/QA_STP_ClickHouse_Backup_FIPS.md new file mode 100644 index 000000000..0b2cd5515 --- /dev/null +++ b/test/testflows/clickhouse_backup/requirements/fips/QA_STP_ClickHouse_Backup_FIPS.md @@ -0,0 +1,358 @@ +# QA-STP ClickHouse Backup FIPS Compatibility Release +# Software Test Plan + +(c) 2026 Altinity Inc. All Rights Reserved. + +**Author:** vsviderskyi + +**Date:** May 19, 2026 + +## Execution Summary + +**Completed:** + +**Test Results:** + +* + +**Build Report:** + +* + +**Summary:** + +Started to execute test plan on TBD and ended on TBD. + +## Table of Contents + +* 1 [Introduction](#introduction) +* 2 [Timeline](#timeline) +* 3 [Human Resources And Assignments](#human-resources-and-assignments) +* 4 [End User Recommendations](#end-user-recommendations) + * 4.1 [Release Notes](#release-notes) + * 4.2 [FIPS Configuration](#fips-configuration) +* 5 [Known Issues](#known-issues) + * 5.1 [Open Issues](#open-issues) + * 5.2 [Summary](#summary) +* 6 [Scope](#scope) + * 6.1 [Test Environment](#test-environment) + * 6.2 [Automated FIPS Regression Tests](#automated-fips-regression-tests) + * 6.2.1 [Connectivity Against ClickHouse FIPS and Non-FIPS Server Versions](#connectivity-against-clickhouse-fips-and-non-fips-server-versions) + * 6.2.2 [`clickhouse-backup-fips --version` Output](#clickhouse-backup-fips---version-output) + * 6.2.3 [GODEBUG `fips140` Modes](#godebug-fips140-modes) + * 6.2.4 [FIPS Integrity Self-test Failure on Tampered Binary](#fips-integrity-self-test-failure-on-tampered-binary) + * 6.2.5 [Forced CAST Failures](#forced-cast-failures) + * 6.2.6 [Inbound TLS — REST API With `openssl s_client`](#inbound-tls--rest-api-with-openssl-s_client) + * 6.2.7 [Outbound TLS to ClickHouse Server With `openssl s_server`](#outbound-tls-to-clickhouse-server-with-openssl-s_server) + * 6.2.8 [Outbound TLS to S3 Endpoint With `openssl s_server`](#outbound-tls-to-s3-endpoint-with-openssl-s_server) + * 6.2.9 [ACVP Tests](#acvp-tests) + +## Introduction + +This test plan covers FIPS 140-3 compatibility testing of the `clickhouse-backup` Altinity FIPS-compatible build (`clickhouse-backup-fips`). + +The main goal of this test plan is to validate that `clickhouse-backup` is FIPS 140-3 compatible. To validate this, the following items SHALL be checked: + +* The FIPS-built `clickhouse-backup` binary starts with the Go FIPS 140-3 cryptographic module enabled and reports it in `--version` output under all three Go FIPS runtime modes (`GODEBUG` unset, `GODEBUG=fips140=on`, `GODEBUG=fips140=only`). +* The FIPS cipher policy is enforced for inbound and outbound TLS when running in strict mode (`GODEBUG=fips140=only`). +* The binary aborts on startup if the FIPS integrity check or any startup cryptographic self-test fails. +* The binary stays operational against both FIPS-compatible and non-FIPS-compatible ClickHouse server versions. +* The ACVP (Automated Cryptographic Validation Protocol) test wrapper bundled with `clickhouse-backup` runs successfully against the FIPS-built binary. + +## Timeline + +The testing of `clickhouse-backup` FIPS builds SHALL be started on May 7, 2026 and be completed by May 25, 2026. + +## Human Resources And Assignments + +The following team members SHALL be dedicated to this release: + +* Vitalii Sviderskyi (regression tests) +* Vitaliy Zakaznikov (manager, regression tests) +* Eugene Klimov (clickhouse-backup, FIPS build and Docker image) + +## End User Recommendations + +### Release Notes + +* https://github.com/Altinity/clickhouse-backup/blob/master/ChangeLog.md +* https://docs.altinity.com/altinitystablebuilds/fips-compatible-altinity-builds/ + +### FIPS Configuration + +For FIPS-compatible operation, users should: + +* Use the FIPS-built `clickhouse-backup-fips` binary. +* Configure TLS for FIPS-compatible ClickHouse server using TLSv1.2 and TLSv1.3 only, with the FIPS-approved cipher suites published in the Altinity FIPS documentation (): + * TLSv1.2 `cipherList`: `ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384` + * TLSv1.3 `cipherSuites`: `TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384` + +`clickhouse-backup-fips` can also be used against non-FIPS ClickHouse servers. + +## Known Issues + +* None known at the time of writing. + +### Open Issues + +* TBD + +### Summary + +> [!NOTE] +> **Pass\*** — tests passed with known fails. + +| Test Suite | Result | Comments | +| --- | --- | --- | +| Connectivity vs FIPS and non-FIPS ClickHouse server | [_TBD_](#connectivity-against-clickhouse-fips-and-non-fips-server-versions) | FIPS `clickhouse-backup` vs FIPS ClickHouse server, and FIPS `clickhouse-backup` vs non-FIPS ClickHouse server | +| `clickhouse-backup-fips --version` output | [_TBD_](#clickhouse-backup-fips---version-output) | | +| Test with `GODEBUG=fips140=on`, `GODEBUG=fips140=only`, and when `GODEBUG` is not set at all | [_TBD_](#godebug-fips140-modes) | | +| FIPS Integrity Self-test Failure on Tampered Binary | [_TBD_](#fips-integrity-self-test-failure-on-tampered-binary) | Tampered `.go.fipsinfo` checksum must abort startup | +| Forced CAST Failures | [_TBD_](#forced-cast-failures) | | +| Inbound TLS — REST API (`openssl s_client`) | [_TBD_](#inbound-tls--rest-api-with-openssl-s_client) | | +| Outbound TLS to ClickHouse server (`openssl s_server`) | [_TBD_](#outbound-tls-to-clickhouse-server-with-openssl-s_server) | | +| Outbound TLS to S3 (`openssl s_server`) | [_TBD_](#outbound-tls-to-s3-endpoint-with-openssl-s_server) | | +| ACVP Tests | [_TBD_](#acvp-tests) | | + +## Scope + +The scope of testing the `clickhouse-backup` FIPS-compatible release SHALL be defined as follows. + +### Test Environment + +The following artifacts and tools will be used: + +* FIPS `clickhouse-backup` binary built with `GOFIPS140=v1.0.0` (`clickhouse-backup-fips`). +* FIPS-compatible Altinity ClickHouse server image: `altinity/clickhouse-server:25.3.8.30001.altinityfips`. +* Non-FIPS Altinity ClickHouse server image: `altinity/clickhouse-server:25.8.16.10002.altinitystable`. +* `openssl` CLI tool on the test host for TLS client and server probes. + +### Automated FIPS Regression Tests + +All FIPS tests below SHALL be executed as automated TestFlows scenarios. The following test suites SHALL be executed. + +#### Connectivity Against ClickHouse FIPS and Non-FIPS Servers + +Results: + +* TBD + +Check that `clickhouse-backup-fips` connects to both versions of the ClickHouse server. + +Run the FIPS `clickhouse-backup` against: + +* FIPS-compatible Altinity ClickHouse server `altinity/clickhouse-server:25.3.8.30001.altinityfips` over the secure native TCP port `9440`. +* Non-FIPS Altinity ClickHouse server `altinity/clickhouse-server:25.8.16.10002.altinitystable` over the standard native TCP port `9000`. + +Expected result: + +* In both cases `clickhouse-backup-fips tables` returns the list of tables without TLS or authentication errors and exits with code `0`. + +#### `clickhouse-backup-fips --version` Output + +Results: + +* TBD + +Check that the FIPS-built binary reports the FIPS 140-3 module as active in its version output, and that the binary itself is built with `GOFIPS140=v1.0.0`. + +Run: + +``` +clickhouse-backup-fips --version +go version -m $(which clickhouse-backup-fips) +``` + +Expected result: + +* `--version` output contains the line `FIPS 140-3: true` (this corresponds to Go's `crypto/fips140.Enabled()` returning true at runtime). +* `go version -m` output contains `build GOFIPS140=v1.0.0`. + +#### GODEBUG `fips140` Modes + +Results: + +* TBD + +Check that `clickhouse-backup-fips` behaves correctly under each of the three Go FIPS runtime modes listed below. + +For every mode run both `--version` and a basic `tables` command against the FIPS-compatible Altinity ClickHouse server `altinity/clickhouse-server:25.3.8.30001.altinityfips`. + +* `GODEBUG` not set — FIPS mode is enabled by build-time default (`GOFIPS140=v1.0.0`). + + Expected result: + * `--version` reports `FIPS 140-3: true`. + * `tables` returns the list of tables. + +* `GODEBUG=fips140=on` — FIPS mode is enabled explicitly without strict enforcement. This is the mode used for the forced CAST test below. + + Expected result: + * `--version` reports `FIPS 140-3: true`. + * `tables` returns the list of tables. + +* `GODEBUG=fips140=only` — FIPS mode is enabled with strict enforcement; any non-approved cryptographic operation triggers an error or panic. This is the mode used for the TLS policy tests below and the default of the FIPS Docker image. + + Expected result: + * `--version` reports `FIPS 140-3: true`. + * `tables` against an approved TLS configuration returns the list of tables. + * Non-approved cryptographic operations cause the binary to fail. + * The full `clickhouse-backup` TestFlows regression suite runs in this mode without panics or strict-FIPS-only regressions. + +#### FIPS Integrity Self-test Failure on Tampered Binary + +Results: + +* TBD + +Check that the FIPS startup integrity self-test stops the binary if the FIPS module bytes have been modified. + +Take a copy of `clickhouse-backup-fips`, corrupt its `.go.fipsinfo` checksum section, and try to run the tampered copy. + +Expected result: + +* The tampered binary panics on startup with `panic: fips140: verification mismatch` and exits with a non-zero exit code. +* The unmodified original binary continues to work normally. + +#### Forced CAST Failures + +Results: + +* TBD + +Check that the FIPS module refuses to start if any startup self-test fails. + +Run the FIPS binary with the `GODEBUG=failfipscast` hook, substituting one self-test name at a time, for example: + +``` +GODEBUG=failfipscast=SHA2-256,fips140=on clickhouse-backup-fips --version +``` + +`SHA2-256` in the command above can be replaced with any name from the list below: + +``` +AES-CBC +CTR_DRBG +CounterKDF +DetECDSA P-256 SHA2-512 sign +ECDH PCT +ECDSA P-256 SHA2-512 sign and verify +ECDSA PCT +Ed25519 sign and verify +Ed25519 sign and verify PCT +HKDF-SHA2-256 +HMAC-SHA2-256 +KAS-ECC-SSC P-256 +ML-DSA sign and verify PCT +ML-DSA-44 +ML-KEM PCT +ML-KEM PCT +ML-KEM-768 +PBKDF2 +RSA sign and verify PCT +RSASSA-PKCS-v1.5 2048-bit sign and verify +SHA2-256 +SHA2-512 +TLSv1.2-SHA2-256 +TLSv1.3-SHA2-256 +cSHAKE128 +``` + +The list is taken directly from the Go FIPS test suite (file `crypto/internal/fips140test/cast_test.go` of the Go release in use). + +Expected result for every name in the list: + +* The process exits with a non-zero code. +* The output contains `fatal error: FIPS 140-3 self-test failed: : simulated CAST failure`. + +How to obtain and refresh this list: + +* Open `$(go env GOROOT)/src/crypto/internal/fips140test/cast_test.go` and copy the entries from the `allCASTs` slice. + +* The list should be refreshed when the Go version used to build `clickhouse-backup-fips` is upgraded, because new algorithms may add/rename/remove entries. + +#### Inbound TLS — REST API With `openssl s_client` + +Results: + +* TBD + +Check that the REST API listener of `clickhouse-backup-fips server` accepts only FIPS-approved TLS handshakes from clients. + +Start `clickhouse-backup-fips server` with `GODEBUG=fips140=only`, generate a local CA and server certificate, and connect with `openssl s_client`. + +Cipher names below are taken from the official Altinity FIPS configuration documentation (). + +FIPS-approved profiles (handshake MUST succeed): + +* `openssl s_client -tls1_3 -ciphersuites TLS_AES_128_GCM_SHA256` — expected result: handshake succeeds (`CONNECTION ESTABLISHED`). +* `openssl s_client -tls1_3 -ciphersuites TLS_AES_256_GCM_SHA384` — expected result: handshake succeeds. +* `openssl s_client -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256` — expected result: handshake succeeds. + +Non-FIPS profiles (handshake MUST be rejected): + +* `openssl s_client -tls1_3 -ciphersuites TLS_CHACHA20_POLY1305_SHA256` — expected result: handshake is rejected (`alert handshake failure` or equivalent). +* `openssl s_client -tls1_2 -cipher ECDHE-RSA-CHACHA20-POLY1305` — expected result: handshake is rejected. +* `openssl s_client -tls1_2 -cipher RC4-SHA` — expected result: handshake is rejected (legacy/weak cipher). +* `openssl s_client -tls1_2 -cipher DES-CBC3-SHA` — expected result: handshake is rejected (legacy/weak cipher). +* `openssl s_client -tls1` — expected result: handshake is rejected (TLSv1.0 is below the FIPS minimum protocol version). +* `openssl s_client -tls1_1` — expected result: handshake is rejected (TLSv1.1 is below the FIPS minimum protocol version). + +#### Outbound TLS to ClickHouse Server With `openssl s_server` + +Results: + +* TBD + +Check that `clickhouse-backup-fips` refuses to connect to a ClickHouse-style TLS endpoint that offers only non-FIPS ciphers. + +Start a local `openssl s_server` on the secure native ClickHouse port `9440` with a chosen cipher profile, then run `clickhouse-backup-fips` with `GODEBUG=fips140=only` and a config that points at this endpoint. Cipher names below come from the official Altinity FIPS configuration documentation. + +FIPS-approved profiles (handshake MUST be accepted by `clickhouse-backup-fips` policy): + +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256` — expected result: TLS handshake is not rejected by the FIPS policy. +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-AES256-GCM-SHA384` — expected result: TLS handshake is not rejected by the FIPS policy. +* `openssl s_server -tls1_3 -ciphersuites TLS_AES_128_GCM_SHA256` — expected result: TLS handshake is not rejected by the FIPS policy. +* `openssl s_server -tls1_3 -ciphersuites TLS_AES_256_GCM_SHA384` — expected result: TLS handshake is not rejected by the FIPS policy. + +Non-FIPS profiles (handshake MUST be rejected): + +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-CHACHA20-POLY1305` — expected result: `clickhouse-backup-fips` fails with `remote error: tls: handshake failure` and `openssl s_server` reports `no shared cipher`. +* `openssl s_server -tls1_2 -cipher DHE-RSA-AES256-GCM-SHA384` — expected result: handshake is rejected as above. +* `openssl s_server -tls1_3 -ciphersuites TLS_CHACHA20_POLY1305_SHA256` — expected result: handshake is rejected as above. + +#### Outbound TLS to S3 Endpoint With `openssl s_server` + +Results: + +* TBD + +Check that `clickhouse-backup-fips` enforces the same TLS policy on outbound HTTPS to an S3-style remote storage endpoint. + +Point `s3.endpoint` in the `clickhouse-backup` config to a local `openssl s_server` on `127.0.0.1:9443` with a chosen cipher profile, then run `clickhouse-backup-fips list remote` with `GODEBUG=fips140=only`. Cipher names below come from the official Altinity FIPS configuration documentation. + +FIPS-approved profiles (handshake MUST be accepted by `clickhouse-backup-fips` policy): + +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-AES128-GCM-SHA256` — expected result: TLS handshake is not rejected by the FIPS policy (downstream HTTP/auth errors from `openssl s_server -www` are acceptable, since this server is not a real S3 API). +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-AES256-GCM-SHA384` — expected result: same as above. +* `openssl s_server -tls1_3 -ciphersuites TLS_AES_128_GCM_SHA256` — expected result: same as above. +* `openssl s_server -tls1_3 -ciphersuites TLS_AES_256_GCM_SHA384` — expected result: same as above. + +Non-FIPS profiles (handshake MUST be rejected): + +* `openssl s_server -tls1_2 -cipher ECDHE-RSA-CHACHA20-POLY1305` — expected result: `clickhouse-backup-fips` fails with `remote error: tls: handshake failure` / `no shared cipher`. +* `openssl s_server -tls1_2 -cipher DHE-RSA-AES256-GCM-SHA384` — expected result: handshake is rejected as above. +* `openssl s_server -tls1_3 -ciphersuites TLS_CHACHA20_POLY1305_SHA256` — expected result: handshake is rejected as above. + +#### ACVP Tests + +Results: + +* TBD + +Run the ACVP (Automated Cryptographic Validation Protocol) wrapper bundled with `clickhouse-backup`. This is a required part of every FIPS test run. + +Invoke `pkg/acvpwrapper/run.sh` against the FIPS-built binary. + +Expected result: + +* The wrapper runs the algorithm test vectors and exits successfully with no failures across the run. diff --git a/test/testflows/clickhouse_backup/requirements/fips/requirements.md b/test/testflows/clickhouse_backup/requirements/fips/requirements.md new file mode 100644 index 000000000..e69de29bb