better error message --resume crash when download.state2 file is missing

[minguyen9988]

Problem

When using --resume on a backup directory that exists but has no download.state2 file, clickhouse-backup crashes with:

Error: backup is already exists

Root cause

The download logic checks if the backup already exists locally:

for i := range localBackups {
    if backupName == localBackups[i].BackupName {
        if !b.resume {
            return ErrBackupIsAlreadyExists
        } else {
            // resume path requires download.state2 to exist
        }
    }
}

When --resume is set but download.state2 doesn't exist (e.g., the previous download crashed before creating the state file, or the state file was manually deleted), the code falls through to an error path.

When this happens

1. Previous download was killed before writing any state
2. User manually deleted the state file but kept the partial data
3. Previous download used a different version that didn't create state files
4. Backup directory was created by a failed download that crashed on metadata

Proposed Fix

Make the state file optional for resume. If it doesn't exist, resume from scratch — the conservative validation (see #1376) will detect which parts are complete and which need re-downloading:

if !b.resume {
    return ErrBackupIsAlreadyExists
} else {
    isResumeExists = true
    _, checkDownloadErr := os.Stat(path.Join(b.DefaultDataPath, "backup", backupName, "download.state2"))
    if errors.Is(checkDownloadErr, os.ErrNotExist) {
        // No state file from a previous download — this is OK.
        // Resume will re-validate and re-download any incomplete parts.
        // The state file is an optimization, not a requirement.
        log.Warn().Msgf("%s already exists but no download.state2 found, will resume download from scratch", backupName)
    } else {
        log.Warn().Msgf("%s already exists will try to resume download", backupName)
    }
}

Behavior after fix

State file exists?	Behavior
Yes	Normal resume — bbolt state tracks completed parts
No	Resume from scratch — validates each part locally, re-downloads incomplete ones
No + manifest	Fast resume — manifest enables local-only validation (zero remote calls)

The key insight is that the state file is an optimization for skipping validation, not a requirement for correctness. The conservative resume validation (checking actual file existence and sizes) is the real safety mechanism.

[minguyen9988]

Reproduction Steps (against master branch, commit 1a5b06b)

Environment

• Any S3-compatible storage (MinIO, AWS S3, GCS via S3)
• Any ClickHouse version
• clickhouse-backup built from master branch

Steps

# 1. Create a backup and upload it
clickhouse-backup create test_resume_crash
clickhouse-backup upload test_resume_crash

# 2. Download it normally (creates the local directory)
clickhouse-backup download test_resume_crash

# 3. Delete ONLY the state file, keep the data
rm /var/lib/clickhouse/backup/test_resume_crash/download.state2

# 4. Try to resume — CRASHES
clickhouse-backup download test_resume_crash --resume
# ERROR: "backup is already exists"

Root cause in code

pkg/backup/download.go lines 88-96 on master:

if !b.resume {
    return ErrBackupIsAlreadyExists
} else {
    _, checkDownloadErr := os.Stat(path.Join(b.DefaultDataPath, "backup", backupName, "download.state2"))
    if errors.Is(checkDownloadErr, os.ErrNotExist) {
        return ErrBackupIsAlreadyExists  // <-- BUG: treats missing state file as fatal
    }
    isResumeExists = true
    log.Warn().Msgf("%s already exists will try to resume download", backupName)
}

When --resume is set but download.state2 doesn't exist, the code returns ErrBackupIsAlreadyExists — the exact same error as when --resume is not set at all. The state file should be optional; its absence should trigger a "resume from scratch" mode, not a crash.

Expected behavior

WARN: test_resume_crash already exists but no download.state2 found, will resume download from scratch

The download should proceed, validating each part's local completeness and re-downloading only incomplete parts.

Additional scenario

This also happens when:

• The previous download was killed before writing any state (OOM, SIGKILL)
• The backup directory was created by mkdir or a failed first attempt
• A different clickhouse-backup version was used previously (different state format)

[Slach]

use_resumable_state: true is the default, and adjustResumeFlag() in pkg/backup/backuper.go:505 forces b.resume=true even when --resume is not passed on CLI. So in practice, if at least one file was downloaded, download.state2 is always created via AppendToState.

The repro rm download.state2 is a manual intervention, not a normal failure mode. Realistic cases when state2 is missing but local data exists:

1. use_resumable_state: false set explicitly in config
2. file removed manually
3. process killed in a narrow window before NewStateFile writes
4. downgrade from a version with different state format

None of these should silently trigger "resume from scratch" — without state2 we don't know which files are complete and which were cut mid-stream, so re-downloading on top of existing files risks silent corruption.

The right user workflow here is explicit:

clickhouse-backup delete local <name>
clickhouse-backup download <name>   # or --resume

What I'd accept as a fix: improve the error message to point at this. Something like:

local backup '<name>' exists but download.state2 is missing.
run `clickhouse-backup delete local <name>` and retry,
or investigate why the resumable state was lost.

Returning ErrBackupIsAlreadyExists with no hint is the only real UX bug here. Auto-fallback to scratch download is not safe and I'm not going to merge that.

[minguyen9988]

ok, got it, I just think the penalty is too harsh to lose a state file it essentially means the whole backup is corrupted right. what if a backup is cancelled midway, would it also trigger a missing state2 file?

[Slach]

Just to close the loop on the kill -9 / mid-cancel concern: download.state2 is a bbolt database and every AppendToState is a full ACID transaction (fsync + atomic meta-page switch with CRC), so SIGKILL/OOM at any point leaves the file consistent — bbolt rolls back the incomplete transaction on next Open. The only realistic windows where state2 can be missing are: the tiny gap between MkdirAll of the backup dir and bolt.Open creating the file, FS-level damage (power loss without write barriers, ENOSPC during file grow), or use_resumable_state: false. None of these are safe to auto-recover from by re-downloading on top of existing files, so the fix here will stay a clearer error message pointing at clickhouse-backup delete local <name>.