Hybrid engine — `currentDatabase()` in engine arguments breaks server restart

[alsugiliazova]

Type of problem

Code bug — currentDatabase() is not resolved at CREATE TABLE time in the Hybrid engine, causing every server restart to fail with UNKNOWN_TABLE when the table was created in a non-default database.

Because of it now stress test fails on every pr (exmaple https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=1918&sha=c726c3f3469a7fa57d214f6959cbbf286b4baeef&name_0=PR&name_1=Stress+test+%28amd_release%29). Same failure on PRs 1895, 1912, 1923 across amd_debug, amd_release, amd_tsan, amd_ubsan, amd_msan, arm_asan, arm_asan (s3) on 2026-06-22 → 2026-06-30.

---

Summary

When a Hybrid table is created in a non-default database using currentDatabase() inside remote() arguments, the function call is stored verbatim in the metadata file. On the next server start, the ATTACH TABLE statement is executed without any session database context, so currentDatabase() evaluates to default instead of the original database. The referenced table is not found and the server cannot start.

This is a recurring Stress test / Cannot start clickhouse-server failure across all antalya-26.3 stress flavors (amd_debug, amd_release, amd_tsan, amd_ubsan, amd_msan, arm_asan), seen continuously on every PR since 2026-06-22.

---

Error

Code: 60. DB::Exception: Table default.local_cold does not exist.
  Maybe you meant test_3.local_cold?
Cannot attach table `test_3`.`t_settings_only` from metadata file
  store/119/119a0976-0350-4f2d-aac5-d499b081fca5/t_settings_only.sql
from query:
  ATTACH TABLE test_3.t_settings_only UUID '66e37911-f721-43e8-b657-6e352221f3ef'
  (`ts` DateTime, `value` UInt64)
  ENGINE = Hybrid(
    remote('localhost:9000', 'test_3', 'local_hot'),
    ts > hybridParam('hybrid_watermark_hot', 'DateTime'),
    remote('localhost:9000', currentDatabase(), 'local_cold'),
    ts <= hybridParam('hybrid_watermark_hot', 'DateTime')
  )
  SETTINGS hybrid_watermark_hot = '2025-09-01'

---

How to reproduce

-- Run in database test_3 (any non-default database)
CREATE DATABASE IF NOT EXISTS test_3;
CREATE TABLE test_3.local_hot  (ts DateTime, value UInt64) ENGINE = MergeTree ORDER BY ts;
CREATE TABLE test_3.local_cold (ts DateTime, value UInt64) ENGINE = MergeTree ORDER BY ts;

CREATE TABLE test_3.t_settings_only (ts DateTime, value UInt64)
ENGINE = Hybrid(
    remote('localhost:9000', 'test_3',         'local_hot'),
    ts > hybridParam('hybrid_watermark_hot', 'DateTime'),
    remote('localhost:9000', currentDatabase(), 'local_cold'),
    ts <= hybridParam('hybrid_watermark_hot', 'DateTime')
)
SETTINGS hybrid_watermark_hot = '2025-09-01';

-- Restart the server. It will fail to start with Code: 60.

---

Root cause

At CREATE TABLE time, currentDatabase() returns test_3 correctly. However, the Hybrid engine stores the engine arguments as-is in the .sql metadata file without resolving the function call to its current value. The stored file therefore contains the literal token currentDatabase().

When the server restarts, it replays all ATTACH TABLE statements from metadata files. At that point there is no active session and no database context, so currentDatabase() returns default. The remote() call resolves to default.local_cold, which does not exist.

The Distributed engine had the identical bug and it was fixed in upstream ClickHouse PR #14211 (v20.10, 2020) by applying AddDefaultDatabaseVisitor to the engine AST before persisting metadata. This substitutes currentDatabase() with the resolved string literal at CREATE time. The Hybrid engine — introduced later and built on StorageDistributed — was never given the same treatment.

The pattern is also actively promoted in the hybridParam documentation and PR #1659 examples:

-- From PR #1659 docs — this pattern breaks on restart
ENGINE = Hybrid(
    cluster('cluster1', currentDatabase(), 'hot'), ...
    cluster('cluster2', currentDatabase(), 'cold'), ...
)

---

Fix

In src/Storages/StorageDistributed.cpp, inside registerStorageHybrid (the Hybrid-specific CREATE path), apply the same AddDefaultDatabaseVisitor pass that StorageDistributed::create uses for Distributed tables. This resolves currentDatabase() in all segment table-function arguments and writes the resolved name to disk.

// Pseudocode — mirrors the existing Distributed fix
AddDefaultDatabaseVisitor visitor(local_context, local_context->getCurrentDatabase());
visitor.visit(args);  // replaces currentDatabase() with the actual db name in-place
// then proceed with metadata write

After this fix, the stored metadata for the table above would read:

remote('localhost:9000', 'test_3', 'local_cold')

instead of:

remote('localhost:9000', currentDatabase(), 'local_cold')

---

Impact

Dimension	Detail
Affected builds	antalya-26.3 (all flavors)
First seen	2026-06-22
Scope	Any Hybrid table created in a non-default database with currentDatabase() in segment arguments
Symptom	Server cannot start after restart — UNKNOWN_TABLE on ATTACH
Workaround	Use explicit database names in remote() / cluster() arguments instead of currentDatabase()
PR that introduced the pattern	Altinity/ClickHouse #1659 — hybridParam / ALTER TABLE MODIFY SETTING for watermarks

---

Workaround (until fix lands)

Replace currentDatabase() with the explicit database name in all Hybrid engine segment arguments:

-- Instead of:
remote('localhost:9000', currentDatabase(), 'local_cold')

-- Write:
remote('localhost:9000', 'my_database', 'local_cold')

---

Identified: 2026-07-01

[mkmkme]

For me, the server does not fail to start, but does show the exception on start. Also I see the exception when connecting via the client. Thanks for the report, I'll fix it!