feat: multi-table support and data dictionary

[cpsievert]

For users

querychat has been single-table: one data frame, one schema, one conversation. If you had orders and customers and products, your options were to pre-join everything into one wide frame (losing the relational structure the LLM could reason about) or to spin up separate querychat sessions (losing the ability to ask cross-table questions). This PR removes that limitation.

You can now register multiple tables and the LLM will reason across all of them — joins, cross-table filters, aggregations:

qc = QueryChat(orders_df, "orders")
qc.add_table(customers_df, "customers")

If your data lives in a SQLAlchemy engine (Python) or DBI connection (R), add_tables() bulk-registers all tables — or a named subset — in a single call, and builds the system prompt exactly once rather than once per table:

qc = QueryChat()
qc.add_tables(engine)                          # all tables
qc.add_tables(engine, ["orders", "customers"]) # specific subset

The LLM's SQL queries can reference any registered table. To consume per-table filter state in a Shiny app, call qc.server() and use table() on the returned values — each table gets its own reactive sql/df/title:

# In your Shiny server function:
qc_vals = qc.server()
qc_vals.table("orders").df()
qc_vals.table("customers").sql()

As a side note: qc_vals.current_table() (R: qc_vals\$current_table()) is also available — a reactive that returns the name of the most recently LLM-queried table, or None/NULL before any query. Useful when you want an output to follow whatever table the LLM just acted on without hard-coding a name.

Under the hood, a new QueryExecutor layer dispatches queries to the right backend. DataFrame sources (pandas, polars, pyarrow) get a shared DuckDB connection; Polars LazyFrames get a shared SQLContext; SQLAlchemy/Ibis tables use the existing shared backend directly. All tables in a session must be the same source type.

But multi-table introduces a scaling problem. The old design embedded full column statistics in the system prompt at startup — every conversation paid the cost of computing stats for every column of every table, whether the LLM needed them or not. With multiple tables, that cost multiplies. So schema fetching is now on-demand: the system prompt contains only a lightweight table overview, and the LLM calls the new querychat_get_schema tool to get column details for specific tables when it actually needs them.

And when you have several tables, column names get ambiguous. What does status mean in the orders table vs. the customers table? A new DataDict type — integrating with the https://data-dict.tidyverse.org/ spec — lets you annotate tables and columns with plain-English descriptions, loaded from a YAML file:

QueryChat(data_dict="data_dict.yaml")

The schema tool uses these annotations directly, skipping the live stats query for any column that already has metadata. A fully-documented table incurs zero stats overhead. DataDict also supports relationships (JOIN hints for multi-table) and a glossary (domain term definitions), both surfaced in the system prompt so the LLM understands your domain language.

These three pieces — multi-table, on-demand schema, and data dictionary — aren't independent features. They're designed as layers: multi-table creates the need for smarter schema handling, on-demand fetching solves the cost problem, and DataDict solves the precision problem.

Breaking changes

Before	After
qc.data_source	qc.table("name").data_source
qc.data_source = new_df	qc.add_table(new_df, "name", replace=True)
qc.server(data_source=df) (Shiny)	qc.add_table(df, "name") before server()
qc.df() / qc.sql() / qc.title() (multi-table)	qc_vals.table("name").df() etc. via server return

Bookmark keys changed: Python uses flat per-table keys (querychat_sql_{name}, querychat_title_{name}); R uses a nested querychat_tables dict. Old single-table flat keys (querychat_sql) are no longer written.

data_description is deprecated in favor of data_dict.

---

For reviewers

The code mirrors the same layered story. Each layer depends only on the ones below it, so reviewing bottom-up avoids forward references:

Layer 1 — The data contract (_datasource.py). DataSource gained two methods: get_column_metas() (cheap — column names and types via LIMIT 0) and populate_column_stats() (expensive — min/max/categories). This split is the foundation everything else builds on: it's what makes on-demand fetching possible and what lets DataDict selectively skip stats queries. ColumnMeta also gained a description field for DataDict annotations.

Layer 2 — Multi-table dispatch (_query_executor.py, new). A QueryExecutor sits between the tools and the data sources. It exposes the same get_column_metas(table) / populate_column_stats(table) / execute_query() surface but routes to the right backend. Three implementations: DataSourceExecutor for single-table or shared-backend multi-table (SQLAlchemy, Ibis); DuckDBExecutor for multi-table DataFrames (registers all tables in one in-memory DuckDB); and PolarsSQLExecutor for multi-table Polars LazyFrames (shared SQLContext). check_source_compatibility() enforces that all tables in a session share the same source type and backend.

Layer 3 — Column metadata (_data_dict.py, new). A Pydantic model hierarchy: DataDict → TableSpec → ColumnSpec. The key method, DataDict.get_table_schema(), merges static annotations onto live ColumnMeta objects and only sends undocumented columns to populate_column_stats(). This is where the "skip stats for documented columns" optimization lives.

Layer 4 — The schema tool (tools.py, prompts/tool-get-schema.md). querychat_get_schema dispatches to either DataDict.get_table_schema() or executor.get_schema() depending on whether a dict is present. It returns a GetSchemaResult subclass that renders a compact status line in Shiny chat rather than dumping the raw schema.

Layer 5 — Prompt rewiring (_system_prompt.py, prompts/prompt.md). The old {{schema}} Mustache variable is gone. Replaced by {{tables_overview}} (lightweight table list), {{relationships}}, {{glossary}}, and a {{multi_table}} flag that drives conditional instructions requiring table= parameters in tool calls.

Layer 6 — Orchestration (_querychat_base.py). The base class now holds dict[str, DataSource]. add_table() uses a staged-rebuild pattern: it builds the new system prompt and executor against a copy of the sources dict, committing only if both succeed, to avoid partial state on failure. add_tables() (Python: SQLAlchemy engine; R: DBI connection) bulk-stages all tables and calls build_system_prompt() exactly once, avoiding the N-1 spurious rebuilds that calling add_table() in a loop would cause. table(name) returns a config-only TableAccessor — data_source and table_name work, but df()/sql()/title() raise with guidance to use the server return value instead.

Layer 7 — Shiny reactive state (_shiny_module.py). ServerValues has per-table state in tables: dict[str, TableState], each with its own reactive sql, title, and df. It also gained a table(name) method that returns a TableAccessor backed by the per-session TableState — this is the correct way to consume per-table reactive results. Also added: current_table(), which tracks the most recently LLM-touched table name across update_dashboard, reset_dashboard, chat_update, and bookmark restore. Single-table apps still get top-level .df, .sql, .title on ServerValues for backward compatibility; multi-table replaces them with _MultiTableBlockedReactive sentinels that raise with migration guidance. Bookmark keys are per-table (Python: flat querychat_sql_{name}; R: nested querychat_tables).

Layer 8 — Public accessors and framework adapters (_table_accessor.py new, _dash.py, _gradio.py, _streamlit.py). TableAccessor is a thin proxy that can be constructed in two modes: config-only (from qc.table(), raises on reactive methods) or state-backed (from qc_vals.table(), fully reactive). StateDictTableAccessor is the Dash/Gradio variant that always raises on reactive methods with instructions to use the state-dict API instead. Framework adapters gained table= parameters on df(), sql(), title().

Test plan

• make py-check passes (format, types, tests)
• Multi-table Shiny app: LLM correctly joins/filters across tables
• Single-table apps still work with no API changes
• DataDict: YAML loads, enriched descriptions reach the LLM, documented columns skip stats queries
• Bookmark round-trip: new per-table keys save/restore correctly; old flat keys still load

[cpsievert]

Thanks for the detailed review. I've addressed the near-term items in sections 2 and 3; thoughts on section 1 below.

Sections 3A–3D

All four simplification items landed:

• 905a8db — replaces StateDictAccessorMixin with StateDictQueryChat, eliminating all 15 type: ignore[attr-defined] comments
• 53a7788 — softens the flat-accessor error to FutureWarning with primary-table fallback, with session-scoped deduplication so Shiny reactives don't spam
• f36db45 — extracts _validate_missing_columns to the QueryExecutor base, removing the ~19-line duplication
• 47d8fae — warns when add_table rebuilds the system prompt after a client already has chat history, covering both the _client_spec and _client_console paths

Section 2 — Bulk table registration

99e011d adds add_tables() for bulk registration in both R and Python, which handles the many-add_table-calls problem. Auto-discovery on construction is a natural next step I'd consider separately.

Section 1 — Coalescing source types into shared DuckDB

I've filed #256 to track loosening the class-identity check so PinSource and DataFrameSource can share a DuckDBExecutor — both already materialize an eager frame, so this is largely a routing change.

For steps 2 and 3 I'm more cautious:

Snapshot mode is hard to recommend as a general solution. Materializing a full remote table into local DuckDB memory at startup works for small tables and silently breaks for production ones, with no reliable way to gate on table size upfront.

Live attach is more interesting but comes with real costs: DuckDB's SQL dialect diverges from Postgres/MySQL enough to introduce compatibility risk (the LLM is already told what flavor to use, and "DuckDB mediating Postgres" isn't the same as talking to Postgres directly); ATTACH scanners only cover Postgres, MySQL, and SQLite, so Snowflake/Databricks users still fall back to snapshots; per-dialect connection-string translation from SQLAlchemy URLs is non-trivial glue; and relaxing enable_external_access is a meaningful security posture change that warrants its own rollout.

I'd keep the register_into contract extensible so steps 2/3 remain additive, but I'd want concrete user demand before pursuing remote-source coalescing.

[cpsievert]

@gadenbuie thanks for the review, ok to merge?