feat: context-aware greeting prompt embeds schema for relevant tables#260
Open
cpsievert wants to merge 7 commits into
Open
feat: context-aware greeting prompt embeds schema for relevant tables#260cpsievert wants to merge 7 commits into
cpsievert wants to merge 7 commits into
Conversation
…tation - Rename _GREETING_BASE_TEXT/_GREETING_EXPLORE_ADDENDUM to drop leading underscores (private module convention) - Widen data_sources param from dict to Mapping in build_greeting_prompt and resolve_greeting_tables, fixing Pyright invariance errors - Clarify GREETING_PROMPT backward-compat comment - Rename misleading test to match what it actually verifies
Em-dash and arrow introduced in GREETING_EXPLORE_ADDENDUM and a comment trigger R CMD check WARNING about non-portable characters.
Contributor
|
I really like the overall idea and think the insight to fix this in the greeting generation is smart. There are a couple things about this design that give me pause:
In talking through these ideas with Claude and reviewing the PR, I landed on an alternative API design that I think is worth considering in parallel. I'm working on a PR and will have something up shortly |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
When querychat gained multi-table support (#195), schema context moved out of the system prompt and into a lazy tool (
tool_get_schema): the LLM acquires data context on demand by calling the tool during a conversation. This works well for user queries — but it introduced a regression in greeting generation.The greeting is produced in a tool-free client (no
update_dashboard, noquerytool), so the LLM has no way to calltool_get_schemato learn about the data. Before multi-table, the schema was always in the system prompt, so the greeting was naturally context-aware. After multi-table, it isn't — the LLM is flying blind when writing the opening message.What this PR does
Restores greeting context-awareness by building a schema-aware prompt at greeting time, rather than relying on the system prompt or tool calls.
New
greeting_tablesparameter onQueryChat(R + Python, all frameworks — Shiny, Streamlit, Gradio) controls which tables' schema to embed:None(default): auto — single-table apps embed the full schema (restoring pre-regression behavior); multi-table apps omit schema and instead prompt the LLM to encourage the user to explore what's availableTrue: embed all tablesFalse: no schema (explorer hint only)New
build_greeting_prompt()function (R + Python) encapsulates the construction logic — resolves which tables to include, fetches their schema, and prepends aGREETING_MARKERsentinel. The sentinel replaces a previous exact-string equality check for filtering the greeting turn out of conversation history; that check would have broken as soon as the prompt content changed.generate_greeting()andmod_serverupdated in both packages to usebuild_greeting_prompt().Test coverage
New tests cover
build_greeting_promptdirectly (single-table, multi-table, explicit list,True/False),generate_greeting()end-to-end with a mock client, andresolve_greeting_tableslogic — in both R and Python.