diff --git a/.changeset/message-queue.md b/.changeset/message-queue.md new file mode 100644 index 000000000..6f91ebae0 --- /dev/null +++ b/.changeset/message-queue.md @@ -0,0 +1,14 @@ +--- +'@tanstack/ai-client': minor +'@tanstack/ai-react': minor +'@tanstack/ai-solid': minor +'@tanstack/ai-vue': minor +'@tanstack/ai-svelte': minor +'@tanstack/ai-preact': minor +--- + +Messages sent while a stream is already in flight are now queued by default and automatically sent once the in-flight stream settles, instead of being silently dropped. **This is a behavior change.** + +The behavior is configurable via a new `queue` option, which accepts `whenBusy: 'queue' | 'drop' | 'interrupt'`, `drain: 'fifo' | 'batch'`, `maxSize`, and `onOverflow`, or a custom strategy function for full control. + +Queued messages are exposed on the hook as `queue` and can be cancelled before they send via `cancelQueued(id)`. `sendMessage` also accepts a per-call `{ whenBusy }` override. diff --git a/docs/chat/streaming.md b/docs/chat/streaming.md index db32def3c..a4159ac50 100644 --- a/docs/chat/streaming.md +++ b/docs/chat/streaming.md @@ -203,6 +203,54 @@ export async function POST(request: Request) { } ``` +## Queueing Messages + +By default, calling `sendMessage` while a stream is already in flight **queues** the message instead of dropping it — it sends automatically once the current run settles. Configure this with the `queue` option, which accepts a `QueueConfig` object, a plain shorthand string, or a strategy function: + +```tsx group=queueing-messages +import { useChat, fetchServerSentEvents } from "@tanstack/ai-react"; + +const { messages, queue, sendMessage, cancelQueued, isLoading } = useChat({ + connection: fetchServerSentEvents("/api/chat"), + queue: { whenBusy: "queue", drain: "fifo", maxSize: 5 }, +}); +``` + +- **`whenBusy`** — what happens to a send that arrives mid-stream: + - `"queue"` (default) — hold the message; it sends once the stream settles. + - `"drop"` — ignore the send. + - `"interrupt"` — abort the current stream (like calling `stop()`) and send the new message immediately. +- **`drain`** — how queued items leave the queue: `"fifo"` (default) sends them one at a time in order; `"batch"` merges everything currently queued into a single send once the stream settles (string contents joined with `\n`, multimodal content concatenated in order). +- **`maxSize`** — caps how many messages can be queued. +- **`onOverflow`** — `"reject"` (default) ignores a send once `maxSize` is reached; `"drop-oldest"` evicts the oldest queued item to make room. + +You can also pass a plain `WhenBusy` string (e.g. `queue: "interrupt"`) as shorthand for `{ whenBusy: "interrupt" }` — this applies for any `WhenBusy` value (`"queue"`, `"drop"`, or `"interrupt"`), not just `"interrupt"` — or a `QueueStrategy` function for full control over the per-send decision (the drain order stays FIFO for the function form). + +`useChat` exposes the pending queue as `queue` so you can render it distinctly from `messages`, along with `cancelQueued(id)` to cancel an item before it sends: + +```tsx group=queueing-messages +function PendingQueue() { + return ( + <> + {queue.map((q) => ( +
+ {strategy.blurb} +
+No messages yet.
+ )} + {visible.map((message) => ( +⏳ streaming…
+ )} ++ Queue ({queue.length}) +
+ {queue.length === 0 ? ( +empty
+ ) : ( +
+ Send a message, then send another while it is still streaming{' '}
+ — each panel reacts differently based on its queue{' '}
+ config.
+
+