Skip to content

optimize title generation speed#1325

Closed
Moshiii wants to merge 1 commit intovercel:main-2026-03-20from
Moshiii:title-generation-optimize
Closed

optimize title generation speed#1325
Moshiii wants to merge 1 commit intovercel:main-2026-03-20from
Moshiii:title-generation-optimize

Conversation

@Moshiii
Copy link
Copy Markdown

@Moshiii Moshiii commented Nov 22, 2025

I notice that the title-generation is block the response flow and become the bottle neck of response speed:
Screenshot 2025-11-23 at 3 40 46 AM

Problem
The chatbot API endpoint was experiencing severe performance issues with response times exceeding 84 seconds for new chat creation. The root cause was blocking title generation - the system waited for an LLM API call to generate a chat title before returning the stream response.

Solution
Made title generation non-blocking using a two-phase approach:
Generate a temporary title immediately from the first 60 characters of the user's message
Generate the LLM-based title asynchronously in the background and update when ready
This allows the chat stream to start immediately while the polished title is generated asynchronously.

Changes
lib/db/queries.ts
Added updateChatTitleById function to update chat titles after creation
app/(chat)/api/chat/route.ts
Replaced blocking await generateTitleFromUserMessage() with immediate temporary title generation
Added asynchronous real title generation that updates the chat when ready
Added imports for updateChatTitleById and getTextFromMessage

Performance Impact
Before: 19+ seconds (blocking on LLM title generation)
After: <1 second (immediate response with temporary title, real title updates in background)

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Nov 22, 2025

@Moshiii is attempting to deploy a commit to the Templates Test vtest314 Team on Vercel.

A member of the Team first needs to authorize it.

@Moshiii
Copy link
Copy Markdown
Author

Moshiii commented Nov 22, 2025

@shaper @rauchg @tomocchino @marcusschiesser Please review this response optimization-related PR

@Moshiii
Copy link
Copy Markdown
Author

Moshiii commented Nov 23, 2025

@MrAllgoodWilson what should I do to get it approved? Please advice.

@MrAllgoodWilson
Copy link
Copy Markdown

I approve

@dancer dancer deleted the branch vercel:main-2026-03-20 March 20, 2026 11:23
@dancer dancer closed this Mar 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants