feat(cli): auto-compress chat history on context window overflow#28488
feat(cli): auto-compress chat history on context window overflow#28488kunalrawat425 wants to merge 1 commit into
Conversation
Add a new `model.autoCompressOnOverflow` boolean setting (default: false) that, when enabled, automatically runs context compression when a prompt is about to exceed the context window limit — instead of stopping with a warning and requiring the user to manually run /compress. When the context-window-will-overflow event fires and the setting is on: - Cancel the current submit (existing behavior) - Show a clear info message that auto-compression is starting - Await geminiClient.tryCompressChat() in the background - Display the compression result (token savings) when done - Prompt the user to re-send their message When the setting is off the existing warning-only behavior is unchanged. The new setting appears in the settings dialog under the Model category and does not require a restart.
|
📊 PR Size: size/M
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces an automated mechanism to handle context window overflows in the CLI. By adding a configurable option, the system can now proactively compress chat history instead of simply halting with an error. This improves the continuity of long-running chat sessions by managing token usage more gracefully. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an autoCompressOnOverflow configuration setting to automatically compress chat history when the context window is close to overflowing. Feedback highlights a critical race condition and usability issue where the background compression task cannot be cancelled and allows user input in the meantime; using an AbortController and setting isResponding is recommended to handle this safely. Additionally, the setting's description should be updated to accurately reflect that users must manually re-send their prompt after compression, rather than it being retried automatically.
| if ( | ||
| isMoreThan25PercentUsed && | ||
| settings.merged.model.autoCompressOnOverflow | ||
| ) { | ||
| addItem({ | ||
| type: 'info', | ||
| text: `Context window limit reached (${remainingTokenCount.toLocaleString()} tokens left). Auto-compressing history — your prompt will be ready to re-send after compression completes.`, | ||
| }); | ||
|
|
||
| void (async () => { | ||
| try { | ||
| const promptId = `auto-compress-overflow-${Date.now()}`; | ||
| const compressed = await geminiClient.tryCompressChat( | ||
| promptId, | ||
| true, | ||
| ); | ||
| if (compressed) { | ||
| addItem({ | ||
| type: MessageType.COMPRESSION, | ||
| compression: { | ||
| isPending: false, | ||
| originalTokenCount: compressed.originalTokenCount, | ||
| newTokenCount: compressed.newTokenCount, | ||
| compressionStatus: compressed.compressionStatus, | ||
| }, | ||
| } as HistoryItemCompression); | ||
| addItem({ | ||
| type: 'info', | ||
| text: 'Auto-compression complete. Please re-send your prompt.', | ||
| }); | ||
| } else { | ||
| addItem({ | ||
| type: 'info', | ||
| text: 'Auto-compression failed. Use `/compress` manually or reduce your message size.', | ||
| }); | ||
| } | ||
| } catch { | ||
| addItem({ | ||
| type: 'info', | ||
| text: 'Auto-compression encountered an error. Use `/compress` manually.', | ||
| }); | ||
| } | ||
| })(); | ||
| return; | ||
| } |
There was a problem hiding this comment.
Race Condition & Un-cancellable Background Task in Auto-Compression
There is a significant race condition and usability issue with the current background auto-compression implementation:
- Un-cancellable Task: When auto-compression starts,
isRespondingis not set totrue. Because of this,streamingStateremainsIdle, andcancelOngoingRequest(triggered by Escape or Ctrl+C) will return early and do nothing. The user cannot cancel the background compression. - Race Condition: Since the input remains active, the user can type and submit a new prompt while the background compression is still running. When the compression finishes, it will call
addItemand inject messages ("Auto-compression complete...") right in the middle of the new streaming prompt's UI/history.
Solution
To resolve this, we should:
- Set
isRespondingtotrueduring compression to disable input and indicate that the system is busy. - Create a new
AbortControllerfor the compression task and assign it toabortControllerRef.currentso that user cancellation (Escape/Ctrl+C) can abort the compression. - Pass the abort signal to
tryCompressChatand checksignal.abortedbefore updating the UI state.
if (
isMoreThan25PercentUsed &&
settings.merged.model.autoCompressOnOverflow
) {
setIsResponding(true);
const compressController = new AbortController();
abortControllerRef.current = compressController;
addItem({
type: 'info',
text: 'Context window limit reached (' + remainingTokenCount.toLocaleString() + ' tokens left). Auto-compressing history — your prompt will be ready to re-send after compression completes.',
});
void (async () => {
try {
const promptId = 'auto-compress-overflow-' + Date.now();
const compressed = await geminiClient.tryCompressChat(
promptId,
true,
compressController.signal,
);
if (compressController.signal.aborted) {
return;
}
if (compressed) {
addItem({
type: MessageType.COMPRESSION,
compression: {
isPending: false,
originalTokenCount: compressed.originalTokenCount,
newTokenCount: compressed.newTokenCount,
compressionStatus: compressed.compressionStatus,
},
} as HistoryItemCompression);
addItem({
type: 'info',
text: 'Auto-compression complete. Please re-send your prompt.',
});
} else {
addItem({
type: 'info',
text: 'Auto-compression failed. Use `/compress` manually or reduce your message size.',
});
}
} catch {
if (compressController.signal.aborted) {
return;
}
addItem({
type: 'info',
text: 'Auto-compression encountered an error. Use `/compress` manually.',
});
} finally {
if (!compressController.signal.aborted) {
setIsResponding(false);
}
}
})();
return;
}References
- Asynchronous operations that can be cancelled by the user should accept and propagate an AbortSignal to ensure cancellability and prevent dangling processes or network requests.
| description: | ||
| 'When the context window is about to overflow, automatically compress the chat history and retry the prompt instead of stopping with an error.', |
There was a problem hiding this comment.
Discrepancy in Setting Description
The setting description states that it will "automatically compress the chat history and retry the prompt", but the actual implementation in useGeminiStream.ts cancels the current submit and prompts the user to manually re-send their prompt ("Auto-compression complete. Please re-send your prompt.").
Updating the description to accurately reflect the manual re-send behavior prevents user confusion. Please also update the corresponding descriptions in schemas/settings.schema.json to match this change.
| description: | |
| 'When the context window is about to overflow, automatically compress the chat history and retry the prompt instead of stopping with an error.', | |
| description: | |
| 'When the context window is about to overflow, automatically compress the chat history and prompt to re-send instead of stopping with an error.', |
gemini-cli
Bot
added
the
status/need-issue
1 participant
Add a new
model.autoCompressOnOverflowsetting that automatically compresses chat history when the context window is about to overflow, instead of stopping with a warning.