[None][fix] serve: surface requests ending inside an unclosed tool call - #17439
Draft
peter941221 wants to merge 1 commit into
Draft
[None][fix] serve: surface requests ending inside an unclosed tool call#17439peter941221 wants to merge 1 commit into
peter941221 wants to merge 1 commit into
Conversation
An active tool call that is never closed at end-of-request is silently returned as an empty or truncated tool_calls. Speculative decoding can truncate a control-token prelude at an interior token, leaving the opened control block unclosed (see NVIDIA#17437). Add an end-of-request guard on HarmonyStreamState.cleanup_stream_state that reports the never-closed tool calls instead of silently dropping them. Signed-off-by: peter941221 <peter941221@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In one-model speculative decoding, the acceptance path commits only the
accepted prefix of a draft. When a draft proposes a rare Harmony control
prelude (a channel open, an interior tool token) and the target rejects an
interior control token, the accepted prefix is left structurally unbalanced:
the opened control block never closes. The adapter then ends the request with
finish_reason=stop and an empty tool_calls, so an intended tool call is
silently dropped (tracked in #17437 and #16377).
This change adds an end-of-request guardrail in HarmonyStreamState. It detects
tool calls that were started but never closed when cleanup_stream_state runs,
and raises a warning that names the affected tool calls. The silent drop
becomes observable and diagnosable at the server boundary, without changing
the emitted tokens, the sampling path, or the response shape.
Scope note
This covers the case where the adapter reached a tool call but never saw it
close (a spec-decode accepted prefix truncated at an interior token). If the
prefix was truncated before the adapter ever reached a tool recipient, the
adapter sees only an ordinary commentary end and this guard plus an empty
tool_calls remains; that sampling-level truncation is tracked separately.
Root cause
the per-request HarmonyStreamState without checking whether any started tool
call was left open, so an unclosed call was returned as an empty/truncated
tool_calls with no signal.
inactive when the stream leaves the tool recipient. A call still active at
end-of-request is precisely the "never closed" case.
Change
calls still active (never closed).
before the stream state is freed.
Testing
call reported, a closed one not reported, a cleanup warning on an
incomplete call, and no warning when the call is complete.
prefix) is reproduced on the real speculative-decoding kernel path.