Skip to content

[None][fix] serve: surface requests ending inside an unclosed tool call - #17439

Draft
peter941221 wants to merge 1 commit into
NVIDIA:mainfrom
peter941221:fix/17437-incomplete-tool-call-guard
Draft

[None][fix] serve: surface requests ending inside an unclosed tool call#17439
peter941221 wants to merge 1 commit into
NVIDIA:mainfrom
peter941221:fix/17437-incomplete-tool-call-guard

Conversation

@peter941221

Copy link
Copy Markdown
Contributor

Description

In one-model speculative decoding, the acceptance path commits only the
accepted prefix of a draft. When a draft proposes a rare Harmony control
prelude (a channel open, an interior tool token) and the target rejects an
interior control token, the accepted prefix is left structurally unbalanced:
the opened control block never closes. The adapter then ends the request with
finish_reason=stop and an empty tool_calls, so an intended tool call is
silently dropped (tracked in #17437 and #16377).

This change adds an end-of-request guardrail in HarmonyStreamState. It detects
tool calls that were started but never closed when cleanup_stream_state runs,
and raises a warning that names the affected tool calls. The silent drop
becomes observable and diagnosable at the server boundary, without changing
the emitted tokens, the sampling path, or the response shape.

Scope note

This covers the case where the adapter reached a tool call but never saw it
close (a spec-decode accepted prefix truncated at an interior token). If the
prefix was truncated before the adapter ever reached a tool recipient, the
adapter sees only an ordinary commentary end and this guard plus an empty
tool_calls remains; that sampling-level truncation is tracked separately.

Root cause

  • tensorrt_llm/serve/harmony_adapter.py: cleanup_stream_state_request removed
    the per-request HarmonyStreamState without checking whether any started tool
    call was left open, so an unclosed call was returned as an empty/truncated
    tool_calls with no signal.
  • Active tool calls are already tracked (tool_info["active"]) and are marked
    inactive when the stream leaves the tool recipient. A call still active at
    end-of-request is precisely the "never closed" case.

Change

  • Add HarmonyStreamState.active_tool_call_names(): returns the names of tool
    calls still active (never closed).
  • In cleanup_stream_state, log a warning naming any never-closed tool calls
    before the stream state is freed.

Testing

  • unit: new tests in test_harmony_parsing.py cover an open (active) tool
    call reported, a closed one not reported, a cleanup warning on an
    incomplete call, and no warning when the call is complete.
  • The acceptance mechanism (interior rejection leaves an unclosed control
    prefix) is reproduced on the real speculative-decoding kernel path.
  • ruff / format / compile checks on the touched files are clean.

An active tool call that is never closed at end-of-request is silently
returned as an empty or truncated tool_calls. Speculative decoding can
truncate a control-token prelude at an interior token, leaving the opened
control block unclosed (see NVIDIA#17437). Add an end-of-request guard on
HarmonyStreamState.cleanup_stream_state that reports the never-closed tool
calls instead of silently dropping them.

Signed-off-by: peter941221 <peter941221@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant