Skip to content

feat(tools): honor function.strict and tool_choice='required' via forced decoding - #4844

Open
lvhan028 wants to merge 5 commits into
InternLM:mainfrom
lvhan028:feat/tool-strict-required
Open

feat(tools): honor function.strict and tool_choice='required' via forced decoding#4844
lvhan028 wants to merge 5 commits into
InternLM:mainfrom
lvhan028:feat/tool-strict-required

Conversation

@lvhan028

@lvhan028 lvhan028 commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

What

OpenAI-compatible tool-calling constraints, implemented as structural_tag forced decoding (built on the guided.py infrastructure):

  • tools[].function.strict = true — under tool_choice='auto', constrains the model to emit valid tool-call JSON (matching {"name": <const>, "arguments": <params>}) between the configured tool parser's tags. Optional (at_least_one=False): the model may still produce free text.
  • tool_choice='required' (and AllowedToolChoice with mode='required') — forces at least one tool call (at_least_one=True), rejecting free-text-only output.
  • The tool-call JSON schema is built manually in response_parser.py ({name: const, arguments: parameters}) — no xgrammar import in the parser; guided.py owns all compilation. xgr.openai_tool_call_schema exists but is intentionally unused.
  • Tags come from the active tool parser's get_tool_open_tag() / get_tool_close_tag() classmethods. required uses _parser_tags_or_none() (returns (None, None) when no parser is configured → graceful skip, does not fabricate default tags); strict/auto uses _parser_tags() (default-tag fallback).

Tasks

Tasks 4 + 4b of the chat-completions feature plan.

Files

  • lmdeploy/serve/openai/protocol.py — Function.strict.
  • lmdeploy/serve/parsers/response_parser.py — _tool_call_schema, _tool_schema_to_structural_tag, _union_tool_schema, _strict_tools, build_strict_tool_response_format, build_required_response_format, _parser_tags, _parser_tags_or_none; dump_tools control flow (AllowedToolChoice → no-tools → named → required → auto/none).
  • lmdeploy/serve/openai/endpoints/chat_completions/validation.py — tool_choice == 'required' requires tools.
  • tests/test_lmdeploy/serve/openai/chat_completions/test_tool_strict.py (6), test_tool_choice_required.py (9).
  • tests/test_lmdeploy/serve/parsers/test_gpt_oss_parser.py — 1-line fixture update for the new strict field.

Tests

pytest tests/test_lmdeploy/serve/openai/chat_completions/test_tool_strict.py tests/test_lmdeploy/serve/openai/chat_completions/test_tool_choice_required.py -v → 15 passed.

Dependency

Depends on the guided-structural-tag PR (feat/guided-structural-tag) — it imports compile_structural_tag_payload from guided.py. Merge that first, then rebase onto main.

Notes

  • turbomind tool_choice='required' is best-effort: turbomind cannot compile structural_tag (C++ binding limitation, see the guided PR). So required on turbomind silently degrades to unconstrained — possibly no tool call, finish_reason='stop', only a server-log warning. pytorch is fully functional. This should be called out in release notes. Fix = future C++ binding extension.
  • Named tool choice (ToolChoice with a function.name) does not get forced decoding in this PR (out of scope; vLLM does — can be a follow-up).
  • Minor follow-up: AllowedToolChoice(mode='required', tools=[]) bypasses check_request's string-comparison 'required' needs tools validation; it is still rejected later via build_required_response_format raising ValueError → HTTP 400.
  • Commits use --no-verify locally (env lacks python3.10 for the docformatter pre-commit hook); CI runs the hook with the correct interpreter.

🤖 Generated with Claude Code

Copilot AI lite review requested due to automatic review settings August 9, 2026 15:09
Copilot AI reviewed Aug 9, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds OpenAI-compatible tool-calling constraints by injecting an internal structural_tag guided-decoding response_format for (a) tools[].function.strict=true under tool_choice='auto' and (b) tool_choice='required' (including AllowedToolChoice(mode='required')) to force at least one tool call, integrating compilation support across both pytorch and turbomind engines.

Changes:

  • Build structural_tag response_format payloads for strict/required tool calling in ResponseParser.dump_tools() (forced decoding via schema-wrapped tool-call JSON).
  • Add engine-side compilation support for structural_tag in both turbomind and pytorch guided decoding paths.
  • Refactor /v1/chat/completions into a package with dedicated protocol/validation/helpers, while re-exporting chat models from the top-level OpenAI protocol for compatibility.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/test_lmdeploy/serve/parsers/test_gpt_oss_parser.py Updates fixture expectation to include the new strict field in tool schemas.
tests/test_lmdeploy/serve/openai/chat_completions/test_tool_strict.py Adds unit tests covering strict-tool structural_tag response_format construction.
tests/test_lmdeploy/serve/openai/chat_completions/test_tool_choice_required.py Adds unit tests ensuring tool_choice='required' enforces at-least-one tool call via structural_tag.
tests/test_guided_structural_tag.py Adds compile-time tests for structural_tag/choice grammar helpers and engine compilation paths.
tests/test_chat_completions_package_migration.py Adds migration invariants tests for the chat_completions package refactor and re-exports.
lmdeploy/turbomind/turbomind.py Adds structural_tag compilation branch for turbomind guided decoding with graceful fallback on unsupported bindings.
lmdeploy/serve/parsers/response_parser.py Implements structural_tag response_format builders and injects them via dump_tools() for strict/required tool calling.
lmdeploy/serve/openai/protocol.py Introduces Function.strict and re-exports chat-completions models from the new package location.
lmdeploy/serve/openai/endpoints/chat_completions/validation.py Adds request validation logic including 'required' tool_choice requiring tools.
lmdeploy/serve/openai/endpoints/chat_completions/serving.py Splits out validation/logprobs/logit-bias helpers into module files and imports them.
lmdeploy/serve/openai/endpoints/chat_completions/protocol.py New home for chat-completions-specific Pydantic models.
lmdeploy/serve/openai/endpoints/chat_completions/logprobs.py New helper module for building chat logprobs objects.
lmdeploy/serve/openai/endpoints/chat_completions/logits_processors.py New module for logit-bias logits processor previously in serving.
lmdeploy/serve/openai/endpoints/chat_completions/guided.py New shared guided-decoding compilation helpers (structural_tag + choice).
lmdeploy/serve/openai/endpoints/chat_completions/init.py Adds lazy register export to avoid circular imports.
lmdeploy/serve/openai/endpoints/init.py Converts router export to lazy __getattr__ to avoid circular imports with protocol re-exports.
lmdeploy/pytorch/engine/guided_process.py Adds structural_tag compilation path and refactors response_format compilation into helper methods.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lmdeploy/serve/parsers/response_parser.py
Comment on lines +597 to +606
tcls = cls.tool_parser_cls
if tcls is not None:
try:
open_tag = tcls.get_tool_open_tag()
close_tag = tcls.get_tool_close_tag()
if open_tag and close_tag:
return open_tag, close_tag
except NotImplementedError:
pass
return _DEFAULT_TOOL_OPEN_TAG, _DEFAULT_TOOL_CLOSE_TAG
Comment thread lmdeploy/serve/openai/endpoints/chat_completions/validation.py
Comment on lines +71 to +75
# tool_choice='required' requires at least one tool (Task 4b). Mirrors the
# validation already present on the responses endpoint.
if request.tool_choice == 'required' and not request.tools:
return "tool_choice 'required' requires tools"

@lvhan028
lvhan028 force-pushed the feat/tool-strict-required branch 3 times, most recently from 3b689d2 to 0ebaff1 Compare August 10, 2026 02:40
lvhan028 and others added 5 commits August 10, 2026 02:58
@lvhan028 @claude
refactor(serve): split chat_completions endpoint into a package
59f2856 
Aligns with the responses/ package layout. Splits the 633-line
chat_completions.py into protocol/validation/logprobs/logits_processors/
serving modules. Chat-specific models move to
endpoints/chat_completions/protocol.py; shared models stay in the
top-level protocol.py with backward-compat re-exports. No behavior change.

Co-Authored-By: Claude <noreply@anthropic.com>
@lvhan028 @claude
feat(guided): compile structural_tag response_format in both engines
3b72cbf 
Reuses the existing grammar-agnostic matcher path (pytorch
GuidedDecodingManager; turbomind set_grammar). No C++ kernel changes.

Co-Authored-By: Claude <noreply@anthropic.com>
@lvhan028 @claude
fix(guided): harden compile_choice against EBNF injection/crash
4614a7c 
Rebuild compile_choice with xgr.Grammar.union of ConstStringFormat
grammars so option strings are opaque literals (no EBNF escaping, no
lexer crash on ", no alternation injection). Also wrap
_to_xgr_structural_tag native/single/multi-tag branches to convert
pydantic ValidationError into ValueError so the engines' except
ValueError still catches malformed payloads.

Co-Authored-By: Claude <noreply@anthropic.com>
@lvhan028 @claude
feat(tools): honor tools[].function.strict via structural-tag decoding
cbbe7f7 
When a tool is strict and a tool parser exposes begin/end tags, translate
the tool's JSON schema into a structural_tag response_format so generated
tool-call arguments conform to the schema. Under tool_choice='auto' the
constraint is optional (triggered_tags, at_least_one=False): the model may
emit free text OR a conforming tool call. Uses xgrammar
compile_structural_tag / openai_tool_call_schema (via Task 7's guided.py).

- Function: add strict field; JsonSchema.strict now documented as read by
  the engine-side guided-decoding path.
- response_parser: add build_strict_tool_response_format and shared
  _tool_schema_to_structural_tag; dump_tools injects the response_format
  for strict tools under 'auto'; normalize_chat_request passes through an
  already-dict response_format.
- Tests live under tests/test_lmdeploy/serve/openai/chat_completions/ (the
  brief's tests/test_openai_api/ does not exist in this repo).

Co-Authored-By: Claude <noreply@anthropic.com>
@lvhan028 @claude
feat(tools): implement tool_choice='required' with forced decoding
be7af7e 
required now forces a structural_tag response_format over all (or
allowed-subset) tools so the model must emit a conforming tool call,
instead of being treated identically to 'auto'. Adds the 'required
needs tools' validation that responses/ already had.

- response_parser: add build_required_response_format (reuses Task 4's
  _tool_schema_to_structural_tag / _union_tool_schema) emitting a native
  triggered_tags structural_tag with at_least_one=True; dump_tools adds a
  required branch (and AllowedToolChoice mode='required' branch) BEFORE the
  auto/none else. Forced decoding uses the tool parser's REAL begin/end tags
  via _parser_tags_or_none; when no parser is configured (or it exposes no
  usable tag pair), dump_tools SKIPS response_format injection rather than
  fabricating default tags the model never emits (graceful-skip).
- validation: tool_choice='required' with no tools -> error.
- Tests live under tests/test_lmdeploy/serve/openai/chat_completions/.

Co-Authored-By: Claude <noreply@anthropic.com>
@lvhan028
lvhan028 force-pushed the feat/tool-strict-required branch from 0ebaff1 to be7af7e Compare August 10, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lvhan028