[DYNAMO] TITO Dynamo nvext transport follow-up#1313
Merged
AmeenP merged 3 commits intoMay 15, 2026
Conversation
AmeenP
force-pushed
the
codex/dynamo-renderer-transport
branch
from
f6b72e9 to
f6f8224
Compare
AmeenP
force-pushed
the
codex/dynamo-tito-nvext-followup
branch
from
985963e to
0ef29e4
Compare
…okenClient
The verifiers TITO client previously only spoke vLLM's TITO surface
(/v1/chat/completions/tokens for the final POST, /tokenize for bridge
tokenization). Dynamo bis/dynamo-rl serves neither route, so multi-turn
TITO against Dynamo silently degraded to MITO every turn-2+ via the
existing fallback path.
This commit teaches the TITO client to read ClientConfig.renderer_transport
(same field RendererClient consults) and route accordingly:
- prime_vllm_generate (default): unchanged - posts to
/v1/chat/completions/tokens and uses /tokenize over HTTP.
- dynamo_chat_nvext: bridge tokenize runs locally via the renderers
package (zero RTTs); final POST goes to /v1/chat/completions with
placeholder messages + nvext.token_data carrying the stitched
prompt_ids + explicit stop_token_ids from renderer.get_stop_token_ids().
Wire shape matches what RendererClient already produces for the
same transport, so a Dynamo deployment validated against renderer
mode automatically accepts TITO traffic too.
Adds two unit tests that assert the dynamo-transport wire shape end-to-
end via a recording client + stub renderer (no real tokenizer download).
OpenAIChatCompletionsTokenClient.get_prompt_ids' prefix-match between the prompt_messages caller-input and the trajectory step messages was asymmetric: - prompt_messages went straight through normalize_for_comparison (which picks up vf.AssistantMessage.model_dump's exhaustive view, including thinking_blocks=None and other defaulted fields). - step_messages went through to_native_prompt FIRST, which produces the slimmer OpenAI-format dict that omits thinking_blocks entirely. The two normalized forms then never compared equal whenever the caller handed the client Pydantic vf.Message types -- the form MultiTurnEnv produces after maybe_normalize_messages -- so the prefix match always returned None and TITO silently fell back to MITO every turn-2+. Probe-3 and the upstream test suite both used raw dict input, so the asymmetry only showed up under real orchestrator rollouts. Fix: drop None-valued keys in normalize_for_comparison. Both sides land on the same shape regardless of whether they came in as Pydantic or as plain OpenAI dicts. Validated end-to-end against bis-dev/5/always-continue-tito (multi-turn TITO + Dynamo bis/dynamo-rl smoke): 348 /v1/chat/completions, 21 SIDECAR-SKIP-TOKENIZE markers, 0 fall- back warnings. Same SIDECAR token-prefix appears across turns, confirming the engine reuses prior-turn ids verbatim. The existing 6 unit tests (4 vanilla TITO + 2 dynamo_chat_nvext) all still pass; their dict-shaped input always normalized to the same shape on both sides, so the symmetric drop-None doesn't change them.
AmeenP
force-pushed
the
codex/dynamo-tito-nvext-followup
branch
from
0ef29e4 to
4bb3da6
Compare
Signed-off-by: AmeenP <ameenp360@gmail.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| encoded = tokenizer(messages, add_special_tokens=False) | ||
| return list(encoded["input_ids"]) | ||
|
|
||
| add_generation_prompt = bool( |
There was a problem hiding this comment.
🟡 Medium clients/openai_chat_completions_token_client.py:611
_local_tokenize extracts only add_generation_prompt from extra_kwargs and silently drops chat_template_kwargs, so the dynamo_chat_nvext transport produces different tokens than prime_vllm_generate when template-affecting options like Qwen3's thinking parameter are passed. The HTTP /tokenize path spreads all extra_kwargs into the request body, but _local_tokenize does not forward them to renderer.render_ids().
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file verifiers/clients/openai_chat_completions_token_client.py around line 611:
`_local_tokenize` extracts only `add_generation_prompt` from `extra_kwargs` and silently drops `chat_template_kwargs`, so the `dynamo_chat_nvext` transport produces different tokens than `prime_vllm_generate` when template-affecting options like Qwen3's `thinking` parameter are passed. The HTTP `/tokenize` path spreads all `extra_kwargs` into the request body, but `_local_tokenize` does not forward them to `renderer.render_ids()`.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the missing TITO-side Dynamo nvext transport support from Biswa's tested verifiers branch on top of #1287, aligned with the updated renderers PR.
OpenAIChatCompletionsTokenClientto userenderer_transport="dynamo_chat_nvext"/chat/completionswithnvext.token_datanvext.extra_fields=["engine_data"]and graftsnvext.engine_data.{prompt_token_ids,completion_token_ids}onto the standard response token fieldsstop_token_idsfor Dynamo compatibilitynvext.extra_fieldswhile addingengine_datavf.Messageinput and forwardschat_template_kwargsto bridge tokenizationRenderedTokensbridge return values and maps verifiers transport names to renderers"vllm"/"dynamo"17005ddContext
This pairs with:
PrimeIntellect-ai/renderers#11ai-dynamo/dynamo#9509renderer_transport="dynamo_chat_nvext"for TITO clients whenclient.backend = "dynamo"Validation
uv lock --checkuv run --python 3.13 ty check verifiers.venv/bin/ruff check pyproject.toml verifiers/types.py verifiers/clients/renderer_client.py verifiers/clients/openai_chat_completions_client.py verifiers/clients/openai_chat_completions_token_client.py tests/test_renderer_client.py tests/test_openai_chat_completions_token_client.py.venv/bin/ruff format --check pyproject.toml verifiers/types.py verifiers/clients/renderer_client.py verifiers/clients/openai_chat_completions_client.py verifiers/clients/openai_chat_completions_token_client.py tests/test_renderer_client.py tests/test_openai_chat_completions_token_client.pyuv run pytest tests/test_openai_chat_completions_token_client.py tests/test_renderer_client.pyNote
Add
dynamo_chat_nvexttransport support toOpenAIChatCompletionsTokenClientdynamo_chat_nvexttransport path inopenai_chat_completions_token_client.pythat posts to/chat/completionswithnvext.token_dataandstop_token_idsinstead of/chat/completions/tokens._local_tokenize_dynamohelper using a cached renderer, bypassing network calls for the dynamo transport._graft_engine_datainopenai_chat_completions_client.pyto surface token IDs fromnvext.engine_dataintochoice.token_idsandresponse.prompt_token_ids.dynamo_chat_nvexttotransport='dynamo'when callinggenerate()inrenderer_client.py, and fixes_get_incremental_prompt_idsto always return a list of ints.get_native_responsenow always setslogprobs=True; nvext merging deduplicatesextra_fieldswhen both caller and default provide nvext.📊 Macroscope summarized 760eede. 4 files reviewed, 2 issues evaluated, 0 issues filtered, 1 comment posted
🗂️ Filtered Issues