fix(llm): fall back when a provider rejects json_schema response_format#6191
Open
HumphreySun98 wants to merge 1 commit into
Open
Conversation
…hat reject it DeepSeek and some other OpenAI-compatible endpoints reject OpenAI's json_schema structured-output response_format with "This response_format type is unavailable now". The native OpenAI provider unconditionally sent it via beta.chat.completions.parse/stream and via a Pydantic response_format, breaking any crew using structured output against those endpoints. Add a supports_native_structured_output() capability (True for OpenAI) and a per-provider supports_json_schema flag on the OpenAI-compatible config (DeepSeek = False). When unsupported, skip the native json_schema paths and fall back to a plain completion; the non-streaming path then validates the result against the requested model client-side, and the Task converter reconciles it otherwise. OpenAI behavior is unchanged. Fixes crewAIInc#5990 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis change adds provider capability checks for native JSON-schema structured output, disables that path for DeepSeek, and updates completion handling to fall back to client-side model validation in sync, async, and streaming flows, with tests covering provider flags and fallback behavior. ChangesStructured output fallback for OpenAI-compatible providers
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/crewai/tests/llms/openai_compatible/test_openai_compatible.py (1)
321-386: ⚡ Quick winAdd a DeepSeek streaming fallback regression test (
stream=True+response_model).Current coverage validates the non-streaming fallback well, but it does not exercise the sync streaming fallback path. A targeted test here would lock in the intended contract and catch the current mismatch early.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/llms/openai_compatible/test_openai_compatible.py` around lines 321 - 386, Add a new test method to the TestStructuredOutputFallback class that validates DeepSeek's streaming fallback behavior when using response_model. Create a test similar to test_deepseek_completion_skips_native_parse_and_validates_client_side but pass stream=True in the _handle_completion parameters to exercise the sync streaming fallback path. Mock the streaming response appropriately (using an iterator or stream chunks) and verify that the client.chat.completions.create is called with stream=True, that client.beta.chat.completions.parse is not called, and that the final result is still a validated _Answer instance.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/llms/providers/openai/completion.py`:
- Line 1914: The sync streaming fallback path for unsupported providers drops
structured parsing validation, causing stream=True with response_model to return
raw str instead of BaseModel. At
lib/crewai/src/crewai/llms/providers/openai/completion.py line 1914, the native
streaming parse is correctly gated by the response_model check, which is the
anchor pattern. The issue is in the sibling fallback path at lines 2039-2054
where _finalize_streaming_response(...) is called without validating
response_model. Add response_model parsing validation in the
unsupported-provider path (2039-2054) to ensure that when response_model is
provided with stream=True on DeepSeek-like providers, the result is properly
parsed into a BaseModel instance, matching the behavior of non-streaming and
async-streaming fallbacks.
---
Nitpick comments:
In `@lib/crewai/tests/llms/openai_compatible/test_openai_compatible.py`:
- Around line 321-386: Add a new test method to the TestStructuredOutputFallback
class that validates DeepSeek's streaming fallback behavior when using
response_model. Create a test similar to
test_deepseek_completion_skips_native_parse_and_validates_client_side but pass
stream=True in the _handle_completion parameters to exercise the sync streaming
fallback path. Mock the streaming response appropriately (using an iterator or
stream chunks) and verify that the client.chat.completions.create is called with
stream=True, that client.beta.chat.completions.parse is not called, and that the
final result is still a validated _Answer instance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 92196d58-34cc-4d5b-b435-b3b948a52c64
📒 Files selected for processing (3)
lib/crewai/src/crewai/llms/providers/openai/completion.pylib/crewai/src/crewai/llms/providers/openai_compatible/completion.pylib/crewai/tests/llms/openai_compatible/test_openai_compatible.py
| tool_calls: dict[int, dict[str, Any]] = {} | ||
|
|
||
| if response_model: | ||
| if response_model and self.supports_native_structured_output(): |
There was a problem hiding this comment.
Sync streaming fallback drops structured parsing when native json_schema is disabled.
Line 1914 correctly gates native streaming parse, but in the unsupported-provider path (Line 2039 onward) the code returns _finalize_streaming_response(...) without validating response_model. For stream=True + response_model on DeepSeek-like providers, this returns raw str instead of BaseModel, unlike your non-streaming and async-streaming fallbacks.
Suggested fix (parse fallback in sync streaming finalize path)
- def _finalize_streaming_response(
+ def _finalize_streaming_response(
self,
full_response: str,
tool_calls: dict[int, dict[str, Any]],
usage_data: dict[str, Any] | None,
params: dict[str, Any],
+ response_model: type[BaseModel] | None = None,
available_functions: dict[str, Any] | None = None,
from_task: Any | None = None,
from_agent: Any | None = None,
finish_reason: str | None = None,
response_id: str | None = None,
- ) -> str | list[dict[str, Any]]:
+ ) -> str | list[dict[str, Any]] | BaseModel:
@@
+ if response_model:
+ try:
+ structured_result = self._validate_structured_output(
+ full_response, response_model
+ )
+ self._emit_call_completed_event(
+ response=structured_result,
+ call_type=LLMCallType.LLM_CALL,
+ from_task=from_task,
+ from_agent=from_agent,
+ messages=params["messages"],
+ usage=usage_data,
+ finish_reason=finish_reason,
+ response_id=response_id,
+ )
+ return structured_result
+ except ValueError as e:
+ logging.warning(f"Structured output validation failed: {e}")
+
full_response = self._apply_stop_words(full_response)
@@
result = self._finalize_streaming_response(
full_response=full_response,
tool_calls=tool_calls,
usage_data=usage_data,
params=params,
+ response_model=response_model,
available_functions=available_functions,
from_task=from_task,
from_agent=from_agent,
finish_reason=stream_finish_reason,
response_id=stream_response_id,
)Also applies to: 2039-2054
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/crewai/src/crewai/llms/providers/openai/completion.py` at line 1914, The
sync streaming fallback path for unsupported providers drops structured parsing
validation, causing stream=True with response_model to return raw str instead of
BaseModel. At lib/crewai/src/crewai/llms/providers/openai/completion.py line
1914, the native streaming parse is correctly gated by the response_model check,
which is the anchor pattern. The issue is in the sibling fallback path at lines
2039-2054 where _finalize_streaming_response(...) is called without validating
response_model. Add response_model parsing validation in the
unsupported-provider path (2039-2054) to ensure that when response_model is
provided with stream=True on DeepSeek-like providers, the result is properly
parsed into a BaseModel instance, matching the behavior of non-streaming and
async-streaming fallbacks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DeepSeek and some other OpenAI-compatible endpoints reject OpenAI's
json_schemastructured-outputresponse_format:The native OpenAI provider (
OpenAICompatibleCompletionextendsOpenAICompletion) unconditionally sent it viabeta.chat.completions.parse/.streamand via a Pydanticresponse_format, so any crew using structured output against those endpoints failed.Fix
Add a capability gate:
OpenAICompletion.supports_native_structured_output()→True.supports_json_schemaflag on the OpenAI-compatibleProviderConfig(DeepSeek =False); the subclass override reads it.When unsupported, the three
json_schema-sending branches are skipped and the call falls back to a plain completion. The non-streaming path then validates the result against the requested model client-side, and theTaskconverter reconciles it otherwise. OpenAI behavior is unchanged. (The async-streaming path already used a plain completion + client-side validation.)Tests
Added
TestStructuredOutputFallbackintest_openai_compatible.py: capability flags,json_schemasuppression in_prepare_completion_params, and a mock asserting DeepSeek skips the native parse and still returns the validated model. Verified failing-without/passing-with the fix; 137 OpenAI/compatible provider tests still pass;ruffandmypyclean.Fixes #5990
This PR was authored with Claude Code. Per
CONTRIBUTING.md, AI-generated contributions require thellm-generatedlabel — I don't have triage permission to set it, so could a maintainer please add it? 🤖 Generated with Claude CodeSummary by CodeRabbit
Release Notes