Skip to content

Add Anthropic structured output support#608

Merged
crmne merged 4 commits intocrmne:mainfrom
hiasinho:feature/anthropic-structured-output
Mar 1, 2026
Merged

Add Anthropic structured output support#608
crmne merged 4 commits intocrmne:mainfrom
hiasinho:feature/anthropic-structured-output

Conversation

@hiasinho
Copy link
Copy Markdown
Contributor

What this does

Adds native structured output support for Anthropic Claude 4.5+ models using the GA output_config API. When with_schema is called on a chat using a supported Anthropic model, the schema is sent via output_config.format with type json_schema, and the response is automatically parsed as JSON.

Because I can't contribute to PR #528, I opened this one.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Required for new features

PRs for new features or enhancements without a prior approved issue will be closed.

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
    • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
    • All tests pass: bundle exec rspec
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

  • I used AI tools to help write this code
  • I have reviewed and understand all generated code (required if above is checked)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

Wire output_config into the Anthropic chat payload when a schema is
provided, using the GA format ({ format: { type: 'json_schema', schema } })
with no beta headers required. The build_output_config method deep-dups the
schema and strips :strict keys that Anthropic rejects.

Fix model detection regexes in capabilities to match Claude 4+ model IDs
(supports_functions?, supports_json_mode?, capabilities_for) and add
supports_structured_output? for Claude 4.5+ models.

Re-record Anthropic VCR cassettes.
@hiasinho hiasinho changed the title Feature/anthropic structured output Add Anthropic structured output support Feb 15, 2026
Copy link
Copy Markdown
Contributor

@llenodo llenodo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR I almost implemented the same thing before finding this.

One suggestion to make it more future proof which assumes future versions of claude (5+) will support structured outputs so we wont need to come back to this later

@KieranP
Copy link
Copy Markdown

KieranP commented Feb 17, 2026

@hiasinho Any chance this PR could also include structured output for anthropic models through AWS Bedrock? The AWS bedrock integration was recently rewritten to use converse API, which support structured output: https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html

@hiasinho
Copy link
Copy Markdown
Contributor Author

Nice PR I almost implemented the same thing before finding this.

One suggestion to make it more future proof which assumes future versions of claude (5+) will support structured outputs so we wont need to come back to this later

Done!

@hiasinho
Copy link
Copy Markdown
Contributor Author

@hiasinho Any chance this PR could also include structured output for anthropic models through AWS Bedrock? The AWS bedrock integration was recently rewritten to use converse API, which support structured output: https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html

Would love to do it, but I don't use AWS. So no creds to create VCRs. Tests will fail. Feel free to build on it.

@hiasinho hiasinho requested a review from llenodo February 18, 2026 07:32
llenodo added a commit to llenodo/ruby_llm that referenced this pull request Feb 18, 2026
Extend the Bedrock Converse API provider to support structured output
via outputConfig.textFormat, building on crmne#608's Anthropic implementation.

Key differences from the direct Anthropic API: schema must be a JSON
string (not a Hash), nested under structure.jsonSchema with a required
name field, and uses camelCase Converse API naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
llenodo added a commit to llenodo/ruby_llm that referenced this pull request Feb 18, 2026
Extend the Bedrock Converse API provider to support structured output
via outputConfig.textFormat, building on crmne#608's Anthropic implementation.

Key differences from the direct Anthropic API: schema must be a JSON
string (not a Hash), nested under structure.jsonSchema with a required
name field, and uses camelCase Converse API naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@llenodo
Copy link
Copy Markdown
Contributor

llenodo commented Feb 18, 2026

@KieranP see #596 for bedrock support

PR is currently closed per community guidelines

end

# Test Anthropic provider
CHAT_MODELS.select { |model_info| model_info[:provider] == :anthropic }.each do |model_info|
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be another test but use the test above.

…uctured-output

# Conflicts:
#	spec/fixtures/vcr_cassettes/activerecord_actsas_extended_thinking_persistence_anthropic_claude-haiku-4-5_persists_thinking_data_and_replays_it_across_turns.yml
#	spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-haiku-4-5_can_handle_multi-turn_conversations.yml
#	spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-haiku-4-5_can_have_a_basic_conversation.yml
#	spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-haiku-4-5_replaces_previous_system_messages_when_replace_true.yml
#	spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-haiku-4-5_returns_raw_responses.yml
#	spec/fixtures/vcr_cassettes/chat_basic_chat_functionality_anthropic_claude-haiku-4-5_successfully_uses_the_system_prompt.yml
#	spec/fixtures/vcr_cassettes/chat_content_object_support_anthropic_claude-haiku-4-5_preserves_content_objects_returned_from_tools.yml
#	spec/fixtures/vcr_cassettes/chat_error_handling_with_anthropic_claude-haiku-4-5_raises_appropriate_auth_error.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_handle_multiple_tool_calls_in_a_single_response.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_parallel_tool_calls.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_tools.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_tools_in_multi-turn_conversations.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_tools_with_multi-turn_streaming_conversations.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_tools_without_parameters.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_can_use_tools_without_parameters_in_multi-turn_streaming_conversations.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_handles_anyof_params.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_handles_array_params.yml
#	spec/fixtures/vcr_cassettes/chat_function_calling_anthropic_claude-haiku-4-5_handles_object_params.yml
#	spec/fixtures/vcr_cassettes/chat_pdf_models_anthropic_claude-haiku-4-5_can_handle_array_of_mixed_files_with_auto-detection.yml
#	spec/fixtures/vcr_cassettes/chat_pdf_models_anthropic_claude-haiku-4-5_handles_multiple_pdfs.yml
#	spec/fixtures/vcr_cassettes/chat_pdf_models_anthropic_claude-haiku-4-5_understands_pdfs.yml
#	spec/fixtures/vcr_cassettes/chat_real_error_scenarios_anthropic_claude-haiku-4-5_handles_context_length_exceeded_errors.yml
#	spec/fixtures/vcr_cassettes/chat_streaming_responses_anthropic_claude-haiku-4-5_reports_consistent_token_counts_compared_to_non-streaming.yml
#	spec/fixtures/vcr_cassettes/chat_streaming_responses_anthropic_claude-haiku-4-5_supports_streaming_responses.yml
#	spec/fixtures/vcr_cassettes/chat_text_models_anthropic_claude-haiku-4-5_can_understand_remote_text.yml
#	spec/fixtures/vcr_cassettes/chat_text_models_anthropic_claude-haiku-4-5_can_understand_text.yml
#	spec/fixtures/vcr_cassettes/chat_vision_models_anthropic_claude-haiku-4-5_can_understand_local_images.yml
#	spec/fixtures/vcr_cassettes/chat_vision_models_anthropic_claude-haiku-4-5_can_understand_remote_images_without_extension.yml
#	spec/fixtures/vcr_cassettes/chat_with_extended_thinking_anthropic_claude-haiku-4-5_preserves_thinking_signatures_between_turns_when_provided.yml
#	spec/fixtures/vcr_cassettes/chat_with_extended_thinking_anthropic_claude-haiku-4-5_returns_thinking_when_available.yml
#	spec/fixtures/vcr_cassettes/chat_with_extended_thinking_anthropic_claude-haiku-4-5_streams_thinking_content_when_available.yml
#	spec/fixtures/vcr_cassettes/chat_with_params_anthropic_claude-haiku-4-5_supports_service_tier_param.yml
@crmne crmne merged commit b5acdbe into crmne:main Mar 1, 2026
19 checks passed
@crmne
Copy link
Copy Markdown
Owner

crmne commented Mar 1, 2026

Fixed them myself to keep momentum.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 1, 2026

Codecov Report

❌ Patch coverage is 50.00000% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.82%. Comparing base (80c3373) to head (2d286e0).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
lib/ruby_llm/providers/anthropic/capabilities.rb 9.09% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #608      +/-   ##
==========================================
- Coverage   80.89%   80.82%   -0.07%     
==========================================
  Files         113      113              
  Lines        5099     5112      +13     
  Branches     1310     1313       +3     
==========================================
+ Hits         4125     4132       +7     
- Misses        974      980       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@crmne crmne linked an issue Mar 2, 2026 that may be closed by this pull request
6 tasks
@hiasinho hiasinho deleted the feature/anthropic-structured-output branch March 2, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Support Anthropic's new structured outputs support

4 participants