Skip to content

test: characterize ambiguous union behavior - #504

Open
jbeckwith-oai wants to merge 3 commits into
mainfrom
codex/add-union-characterization-corpus
Open

test: characterize ambiguous union behavior#504
jbeckwith-oai wants to merge 3 commits into
mainfrom
codex/add-union-characterization-corpus

Conversation

@jbeckwith-oai

@jbeckwith-oai jbeckwith-oai commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a real Responses, Beta Responses, and Realtime union corpus covering missing, duplicate-known, unique-known, and explicitly unknown discriminator evidence
  • record all exact and structurally viable candidates, the current compatibility-selected class, exactness, and serialized request/event bodies
  • pair wire-string and Ruby-symbol forms so key/value normalization differences remain visible
  • add an executable differential that regenerates observations from v0.78.0, v0.80.0, and the current checkout without checking in generated snapshots

This PR is characterization-only. It does not change Union, Converter, BaseModel, generated models, signatures, or public runtime behavior.

What the corpus shows

  • v0.78.0 and v0.80.0 are identical for every recorded case.
  • Symbol-keyed typed Responses inputs have multiple exact candidates, including EasyInputMessage and ResponseInputItem::Message; type: :message is also registered by three variants and does not uniquely identify one Ruby class.
  • Wire-string hashes expose a different candidate set while retaining the same serialized request body.
  • An explicit unknown Responses/Beta stream tag is still structurally materialized as a known event by the shared union. The winning class changed between the published releases and current generated schemas, demonstrating that generated variant evolution can change the guess without a runtime edit.
  • Current Realtime preserves the same unknown payload as UnknownServerEvent at its connection boundary even though direct shared-union coercion can still guess a known event for symbolized input.
  • Request shorthand, extra unknown fields, and stream payloads retain their recorded wire bodies regardless of materialization.

The field is deliberately named compatibility_selected: an ambiguous or unknown selection is recorded behavior, not a claim that the class is semantically correct. Focused assertions retain the compatibility-sensitive facts; the full historical differential is generated on demand.

Generator ownership

The corpus exercises real generated models without editing them. All changes are SDK-owned tests, fixtures, and developer scripts.

Validation

  • focused characterization tests: 7 runs, 62 assertions, 0 failures
  • v0.78.0 / v0.80.0 / current-checkout differential
  • bundle exec rake lint: 2,730 RuboCop files, Sorbet, and 1,239 RBS files green
  • canonical primary suite: 1,124 runs and 10,210 assertions; sole error is the sandbox-only proxy invariant described below
  • remaining Realtime network invariants: 6 runs, 26 assertions, 0 failures
  • Bedrock suite: 38 runs, 377 assertions, 0 failures, 1 existing skip
  • gem package build
  • thermo-nuclear code-quality review
  • git diff --check

The canonical local suite's sole failure is RealtimeNetworkInvariantsTest#test_http_proxy_uses_connect_and_isolates_proxy_and_origin_credentials; the sandbox proxy stack closes the local CONNECT listener before acceptance. The same invariant was already reproduced unchanged on untouched main during the preceding #376 work, and the other six tests in that file pass.

Refs #287 and #376.

@jbeckwith-oai
jbeckwith-oai requested a review from a team as a code owner August 21, 2026 04:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-21T04:56:26.576623Z e4cc7dc New commits
🔒 Security Review Completed 2026-08-21T04:56:35.245762Z e4cc7dc New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openai-sdks

openai-sdks Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

OkTest Summary

237/237 SDK tests passed in 12.181s for Ruby SDK PR #504.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 323ms
tests/chat-completions-create.test.ts ✅ Passed 369ms
tests/chat-completions-stream.test.ts ✅ Passed 160ms
tests/files-content-binary.test.ts ✅ Passed 351ms
tests/files-create-multipart.test.ts ✅ Passed 243ms
tests/files-list-pagination.test.ts ✅ Passed 189ms
tests/initialize-config.test.ts ✅ Passed 338ms
tests/instance-isolation.test.ts ✅ Passed 351ms
tests/models-list.test.ts ✅ Passed 222ms
tests/responses-background-lifecycle.test.ts ✅ Passed 299ms
tests/responses-body-method-errors.test.ts ✅ Passed 579ms
tests/responses-cancel-timeout.test.ts ✅ Passed 258ms
tests/responses-cancel.test.ts ✅ Passed 334ms
tests/responses-compact-retries.test.ts ✅ Passed 342ms
tests/responses-compact.test.ts ✅ Passed 363ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 226ms
tests/responses-create-advanced.test.ts ✅ Passed 269ms
tests/responses-create-disconnect.test.ts ✅ Passed 1.031s
tests/responses-create-errors.test.ts ✅ Passed 423ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 291ms
tests/responses-create-retries.test.ts ✅ Passed 417ms
tests/responses-create-stream-failures.test.ts ✅ Passed 161ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 242ms
tests/responses-create-stream-wire.test.ts ✅ Passed 2.494s
tests/responses-create-stream.test.ts ✅ Passed 177ms
tests/responses-create-terminal-states.test.ts ✅ Passed 420ms
tests/responses-create-timeout.test.ts ✅ Passed 273ms
tests/responses-create.test.ts ✅ Passed 432ms
tests/responses-delete.test.ts ✅ Passed 511ms
tests/responses-input-items-errors.test.ts ✅ Passed 196ms
tests/responses-input-items-list.test.ts ✅ Passed 216ms
tests/responses-input-items-options.test.ts ✅ Passed 361ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 383ms
tests/responses-input-tokens-count.test.ts ✅ Passed 399ms
tests/responses-malformed-inputs.test.ts ✅ Passed 2.258s
tests/responses-not-found-errors.test.ts ✅ Passed 385ms
tests/responses-parse.test.ts ✅ Passed 263ms
tests/responses-retrieve-retries.test.ts ✅ Passed 349ms
tests/responses-retrieve.test.ts ✅ Passed 360ms
tests/responses-stored-method-errors.test.ts ✅ Passed 1.064s
tests/retry-behavior.test.ts ✅ Passed 3.215s
tests/sdk-error-shape.test.ts ✅ Passed 561ms

View OkTest run #32448635177

SDK merge (6f44a1c34792) · head (e4cc7dcd44f4) · base (851f53438f5a) · OkTest (2b1bdfd25e98)

@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Castiron custom code

✅ No new custom-code files detected.

48 mixed files remain; 0 existing customizations changed.

Compared 851f53438f5ae4cc7dcd44f4. Generated baselines verified.

48 existing customizations unchanged
  • lib/openai.rb
  • lib/openai/client.rb
  • lib/openai/models/chat/chat_completion_message.rb
  • lib/openai/models/chat/chat_completion_message_function_tool_call.rb
  • lib/openai/models/chat/completion_create_params.rb
  • lib/openai/models/response_format_json_schema.rb
  • lib/openai/models/responses/function_tool.rb
  • lib/openai/models/responses/response.rb
  • lib/openai/models/responses/response_create_params.rb
  • lib/openai/models/responses/response_format_text_config.rb
  • lib/openai/models/responses/response_format_text_json_schema_config.rb
  • lib/openai/models/responses/response_function_tool_call.rb
  • lib/openai/models/responses/response_function_web_search.rb
  • lib/openai/models/responses/response_output_text.rb
  • lib/openai/models/responses/tool.rb
  • lib/openai/resources/beta/responses.rb
  • lib/openai/resources/beta/threads.rb
  • lib/openai/resources/chat/completions.rb
  • lib/openai/resources/files.rb
  • lib/openai/resources/responses.rb
  • lib/openai/resources/vector_stores/file_batches.rb
  • lib/openai/resources/vector_stores/files.rb
  • lib/openai/resources/webhooks.rb
  • rbi/openai/client.rbi
  • rbi/openai/models/chat/chat_completion_message.rbi
  • rbi/openai/models/chat/chat_completion_message_function_tool_call.rbi
  • rbi/openai/models/chat/completion_create_params.rbi
  • rbi/openai/models/response_format_json_schema.rbi
  • rbi/openai/models/responses/response.rbi
  • rbi/openai/models/responses/response_create_params.rbi
  • rbi/openai/models/responses/response_function_tool_call.rbi
  • rbi/openai/models/responses/response_function_web_search.rbi
  • rbi/openai/models/responses/response_output_text.rbi
  • rbi/openai/resources/chat/completions.rbi
  • rbi/openai/resources/files.rbi
  • rbi/openai/resources/responses.rbi
  • rbi/openai/resources/vector_stores/file_batches.rbi
  • rbi/openai/resources/vector_stores/files.rbi
  • scripts/castiron/README.md
  • scripts/castiron/custom_code_report.py

8 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 32448634742 --repo openai/openai-ruby \
  --name castiron-custom-code-32448634742-1 --dir /tmp/castiron-custom-code-32448634742-1
git apply --stat /tmp/castiron-custom-code-32448634742-1/custom-code.patch
cat /tmp/castiron-custom-code-32448634742-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin 851f53438f5afc7f32e3ccbe866fd03bbade576a e4cc7dcd44f4f1e3e33338a8579fdf9648eb828d
python3 scripts/castiron/custom_code_report.py report \
  --base 851f53438f5afc7f32e3ccbe866fd03bbade576a \
  --head e4cc7dcd44f4f1e3e33338a8579fdf9648eb828d --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-e4cc7dcd44f4
cat /tmp/castiron-custom-code-e4cc7dcd44f4/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d90571e548

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/union-characterization Outdated
Comment thread scripts/union-characterization-differential Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant