Skip to content

test: harden mock server lifecycle checks - #507

Open
jbeckwith-oai wants to merge 9 commits into
mainfrom
codex/harden-mock-server-lifecycle
Open

test: harden mock server lifecycle checks#507
jbeckwith-oai wants to merge 9 commits into
mainfrom
codex/harden-mock-server-lifecycle

Conversation

@jbeckwith-oai

Copy link
Copy Markdown
Contributor

Summary

  • treat HTTP error responses from the Steady health endpoint as unhealthy
  • stop every process listening on the mock port while tolerating stale PIDs
  • add focused regression coverage for startup, reuse, override, and cleanup behavior

Evidence

Without curl --fail, an HTTP error response can be mistaken for a healthy mock server. lsof -t can also return multiple newline-delimited PIDs, and listeners can exit between discovery and cleanup.

Blast radius and ownership

This changes only the handwritten local test launcher and its new workflow tests. It does not affect generated code, SDK runtime behavior, public APIs, type signatures, dependencies, serialization, transport, or release behavior.

Validation

  • bash -n scripts/test
  • bundle exec ruby -w test/scripts/test_workflow_test.rb — 4 tests, 20 assertions
  • script subsystem — 78 tests, 626 assertions
  • bundle exec rake lint — RuboCop, rubyfmt, Sorbet, RBS, and directive validation passed
  • ./scripts/test — 1,091 tests, 9,822 assertions, 0 failures, 0 errors
  • git diff --check
  • thermo-nuclear code-quality review — no remaining findings

@jbeckwith-oai jbeckwith-oai added the codex-maintenance Low-risk maintenance changes created by Codex label Aug 21, 2026
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Castiron custom code

✅ No new custom-code files detected.

48 mixed files remain; 0 existing customizations changed.

Compared 0491bb7ae7ddcc05d64deb4e. Generated baselines verified.

48 existing customizations unchanged
  • lib/openai.rb
  • lib/openai/client.rb
  • lib/openai/models/chat/chat_completion_message.rb
  • lib/openai/models/chat/chat_completion_message_function_tool_call.rb
  • lib/openai/models/chat/completion_create_params.rb
  • lib/openai/models/response_format_json_schema.rb
  • lib/openai/models/responses/function_tool.rb
  • lib/openai/models/responses/response.rb
  • lib/openai/models/responses/response_create_params.rb
  • lib/openai/models/responses/response_format_text_config.rb
  • lib/openai/models/responses/response_format_text_json_schema_config.rb
  • lib/openai/models/responses/response_function_tool_call.rb
  • lib/openai/models/responses/response_function_web_search.rb
  • lib/openai/models/responses/response_output_text.rb
  • lib/openai/models/responses/tool.rb
  • lib/openai/resources/beta/responses.rb
  • lib/openai/resources/beta/threads.rb
  • lib/openai/resources/chat/completions.rb
  • lib/openai/resources/files.rb
  • lib/openai/resources/responses.rb
  • lib/openai/resources/vector_stores/file_batches.rb
  • lib/openai/resources/vector_stores/files.rb
  • lib/openai/resources/webhooks.rb
  • rbi/openai/client.rbi
  • rbi/openai/models/chat/chat_completion_message.rbi
  • rbi/openai/models/chat/chat_completion_message_function_tool_call.rbi
  • rbi/openai/models/chat/completion_create_params.rbi
  • rbi/openai/models/response_format_json_schema.rbi
  • rbi/openai/models/responses/response.rbi
  • rbi/openai/models/responses/response_create_params.rbi
  • rbi/openai/models/responses/response_function_tool_call.rbi
  • rbi/openai/models/responses/response_function_web_search.rbi
  • rbi/openai/models/responses/response_output_text.rbi
  • rbi/openai/resources/chat/completions.rbi
  • rbi/openai/resources/files.rbi
  • rbi/openai/resources/responses.rbi
  • rbi/openai/resources/vector_stores/file_batches.rbi
  • rbi/openai/resources/vector_stores/files.rbi
  • scripts/castiron/README.md
  • scripts/castiron/custom_code_report.py

8 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 32527425563 --repo openai/openai-ruby \
  --name castiron-custom-code-32527425563-1 --dir /tmp/castiron-custom-code-32527425563-1
git apply --stat /tmp/castiron-custom-code-32527425563-1/custom-code.patch
cat /tmp/castiron-custom-code-32527425563-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin 0491bb7ae7dd465c48aa1b84496b2e6d7e3597ea cc05d64deb4e51b74ea440563cad174d7b2c7801
python3 scripts/castiron/custom_code_report.py report \
  --base 0491bb7ae7dd465c48aa1b84496b2e6d7e3597ea \
  --head cc05d64deb4e51b74ea440563cad174d7b2c7801 --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-cc05d64deb4e
cat /tmp/castiron-custom-code-cc05d64deb4e/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@jbeckwith-oai
jbeckwith-oai marked this pull request as ready for review August 21, 2026 04:34
@jbeckwith-oai
jbeckwith-oai requested a review from a team as a code owner August 21, 2026 04:34
@jbeckwith-oai jbeckwith-oai changed the title fix(tests): harden mock server lifecycle checks test: harden mock server lifecycle checks Aug 21, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-21T21:14:00.244692Z cc05d64 New commits
🔒 Security Review Completed 2026-08-21T21:14:24.864989Z cc05d64 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openai-sdks

openai-sdks Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

OkTest Summary

237/237 SDK tests passed in 10.583s for Ruby SDK PR #507.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 224ms
tests/chat-completions-create.test.ts ✅ Passed 240ms
tests/chat-completions-stream.test.ts ✅ Passed 145ms
tests/files-content-binary.test.ts ✅ Passed 155ms
tests/files-create-multipart.test.ts ✅ Passed 135ms
tests/files-list-pagination.test.ts ✅ Passed 172ms
tests/initialize-config.test.ts ✅ Passed 219ms
tests/instance-isolation.test.ts ✅ Passed 227ms
tests/models-list.test.ts ✅ Passed 278ms
tests/responses-background-lifecycle.test.ts ✅ Passed 234ms
tests/responses-body-method-errors.test.ts ✅ Passed 558ms
tests/responses-cancel-timeout.test.ts ✅ Passed 309ms
tests/responses-cancel.test.ts ✅ Passed 265ms
tests/responses-compact-retries.test.ts ✅ Passed 270ms
tests/responses-compact.test.ts ✅ Passed 258ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 205ms
tests/responses-create-advanced.test.ts ✅ Passed 191ms
tests/responses-create-disconnect.test.ts ✅ Passed 960ms
tests/responses-create-errors.test.ts ✅ Passed 351ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 132ms
tests/responses-create-retries.test.ts ✅ Passed 290ms
tests/responses-create-stream-failures.test.ts ✅ Passed 147ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 224ms
tests/responses-create-stream-wire.test.ts ✅ Passed 2.41s
tests/responses-create-stream.test.ts ✅ Passed 175ms
tests/responses-create-terminal-states.test.ts ✅ Passed 417ms
tests/responses-create-timeout.test.ts ✅ Passed 284ms
tests/responses-create.test.ts ✅ Passed 309ms
tests/responses-delete.test.ts ✅ Passed 244ms
tests/responses-input-items-errors.test.ts ✅ Passed 195ms
tests/responses-input-items-list.test.ts ✅ Passed 201ms
tests/responses-input-items-options.test.ts ✅ Passed 177ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 315ms
tests/responses-input-tokens-count.test.ts ✅ Passed 228ms
tests/responses-malformed-inputs.test.ts ✅ Passed 2.107s
tests/responses-not-found-errors.test.ts ✅ Passed 430ms
tests/responses-parse.test.ts ✅ Passed 196ms
tests/responses-retrieve-retries.test.ts ✅ Passed 351ms
tests/responses-retrieve.test.ts ✅ Passed 278ms
tests/responses-stored-method-errors.test.ts ✅ Passed 921ms
tests/retry-behavior.test.ts ✅ Passed 3.229s
tests/sdk-error-shape.test.ts ✅ Passed 434ms

View OkTest run #32527388394

SDK merge (841c873bedf9) · head (cc05d64deb4e) · base (0491bb7ae7dd) · OkTest (2b1bdfd25e98)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bd221ed42d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/test Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b97fc07c97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/test

@apcha-oai apcha-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[sdk codex review] Completed two independent reviews of 5dffc71. The previously reported connected-client and failed-bind cases are fixed, but two distinct P2 lifecycle cases remain: cleanup can kill an unrelated listener on another local address, and readiness-timeout failure can leave an owned mock listener running. Both have isolated base/head reproductions; no live host listener was killed. The supplied 6-test/26-assertion workflow suite and hosted checks pass but do not cover these cases. Please preserve the existing fixes while making cleanup ownership-aware. Head/base/author/draft/mergeability and feedback were rechecked; newer main changes only fast-format tooling.

Comment thread scripts/test Outdated
Comment thread scripts/test Outdated
@jbeckwith-oai
jbeckwith-oai force-pushed the codex/harden-mock-server-lifecycle branch from 5dffc71 to 81742d3 Compare August 21, 2026 19:53
Comment thread test/scripts/test_workflow_test.rb Fixed
@jbeckwith-oai
jbeckwith-oai force-pushed the codex/harden-mock-server-lifecycle branch from 81742d3 to cacf201 Compare August 21, 2026 19:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cacf2012e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/scripts/test_workflow_test.rb
@jbeckwith-oai
jbeckwith-oai force-pushed the codex/harden-mock-server-lifecycle branch from cacf201 to a5b5239 Compare August 21, 2026 20:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5b5239208

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/mock Outdated
@jbeckwith-oai
jbeckwith-oai force-pushed the codex/harden-mock-server-lifecycle branch from a5b5239 to 8935263 Compare August 21, 2026 20:17

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8935263309

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/mock
Comment thread scripts/mock Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4332423b13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/mock Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ba23d4210f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/scripts/test_workflow_test.rb
@jbeckwith-oai
jbeckwith-oai force-pushed the codex/harden-mock-server-lifecycle branch from ba23d42 to b1bc81c Compare August 21, 2026 20:50

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b1bc81c581

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/mock
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-maintenance Low-risk maintenance changes created by Codex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants