Skip to content

DO NOT MERGE: test graceful OOM shutdown via intentional memory leak#969

Open
devin-ai-integration[bot] wants to merge 2 commits intodevin/1774478445-memory-failfastfrom
devin/1774888252-memory-failfast-feature-1
Open

DO NOT MERGE: test graceful OOM shutdown via intentional memory leak#969
devin-ai-integration[bot] wants to merge 2 commits intodevin/1774478445-memory-failfastfrom
devin/1774888252-memory-failfast-feature-1

Conversation

@devin-ai-integration
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot commented Mar 30, 2026

This PR targets the following PR:


Summary

Adds an intentional memory leak in AirbyteEntrypoint.read() to test the memory monitor's graceful shutdown behavior (introduced in #962). Each consumed record is duplicated 500× into a growing list to rapidly inflate Python process anonymous RSS, which should trigger the dual-condition fail-fast check.

This tests the positive case: the memory monitor should detect and raise AirbyteTracedException when both cgroup usage and process anonymous RSS exceed their respective thresholds.

DO NOT MERGE — test-only change to validate memory monitor behavior.

Companion PR: #970 (tests the negative case — cgroup pressure without process RSS growth)

Review & Testing Checklist for Human

  • Verify this PR is never merged to a release branch
  • Deploy a connector using this branch in a memory-bounded container and observe warning log at ~95% cgroup usage
  • Confirm graceful shutdown with AirbyteTracedException at ~98% cgroup + ~90% RssAnon
  • Verify queued state messages are flushed before the exception propagates

Notes

Link to Devin session: https://app.devin.ai/sessions/070ecb51ceee4f9189e1c09a83ba31cb

Co-Authored-By: patrick.nilan@airbyte.io <patrick.nilan@airbyte.io>
@devin-ai-integration
Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1774888252-memory-failfast-feature-1#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1774888252-memory-failfast-feature-1

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

Co-Authored-By: patrick.nilan@airbyte.io <patrick.nilan@airbyte.io>
@devin-ai-integration devin-ai-integration bot changed the title feat(cdk): memory fail-fast feature branch 1 DO NOT MERGE: test graceful OOM shutdown via intentional memory leak Mar 30, 2026
@github-actions
Copy link
Copy Markdown

PyTest Results (Fast)

3 943 tests  ±0   3 932 ✅ ±0   7m 31s ⏱️ -16s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 5aac720. ± Comparison against base commit 6c2cd9c.

@github-actions
Copy link
Copy Markdown

PyTest Results (Full)

3 946 tests  ±0   3 934 ✅ ±0   11m 7s ⏱️ +10s
    1 suites ±0      12 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 5aac720. ± Comparison against base commit 6c2cd9c.

@pnilan
Copy link
Copy Markdown
Contributor

Patrick Nilan (pnilan) commented Mar 30, 2026

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/23757385016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant