Skip to content

fix(otlp-http): reset requests session after fork#4995

Open
dshivashankar1994 wants to merge 10 commits into
open-telemetry:mainfrom
dshivashankar1994:fix_stale_session
Open

fix(otlp-http): reset requests session after fork#4995
dshivashankar1994 wants to merge 10 commits into
open-telemetry:mainfrom
dshivashankar1994:fix_stale_session

Conversation

@dshivashankar1994

@dshivashankar1994 dshivashankar1994 commented Mar 17, 2026

Copy link
Copy Markdown

Register an after_in_child fork handler for OTLP HTTP metric and trace exporters. In the child process, copy session headers → close existing requests.Session → create a new session → restore headers. Prevents child processes from reusing parent connection/session state after fork(). Avoids requests session finalizer warnings when inherited file descriptors are closed.

Fixes #4994

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Does This PR Require a Contrib Repo Change?

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

Register an after_in_child fork handler for OTLP HTTP metric and trace exporters.
In the child process, copy session headers → close existing requests.Session → create a new session → restore headers.
Prevents child processes from reusing parent connection/session state after fork().
Avoids requests session finalizer warnings when inherited file descriptors are closed.
@dshivashankar1994 dshivashankar1994 requested a review from a team as a code owner March 17, 2026 19:21
@xrmx xrmx moved this to Ready for review in Python PR digest Mar 18, 2026
Co-authored-by: Riccardo Magliocchetti <riccardo.magliocchetti@gmail.com>
@tammy-baylis-swi

Copy link
Copy Markdown
Contributor

Thanks for this! Please also do a tox -e precommit and commit any fixes.

@themavik themavik left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: Resets the requests Session in the child process after fork via os.register_at_fork(after_in_child=...) to avoid reusing the parent's connection state.

Done well: hasattr(os, "register_at_fork") guards for platforms without it (e.g. Windows). The _reset_session_after_fork logic correctly copies headers, closes the old session, and creates a new one. The test mocks register_at_fork and verifies after_in_child closes the initial session and that the new session gets the expected headers including Content-Type.

Suggestion: The bare except Exception in _reset_session_after_fork swallows all errors; exc_info=True helps debugging but consider logging at WARNING level so failures are visible. Also, do the trace and log OTLP HTTP exporters need the same fork handling? If so, a follow-up could align them.

@xrmx

xrmx commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

The changes looks good to me, before approving I would like to reproduce with a plain gunicorn web app (or similar) instead of the provided hand crafted one.

@dshivashankar1994

dshivashankar1994 commented Mar 19, 2026

Copy link
Copy Markdown
Author

The changes looks good to me, before approving I would like to reproduce with a plain gunicorn web app (or similar) instead of the provided hand crafted one.

This didn't occur as part of Gunicorn or web apps but a generic fork‑and‑daemon execution model that is common in batch and job‑runner systems. The reproducer I shared is intentionally minimal to isolate the fork‑safety problem itself, independent of any specific framework.

Our workloads involve short‑lived parent processes that fork and immediately daemonize. As part of daemon hygiene, the child process explicitly closes all inherited file descriptors shortly after fork. This is a key trigger for the issue. The failure manifests specifically when resources initialized in the parent (HTTP sessions, background threads, and their shutdown finalizers) are inherited by the child and later cleaned up after OS‑level FD closure.

@xrmx I hope the above answers your comment ?

@dshivashankar1994 dshivashankar1994 requested a review from xrmx March 19, 2026 15:48
headers = self._session.headers.copy()
self._session.close()

self._session = requests.Session()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Session() can be customized / modified beyond just the headers.. This sort of breaks people if they are passing in a custom session to the constructor or using the _load_session_from_envvar thing I created.. Both of those features are pretty new and I doubt anyone is really using them yet so not a huge deal..

You could at leastcheck if _load_session_from_envvar returns None or not, and if doesn't use what it returns as the Session..

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could at leastcheck if _load_session_from_envvar returns None or not, and if doesn't use what it returns as the Session..

Are you suggesting to use the Session returned by _load_session_from_envvar as is in the forked process ? I believe this is not correct as both parent and forked process will use the same Session, which is what we are trying to avoid here

@dshivashankar1994

Copy link
Copy Markdown
Author

@xrmx Thoughts on #4995 (comment) ?

@dshivashankar1994

Copy link
Copy Markdown
Author

@MikeGoldsmith, adding you incase you'd help in reviewing

@MikeGoldsmith MikeGoldsmith left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @dshivashankar1994. I think the fix direction is right but there are a couple of things to address before we can accept:

  1. All three HTTP exporters need this fix — the trace exporter and log exporter have the same requests.Session pattern and the same fork safety issue. Can you apply the same fix to those as well?

  2. Session recreation should match original creation logic@DylanRussell's point is valid. The current fix only preserves headers, but if a custom session was passed via session= or loaded via _load_session_from_envvar, that config (adapters, auth, verify, cert, etc.) is lost. Rather than just copying headers, the fork handler should recreate the session the same way it was originally created — e.g. calling _load_session_from_envvar() again to get a new session with the same config.

  3. Missing CHANGELOG entry.

Also, @xrmx mentioned wanting to reproduce the issue before moving forward — has that been done?

@github-project-automation github-project-automation Bot moved this from Ready for review to Reviewed PRs that need fixes in Python PR digest Apr 21, 2026
@dshivashankar1994

Copy link
Copy Markdown
Author

Apologies for the delay. I've extended the changes to traces, logs too and incorporated the comments. @MikeGoldsmith Can you check now ?

Re,

Also, @xrmx mentioned wanting to reproduce the issue before moving forward — has that been done?

The issue didn't occur in web app, but in our code flow that tries to daemonize. The reproducer I have shared is exact replica of our side flow. Response in #4995

@MikeGoldsmith

Copy link
Copy Markdown
Member

Looks like we have some lint and CI failures, please can you run tox run precommit to clean them up. Thanks!

@dshivashankar1994

dshivashankar1994 commented May 10, 2026

Copy link
Copy Markdown
Author

@MikeGoldsmith

> tox -e precommit
ROOT: will run in automatically provisioned tox, host /usr/local/python/python-3.11/std/bin/python3 is missing [requires (has)]: tox-uv>=1
ROOT: provision> .tox/.tox/bin/python -m tox -e precommit
precommit: commands[0]> pre-commit run --color=always --all-files
ruff (legacy alias)......................................................Passed
ruff format..............................................................Passed
uv-lock..................................................................Passed
rstcheck.................................................................Passed
  precommit: OK (3.13=setup[0.02]+cmd[3.10] seconds)
  congratulations :) (3.35 seconds)

The lint error opentelemetry-exporter-otlp-proto-http/tests/metrics/test_otlp_metrics_exporter.py:1:0: C0302: Too many lines in module (1573/1000) (too-many-lines) seem to be an error even before my change. Do you suggest to split it logically or just move my tests outside ?

@emdneto

emdneto commented May 12, 2026

Copy link
Copy Markdown
Member

Thanks for the PR!

Just a heads-up: we no longer update CHANGELOG.md directly. The changelog is now generated from changelog fragments using Towncrier.

Please add the appropriate changelog fragment for this change instead of editing CHANGELOG.md manually. You can find the instructions and expected format in CONTRIBUTING.md.

@dshivashankar1994

Copy link
Copy Markdown
Author

Ping for #4995 (comment) @MikeGoldsmith

@linux-foundation-easycla

linux-foundation-easycla Bot commented May 19, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

@github-actions github-actions Bot added the Stale label Jun 3, 2026
@dshivashankar1994

Copy link
Copy Markdown
Author

@MikeGoldsmith Let know if you need anything from my side.

@github-actions github-actions Bot removed the Stale label Jun 4, 2026
@github-actions

Copy link
Copy Markdown

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

@github-actions github-actions Bot added the Stale label Jun 19, 2026
@dshivashankar1994

Copy link
Copy Markdown
Author

Ping @MikeGoldsmith for above

@github-actions github-actions Bot removed the Stale label Jun 25, 2026
@xrmx xrmx moved this from Reviewed PRs that need fixes to Ready for review in Python PR digest Jul 6, 2026
@dshivashankar1994

Copy link
Copy Markdown
Author

@MikeGoldsmith Let know if you need anything from my side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Ready for review

Development

Successfully merging this pull request may close these issues.

Fork-Safety Issue Causing EBADF and Telemetry Data Loss

7 participants