Fix inotify file descriptor leak during beacon refresh#68838
Open
sujitdb wants to merge 2 commits intosaltstack:3006.xfrom
Open
Fix inotify file descriptor leak during beacon refresh#68838sujitdb wants to merge 2 commits intosaltstack:3006.xfrom
sujitdb wants to merge 2 commits intosaltstack:3006.xfrom
Conversation
dwoz
requested changes
Mar 20, 2026
twangboy
previously approved these changes
Mar 20, 2026
This was
linked to
issues
Mar 20, 2026
When beacons_refresh() creates a new Beacon instance, the old instance's beacon modules (e.g. inotify) held open file descriptors that were never closed. Each new Beacon gets a fresh empty __context__ dict via LazyLoader, so inotify's _get_notifier() creates a new pyinotify.Notifier while the old one is orphaned with its fds still open. Over repeated refreshes, this exhausts the inotify instance limit (default 128 on RHEL8), causing "Too many open files (EMFILE)" errors. Add close_beacons() to the Beacon class that calls close() on each beacon module before the instance is replaced. Also add __del__() as a safety net for garbage collection, and call close_beacons() explicitly from Minion.beacons_refresh() before creating a new Beacon instance. Fixes: saltstack#66449 Fixes: saltstack#58907 Made-with: Cursor
twangboy
approved these changes
Mar 20, 2026
dwoz
approved these changes
Mar 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When beacons_refresh() creates a new Beacon instance, the old instance's beacon modules (e.g. inotify) held open file descriptors that were never closed. Each new Beacon gets a fresh empty context dict via LazyLoader, so inotify's _get_notifier() creates a new pyinotify.Notifier while the old one is orphaned with its fds still open. Over repeated refreshes, this exhausts the inotify instance limit (default 128 on RHEL8), causing "Too many open files (EMFILE)" errors.
Add close_beacons() to the Beacon class that calls close() on each beacon module before the instance is replaced.
Fixes: #66449
Fixes: #58907
What does this PR do?
Fixes inotify file descriptor leak in the beacon subsystem. When beacons are refreshed
(e.g. during
module_refresh,pillar_refresh, orsaltutil.refresh_beacons), the oldBeaconinstance's modules were never closed before being replaced, causing inotify filedescriptors to leak until the kernel limit is exhausted.
What issues does this PR fix or reference?
Fixes #66449
Fixes #58907
Root Cause
When
beacons_refresh()creates a newBeaconinstance,salt.loader.beacons()createsa new
LazyLoaderwith a fresh empty__context__ = {}. The inotify beacon's_get_notifier()checksif notifier not in __context__— since the context is empty, itcreates a new
pyinotify.Notifierwith new inotify file descriptors. The oldnotifier from the previous
Beaconinstance's__context__is never stopped, so itsinotify fds are orphaned. Over repeated refreshes, this exhausts the inotify instance limit
(default 128 on RHEL8), producing:
Unable to start beacon, Cannot initialize new instance of inotify,
Errno=Too many open files (EMFILE)
Changes
salt/beacons/__init__.py: Addedclose_beacons()method to theBeaconclass thatiterates all configured beacon modules and calls their
close()function (if one exists)to release resources.
salt/minion.py: UpdatedMinion.beacons_refresh()to explicitly callclose_beacons()on the oldBeaconinstance before replacing it with a new one.tests/pytests/unit/test_beacons.py: Added 8 unit tests coveringclose_beacons()behavior.
tests/pytests/unit/test_minion.py: Added 1 unit test verifyingbeacons_refresh()calls
close_beacons().changelog/66449.fixed.md: Added changelog entry.Tests
All new tests pass:
tests/pytests/unit/test_beacons.py::test_close_beacons_calls_close_on_modules PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_with_beacon_module_override PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_skips_modules_without_close PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_handles_close_exception PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_multiple_beacons PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_skips_enabled_key PASSED
tests/pytests/unit/test_beacons.py::test_del_calls_close_beacons PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_dict_config PASSED
tests/pytests/unit/test_minion.py::test_beacons_refresh_closes_old_beacons PASSED
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No