Skip to content

Fix inotify file descriptor leak during beacon refresh#68838

Open
sujitdb wants to merge 2 commits intosaltstack:3006.xfrom
sujitdb:fix/66449-beacon-inotify-fd-leak
Open

Fix inotify file descriptor leak during beacon refresh#68838
sujitdb wants to merge 2 commits intosaltstack:3006.xfrom
sujitdb:fix/66449-beacon-inotify-fd-leak

Conversation

@sujitdb
Copy link
Copy Markdown
Collaborator

@sujitdb sujitdb commented Mar 20, 2026

When beacons_refresh() creates a new Beacon instance, the old instance's beacon modules (e.g. inotify) held open file descriptors that were never closed. Each new Beacon gets a fresh empty context dict via LazyLoader, so inotify's _get_notifier() creates a new pyinotify.Notifier while the old one is orphaned with its fds still open. Over repeated refreshes, this exhausts the inotify instance limit (default 128 on RHEL8), causing "Too many open files (EMFILE)" errors.

Add close_beacons() to the Beacon class that calls close() on each beacon module before the instance is replaced.

Fixes: #66449
Fixes: #58907

What does this PR do?

Fixes inotify file descriptor leak in the beacon subsystem. When beacons are refreshed
(e.g. during module_refresh, pillar_refresh, or saltutil.refresh_beacons), the old
Beacon instance's modules were never closed before being replaced, causing inotify file
descriptors to leak until the kernel limit is exhausted.

What issues does this PR fix or reference?

Fixes #66449
Fixes #58907

Root Cause

When beacons_refresh() creates a new Beacon instance, salt.loader.beacons() creates
a new LazyLoader with a fresh empty __context__ = {}. The inotify beacon's
_get_notifier() checks if notifier not in __context__ — since the context is empty, it
creates a new pyinotify.Notifier with new inotify file descriptors. The old
notifier from the previous Beacon instance's __context__ is never stopped, so its
inotify fds are orphaned. Over repeated refreshes, this exhausts the inotify instance limit
(default 128 on RHEL8), producing:
Unable to start beacon, Cannot initialize new instance of inotify,
Errno=Too many open files (EMFILE)

Changes

  • salt/beacons/__init__.py: Added close_beacons() method to the Beacon class that
    iterates all configured beacon modules and calls their close() function (if one exists)
    to release resources.
  • salt/minion.py: Updated Minion.beacons_refresh() to explicitly call
    close_beacons() on the old Beacon instance before replacing it with a new one.
  • tests/pytests/unit/test_beacons.py: Added 8 unit tests covering close_beacons()
    behavior.
  • tests/pytests/unit/test_minion.py: Added 1 unit test verifying beacons_refresh()
    calls close_beacons().
  • changelog/66449.fixed.md: Added changelog entry.

Tests

All new tests pass:
tests/pytests/unit/test_beacons.py::test_close_beacons_calls_close_on_modules PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_with_beacon_module_override PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_skips_modules_without_close PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_handles_close_exception PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_multiple_beacons PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_skips_enabled_key PASSED
tests/pytests/unit/test_beacons.py::test_del_calls_close_beacons PASSED
tests/pytests/unit/test_beacons.py::test_close_beacons_dict_config PASSED
tests/pytests/unit/test_minion.py::test_beacons_refresh_closes_old_beacons PASSED

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

Yes/No

twangboy
twangboy previously approved these changes Mar 20, 2026
When beacons_refresh() creates a new Beacon instance, the old instance's
beacon modules (e.g. inotify) held open file descriptors that were never
closed. Each new Beacon gets a fresh empty __context__ dict via LazyLoader,
so inotify's _get_notifier() creates a new pyinotify.Notifier while the
old one is orphaned with its fds still open. Over repeated refreshes, this
exhausts the inotify instance limit (default 128 on RHEL8), causing
"Too many open files (EMFILE)" errors.

Add close_beacons() to the Beacon class that calls close() on each beacon
module before the instance is replaced. Also add __del__() as a safety net
for garbage collection, and call close_beacons() explicitly from
Minion.beacons_refresh() before creating a new Beacon instance.

Fixes: saltstack#66449
Fixes: saltstack#58907
Made-with: Cursor
@dwoz dwoz removed the test:full Run the full test suite label Mar 24, 2026
@dwoz dwoz added the test:full Run the full test suite label Mar 24, 2026
@dwoz dwoz removed the test:full Run the full test suite label Mar 24, 2026
@dwoz dwoz added the test:full Run the full test suite label Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:full Run the full test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Increase the number of inotify when changes are applied repeatedly [BUG] refreshing beacons leaks inotify handles

4 participants