[do not merge] Memory leak debugging - #103
Draft
awilfox wants to merge 1 commit into
Draft
Conversation
With this commit, you can run: `podman compose exec -e TEST_COUNT=1_000 -e PATH_TO_HIT=/health app bundle exec derailed exec perf:mem_over_time 1>out1.log 2>err1.log` and see a memory leak when parallel checks are enabled.
awilfox
commented
Aug 3, 2026
| # Uncomment if you wish to allow Action Cable access from any origin. | ||
| # config.action_cable.disable_request_forgery_protection = true | ||
| config.hosts << 'app.test' | ||
| config.hosts = [] |
Member
Author
There was a problem hiding this comment.
This was necessary to make the derailed tests work.
awilfox
commented
Aug 3, 2026
| @@ -1,5 +1,7 @@ | |||
| # Read Docker secrets into the environment | |||
| Dir['/run/secrets/*'].each do |filepath| | |||
| next unless File.file?(filepath) | |||
Member
Author
There was a problem hiding this comment.
This was so that the app ran in Podman.
awilfox
commented
Aug 3, 2026
Comment on lines
+29
to
+31
| extra_hosts: | ||
| - geoservices.lib.berkeley.edu:128.32.10.252 | ||
| - spatial.lib.berkeley.edu:128.32.10.252 |
Member
Author
There was a problem hiding this comment.
As discussed on our Zoom call, this is a workaround for the fact that Verizon's DNS server was rate limiting my resolution of these domains and eventually returning NODATA.
awilfox
commented
Aug 3, 2026
| - SOLR_URL=http://solr:8983/solr/geodata-test | ||
| - GEODATA_SPATIAL_HEALTHCHECK_URL=https://spatial.lib.berkeley.edu/public/berkeley-status/data.zip | ||
| - GEODATA_GEOSERVER_PUBLIC_HEALTHCHECK_URL=https://geoservices.lib.berkeley.edu/geoserver/wms?service=WMS&request=GetCapabilities | ||
| - GEODATA_GEOSERVER_SECURE_HEALTHCHECK_URL=https://geoservices.lib.berkeley.edu/geoserver/wms?service=WMS&request=GetCapabilities |
Member
Author
There was a problem hiding this comment.
Specifically overrode the secure URL with the public one because any error from the health check kills the memory test early.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With this commit, you can run:
podman compose exec -e TEST_COUNT=500 -e PATH_TO_HIT=/health app bundle exec derailed exec perf:mem_over_time 1>out1.log 2>err1.logand see a memory leak when parallel checks are enabled.
You can increase
TEST_COUNTif you want, but there's not a whole lot of benefit unless you think you've solved the problem and want to see if it stays fixed after a number of additional requests.Lower
TEST_COUNTisn't helpful. The first fifty requests are still warming the app, and growth is expected. At 100, you can start to see the leaked objects, but there's still a lot of noise in the retained object list. At 500, you can really see the leak is obvious.Redirection of stdout is recommended because otherwise you get a Rails firehose directed at you. Redirection of stderr is not necessary but it just suppressed this noise:
which didn't seem particularly helpful anyway.
This process creates a file in
tmp/with the format$(date -Iseconds)-$(ruby --version)-memory-$(times)-times.txt. This file contains each memory usage reading in MiB. For example, my local filetmp/2026-08-01T02:12:08+00:00-ruby 3.4.10 (2026-06-30 revision 2b0b7728dc) +PRISM [aarch64-linux]-memory-1000-times.txtlooks like:log file contents
which shows that the leak is ~stable/not happening (this example run was with parallel checks disabled)
To derive a list of retained objects (i.e. to see the actual leaked
Fiberobjects), run:podman compose exec -e TEST_COUNT=500 -e PATH_TO_HIT=/health app bundle exec derailed exec perf:objects > objectsSome basic differential analysis on running it 1/10/50 times showed the leak pretty easily:
And "retained memory by class" showed all of these were:
28440 FiberIf you need any further information, please comment/@ me, happy to help.