[feat][Ruby][meta] - Overall Ruby interpreter improvements

[dalehamel]

I have submitted a number of issues and PRs related to improving Ruby profiling, and have additional ones queued up.

I thought it would be helpful to create this meta issue to describe the deficiencies that I'm addressing, and paint a more complete picture of the desired end state.

• The largest single improvement is Mimic Ruby's backtrace logic in ruby interpreter and tracer, add support for ruby CMEs #907 which brings the ruby unwinding and symbolization algorithm more in-line with ruby's own approach used by rb_profile_frames which other ruby profilers such as stackprof and vernier use, this addresses [feat] Include class name for ruby method calls #714
  • This introduces an issue with line numbers, which I have opened Allow pushing an "extra" value in Frame in padding #931 to address so that we can push an additional value to fix it (as is done in Fix leaf lineno Shopify/opentelemetry-ebpf-profiler#4), (superseded by Convert ebpf trace struct frame to variable length data #940)
  • Building on this, we need to detect and mark GC frames so that we don't erroneously charge whatever is on the ruby stack for the GC that the ruby VM has decided to run [feat][Ruby] Support detecting GC state and handle it accordingly #936
  • Building on this, support for handling of ruby JIT frames is necessary for many production use cases as ruby jit provides a substantial speedup, but currently breaks the unwinding strategy used by the ruby tracer [feat][Ruby] Support for ruby JIT frames #937
• Ruby stores its execution context in thread local storage. We need a reliable way to read this, currently it is only supported if TLS Descriptors are used
  • feature: strategy for reading TLS when not using TLS descriptors #883 , and Extract DTV info from __tls_get_addr, add to LibcInfo #929 which partially addresses this
  • [feat] Support static bin/ruby builds for ruby interpreter #884 is still needed for static ruby (ie, no libruby.so) support, nothing implemented here yet
• There is also an issue where ruby stacks are typically a lot deeper than other runtimes, hence why I've opened Increase frame buffer to max 1024 frames per trace #908 to address Max stack depth of 128 is insufficient for a variety of cases #760 .

Demonstration branch

At Shopify we've been using a branch composed of a bunch of these fixes and a few other tweaks, it is available at Shopify#8 and should be more or less suitable for production use (assuming production is x86_64, and ruby is compiled with --enable-shared for now). We're using it internally, and probably have one of the largest and most demanding ruby stacks in the world, so I'd expect if it works for us it is sufficient for others.

Here is an example of it working with JIT, detecting GC, and appropriately charging the leaf frames to the appropriate native functions, while also providing frame labels that match what is seen in stackprof or other ruby profilers:

End to end Ruby to native to kernel:

Thanks to #907 we can see the cfunc call that initiated the native calls and it makes sense (here is the native code servicing request to look up memory usage, which is ultimately served by the kernel doing smaps_rollup)

Here is the overall ruby process with native profiling unwinding:

It looks identical to the stackprof request profiles for the same thing, albeit with more detail about the native calls:

Zooming in on GC, we can see exactly what is happening during the marking and sweeping stages:

And zooming in on the leaf nodes, we can see where time is being spent in native functions to service cfunc calls:

[dalehamel]

The path forward here i think is:

• Get Mimic Ruby's backtrace logic in ruby interpreter and tracer, add support for ruby CMEs #907 reviewed and work on the feedback for that
• Hopefully Convert ebpf trace struct frame to variable length data #940 work lands soon, and rebase Mimic Ruby's backtrace logic in ruby interpreter and tracer, add support for ruby CMEs #907 on that
• Pull in the necessary fixes from Shopify@6a58b5e to fix the line number, using the new mechanism to push the address for the cfp-iseq pointer needed
• Once Mimic Ruby's backtrace logic in ruby interpreter and tracer, add support for ruby CMEs #907 lands, work on integrating GC frames [feat][Ruby] Support detecting GC state and handle it accordingly #936 , PR is basically ready to go
• Likewise, for JIT [feat][Ruby] Support for ruby JIT frames #937 can be put up at the same time for review

And in the meantime, I'll push forward on

• [feat] Support static bin/ruby builds for ruby interpreter #884 support for profiling just bin/ruby and getting the EC from TLS on that in static mode
• Extract DTV info from __tls_get_addr, add to LibcInfo #929 to get the DTV information plumbed through to the interpreter for TLS with libruby, but no TLS descriptors
  • Once this lands, I next need to submit the code to use this offset information, currently rough cut is in Shopify@38f71f5
• New work that I promised in Mimic Ruby's backtrace logic in ruby interpreter and tracer, add support for ruby CMEs #907 to handle anonymous classes and modules when symbolizing the classpath

Once this is all done, a test like I have in Rbspy for symbolizing a backtracie snapshot stack would be good to add to the coredump test cases, to verify we can symbolize ruby to spec for the various cases covered there.

[dalehamel]

additionally, ruby 4.0 has been released. We have a patch that updates the struct offsets for this too, but need to land these other PRs first.

There are some additional things that have come up in the review #907 that we'll want to address and I don't want to forget about, so adding a comment here:

• Flags are being read as a Ptr but should be read as Uint64
• Flags should explicitly use the rbasic.flags offset, rather than implicitly reading at 0
• Feature flag for cfuncs should be added rather than static version check