aya: add multi-uprobe attach support#1417
Conversation
✅ Deploy Preview for aya-rs-docs ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
374eba6 to
bd05ab8
Compare
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 8 of 13 files at r1.
Reviewable status: 8 of 14 files reviewed, 3 unresolved discussions (waiting on @swananan)
test/integration-test/src/tests/uprobe_multi.rs line 34 at r1 (raw file):
Some(map) => map, None => { eprintln!("skipping test because RING_BUF map is unavailable");
how can this happen? if it can't, please fail the test on it.
test/integration-ebpf/src/uprobe_multi.rs line 21 at r1 (raw file):
let cookie = unsafe { helpers::bpf_get_attach_cookie(ctx.as_ptr()) }; let cookie_bytes = cookie.to_le_bytes(); let _ = RING_BUF.output::<[u8]>(cookie_bytes, 0);
let _res to match existing style plz
aya/src/programs/uprobe.rs line 198 at r1 (raw file):
/// cookies; the list must be non-empty or the call fails with /// [`UProbeError::InvalidLocations`]. pub fn attach_multi(
does this need to be a separate function? The current attach already has an abstraction we can use (UProbeAttachLocation). We can change that type to allow it to model multiple attachment and implement conversion (From or Into) such that a user could pass a list to indicate multiple attachment. WDYT?
|
@tamird I used a separate That said, these are just instincts from a Rust newbie (I’ve been around Rust for a while now, but I still feel like a newbie 😂). Yeah, I’ve noticed the current If we go with a single |
|
Bundling the two makes sense to me. Perhaps that can happen in a standalone PR ahead of this one. |
This follows the aya-rs#1417 review discussion: by bundling location + cookie into a UProbeAttachPoint we get a more idiomatic Into<_> entry point, keep the one-to-one relationship enforced by the type system, and make it easier to extend attach with multi-location support without introducing parallel arrays or a brand new API.
This follows the aya-rs#1417 review discussion: by bundling location + cookie into a UProbeAttachPoint we get a more idiomatic Into<_> entry point, keep the one-to-one relationship enforced by the type system, and make it easier to extend attach with multi-location support without introducing parallel arrays or a brand new API.
This follows the aya-rs#1417 review discussion: by bundling location + cookie into a UProbeAttachPoint we get a more idiomatic Into<_> entry point, keep the one-to-one relationship enforced by the type system, and make it easier to extend attach with multi-location support without introducing parallel arrays or a brand new API.
This follows the #1417 review discussion: by bundling location + cookie into a UProbeAttachPoint we get a more idiomatic Into<_> entry point, keep the one-to-one relationship enforced by the type system, and make it easier to extend attach with multi-location support without introducing parallel arrays or a brand new API.
b31163b to
e36164d
Compare
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed 3 files and all commit messages, and made 9 comments.
Reviewable status: 9 of 25 files reviewed, 12 unresolved discussions (waiting on @swananan).
aya/src/bpf.rs line 466 at r3 (raw file):
ProgramSection::KRetProbe | ProgramSection::KProbe | ProgramSection::UProbe { .. }
can you please leave this exhaustive as it was before?
test/integration-test/src/tests/uprobe_multi.rs line 15 at r3 (raw file):
const PROG_A: &str = "uprobe_multi_trigger_program_a"; const PROG_B: &str = "uprobe_multi_trigger_program_b"; const PROG_NO_COOKIE: &str = "uprobe_multi_trigger_program_no_cookie";
are these constants useful? each is only used once
Code quote:
const PROG_A: &str = "uprobe_multi_trigger_program_a";
const PROG_B: &str = "uprobe_multi_trigger_program_b";
const PROG_NO_COOKIE: &str = "uprobe_multi_trigger_program_no_cookie";test/integration-test/src/tests/uprobe_multi.rs line 26 at r3 (raw file):
return; } const RING_BUF_BYTE_SIZE: u32 = 512;
comment for magic number?
test/integration-test/src/tests/uprobe_multi.rs line 72 at r3 (raw file):
prog.detach(link_id).unwrap(); // Fire another probe to ensure detach removed all active links.
please rephrase: this won't fire a probe
test/integration-test/src/tests/uprobe_multi.rs line 119 at r3 (raw file):
aya::programs::uprobe::UProbeError::UProbeMultiNotSupported, )) => { eprintln!("skipping test on kernel {kernel_version:?}, uprobe_multi not supported");
do we need this and the version-related skip above? it would be better to remove all the version-related skips and instead encode the expected behavior on old kernels. in other words we should expect this failure and then assert that the kernel version is indeed older than 6.6
test/integration-test/src/tests/uprobe_multi.rs line 123 at r3 (raw file):
} Err(err) => panic!("attach returned unexpected error: {err:?}"), Ok(_) => panic!("attach succeeded for non-multi program"),
can you please ascribe the type here instead of _?
test/integration-test/src/tests/uprobe_multi.rs line 160 at r3 (raw file):
#[unsafe(no_mangle)] #[inline(never)] extern "C" fn uprobe_multi_trigger_program_a(arg: u64) {
do we need these functions to take arguments? our probes don't use them. so: either remove the arguments or make the probes use them, please
test/integration-test/src/tests/array.rs line 58 at r3 (raw file):
let prog: &mut UProbe = ebpf.program_mut(prog_name).unwrap().try_into().unwrap(); prog.load().unwrap(); prog.attach([symbol], "/proc/self/exe", None).unwrap();
is it necessary for this to change? can't we allow the single-attach API to remain the same?
aya/src/programs/uprobe.rs line 139 at r3 (raw file):
/// in one syscall. Passing more than one attach point with a non-multi program returns /// [`UProbeError::ProgramNotMulti`]. Kernels without multi-uprobe support return /// [`UProbeError::UProbeMultiNotSupported`].
this is interesting. instead of adding this into UProbe as runtime data, should we make this a compile-time thing? it would require the user to know at load-time whether or not the program supports multiple attachment. WDYT?
a8f3191 to
2ef4921
Compare
|
@tamird Thanks for the review—I’ve addressed the comments, with a few notes:
|
2ef4921 to
d441b73
Compare
|
Sorry for the delay.
Yes, we cannot have compile-time safety regarding whether the BPF was built with multi attach or not. But we can have compile time safety about how it is used. In other words, the API I am envisioning forces the user to write down whether they expect multi attach or not, and only then can they use the multi attachment APIs. |
For the new uprobe attach API, do you want a simple "enum-wrapped spec" (e.g., Or should we go with a type-state design (e.g.,
|
|
The type-state design is what I was imagining. |
e3fd6e3 to
b3fbabc
Compare
|
@tamird I added the new expect_* API, giving users the option to customize this behavior, could you review these changes? thanks |
tamird
left a comment
There was a problem hiding this comment.
@tamird reviewed all commit messages and made 1 comment.
Reviewable status: 7 of 26 files reviewed, 13 unresolved discussions (waiting on @swananan).
aya/src/programs/uprobe.rs line 250 at r6 (raw file):
/// `#[uprobe(multi)]`. Programs restored from pin/info do not encode multi metadata; in that /// case we allow the conversion but log at debug. pub fn expect_single(&mut self) -> Result<&mut UProbe<Single>, ProgramError> {
these methods are sort of reinventing enums. what if instead of this we turned UProbe into an enum? Then the user can do this unpacking themselves.
I see expect_* APIs as giving callers an explicit choice between single and multi, switching to an enum returned by the loader/pin would mean the mode is decided for them (and Unspecified when coming from pin/info), so the user lose that choice. If I’m misunderstanding your idea, could you show how you’d model the API with an enum so the user can still pick the mode they want. |
|
Thanks for working on this! API wise (edit: in the aya crate, the API changes in -ebpf make sense ofc), do we even need to surface whether something is legacy uprobe or uprobe.multi? We already have the link abstraction, that seems like all we need to expose? When calling attach_multi() on a regular single uprobe, we could either create a new link which is a list of inner links, or just fail the operation at runtime. Not clear to me that complicating the API brings any benefits? |
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink, KProbeLink and TracePointLink. Consolidate all TryFrm<FdLink> implementations into an impl_try_from_fdlink! macro. Strengthen pin_lifecycle tests to verify the FdLink to link conversion, covering this previously untested path. Found while working on aya-rs#1417.
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink, KProbeLink and TracePointLink. Consolidate all TryFrom<FdLink> implementations into an impl_try_from_fdlink! macro. Strengthen pin_lifecycle tests to verify the FdLink to link conversion, covering this previously untested path. Found while working on aya-rs#1417.
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink, KProbeLink and TracePointLink. Consolidate all TryFrom<FdLink> implementations into an impl_try_from_fdlink! macro. Strengthen pin_lifecycle tests to verify the FdLink to link conversion, covering this previously untested path. Found while working on #1417.
alessandrod
left a comment
There was a problem hiding this comment.
thanks! almost there
| point: Point, | ||
| points: P, | ||
| target: T, | ||
| pid: Option<u32>, |
There was a problem hiding this comment.
now as far as I understand it
Some(0) with single attach => attach to current process
Some(0) with multi attach => attach to any process
Obvs this isn't good. I think we should special case Some(0) and transform it to
Some(current_pid)?
There was a problem hiding this comment.
There was a problem hiding this comment.
Thanks for the reminder, I realized this was real issue here.
Also, passing 0 to the previous uprobe API was not properly supported, so I split that fix out into a separate PR: #1543
With that in place, this PR now keeps the semantics explicit: CallingProcess uses 0 only for the legacy single attach path, and uses the real current pid for uprobe_multi.
| additional: u64, | ||
| } | ||
|
|
||
| fn resolve_symbol_requests( |
There was a problem hiding this comment.
this whole code is peak LLM 😆
build temporaries of temporaries of temporaries then finally bottom up.
Doubtful that we need this separate step and SymbolRequests at all. Seems like
we can do the work in resolve_multi_locations?
There was a problem hiding this comment.
Fair, the version did start from the LLM-generated code. But I had tried to clean it up, I was trying to preserve the existing symbol-resolution behavior where the object file is mapped once and all pending symbols are resolved against that mapping. That pushed me toward collecting requests first, but the implementation ended up with too many temporary layers and made the control flow hard to follow.
I reworked this part more thoroughly now. The goal is still to avoid mapping the object file once per symbol: we collect the points, defer only the symbols that need object lookup, resolve those against a single object mapping, and then apply the additional offsets.
There is still an unavoidable small request-collection step here: we need to know all symbol-based attach points before resolving them, so one object scan can satisfy all pending symbol lookups.
421c105 to
ab81443
Compare
swananan
left a comment
There was a problem hiding this comment.
@swananan made 3 comments and resolved 1 discussion.
Reviewable status: 1 of 42 files reviewed, 14 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).
aya/src/programs/links.rs line 589 at r20 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
Ah, I see now that this form is only used once. Maybe you'd reconsider and just inline this impl of TryFrom now?
This is not UProbe-specific anymore. About five link wrappers now need this same conversion, so I kept it in the macro instead of inlining the UProbe impl.
I also changed the macro arm to method = into_fd_link, which removes the unnecessary $inner parameter.
impl_try_into_fdlink!(IterLink, method = into_fd_link);
impl_try_into_fdlink!(KProbeLink, method = into_fd_link);
impl_try_into_fdlink!(PerfEventLink, method = into_fd_link);
impl_try_into_fdlink!(TracePointLink, method = into_fd_link);
impl_try_into_fdlink!(UProbeLink, method = into_fd_link);
aya/src/programs/perf_attach.rs line 183 at r30 (raw file):
Previously, tamird (Tamir Duberstein) wrote…
this function has a single caller and its logic is basically trivial. i understand the LLM extracted it for testing, but are those tests carrying their weight?
I reworked this area instead of keeping collect_link_detach_errors.
The previous version had an awkward layer issue: perf_attach was collecting the errors but returned a Uprobe-specifical detach error, since Kprobe can also support the multi attach/detach in the future. So I moved this to a generic PerfLinkDetachError owned by perf_attach.
And agreed on the test concern. I’ll make sure future tests carry their weight and focus on meaningful behavior.
| /// - a `Vec<_>` or array of either of the above. | ||
| /// | ||
| /// For other iterator-like inputs, collect into `Vec<_>` first. | ||
| pub struct UProbeAttachPoints<'a>(Vec<UProbeAttachPoint<'a>>); |
There was a problem hiding this comment.
Thanks for the reminder, I was initially hesitant because this loses the convenience of passing a bare u64 or &str for a single attach point, but avoiding unnecessary Vec allocation and keeping the API unsurprising is more important.
The current IntoIterator<Item = Into<UProbeAttachPoint>> API lets callers pass arrays/slices directly, with single-point attaches written explicitly as ["symbol"] or [offset].
|
Sorry this PR has been open for a while, I realize that makes the review more expensive. Here is a quick reset of where things stand. To keep this PR focused, I split out several related changes first:
There is still one split-out PR waiting for review from @alessandrod : #1547. If merged, I will rebase this change. |
358d7c2 to
d9d703e
Compare
|
Reworked this a bit: updated the commit message to explain the motivation, and adjusted the integration tests to cover unknown-mode fallback using a regular uprobe program in the uprobe_multi test object. |
Opening many uprobes through perf_event_open is slow because each attach point requires its own perf event and link setup. Kernels with uprobe_multi (Linux 6.6+) can attach one BPF program to many user-space locations with a single BPF link, which makes broad uprobe attachment much cheaper. Allow UProbe::attach to attach one program to multiple uprobe locations. Use uprobe_multi links for programs loaded from uprobe.multi sections. For handles reconstructed from program info or pinned programs, probe the attach mode at attach time: try uprobe_multi first, then fall back to legacy per-point perf links if the kernel does not support uprobe_multi or the loaded program was not loaded for the uprobe_multi attach type. Add per-point attach cookies, grouped perf-link detach handling, and fd-link conversion support for logical links backed by multiple perf attachments. Cover the multi path and unknown-mode fallback in integration tests. Fixes aya-rs#992.
This PR tackles #992 and focuses on multi-attach uprobes.
Main talking points:
Refactors symbol resolution to batch lookups over a single object::File mmap, since repeated mmap/parse per location was too expensive for multi-attach workloads. I haven’t split out a dedicated symbol resolver yet because that felt heavier, but I’m open to doing this.
Introduces
#[uprobe(multi)]expectations, add attach-type checks, and emit friendlyProgramNotMulti/UProbeMultiNotSupportederrors. To keepcompiled_for_multiin sync,UProbecan no longer rely on the sharedimpl_from_prog_info!macro, so it now hand-rollsfrom_program_info(open to suggestions for a more reusable pattern).This change is