Skip to content

aya: add multi-uprobe attach support#1417

Open
swananan wants to merge 1 commit intoaya-rs:mainfrom
swananan:feature/uprobe-multi-attach
Open

aya: add multi-uprobe attach support#1417
swananan wants to merge 1 commit intoaya-rs:mainfrom
swananan:feature/uprobe-multi-attach

Conversation

@swananan
Copy link
Copy Markdown
Contributor

@swananan swananan commented Dec 15, 2025

This PR tackles #992 and focuses on multi-attach uprobes.

Main talking points:

  1. Refactors symbol resolution to batch lookups over a single object::File mmap, since repeated mmap/parse per location was too expensive for multi-attach workloads. I haven’t split out a dedicated symbol resolver yet because that felt heavier, but I’m open to doing this.

  2. Introduces #[uprobe(multi)] expectations, add attach-type checks, and emit friendly ProgramNotMulti/UProbeMultiNotSupported errors. To keep compiled_for_multi in sync, UProbe can no longer rely on the shared impl_from_prog_info! macro, so it now hand-rolls from_program_info (open to suggestions for a more reusable pattern).


This change is Reviewable

@netlify
Copy link
Copy Markdown

netlify Bot commented Dec 15, 2025

Deploy Preview for aya-rs-docs ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit ebce817
🔍 Latest deploy log https://app.netlify.com/projects/aya-rs-docs/deploys/69f6c7310c424a0008d05588
😎 Deploy Preview https://deploy-preview-1417--aya-rs-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@swananan swananan force-pushed the feature/uprobe-multi-attach branch 3 times, most recently from 374eba6 to bd05ab8 Compare December 15, 2025 16:33
Copy link
Copy Markdown
Member

@tamird tamird left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tamird reviewed 8 of 13 files at r1.
Reviewable status: 8 of 14 files reviewed, 3 unresolved discussions (waiting on @swananan)


test/integration-test/src/tests/uprobe_multi.rs line 34 at r1 (raw file):

        Some(map) => map,
        None => {
            eprintln!("skipping test because RING_BUF map is unavailable");

how can this happen? if it can't, please fail the test on it.


test/integration-ebpf/src/uprobe_multi.rs line 21 at r1 (raw file):

    let cookie = unsafe { helpers::bpf_get_attach_cookie(ctx.as_ptr()) };
    let cookie_bytes = cookie.to_le_bytes();
    let _ = RING_BUF.output::<[u8]>(cookie_bytes, 0);

let _res to match existing style plz


aya/src/programs/uprobe.rs line 198 at r1 (raw file):

    ///   cookies; the list must be non-empty or the call fails with
    ///   [`UProbeError::InvalidLocations`].
    pub fn attach_multi(

does this need to be a separate function? The current attach already has an abstraction we can use (UProbeAttachLocation). We can change that type to allow it to model multiple attachment and implement conversion (From or Into) such that a user could pass a list to indicate multiple attachment. WDYT?

@swananan
Copy link
Copy Markdown
Contributor Author

@tamird I used a separate attach_multi for two reasons. First, the uprobe multi lets every location carry its own cookie, I wanted the type system to enforce that one-to-one relationship instead of juggling parallel arrays. Bundling location + cookie into the existing attach signature felt like churn—people upgrading Aya would have to touch their code even if they never use uprobe multi. Second, having a dedicated entry point keeps the docs straightforward and makes it harder to miss details like “your eBPF program must use #[uprobe(multi)]”. Even with good error messages that requirement can be surprising.

That said, these are just instincts from a Rust newbie (I’ve been around Rust for a while now, but I still feel like a newbie 😂). Yeah, I’ve noticed the current attach_multi API isn’t very idiomatic Rust either.

If we go with a single attach interface, I’d prefer to bundle the location and cookie together so we keep that one-one relationship. If that approach sounds ok to you, I'm happy to refactor it.

@tamird
Copy link
Copy Markdown
Member

tamird commented Dec 16, 2025

Bundling the two makes sense to me. Perhaps that can happen in a standalone PR ahead of this one.

swananan added a commit to swananan/aya that referenced this pull request Dec 17, 2025
This follows the aya-rs#1417 review discussion: by bundling location
+ cookie into a UProbeAttachPoint we get a more idiomatic Into<_>
entry point, keep the one-to-one relationship enforced by the type
system, and make it easier to extend attach with multi-location
support without introducing parallel arrays or a brand new API.
swananan added a commit to swananan/aya that referenced this pull request Dec 23, 2025
This follows the aya-rs#1417 review discussion: by bundling location
+ cookie into a UProbeAttachPoint we get a more idiomatic Into<_>
entry point, keep the one-to-one relationship enforced by the type
system, and make it easier to extend attach with multi-location
support without introducing parallel arrays or a brand new API.
swananan added a commit to swananan/aya that referenced this pull request Dec 23, 2025
This follows the aya-rs#1417 review discussion: by bundling location
+ cookie into a UProbeAttachPoint we get a more idiomatic Into<_>
entry point, keep the one-to-one relationship enforced by the type
system, and make it easier to extend attach with multi-location
support without introducing parallel arrays or a brand new API.
tamird pushed a commit that referenced this pull request Dec 23, 2025
This follows the #1417 review discussion: by bundling location
+ cookie into a UProbeAttachPoint we get a more idiomatic Into<_>
entry point, keep the one-to-one relationship enforced by the type
system, and make it easier to extend attach with multi-location
support without introducing parallel arrays or a brand new API.
@swananan swananan force-pushed the feature/uprobe-multi-attach branch 2 times, most recently from b31163b to e36164d Compare December 25, 2025 16:00
Copy link
Copy Markdown
Member

@tamird tamird left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tamird reviewed 3 files and all commit messages, and made 9 comments.
Reviewable status: 9 of 25 files reviewed, 12 unresolved discussions (waiting on @swananan).


aya/src/bpf.rs line 466 at r3 (raw file):

                                ProgramSection::KRetProbe
                                | ProgramSection::KProbe
                                | ProgramSection::UProbe { .. }

can you please leave this exhaustive as it was before?


test/integration-test/src/tests/uprobe_multi.rs line 15 at r3 (raw file):

const PROG_A: &str = "uprobe_multi_trigger_program_a";
const PROG_B: &str = "uprobe_multi_trigger_program_b";
const PROG_NO_COOKIE: &str = "uprobe_multi_trigger_program_no_cookie";

are these constants useful? each is only used once

Code quote:

const PROG_A: &str = "uprobe_multi_trigger_program_a";
const PROG_B: &str = "uprobe_multi_trigger_program_b";
const PROG_NO_COOKIE: &str = "uprobe_multi_trigger_program_no_cookie";

test/integration-test/src/tests/uprobe_multi.rs line 26 at r3 (raw file):

        return;
    }
    const RING_BUF_BYTE_SIZE: u32 = 512;

comment for magic number?


test/integration-test/src/tests/uprobe_multi.rs line 72 at r3 (raw file):

    prog.detach(link_id).unwrap();
    // Fire another probe to ensure detach removed all active links.

please rephrase: this won't fire a probe


test/integration-test/src/tests/uprobe_multi.rs line 119 at r3 (raw file):

            aya::programs::uprobe::UProbeError::UProbeMultiNotSupported,
        )) => {
            eprintln!("skipping test on kernel {kernel_version:?}, uprobe_multi not supported");

do we need this and the version-related skip above? it would be better to remove all the version-related skips and instead encode the expected behavior on old kernels. in other words we should expect this failure and then assert that the kernel version is indeed older than 6.6


test/integration-test/src/tests/uprobe_multi.rs line 123 at r3 (raw file):

        }
        Err(err) => panic!("attach returned unexpected error: {err:?}"),
        Ok(_) => panic!("attach succeeded for non-multi program"),

can you please ascribe the type here instead of _?


test/integration-test/src/tests/uprobe_multi.rs line 160 at r3 (raw file):

#[unsafe(no_mangle)]
#[inline(never)]
extern "C" fn uprobe_multi_trigger_program_a(arg: u64) {

do we need these functions to take arguments? our probes don't use them. so: either remove the arguments or make the probes use them, please


test/integration-test/src/tests/array.rs line 58 at r3 (raw file):

            let prog: &mut UProbe = ebpf.program_mut(prog_name).unwrap().try_into().unwrap();
            prog.load().unwrap();
            prog.attach([symbol], "/proc/self/exe", None).unwrap();

is it necessary for this to change? can't we allow the single-attach API to remain the same?


aya/src/programs/uprobe.rs line 139 at r3 (raw file):

    /// in one syscall. Passing more than one attach point with a non-multi program returns
    /// [`UProbeError::ProgramNotMulti`]. Kernels without multi-uprobe support return
    /// [`UProbeError::UProbeMultiNotSupported`].

this is interesting. instead of adding this into UProbe as runtime data, should we make this a compile-time thing? it would require the user to know at load-time whether or not the program supports multiple attachment. WDYT?

@swananan swananan force-pushed the feature/uprobe-multi-attach branch 4 times, most recently from a8f3191 to 2ef4921 Compare December 29, 2025 07:35
@swananan
Copy link
Copy Markdown
Contributor Author

@tamird Thanks for the review—I’ve addressed the comments, with a few notes:

  1. uprobe_multi.rs has three tests sharing PROG_A and other constants, so I kept those constants at module scope.
  2. Compile-time errors would be nice, but UProbeMultiNotSupported can only be detected at runtime: the build host’s kernel may differ from the runtime kernel. Same for ProgramNotMulti: whether the BPF object was built with multi is only known when the program is loaded, and pinned programs make compile-time detection impossible. I’m not aware of a way to make these compile-time guarantees.

@swananan swananan force-pushed the feature/uprobe-multi-attach branch from 2ef4921 to d441b73 Compare December 29, 2025 07:57
@tamird
Copy link
Copy Markdown
Member

tamird commented Jan 1, 2026

Sorry for the delay.

@tamird Thanks for the review—I’ve addressed the comments, with a few notes:

  1. uprobe_multi.rs has three tests sharing PROG_A and other constants, so I kept those constants at module scope.
  2. Compile-time errors would be nice, but UProbeMultiNotSupported can only be detected at runtime: the build host’s kernel may differ from the runtime kernel. Same for ProgramNotMulti: whether the BPF object was built with multi is only known when the program is loaded, and pinned programs make compile-time detection impossible. I’m not aware of a way to make these compile-time guarantees.

Yes, we cannot have compile-time safety regarding whether the BPF was built with multi attach or not. But we can have compile time safety about how it is used. In other words, the API I am envisioning forces the user to write down whether they expect multi attach or not, and only then can they use the multi attachment APIs.

@swananan
Copy link
Copy Markdown
Contributor Author

swananan commented Jan 1, 2026

Sorry for the delay.

@tamird Thanks for the review—I’ve addressed the comments, with a few notes:

  1. uprobe_multi.rs has three tests sharing PROG_A and other constants, so I kept those constants at module scope.
  2. Compile-time errors would be nice, but UProbeMultiNotSupported can only be detected at runtime: the build host’s kernel may differ from the runtime kernel. Same for ProgramNotMulti: whether the BPF object was built with multi is only known when the program is loaded, and pinned programs make compile-time detection impossible. I’m not aware of a way to make these compile-time guarantees.

Yes, we cannot have compile-time safety regarding whether the BPF was built with multi attach or not. But we can have compile time safety about how it is used. In other words, the API I am envisioning forces the user to write down whether they expect multi attach or not, and only then can they use the multi attachment APIs.

For the new uprobe attach API, do you want a simple "enum-wrapped spec" (e.g., AttachSpec::Single(UProbeAttachPoint) or AttachSpec::Multi(Vec<UProbeAttachPoint>) ), so the caller explicitly picks single vs multi per call.

Or should we go with a type-state design (e.g., UProbe<Single> / UProbe<Multi>) that enforces the same mode across the probe’s lifecycle and only exposes the matching APIs?

MultiCompileState::Unknown => points.len() > 1,
I suddenly realized that the previous API design was indeed somewhat unreasonable, especially in this case. When the eBPF program’s attach type is uncertain, we shouldn’t make a hard choice in the logic based solely on the number of attachment points, because a program with a multi attach type may still need to attach to only a single point. This decision should be left to the user.

@tamird
Copy link
Copy Markdown
Member

tamird commented Jan 3, 2026

The type-state design is what I was imagining.

@swananan swananan force-pushed the feature/uprobe-multi-attach branch 2 times, most recently from e3fd6e3 to b3fbabc Compare January 10, 2026 05:49
@swananan
Copy link
Copy Markdown
Contributor Author

@tamird I added the new expect_* API, giving users the option to customize this behavior, could you review these changes? thanks

Copy link
Copy Markdown
Member

@tamird tamird left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tamird reviewed all commit messages and made 1 comment.
Reviewable status: 7 of 26 files reviewed, 13 unresolved discussions (waiting on @swananan).


aya/src/programs/uprobe.rs line 250 at r6 (raw file):

    /// `#[uprobe(multi)]`. Programs restored from pin/info do not encode multi metadata; in that
    /// case we allow the conversion but log at debug.
    pub fn expect_single(&mut self) -> Result<&mut UProbe<Single>, ProgramError> {

these methods are sort of reinventing enums. what if instead of this we turned UProbe into an enum? Then the user can do this unpacking themselves.

@swananan
Copy link
Copy Markdown
Contributor Author

these methods are sort of reinventing enums. what if instead of this we turned UProbe into an enum? Then the user can do this unpacking themselves.

I see expect_* APIs as giving callers an explicit choice between single and multi, switching to an enum returned by the loader/pin would mean the mode is decided for them (and Unspecified when coming from pin/info), so the user lose that choice. If I’m misunderstanding your idea, could you show how you’d model the API with an enum so the user can still pick the mode they want.

@alessandrod
Copy link
Copy Markdown
Collaborator

alessandrod commented Feb 1, 2026

Thanks for working on this!

API wise (edit: in the aya crate, the API changes in -ebpf make sense ofc), do we even need to surface whether something is legacy uprobe or uprobe.multi? We already have the link abstraction, that seems like all we need to expose?

When calling attach_multi() on a regular single uprobe, we could either create a new link which is a list of inner links, or just fail the operation at runtime.

Not clear to me that complicating the API brings any benefits?

swananan added a commit to swananan/aya that referenced this pull request Apr 14, 2026
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink,
KProbeLink and TracePointLink. Consolidate all TryFrm<FdLink>
implementations into an impl_try_from_fdlink! macro.

Strengthen pin_lifecycle tests to verify the FdLink to link
conversion, covering this previously untested path.

Found while working on aya-rs#1417.
swananan added a commit to swananan/aya that referenced this pull request Apr 14, 2026
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink,
KProbeLink and TracePointLink. Consolidate all TryFrom<FdLink>
implementations into an impl_try_from_fdlink! macro.

Strengthen pin_lifecycle tests to verify the FdLink to link
conversion, covering this previously untested path.

Found while working on aya-rs#1417.
tamird pushed a commit that referenced this pull request Apr 14, 2026
Fix wrong bpf_link_type checks in TryFrom<FdLink> for UProbeLink,
KProbeLink and TracePointLink. Consolidate all TryFrom<FdLink>
implementations into an impl_try_from_fdlink! macro.

Strengthen pin_lifecycle tests to verify the FdLink to link
conversion, covering this previously untested path.

Found while working on #1417.
Copy link
Copy Markdown
Collaborator

@alessandrod alessandrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! almost there

Comment thread aya/src/programs/uprobe.rs Outdated
point: Point,
points: P,
target: T,
pid: Option<u32>,
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now as far as I understand it

Some(0) with single attach => attach to current process
Some(0) with multi attach => attach to any process

Obvs this isn't good. I think we should special case Some(0) and transform it to
Some(current_pid)?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder, I realized this was real issue here.

Also, passing 0 to the previous uprobe API was not properly supported, so I split that fix out into a separate PR: #1543

With that in place, this PR now keeps the semantics explicit: CallingProcess uses 0 only for the legacy single attach path, and uses the real current pid for uprobe_multi.

Comment thread aya/src/programs/uprobe.rs Outdated
additional: u64,
}

fn resolve_symbol_requests(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this whole code is peak LLM 😆

build temporaries of temporaries of temporaries then finally bottom up.

Doubtful that we need this separate step and SymbolRequests at all. Seems like
we can do the work in resolve_multi_locations?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, the version did start from the LLM-generated code. But I had tried to clean it up, I was trying to preserve the existing symbol-resolution behavior where the object file is mapped once and all pending symbols are resolved against that mapping. That pushed me toward collecting requests first, but the implementation ended up with too many temporary layers and made the control flow hard to follow.

I reworked this part more thoroughly now. The goal is still to avoid mapping the object file once per symbol: we collect the points, defer only the symbols that need object lookup, resolve those against a single object mapping, and then apply the additional offsets.

There is still an unavoidable small request-collection step here: we need to know all symbol-based attach points before resolving them, so one object scan can satisfy all pending symbol lookups.

@swananan swananan force-pushed the feature/uprobe-multi-attach branch 12 times, most recently from 421c105 to ab81443 Compare April 30, 2026 15:25
Copy link
Copy Markdown
Contributor Author

@swananan swananan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@swananan made 3 comments and resolved 1 discussion.
Reviewable status: 1 of 42 files reviewed, 14 unresolved discussions (waiting on alessandrod, tamird, and vadorovsky).


aya/src/programs/links.rs line 589 at r20 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

Ah, I see now that this form is only used once. Maybe you'd reconsider and just inline this impl of TryFrom now?

This is not UProbe-specific anymore. About five link wrappers now need this same conversion, so I kept it in the macro instead of inlining the UProbe impl.

I also changed the macro arm to method = into_fd_link, which removes the unnecessary $inner parameter.

  impl_try_into_fdlink!(IterLink, method = into_fd_link);
  impl_try_into_fdlink!(KProbeLink, method = into_fd_link);
  impl_try_into_fdlink!(PerfEventLink, method = into_fd_link);
  impl_try_into_fdlink!(TracePointLink, method = into_fd_link);
  impl_try_into_fdlink!(UProbeLink, method = into_fd_link);

aya/src/programs/perf_attach.rs line 183 at r30 (raw file):

Previously, tamird (Tamir Duberstein) wrote…

this function has a single caller and its logic is basically trivial. i understand the LLM extracted it for testing, but are those tests carrying their weight?

I reworked this area instead of keeping collect_link_detach_errors.

The previous version had an awkward layer issue: perf_attach was collecting the errors but returned a Uprobe-specifical detach error, since Kprobe can also support the multi attach/detach in the future. So I moved this to a generic PerfLinkDetachError owned by perf_attach.

And agreed on the test concern. I’ll make sure future tests carry their weight and focus on meaningful behavior.

Comment thread aya/src/programs/uprobe.rs Outdated
/// - a `Vec<_>` or array of either of the above.
///
/// For other iterator-like inputs, collect into `Vec<_>` first.
pub struct UProbeAttachPoints<'a>(Vec<UProbeAttachPoint<'a>>);
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reminder, I was initially hesitant because this loses the convenience of passing a bare u64 or &str for a single attach point, but avoiding unnecessary Vec allocation and keeping the API unsurprising is more important.

The current IntoIterator<Item = Into<UProbeAttachPoint>> API lets callers pass arrays/slices directly, with single-point attaches written explicitly as ["symbol"] or [offset].

@swananan
Copy link
Copy Markdown
Contributor Author

Sorry this PR has been open for a while, I realize that makes the review more expensive. Here is a quick reset of where things stand.

To keep this PR focused, I split out several related changes first:

There is still one split-out PR waiting for review from @alessandrod : #1547. If merged, I will rebase this change.

@swananan swananan requested review from alessandrod and tamird April 30, 2026 15:38
@swananan swananan force-pushed the feature/uprobe-multi-attach branch 5 times, most recently from 358d7c2 to d9d703e Compare May 1, 2026 02:43
@swananan
Copy link
Copy Markdown
Contributor Author

swananan commented May 1, 2026

Reworked this a bit: updated the commit message to explain the motivation, and adjusted the integration tests to cover unknown-mode fallback using a regular uprobe program in the uprobe_multi test object.

Opening many uprobes through perf_event_open is slow because each
attach point requires its own perf event and link setup. Kernels with
uprobe_multi (Linux 6.6+) can attach one BPF program to many
user-space locations with a single BPF link, which makes broad uprobe
attachment much cheaper.

Allow UProbe::attach to attach one program to multiple uprobe
locations. Use uprobe_multi links for programs loaded from
uprobe.multi sections. For handles reconstructed from program info or
pinned programs, probe the attach mode at attach time: try uprobe_multi
first, then fall back to legacy per-point perf links if the kernel does
not support uprobe_multi or the loaded program was not loaded for the
uprobe_multi attach type.

Add per-point attach cookies, grouped perf-link detach handling, and
fd-link conversion support for logical links backed by multiple perf
attachments. Cover the multi path and unknown-mode fallback in
integration tests.

Fixes aya-rs#992.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants