Skip to content

MSI-X support #2101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cagatay-y wants to merge 1 commit into
base: main
Choose a base branch
from
Open

MSI-X support #2101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,28 +42,28 @@ jobs:
- run: rustup component add clippy
- name: cargo hack clippy (x86_64)
run: |
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,rtl8139 --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,rtl8139,msix --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,rtl8139 --features pci,tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,virtio-net --features tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,virtio-net --features pci,tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,virtio-net,msix --features tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target x86_64-unknown-none --exclude-features gem-net,virtio-net,msix --features pci,tcp,rtl8139
- name: cargo hack clippy (aarch64)
run: |
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,rtl8139 --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,rtl8139 --features pci,tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,virtio-net --features tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,virtio-net --features pci,tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,rtl8139,msix --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,rtl8139,msix --features pci,tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,virtio-net,msix --features tcp,rtl8139
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target aarch64-unknown-none-softfloat --exclude-features gem-net,virtio-net,msix --features pci,tcp,rtl8139
- name: cargo hack clippy (riscv64)
run: |
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features gem-net,rtl8139 --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features gem-net,rtl8139 --features pci,tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features rtl8139,virtio-net --features tcp,gem-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features rtl8139,virtio-net --features pci,tcp,gem-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features dhcpv4,dns,gem-net,net,rtl8139,virtio-net,msix --features pci
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features gem-net,rtl8139,msix --features tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features gem-net,rtl8139,msix --features pci,tcp,virtio-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features rtl8139,virtio-net,msix --features tcp,gem-net
cargo hack clippy --package hermit-kernel --each-feature --no-dev-deps --target riscv64gc-unknown-none-elf --exclude-features rtl8139,virtio-net,msix --features pci,tcp,gem-net

format:
name: Format
Expand Down
6 changes: 6 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,12 @@ pci = ["virtio?/pci"]
## [PCI ID]: https://pci-ids.ucw.cz/
pci-ids = ["dep:pci-ids"]

## Causes the drivers to use _MSI-X (Message Signaled Interrupt)_ when
## the PCI device and the driver (currently only virtio-net) supports it.
## It is only supported on x86_64 and is known to not work with all virtual
## devices.
msix = ["volatile/derive"]

## Enables semihosting support.
##
## Semihosting allows communicating with the host for
Expand Down
2 changes: 1 addition & 1 deletion src/drivers/net/virtio/mmio.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ impl VirtioNetDriver<Uninit> {
notif_cfg,
inner: Uninit,
num_vqs: 0,
irq,
irq: Some(irq),
checksums: ChecksumCapabilities::default(),
})
}
Expand Down
31 changes: 29 additions & 2 deletions src/drivers/net/virtio/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ use self::error::VirtioNetError;
use crate::config::VIRTIO_MAX_QUEUE_SIZE;
use crate::drivers::net::virtio::constants::BUFF_PER_PACKET;
use crate::drivers::net::{NetworkDriver, mtu};
#[cfg(feature = "msix")]
use crate::drivers::pci::MsixTableEntry;
use crate::drivers::virtio::ControlRegisters;
#[cfg(not(feature = "pci"))]
use crate::drivers::virtio::transport::mmio::{ComCfg, IsrStatus, NotifCfg};
Expand Down Expand Up @@ -241,11 +243,13 @@ pub(crate) struct VirtioNetDriver<T = Init> {
pub(super) com_cfg: ComCfg,
pub(super) isr_stat: IsrStatus,
pub(super) notif_cfg: NotifCfg,
#[cfg(feature = "msix")]
pub(super) msix_table: Option<VolatileRef<'static, [MsixTableEntry]>>,

pub(super) inner: T,

pub(super) num_vqs: u16,
pub(super) irq: InterruptLine,
pub(super) irq: Option<InterruptLine>,
pub(super) checksums: ChecksumCapabilities,
}

Expand Down Expand Up @@ -485,7 +489,7 @@ impl smoltcp::phy::Device for VirtioNetDriver {

impl Driver for VirtioNetDriver<Init> {
fn get_interrupt_number(&self) -> InterruptLine {
self.irq
self.irq.unwrap()
}

fn get_name(&self) -> &'static str {
Expand Down Expand Up @@ -741,11 +745,34 @@ impl VirtioNetDriver<Uninit> {
}
debug!("{:?}", self.checksums);

#[cfg(feature = "msix")]
if let Some(msix_table) = self.msix_table.as_mut() {
warn!(
"Setting up message signaled interrupts. MSI-X is known to not work on all platforms (e.g. QEMU with TAP)."
);
// Chosen arbitatrily. Ideally does not clash with another interrupt line.
const MSIX_VECTOR: u8 = 112;
let msix_entry = unsafe {
msix_table
.as_mut_ptr()
.map(|table| table.get_unchecked_mut(0))
};
MsixTableEntry::configure(msix_entry, MSIX_VECTOR);
self.com_cfg.select_vq(0).unwrap().set_msix_table_index(0);
self.irq = Some(MSIX_VECTOR);
};

if self.irq.is_none() {
warn!("No interrupt methods found for virtio-net.");
}

Ok(VirtioNetDriver {
dev_cfg: self.dev_cfg,
com_cfg: self.com_cfg,
isr_stat: self.isr_stat,
notif_cfg: self.notif_cfg,
#[cfg(feature = "msix")]
msix_table: self.msix_table,
inner,
num_vqs: self.num_vqs,
irq: self.irq,
Expand Down
8 changes: 7 additions & 1 deletion src/drivers/net/virtio/pci.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ impl VirtioNetDriver<Uninit> {
notif_cfg,
isr_cfg,
dev_cfg_list,
#[cfg(feature = "msix")]
msix_table,
..
} = caps_coll;

Expand All @@ -44,14 +46,18 @@ impl VirtioNetDriver<Uninit> {
return Err(error::VirtioNetError::NoDevCfg(device_id));
};

let irq = device.get_irq();

Ok(VirtioNetDriver {
dev_cfg,
com_cfg,
isr_stat: isr_cfg,
notif_cfg,
#[cfg(feature = "msix")]
msix_table,
inner: Uninit,
num_vqs: 0,
irq: device.get_irq().unwrap(),
irq,
checksums: ChecksumCapabilities::default(),
})
}
Expand Down
43 changes: 43 additions & 0 deletions src/drivers/pci.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,49 @@ pub(crate) fn init() {
});
}

/// MSI-X Table entry.
#[cfg(feature = "msix")]
#[repr(C)]
#[derive(volatile::VolatileFieldAccess)]
pub(crate) struct MsixTableEntry {
/// Message Address
message_address: u32,

/// Message Upper Address
message_upper_address: u32,

/// Message Data
message_data: u32,

/// Vector Control
vector_control: u32,
}
Comment thread
cagatay-y marked this conversation as resolved.
Comment thread
cagatay-y marked this conversation as resolved.
Comment on lines +542 to +556
Copy link
Copy Markdown
Member

@mkroening mkroening Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to upstream this to pci_types?

That crate currently does not use any volatile operations at all, though. Is volatile necessary here? Should pci_types use volatile for some current operations? Or does this live in the PCI config region and should be accessed via pci_types::ConfigRegionAccess instead?

Copy link
Copy Markdown
Contributor Author

@cagatay-y cagatay-y Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is part of the PCI-e specification, so I may make sense to.

Volatile is necessary and ConfigRegionAccess does not apply as it is an object that is mapped to the BAR space. I am not aware of any pci_types operations that are missing volatile accesses.


#[cfg(feature = "msix")]
impl MsixTableEntry {
#[cfg(target_arch = "x86_64")]
pub fn configure(msix_entry: volatile::VolatilePtr<'_, Self>, vector: u8) {
use MsixTableEntryVolatileFieldAccess;
use bit_field::BitField;

// Mask the entry because "[s]oftware must not modify the Address, Data, or Steering Tag fields
// of an entry while it is unmasked." (PCIe spec. 6.1.4.2)
msix_entry
.vector_control()
.update(|mut control| *control.set_bit(0, true));

msix_entry
.message_address()
.update(|mut addr_low| *addr_low.set_bits(20..32, 0xfee));
msix_entry
.message_data()
.update(|mut data| *data.set_bits(0..8, u32::from(vector) + 32));
msix_entry
.vector_control()
.update(|mut control| *control.set_bit(0, false));
}
}

/// A module containing PCI specific errors
///
/// Errors include...
Expand Down
Loading
Loading