Describe the bug
I am seeing silent and non-deterministic NVMe read corruption on a Raspberry Pi Compute Module 5 when running kernel 6.18.34+rpt-rpi-2712.
The same exact hardware, NVMe filesystem and test file work correctly when booted with kernel 6.12.75+rpt-rpi-2712.
The NVMe device remains visible and operational. There are no NVMe I/O errors, PCIe AER errors or SMART media errors. However, repeated normal buffered reads of the same unchanged file return different data.
O_DIRECT reads are much more reliable, although one incorrect full-file O_DIRECT read was also observed on 6.18.34.
This appears possibly related to #7492, but this is a different NVMe controller and a different failure mode: the PCIe/NVMe device remains present, while data read through it can be silently corrupted.
Hardware
Raspberry Pi Compute Module 5
8 GB RAM
Official Raspberry Pi Compute Module IO Board
Official Raspberry Pi SSD 512 GB
SSD manufacturer: BIWIN
NVMe controller:
0001:01:00.0 Non-Volatile memory controller [0108]:
Biwin Storage Technology Co., Ltd. KingSpec NX series NVMe SSD (DRAM-less)
[1dee:5216] (rev 01)
NVMe model:
BIWIN CE430T5D100-512G
Firmware: 1.4.7.70
PCIe link:
LnkSta: Speed 5GT/s, Width x1
AER status is clean:
UESta: all clear
CESta: all clear
LaneErrStat: 0
NVMe SMART:
critical_warning=0
media_errors=0
num_err_log_entries=0
SMART overall: PASSED
Broken configuration
Raspberry Pi OS image:
raspios_arm64-2026-06-19
Kernel:
Linux 6.18.34+rpt-rpi-2712
Working configuration
Raspberry Pi OS image:
raspios_arm64-2026-04-21
Kernel:
Linux raspberrypi 6.12.75+rpt-rpi-2712
#1 SMP PREEMPT Debian 1:6.12.75-1+rpt1
No hardware was changed between the tests.
The same NVMe filesystem and the same exact 10 GiB file were used.
Steps to reproduce the behaviour
Reproduction
A 10 GiB random file was created on the CM5 eMMC and copied normally to the NVMe.
Source file SHA-256:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
The eMMC source remains completely stable across repeated reads.
On kernel 6.18.34, repeated normal reads of the NVMe copy return different SHA-256 values each time.
Example:
915ccf90d89fdac13d6e398a437f8b374d832d6996ea90f5de80ed93853287f0
e031ed5ab170a2d41ebc26f42e2e4d176551f390bb006d2d818577c87be0063d
13044bd67830be6e07e6bad467273baf515cc88d187f5cf40b70f059e7d5b3ed
After another clean reboot on the same kernel:
e8526c2929172b08c120bb7b05e9e3fadca18362fb0a52c14958d5b05f552a55
27eb942bba7846b6fffe1a598e9ef922d2a18ff1be5ae0e09e6cc3169ea622b1
a6955d608ac6a719966403202c413a064cfacd2ec9a0e37659b5e19c88388bbf
The file inode, size, mtime and ctime remain unchanged.
Test command:
for i in 1 2 3; do
sha256sum /srv/rpi-sb-provisioner/.nvme-integrity-test/test-10g.bin
done
O_DIRECT comparison on 6.18.34
Using:
for i in 1 2 3; do
dd if=/srv/rpi-sb-provisioner/.nvme-integrity-test/test-10g.bin
bs=4M iflag=direct,fullblock status=none | sha256sum
done
Several runs produced the correct SHA every time:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
However, one clean-boot test produced one incorrect O_DIRECT read followed by two correct reads:
f1cf3488f8904d95c5bd46fa655ed77ef39e03d3fedebcc6c2c728516b538805
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
Kernel 6.12.75 control test
I then booted the same CM5 with Raspberry Pi OS 2026-04-21, kernel:
6.12.75+rpt-rpi-2712
The existing NVMe filesystem was mounted read-only:
mount -o ro,noload /dev/nvme0n1p1 /srv/rpi-sb-provisioner
The same exact 10 GiB file was tested without rewriting it.
Three buffered reads:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
Three O_DIRECT reads:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
So all six full 10 GiB reads return the correct data on kernel 6.12.75.
Other checks
Power is not currently an issue:
vcgencmd get_throttled
throttled=0x0
The corruption was reproduced with temperatures around 60-66°C without throttling.
Kernel logs show no:
NVMe reset
NVMe timeout
PCIe AER error
I/O error
EXT4 error
Limiting RAM to 2 GiB with:
mem=2G
does not fix the issue on kernel 6.18.34.
With mem=2G:
eMMC reads remain stable
NVMe buffered reads remain non-deterministic
O_DIRECT reads return the correct file SHA
Summary
Same hardware, same NVMe, same filesystem and same exact file:
6.12.75:
buffered reads PASS
O_DIRECT reads PASS
6.18.34:
buffered reads FAIL / non-deterministic silent corruption
O_DIRECT reads mostly correct, but at least one incorrect full-file read observed
No PCIe AER, NVMe SMART or kernel I/O errors are reported while the corrupted data is being returned.
Possibly related to #7492 because both regress between kernel 6.12 and 6.18 on BCM2712 PCIe/NVMe, although the controller and failure mode are different.
Device (s)
Raspberry Pi CM5
System
kernel 6.18.34+rpt-rpi-2712
Logs
No response
Additional context
No response
Describe the bug
I am seeing silent and non-deterministic NVMe read corruption on a Raspberry Pi Compute Module 5 when running kernel 6.18.34+rpt-rpi-2712.
The same exact hardware, NVMe filesystem and test file work correctly when booted with kernel 6.12.75+rpt-rpi-2712.
The NVMe device remains visible and operational. There are no NVMe I/O errors, PCIe AER errors or SMART media errors. However, repeated normal buffered reads of the same unchanged file return different data.
O_DIRECT reads are much more reliable, although one incorrect full-file O_DIRECT read was also observed on 6.18.34.
This appears possibly related to #7492, but this is a different NVMe controller and a different failure mode: the PCIe/NVMe device remains present, while data read through it can be silently corrupted.
Hardware
Raspberry Pi Compute Module 5
8 GB RAM
Official Raspberry Pi Compute Module IO Board
Official Raspberry Pi SSD 512 GB
SSD manufacturer: BIWIN
NVMe controller:
0001:01:00.0 Non-Volatile memory controller [0108]:
Biwin Storage Technology Co., Ltd. KingSpec NX series NVMe SSD (DRAM-less)
[1dee:5216] (rev 01)
NVMe model:
BIWIN CE430T5D100-512G
Firmware: 1.4.7.70
PCIe link:
LnkSta: Speed 5GT/s, Width x1
AER status is clean:
UESta: all clear
CESta: all clear
LaneErrStat: 0
NVMe SMART:
critical_warning=0
media_errors=0
num_err_log_entries=0
SMART overall: PASSED
Broken configuration
Raspberry Pi OS image:
raspios_arm64-2026-06-19
Kernel:
Linux 6.18.34+rpt-rpi-2712
Working configuration
Raspberry Pi OS image:
raspios_arm64-2026-04-21
Kernel:
Linux raspberrypi 6.12.75+rpt-rpi-2712
#1 SMP PREEMPT Debian 1:6.12.75-1+rpt1
No hardware was changed between the tests.
The same NVMe filesystem and the same exact 10 GiB file were used.
Steps to reproduce the behaviour
Reproduction
A 10 GiB random file was created on the CM5 eMMC and copied normally to the NVMe.
Source file SHA-256:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
The eMMC source remains completely stable across repeated reads.
On kernel 6.18.34, repeated normal reads of the NVMe copy return different SHA-256 values each time.
Example:
915ccf90d89fdac13d6e398a437f8b374d832d6996ea90f5de80ed93853287f0
e031ed5ab170a2d41ebc26f42e2e4d176551f390bb006d2d818577c87be0063d
13044bd67830be6e07e6bad467273baf515cc88d187f5cf40b70f059e7d5b3ed
After another clean reboot on the same kernel:
e8526c2929172b08c120bb7b05e9e3fadca18362fb0a52c14958d5b05f552a55
27eb942bba7846b6fffe1a598e9ef922d2a18ff1be5ae0e09e6cc3169ea622b1
a6955d608ac6a719966403202c413a064cfacd2ec9a0e37659b5e19c88388bbf
The file inode, size, mtime and ctime remain unchanged.
Test command:
for i in 1 2 3; do
sha256sum /srv/rpi-sb-provisioner/.nvme-integrity-test/test-10g.bin
done
O_DIRECT comparison on 6.18.34
Using:
for i in 1 2 3; do
dd if=/srv/rpi-sb-provisioner/.nvme-integrity-test/test-10g.bin
bs=4M iflag=direct,fullblock status=none | sha256sum
done
Several runs produced the correct SHA every time:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
However, one clean-boot test produced one incorrect O_DIRECT read followed by two correct reads:
f1cf3488f8904d95c5bd46fa655ed77ef39e03d3fedebcc6c2c728516b538805
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
Kernel 6.12.75 control test
I then booted the same CM5 with Raspberry Pi OS 2026-04-21, kernel:
6.12.75+rpt-rpi-2712
The existing NVMe filesystem was mounted read-only:
mount -o ro,noload /dev/nvme0n1p1 /srv/rpi-sb-provisioner
The same exact 10 GiB file was tested without rewriting it.
Three buffered reads:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
Three O_DIRECT reads:
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
4c4f8562fea6791da606149fc48f37f0788956603748cd2fe1336bc066e280bd
So all six full 10 GiB reads return the correct data on kernel 6.12.75.
Other checks
Power is not currently an issue:
vcgencmd get_throttled
throttled=0x0
The corruption was reproduced with temperatures around 60-66°C without throttling.
Kernel logs show no:
NVMe reset
NVMe timeout
PCIe AER error
I/O error
EXT4 error
Limiting RAM to 2 GiB with:
mem=2G
does not fix the issue on kernel 6.18.34.
With mem=2G:
eMMC reads remain stable
NVMe buffered reads remain non-deterministic
O_DIRECT reads return the correct file SHA
Summary
Same hardware, same NVMe, same filesystem and same exact file:
6.12.75:
buffered reads PASS
O_DIRECT reads PASS
6.18.34:
buffered reads FAIL / non-deterministic silent corruption
O_DIRECT reads mostly correct, but at least one incorrect full-file read observed
No PCIe AER, NVMe SMART or kernel I/O errors are reported while the corrupted data is being returned.
Possibly related to #7492 because both regress between kernel 6.12 and 6.18 on BCM2712 PCIe/NVMe, although the controller and failure mode are different.
Device (s)
Raspberry Pi CM5
System
kernel 6.18.34+rpt-rpi-2712
Logs
No response
Additional context
No response