Repeated EXT4 metadata corruption on Raspberry Pi 5 with Penta SATA HAT (JMB585) despite healthy SSDs and RAID1

Hardware and peripherals SATA HAT
1 


Hello,

I am trying to diagnose repeated EXT4 filesystem corruption on a Raspberry Pi 5 using a Penta SATA HAT with the JMicron JMB585 controller.

I have spent about 1.5 days testing this and I would really appreciate input from people familiar with the Raspberry Pi 5 PCIe stack, JMB585, Linux MD RAID, or EXT4.

## Hardware

- Raspberry Pi 5, 8 GB RAM
- Penta SATA HAT
- SATA controller: JMicron JMB58x / JMB585
  - PCI ID: `197b:0585`
  - Driver: `ahci`
- 2x Kingston SEDC500R960G 960 GB SSD
- The two Kingston SSDs form a Linux software RAID1:
  - `/dev/sdb1`
  - `/dev/sdc1`
  - `/dev/md0`
- EXT4 filesystem on `/dev/md0`
- `/dev/md0` is used as the root filesystem `/`
- Separate Samsung 850 EVO 250 GB SSD is also connected to the HAT
- An external eSATA RAID enclosure was previously connected as well, but it has now been completely disconnected for testing.

The RAID itself always remains healthy:

`[UU]`

## Software

Current kernel:

`6.12.75+rpt-rpi-2712`

The Raspberry Pi 5 32-bit PCIe DMA overlay is enabled:

`dtoverlay=pcie-32bit-dma-pi5`

I also verified the JMB585 DMA masks at runtime:

```text
/sys/bus/pci/devices/0001:01:00.0/dma_mask_bits
31

/sys/bus/pci/devices/0001:01:00.0/consistent_dma_mask_bits
31

So the controller does appear to be operating with restricted DMA addressing.

The problem

EXT4 repeatedly reports metadata checksum errors.

Earlier examples included:



EXT4-fs error (device md0):
htree_dirblock_to_tree:1083:
inode #141793:
comm find:
Directory block failed checksum

Other affected inodes included:



140515
58443782

Earlier in the investigation there were also different inode numbers, so the problem is not limited to one directory or one inode.

After repairing the filesystem offline with e2fsck, the filesystem can be completely clean again.

For example, after an offline repair I ran e2fsck -f -v /dev/md0 a second time and it completed without errors.

After rebooting:



errors_count = 0

The previously failing directory could then be read successfully, and a full directory scan also completed without errors.

The system then ran for several hours with Docker containers active and errors_count remained at 0.

Later, however, a more serious EXT4 error appeared:



bad block bitmap checksum
Delayed block allocation failed
This should not happen!! Data will be lost

This involved a different inode (13404678), and another inode (542063) was also involved afterwards.

So the corruption is not always the same EXT4 structure:

  • sometimes directory metadata/checksums

  • later block bitmap / allocation metadata

Offline filesystem checks

I use a separate SD-card installation as a rescue root so that /dev/md0 can be checked completely offline.

After the latest failure:



/ = /dev/mmcblk0p2
md0 = active (auto-read-only)
RAID1 = [UU]

md0 was not mounted.

The first e2fsck found:



Inode 1051162 extent tree (at level 1) could be shorter.
Inode 1051189 extent tree (at level 1) could be shorter.
Inode 1051243 extent tree (at level 1) could be shorter.

Free blocks count wrong (196949284, counted=196950308).

The free block count was repaired.

After that, a second full:



e2fsck -f -v /dev/md0

completed completely clean.

It also reports:



0 bad blocks

SSD health

Both Kingston SSDs report:



SMART overall-health: PASSED
SSD Life Left: 99%

Relevant SMART values are all clean:



Raw_Read_Error_Rate = 0
SATA_Phy_Error_Count = 0
Bad_Block_Rate = 0
Program_Fail_Count = 0
Reported_Uncorrect = 0
Reallocated_Event_Count = 0
SATA_CRC_Error_Count = 0
CRC_Error_Count = 0
Read_Fail_Count = 0

Both SMART error logs say:



No Errors Logged

There are no obvious media errors or SATA CRC errors.

RAID1 comparison

I also investigated one specific directory checksum error in more detail.

For inode 141793:



Type: directory
Size: 4096
EXTENTS:
(0):136927624

Both RAID members use:



Data Offset : 264192 sectors
Super Offset : 8 sectors

I read that exact 4 KiB filesystem block directly from both physical RAID members and calculated SHA256 hashes.

Both SSDs returned exactly the same hash:



968a6654dbdb45561f71740f5a852470b67c768b4a0259bef2175612032ff3df

So at least for this known failing EXT4 block, the data stored on both RAID1 members was bit-for-bit identical.

mismatch_cnt was also:



0

before any full RAID consistency scan.

External eSATA enclosure

The external eSATA enclosure previously showed some suspicious link behaviour during boot:



ata5: link is slow to respond, please be patient
ata5.00: hard resetting link
ata5.01: hard resetting link
...

Because of that, I completely unmounted it, powered it down and physically removed the eSATA cable.

The EXT4 corruption still returned later while the external enclosure was physically disconnected.

So it may have been an additional problem, but it cannot be the only cause of the EXT4 corruption.

Important observation

Before using this SATA HAT, this Raspberry Pi ran from a microSD card without these filesystem problems.

The strange part is that the corruption often becomes visible during periods of heavier filesystem activity: scanning directories, reading many files, or other active work.

The machine can also run for hours with Docker containers running without an immediately visible error.

This makes me suspect that heavier I/O may be triggering an underlying problem rather than EXT4 itself simply being damaged once.

What I am currently testing

A RAM test is currently running with memtester.

I have not yet tested the two Kingston SSDs without Linux MD RAID1.

That is one of the next tests I am considering:

  1. One Kingston SSD

  2. EXT4 directly on the SSD

  3. Same Penta SATA HAT / JMB585

  4. No mdadm

  5. Same workload and monitoring

If corruption still occurs, that would make MD RAID1 much less likely to be involved.

A further comparison could be the exact same SSD through USB-SATA, but I would prefer to understand and fix the Penta SATA HAT / PCIe path rather than use USB as a permanent workaround.

Questions

I would especially appreciate help with these questions:

  1. Has anyone seen repeated EXT4 metadata checksum corruption with the JMB585 / Penta SATA HAT on Raspberry Pi 5?

  2. Are there any known JMB585, AHCI, PCIe, DMA or Raspberry Pi 5 kernel issues that could cause silent data corruption without SATA CRC or SMART errors?

  3. Is kernel 6.12.75+rpt-rpi-2712 known to be problematic with this controller?

  4. Is there a specific Raspberry Pi kernel version currently recommended for the JMB585?

  5. Is dtoverlay=pcie-32bit-dma-pi5 sufficient, or are additional PCIe/DMA settings recommended?

  6. Is there firmware available for the JMB585 controller on these Penta SATA HAT boards?

  7. Could Linux MD RAID1 combined with this controller expose a timing/DMA issue that would not appear with a single SSD?

  8. Are there useful kernel debug options, tracing tools or tests I can enable to catch corruption closer to where it actually happens?

  9. Is there a good way to distinguish between:

    • faulty Penta SATA HAT hardware

    • JMB585/controller problem

    • PCIe/DMA problem

    • RAM corruption

    • Linux MD RAID issue

    • EXT4/kernel issue

I am happy to provide full dmesg, SMART data, lspci -vv, mdadm --detail, EXT4 logs or other diagnostics if useful.

My main goal is not simply to work around the issue, but to identify the actual cause and make this storage setup reliable.

Thank you.

2

Your 6.12 kernel might be affected by this bug, which is fixed in Linux 7.0:

3

Thank you for pointing me to this JMB585 DMA bug. I have now followed your recommendation and moved the system to Raspberry Pi kernel 7.2.2.

I want to provide the complete update because I have spent the last four days investigating this, testing multiple kernels and storage paths, repairing the filesystem several times, and trying to separate hardware, RAID, ext4, Docker and kernel-related causes.

Hardware and storage configuration

The system is:

  • Raspberry Pi 5, 8 GB

  • Radxa Penta SATA HAT

  • JMicron JMB585, PCI ID 197b:0585

  • Linux ahci driver

  • Penta and Pi powered through the Penta 12 V input using a 12 V / 5 A / 60 W PSU

  • Raspberry Pi 5 A current configuration verified:

    • /proc/device-tree/chosen/power/max_current = 5000

    • vcgencmd get_throttled = 0x0

  • Two Kingston DC500R 960 GB SSDs in Linux RAID1

  • RAID device: /dev/md127

  • ext4 filesystem, approximately 894 GiB

  • Separate Samsung 850 EVO attached through the Penta

  • Separate Samsung 860 EVO attached through USB for comparison

  • The running OS is now on a separate SanDisk USB device

  • The RAID is no longer the root filesystem; it contains Docker and persistent data

The external IcyBox RAID5 previously connected to the system was physically disconnected during later tests. The ext4 corruption still occurred afterward, so that device cannot be the sole cause.

Kernels used during the investigation

Over the four-day investigation, the system has run several kernels:

  • Raspberry Pi 6.18.39

  • Raspberry Pi 6.12.75+rpt-rpi-2712

  • Raspberry Pi 6.12.96+rpt-rpi-2712

  • Now Raspberry Pi 7.2.2+

The current kernel was built from the unmodified Raspberry Pi rpi-7.2.y source:

Linux raspi 7.2.2+ #3 SMP PREEMPT ... aarch64 GNU/Linux
source commit: 003a7d95a5c97f699cb62ffcda6d618130855052
git status: clean
source diff: empty

The current Raspberry Pi 7.2 source still contains:

0*AHCI_HFLAG_32BIT_ONLY

Raspberry Pi partially reverted the direct upstream quirk after Pi 5/JMB585 systems could fail AHCI probe with errno=-12, as shown in raspberrypi/linux#7355.

I am using the workaround documented by Radxa for the Penta SATA HAT:

dtoverlay=pcie-32bit-dma-pi5

This overlay was active under 6.12.96 and remains active under 7.2.2.

The runtime values are:

dma_mask_bits=31
consistent_dma_mask_bits=31

The AHCI boot message still reports:

flags: 64bit ncq sntf stag pm led clo pmp fbs pio slum part ccc apst boh

There is no:

controller can't do 64bit DMA, forcing 32bit

I understand that the 64bit banner describes the AHCI capability and does not necessarily mean that DMA above the restricted address range is actually being used. The sysfs masks indicate that the Raspberry Pi overlay is constraining the DMA window.

However, this distinction is why I would like confirmation that Raspberry Pi 7.2.2 plus pcie-32bit-dma-pi5 is the exact configuration Radxa expects to solve the silent-corruption problem described in upstream commit 105c42566a55.

Filesystem corruption observed

The RAID1 ext4 filesystem has repeatedly produced real metadata errors, including:

Directory block failed checksum
No space for directory leaf checksum. Please run e2fsck -D.
bad block bitmap checksum
inode checksum mismatch
Delayed block allocation failed
This should not happen!! Data will be lost

The corruption has returned after previous offline fsck repairs and initially clean verification passes.

One recurrence under 6.12.96 involved:

inode 35616007
/smv/docker/appdata/qui/backups/torrents/4e/0c/89

The kernel reported:

No space for directory leaf checksum
Directory block failed checksum

This happened after an earlier offline repair and clean verification.

During the later Raspberry Pi 7.2.2 run, a read-only filesystem traversal found three bad directory inodes:

40487748
/smv/docker/engine/overlay2/.../homeassistant/components/switchbot/__pycache__

34702193
/usr_old/share/icons/PiXtrix/scalable/devices

34691805
/usr_old/share/nodejs/set-function-length

A normal manual SSH find reproduced these errors. Codex was therefore not required to expose the corruption.

Two of the three directories were outside Docker, so the problem was also not limited to Docker or OverlayFS.

The same inode could produce another error whenever it was traversed again. Therefore ext4 errors_count counted error events, not unique newly corrupted directories.

Throughout these events:

  • md RAID1 remained [UU]

  • no RAID member dropped

  • no corresponding ATA error appeared

  • no AHCI reset appeared

  • no block I/O error appeared

  • no PCIe/AER error appeared

Important uncertainty about the errors found after moving to 7.2.2

The three corrupt directories were discovered while running 7.2.2, but this does not prove that 7.2.2 created the corruption.

The filesystem had already shown recurring corruption under 6.12.96. Therefore these directories may have been latent damage that was only discovered when the broader filesystem traversal reached them.

That creates an unavoidable overlap between the old filesystem state and the move to 7.2.2:

  • The errors were discovered under 7.2.2.

  • But they may have been created earlier under 6.12.96.

  • Therefore the discovery cannot prove either that 7.2.2 failed or that 7.2.2 caused them.

  • It also cannot yet prove that 7.2.2 plus the DMA overlay has solved the underlying problem.

SSD, RAID, RAM and PCIe checks

Both Kingston RAID SSDs report:

  • SMART overall PASSED

  • approximately 99% remaining life

  • no reallocated sectors

  • no CRC errors

  • no reported uncorrectable errors

  • no read/program/erase failures

  • low power-on hours

A memtester 5G 1 run completed all 18 patterns successfully.

For one previously corrupt directory block, both RAID1 members contained identical data and produced the same SHA256:

968a6654dbdb45561f71740f5a852470b67c768b4a0259bef2175612032ff3df

This makes an isolated failure of one RAID SSD less likely, although it does not prove which higher layer created the bad data.

Other checks included:

  • md RAID remained [UU]

  • previous mismatch_cnt=0

  • both members reported in_sync

  • md RAID1 serialize_policy=0

  • JMB585 endpoint AER counters: zero

  • Raspberry Pi root-port AER counters: zero

  • no PCIe Bus Error in the kernel log

  • active PCIe link: 5 GT/s, x1

  • ASPM disabled

  • no continuous discard mount option

  • weekly fstrim.timer enabled

  • ext4 inline_data feature not enabled

  • known affected directory did not have the ext4 inline-data inode flag

These results reduce several possibilities but do not completely exclude a controller, power, kernel, md RAID or filesystem-layer problem.

Controlled storage tests under 6.12.96

Under Raspberry Pi 6.12.96, the following tests completed without corruption:

  • 50 GiB direct write plus CRC verification on the Penta Samsung SSD

  • 50 GiB direct write plus CRC verification on the USB Samsung SSD

  • 50 GiB direct write plus CRC verification on md RAID1

A simultaneous three-filesystem metadata workload was then run:

  • 51,200 files on the Penta Samsung

  • 51,200 files on the USB Samsung

  • 51,200 files on RAID1

  • each file 1 MiB

  • 153,600 files / 150 GiB total

  • fsync_on_close=1

All three jobs returned err=0.

All 153,600 files were then removed. The deletion phase also remained clean.

Afterward:

Penta Samsung errors_count: 0
USB Samsung errors_count:   0
RAID1 errors_count:         0

No relevant ATA, AHCI, I/O, AER or ext4 errors appeared.

Therefore generic sustained I/O and generic mass file creation/deletion did not reproduce the corruption.

Repeating the tests under Raspberry Pi 7.2.2

After moving to the current Raspberry Pi 7.2.2 kernel, I repeated the same three-way metadata workload:

  • Penta Samsung: 50 GiB / 51,200 files

  • USB Samsung: 50 GiB / 51,200 files

  • RAID1: 50 GiB / 51,200 files

  • 153,600 files / 150 GiB total

Results:

Penta Samsung: err=0
USB Samsung:   err=0
RAID1:         err=0

The complete parallel deletion phase was also clean.

All ext4 error counters remained zero, and there were no corresponding kernel storage errors.

Docker was then enabled again. The complete 41-container startup storm initially remained clean:

  • RAID [UU]

  • ext4 errors_count=0

  • no ATA/AHCI/I/O/AER errors

  • low disk I/O wait

Only the later broad read-only filesystem traversal discovered the three corrupt directories described above. Again, these may have been pre-existing.

Latest offline repair

After finding those errors, I stopped Docker, containerd and all remaining container processes.

Stopping the systemd Docker services alone was not enough because surviving container processes and dozens of OverlayFS mounts still referenced the RAID. Those processes and mounts had to be removed before the RAID could be unmounted.

Before running fsck, I verified that:

findmnt -rn -S /dev/md127

returned no mounts.

I then ran:

sudo e2fsck -f -D /dev/md127

The result was:

Inode 44522248 extent tree could be shorter. Optimize: yes
Inode 44522256 extent tree could be shorter. Optimize: yes
Inode 44522271 extent tree could be shorter. Optimize: yes

Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 3A: Optimizing directories
Pass 4: Checking reference counts
Pass 5: Checking group summary information

raspi-root: ***** FILE SYSTEM WAS MODIFIED *****
raspi-root: 2872183/58711824 files (0.1% non-contiguous)
raspi-root: 96172150/234389504 blocks

A second forced, non-writing verification was then performed:

sudo e2fsck -f -n /dev/md127

It completed all five passes clean with exit status 0:

raspi-root: 2872183/58711824 files (0.1% non-contiguous)
raspi-root: 96172150/234389504 blocks

Current situation

After the clean verification:

  • /dev/md127 was mounted again

  • /smv and the monitoring bind mounts were restored

  • RAID remained [UU]

  • Docker and containerd were restarted

  • all 41 existing containers were started

  • all 41 containers are currently running

  • no containers are exited

  • no containers are unhealthy

  • rasdaemon, prometheus-node-exporter and sysstat are active

  • ext4 errors_count=0

With all real Docker workloads running, I repeated a complete filesystem namespace/metadata traversal:

sudo find /mnt/raid -xdev -ls >/dev/null

Result:

exit status: 0
elapsed time: 114 seconds
ext4 errors_count before: 0
ext4 errors_count after:  0

The kernel log contained no:

  • EXT4 warning or error

  • checksum failure

  • ATA reset or error

  • block I/O error

  • PCIe Bus Error

  • AER event

This is now the first fully verified clean post-repair baseline under Raspberry Pi 7.2.2 plus dtoverlay=pcie-32bit-dma-pi5.

Where this leaves the investigation

At this moment the system is repaired, mounted and operating normally.

However, I cannot yet say with certainty that the problem is solved:

  • The corruption found after moving to 7.2.2 may have been old damage from before the kernel change.

  • Because that damage has now been repaired, the current clean state finally provides a trustworthy baseline.

  • If a new ext4 checksum error appears after this point, it will be much stronger evidence that Raspberry Pi 7.2.2 plus pcie-32bit-dma-pi5 did not prevent fresh corruption.

  • If the system remains clean over time under the real workload, that would support—but still not immediately prove—that the new kernel/configuration resolved the issue.

After four days of testing, repairs, kernel changes and storage comparisons, my most important question is therefore:

Can Radxa confirm that the exact configuration I am now running—unmodified Raspberry Pi rpi-7.2.y 7.2.2 plus dtoverlay=pcie-32bit-dma-pi5—is expected to provide the complete protection against the JMB585 silent-corruption bug from upstream commit 105c42566a55?

In particular:

  1. Is the Pi DMA overlay intended to provide the same data-integrity protection as the upstream AHCI_HFLAG_32BIT_ONLY quirk?

  2. Is the continued flags: 64bit AHCI banner expected and safe when the sysfs DMA masks are 31/31?

  3. What exact Raspberry Pi kernel, firmware and overlay combination has Radxa validated for data integrity with multiple Penta-attached SSDs and Linux md RAID1?

  4. Is there a runtime check beyond the sysfs DMA masks that can confirm all JMB585 DMA transactions remain within the safe address range?

  5. Are there any JMB585 firmware, controller or Penta power-path diagnostics Radxa recommends?

  6. Based on the evidence above, is there any additional controlled test Radxa wants me to run now that the filesystem has a verified clean baseline?

I can provide the full command log, fsck outputs, kernel configuration and complete investigation timeline if required.

4

I resolved the affected inode numbers with debugfs:

sudo debugfs -R ‘ncheck 278049’ /dev/md127
278049 /smv_old/docker/appdata/qui/backups/torrents/76/72

sudo debugfs -R ‘ncheck 401137’ /dev/md127
401137 /smv_old/docker/appdata/qui/backups/torrents/d7/77

Both affected directories are inside an old copied application-data tree:

/smv_old/docker/appdata/qui/backups/torrents/

They are not directories belonging to the currently active Docker tree under /smv/docker.

This may explain why normal application use did not immediately encounter them, but it does not explain why their EXT4 directory checksums failed after a successful offline e2fsck -f -D and subsequent clean verification.

The errors are therefore located in old/duplicate data, but they are still real metadata checksum failures on the same active EXT4 filesystem on md127.

5

You have done extensive testing, however certain things can be done more efficiently, like using just a non-RAID setup with a single mode Btrfs filesystem, like the developers who created the upstream bug-report w.r.t. this failing 64-bit DMA of the JMB chip have mentioned. Btrfs supports online scrub, it will checksum every storage block, not only metadata. So you can keep your system/dockers running and filesystem mounted for this kind of topic reporting. And it is transactional with a monotonous counter principle, so you can track back when a corruption happened. If you yourself already question that of your own Ext4 filesystem, how should others know or be motivated to reproduce.

But most important, I do not see which version of firmware is used (RPI EEPROM contents). As this is closed-source, other than RPL themselves have no good option to do proper analysis, also not you in fact. So you need to ask RPL about this issue you have, it is their base platform, that seems to need to actions, namely a HW/firmware config change (the overlay) and the upstream kernel patch on their downstream kernel.

Just think: Does a PC or any other computer platform using this JMB chip need a firmware/BIOS/config settings change in addition to just the upstream kernel patch (on upstream/mainline kernel)?

Your RPi 7.2.2+ kernel is a latest own downstream kernel testing/development vehicle, normally only to be used on request of RPL engineers. And by the time someone might read all this, the version on mainline kernel.org will be already higher than 7.2.2. Or people already take 7.3-rcX, that is at least what I mostly do in such cases.

About the RAID and EXT4: Only the A is maybe valid in RAID; It are not ‘Disks’ (floppy disk or hard disk) but flash storage, also not ‘Inexpensive’ these days and ‘Redundant’ is questionable as the storage devices transport van the same channel (the buggy 64-bit DMA transport). So no option to do multi-path.

6

Please see:

If you can run the fio reproducer on JMicron JMB585 without the
dtoverlay=pcie-32bit-dma-pi5 overlay and without the AHCI_HFLAG_32BIT_ONLY
flag, and report the result to the mailing list (in plain text, no HTML), that would be
appreciated.

7

Thanks for pointing me toward the JMB585 DMA issue. I have now done quite a bit more testing and also followed the current upstream linux-ide discussion.

I set up a completely separate Raspberry Pi OS test installation and used a dedicated Kingston SEDC500R960G SSD behind the same Penta SATA HAT / JMB585.

Test system:

  • Raspberry Pi 5, 8 GB

  • JMicron JMB585 (197b:0585), AHCI

  • Linux 6.18.34+rpt-rpi-2712

  • fio 3.39

  • PCIe Gen2

I ran the 256 GiB CRC32C fio canary reproducer that is currently being discussed on the linux-ide mailing list.

Test 1 – without dtoverlay=pcie-32bit-dma-pi5

Runtime showed:



dma_mask_bits: 64
consistent_dma_mask_bits: 64
iommu_group: NONE

AHCI also reported flags: 64bit.

I wrote the 256 GiB canary once and then completed more than 20 full verification passes, so approximately 5 TiB of verified reads.

Result: no CRC32C verification failure and no corruption reproduced.

SMART remained clean, with no SATA CRC errors, uncorrectable errors or read failures, and there were no relevant ATA/AHCI/EXT4 errors in the kernel log.

Test 2 – with dtoverlay=pcie-32bit-dma-pi5

After enabling the overlay and rebooting:



dma_mask_bits: 31
consistent_dma_mask_bits: 31
iommu_group: NONE

First I verified the original canary that had been written under the 64-bit DMA configuration. The complete 256 GiB verification passed.

I then wrote a new 256 GiB canary under the 31-bit DMA configuration and ran exactly 20 verification passes.

Final result:



original 64-bit-written canary cross-check: PASS
new 32-bit canary write: PASS
verify passes: 20/20 PASS
reads: 5376 GiB
writes: 256.02 GiB

SMART/error counters were still all clean afterwards.

So, somewhat surprisingly, I have not been able to reproduce the corruption with this fio test in either configuration.

There is an important limitation: on the Raspberry Pi 5 the JMB585 shows:



iommu_group: NONE

The current upstream discussion specifically asks for testing a JMB582/JMB585 behind an Intel or ARM IOMMU in enforcing mode, so my Pi cannot fully satisfy that particular test request.

I have just sent my results and the links to this thread and my Raspberry Pi forum thread to the linux-ide mailing list as an additional data point and asked them what would be most useful to test next.

What still concerns me is that the original problem was real and repeatable: EXT4 metadata corruption occurred multiple times on the two-disk Linux md RAID1 while SMART, SATA CRC counters and RAID health remained clean. In one known corrupt EXT4 block, both physical RAID members even contained the exact same data.

The major difference is therefore:

original setup: two Kingston SSDs → JMB585 → md RAID1 → EXT4

fio reproducer: one Kingston SSD → JMB585 → EXT4/test file

I have kept the complete test installation, canary files and logs.

@RadxaYuntian, given these results, what would you recommend testing next?

In particular, would it make sense to recreate the original two-SSD md RAID1 setup on the test installation and run a comparable long checksumming/write workload there?

I would still like to find the actual cause rather than simply assume that the DMA overlay solves it, especially since the original corruption occurred even while dtoverlay=pcie-32bit-dma-pi5 was already enabled.

8

This seems is a slightly older kernel version as on one of my Raspberry Pi updated some time ago just from standard repo: 6.18.39+rpt-rpi-v8

Which now could easily be that your RPi (the HW but including the programmable EEPROM) has used a newer root filesystem and as by default rpi-eeprom-update.service is enabled, the EEPROM can contain a newer and unknown version as this whole topic nowhere mentions that.

RPi5 is composed of a BCM2712 from Broadcom and an RP1 chip from Raspberry Pi Ltd. That latter chip has 4-lane PCIE connectivity AFAR, it also has 2x Cortex-M3 CPU’s ‘doing things’. Besides that, there is the VideoCore still around although that does not play such an active role anymore as it does in older RPi’s.

So which version is in your board?

cat /usr/lib/firmware/raspberrypi/bootloader-2712/versions.txt

# firmware-2712 firmware versions
#
# Maps each pieeprom-YYYY-MM-DD.bin to the BUILD_TIMESTAMP
# and source git hash embedded in the binary, plus the
# release (default / latest / old) the file lives in. When
# a binary appears in more than one release the highest-
# precedence one wins:
#     default > latest > old   (old = archived)
#
# Generated by tools/update-versions-txt. Sorted newest-first.
#
# version   build_epoch  fw_git_hash  release  mfg_ver
2026-08-12  1786493216   f5e89631     latest   1
2026-08-04  1785847198   0a7ef05f     latest   1
2026-06-29  1782691621   f68405bc     latest   1
2026-06-17  1781654813   6219bf00     latest   1
2026-05-26  1779807685   086b83e3     default  1
2026-05-22  1779408415   7dcdc4b8     old      1
2026-05-17  1778976445   1abffaec     old      1
2026-05-13  1778671631   4c8cdd3d     old      1
2026-05-11  1778498402   66f33f7e     old      1
2026-04-30  1777551683   1a17f6cb     old      
2026-04-27  1777248418   c2a7b9d3     old      
2026-04-14  1776201624   9cd61c53     old      
2026-02-23  1771840899   85353ce4     old      
2026-02-06  1770388300   8124798b     old      
2026-01-21  1769002727   ab8a9dde     old      
2026-01-16  1768585427   4aec3e13     old      
2025-12-08  1765222194   2226a853     old      
2025-11-27  1764250826   999d0ec9     old      
2025-11-21  1763732176   b8803387     old      
2025-11-05  1762364238   57db150d     old      
2025-10-17  1760694517   b66568da     old      
2025-10-08  1759940358   a06c733f     old      
2025-09-25  1758829114   f6ca7458     old      
2025-09-23  1758625555   02da46a3     old      
2025-09-22  1758541389   9652cde4     old      
2025-08-27  1756321307   000d3ca2     old      
2025-08-20  1755703318   cd866525     old      
2025-08-13  1755094299   bb80e517     old      
2025-07-17  1752769512   a668b6e6     old      
2025-07-03  1751539154   ab660caa     old      
2025-06-29  1751239011   6672d86c     old      
2025-06-20  1750421313   960432b6     old      
2025-06-13  1749807566   5855b10b     old      
2025-06-09  1749461452   3a4915a4     old      
2025-05-08  1746713597   69471177     old      
2025-04-07  1744067807   50beecf0     old      
2025-03-27  1743034604   897f94ed     old      
2025-03-19  1742391686   cec1d3ae     old      
2025-03-10  1741626637   2bb2ae64     old      
2025-03-03  1741014903   9116015f     old      
2025-02-12  1739357512   f788aab6     old      
2025-02-11  1739293519   75c1e570     old      
2025-01-27  1737983339   76eda9d7     old      
2025-01-22  1737505011   a7753063     old      
2025-01-14  1736813808   0451f142     old      
2025-01-13  1736727407   fa261b3c     old      
2025-01-08  1736358768   97facbf4     old      
2025-01-07  1736263931   e10f823b     old      
2025-01-06  1736182835   f49ef7cb     old      
2024-12-19  1734609433   ccf64a4f     old      
2024-12-15  1734221810   461827f2     old      
2024-12-07  1733575343   3858f977     old      
2024-11-27  1732717699   0b72471b     old      
2024-11-12  1731427844   4b019946     old      
2024-11-05  1730810292   3c4fc886     old      
2024-10-21  1729520869   951e1cc9     old      
2024-10-10  1728517007   0e63dd5e     old      
2024-09-23  1727096576   26826259     old      
2024-09-10  1725975630   5be4f304     old      
2024-09-05  1725562503   7fe8d029     old      
2024-07-30  1722349546   790da7ef     old      
2024-07-25  1721921872   622dd3d8     old      
2024-06-05  1717602109   6fe0b091     old      
2024-06-04  1717489297   503a909a     old      
2024-05-17  1715945383   72caf667     old      
2024-05-13  1715613301   83b5e5e6     old      
2024-04-20  1713610410   d1744d21     old      
2024-04-18  1713429900   86ccc427     old      
2024-04-17  1713358463   0b7b6f28     old      
2024-04-05  1712313679   a482d84b     old      
2024-02-16  1708097321   4c845bd3     old      
2024-02-14  1707895062   c1b99e8d     old      
2024-02-08  1707392087   42e0a50b     old      
2024-02-05  1707143914   0e79493d     old      
2024-01-24  1706098561   1aeae380     old      
2024-01-22  1705934676   e891ded6     old      
2024-01-15  1705345348   f0aa0715     old      
2024-01-05  1704470260   30cc5f37     old      
2023-12-14  1702572205   0f881224     old      
2023-12-06  1701887365   e02d33b3     old      
2023-11-20  1700509217   361205c6     old      
2023-10-30  1698684310   30de0ba5     old      
2023-10-18  1697650217   c2da2ae7     old      
2023-09-28  1695896697   b74d18ae     old      
2023-09-21  1695315523   3094eda5     old      
2023-09-13  1694601426   9d494316     old

This problem with the RPi 6.18 kernel and JMicron chip is now known for several months including the workaround, so it could even be that there is some (hidden) fix/workaround in the EEPROM code done in the meantime. You simply cannot track back anymore why you had data corruption earlier. It could even be that an additional cause was a powering issue. You now have 1 SSD (less risk for burst/peak power) and also a single path for filesystem level I/O (no kernel device mapper).

9

Thanks, this is useful. I checked the actual EEPROM currently installed, rather than only the list of firmware files available in the root filesystem.

The board currently reports:



$ vcgencmd bootloader_version
2026/05/26 16:01:25
version 086b83e3332dfc8927c56762771d082f3077a1ae (release)
timestamp 1779807685
update-time 1787845889
capabilities 0x0000007f

$ sudo rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Tue 26 May 15:01:25 UTC 2026 (1779807685)
    LATEST: Tue 26 May 15:01:25 UTC 2026 (1779807685)
   RELEASE: default

$ systemctl is-enabled rpi-eeprom-update.service
enabled

The update-time corresponds to 27 August 2026.

This is interesting for the test I just performed: the completely separate Raspberry Pi OS test installation was not used until 5 September. Therefore that fresh test root filesystem did not install a newer EEPROM immediately before the successful fio test. The May 26 EEPROM was already installed on the Pi before that test.

So although I agree that EEPROM state is another variable which should be recorded, a newly installed EEPROM from the fresh test image does not appear to explain why the current fio reproducer did not show corruption.

There is also another important part of my investigation that I probably did not make clear enough in my previous post:

The corruption was not limited to the two-SSD md RAID1 + EXT4 configuration.

Later in the investigation I also had a single Kingston SEDC500R960G connected through the same Penta SATA HAT / JMB585, without md RAID1. That SSD was formatted as a single-device Btrfs filesystem (KINGSTON2):



data:     single
metadata: DUP
system:   DUP

I first ran a complete read-only Btrfs scrub of the SSD:



62.83 GiB checked
no errors found

The corruption counters were then zero.

I started the containers afterwards. Later a new checksum error appeared in the active Qui SQLite database:



BTRFS warning (device sdf1):
csum failed root 5 ino 459056 off 12533760
csum 0x8bc3de90 expected csum 0x1ddbf15d mirror 1

BTRFS error (device sdf1):
bdev /dev/sdf1 errs:
wr 0, rd 0, flush 0, corrupt 1, gen 0

mirror 1 in this case was not a second disk copy: the Btrfs data profile was single.

The sequence was roughly:



19:33  complete Btrfs scrub: clean
19:45  system/containers running
20:33  qui.db last modified
20:53  checksum mismatch detected

This Kingston was connected through:



JMicron JMB585 197b:0585
driver: ahci
PCIe: 5.0 GT/s x1
dma_mask_bits: 31
consistent_dma_mask_bits: 31

This was also tested with the 31-bit DMA restriction active (including during later custom-kernel testing).

SMART was again clean: no SATA CRC errors, read errors, reported uncorrectable errors, reallocations, etc.

So md RAID1 and EXT4 do not seem to be required for the corruption to occur. The single-device Btrfs test produced a newly detected checksum mismatch after a previously clean full scrub.

That is one reason why I do not think the difference between “two SSDs + md RAID1” and “one SSD” alone explains why Mikael’s fio reproducer was clean.

Regarding power: I agree that it should remain a possible variable.

The system is powered through the Penta SATA HAT using:



LEICKE NT03012
12 V / 5 A / 60 W

I previously verified:



/proc/device-tree/chosen/power/max_current = 5000
vcgencmd get_throttled = 0x0

and I have not seen SATA CRC errors, ATA I/O errors or PCIe AER errors corresponding with the corruption events.

Of course, get_throttled=0x0 cannot exclude a very short transient or a problem in the Penta’s own 12V → 5V conversion/distribution path. I have not measured that with an oscilloscope, so I do not want to claim that power has been ruled out.

But since a newly created checksum error also occurred with a single SSD, a peak-power problem caused specifically by having two Kingston SSDs active would not seem to explain all observations.

Regarding kernel 6.18.34 vs 6.18.39: you are correct that 6.18.39 is newer. The clean Raspberry Pi OS test image contained 6.18.34. I deliberately kept exactly the same kernel, hardware and SSD for both halves of the A/B test:



A: normal configuration, DMA mask 64/64
B: pcie-32bit-dma-pi5, DMA mask 31/31

so the DMA configuration was the main changed variable.

If there is a specific change between 6.18.34 and 6.18.39 that could be relevant to JMB585/PCIe/DMA, I am happy to repeat the reproducer on 6.18.39 as well.

I would also be interested if you know of a particular Raspberry Pi EEPROM commit/change related to JMB585, external PCIe DMA or coherency. Since this board is currently running the May 26 EEPROM and that EEPROM was already installed before the latest tests, a specific firmware change/date would help a lot in correlating it with the earlier failures.

At this point the puzzling part for me is:



md RAID1 + EXT4                -> corruption observed
single SSD + Btrfs + JMB585   -> new checksum corruption observed
single SSD + fio canary       -> >5 TiB verified, no corruption
same fio test with 31-bit DMA -> >5 TiB verified, no corruption

That makes me wonder whether Mikael’s sequential/direct fio workload simply does not exercise whatever workload triggers my failure. The actual errors have tended to appear with real application activity involving many small writes/metadata operations (Docker/appdata/SQLite).

If you have a suggestion for a more representative stress test — or for a controlled power-path A/B test using a power arrangement that Radxa considers supported — I would be very interested to try it.

I have kept the test OS, the canary files and all fio/SMART/kernel logs, so I can still run additional controlled tests.

I have also sent the fio result to the current linux-ide discussion, because Mikael’s reproducer came from that thread:

https://lore.kernel.org/linux-ide/apqn-bdMi4y4l1gE@ryzen/T/

Thanks for looking into this.

10

Final update: reproducible trigger isolated and practical workaround qualified

Thanks everyone for the suggestions in this thread. I have continued testing this quite extensively, and I now have enough controlled results to document the final practical outcome.

I do not think I have proven the exact low-level root cause, but I have been able to:

  • reproduce the silent corruption reliably on a single SSD without md RAID;

  • isolate queueing / multiple outstanding commands as the trigger in my reproducer;

  • test queue_depth=1 for more than 10 hours without corruption;

  • test real libata.force=noncq for a continuous 7+ hours without corruption;

  • measure the performance cost of the mitigations;

  • choose a conservative production configuration.

Raspberry Pi EEPROM

To answer the earlier EEPROM question, the board is using:



BOOTLOADER:
CURRENT: Tue May 26 15:01:25 UTC 2026
build epoch: 1779807685
firmware git hash: 086b83e3
release: default

So the subsequent fresh reproducer and queueing tests were not performed on an unknown newer EEPROM revision.

Important change: I found a reliable reproducer

The earlier 256 GiB sequential CRC32C canary tests did not reproduce the corruption, despite several TiB of verified reads.

I therefore changed the workload shape rather than simply increasing the amount of sequential I/O.

I used a dedicated Kingston SEDC500R960G behind the same Penta/JMB585, with a fresh single-device Btrfs filesystem and no md RAID:



fio --name=sqlite_sim --directory=/mnt/jmbtest \
    --rw=randrw --rwmixread=50 \
    --bs=4k --iodepth=8 \
    --ioengine=io_uring \
    --direct=1 --fsync=8 \
    --size=8G --time_based \
    --verify=crc32c \
    --numjobs=2

The Raspberry Pi DMA restriction was active:



dma_mask_bits:             31
consistent_dma_mask_bits:  31

With the normal SATA device queue depth:



queue_depth = 32

the workload reproduced Btrfs checksum corruption very quickly.

Early repetitions failed after approximately:



~80 seconds
~41 seconds

The important point is that the fio EIO was generated because Btrfs detected incorrect data via its checksum. There was still no corresponding low-level device failure.

During the corruption:



write_io_errs   = 0
read_io_errs    = 0
flush_io_errs   = 0
generation_errs = 0

SMART remained clean and I still found no corresponding:



ATA error
AHCI reset/error
SATA CRC error
PCIe/AER error

So this reproduces essentially the same “silent below the filesystem” behaviour that originally appeared as EXT4 metadata corruption.

60-minute confirmation at queue_depth=32

I then allowed the same workload to continue despite checksum errors.

After 30 minutes:



Btrfs corruption_errs delta: 102

After the full 60-minute run:



Btrfs corruption_errs delta: 443

write_io_errs:    0
read_io_errs:     0
flush_io_errs:    0
generation_errs:  0

ATA/AHCI/AER error matches: 0
SMART: clean

The kernel printed fewer literal csum failed messages because Btrfs/kernel logging was rate-limited. The suppressed callbacks exactly accounted for the difference, so I used the Btrfs corruption_errs delta as the primary PASS/FAIL counter.

This also established something important:

md RAID1 is not required to reproduce the problem.

A single SSD through the JMB585 is sufficient.

Queue-depth sweep

I then tested the same reproducer while limiting the SATA device queue depth.

The most important results were:

queue_depth result
32 FAIL after 44 s
1 PASS
2 FAIL after 103 s
3 FAIL after 57 s
4 FAIL after 58 s
7 FAIL after 48 s
8 FAIL after 63 s
9 FAIL after 66 s
10 FAIL after 47 s
12 FAIL after 76 s
13 FAIL after 77 s
14 FAIL after 44 s
15 FAIL after 66 s
16 FAIL after 58 s

Depths 6 and 11 happened to survive their short 90-second screening runs, but I do not interpret those as safe values. The failure is stochastic and several neighbouring/higher depths fail quickly.

The important boundary is that queue_depth=2 already reproduced corruption, while queue_depth=1 survived all extended qualification runs.

For queue_depth=1 I recorded:



300 s      PASS
1800 s     PASS
7200 s     PASS
26226 s    PASS

Including the earlier 5- and 30-minute tests, the accumulated clean test time at queue_depth=1 is:



37,626 seconds
= 10 hours 27 minutes 6 seconds

with:



corruption_errs = 0

throughout those clean runs.

The continuous overnight portion alone ran for approximately:



7 hours 17 minutes

without corruption.

Important terminology correction: queue_depth=1 is not the same as NCQ disabled

Earlier in the investigation I sometimes described queue_depth=1 as “NCQ off”.

That is technically imprecise.

Setting:



queue_depth=1

limits Linux to one outstanding command, but the ATA command can still use the NCQ/FPDMA command path.

Actual NCQ disablement requires something such as:



libata.force=noncq

This turned out to be useful because it allowed me to distinguish:



A) multiple outstanding commands are required to trigger the problem

versus

B) merely using the NCQ/FPDMA protocol is enough to trigger it

Real libata.force=noncq test

I therefore enabled:



libata.force=noncq

on the actual active kernel command line.

After reboot the kernel explicitly confirmed:



ata1.00: FORCE: modified (noncq)
ata1.00: ... NCQ (not used)

and:



dma_mask_bits:             31
consistent_dma_mask_bits:  31
queue_depth:                1

I first ran the same reproducer for 15 minutes.

Result:



PASS
write_io_errs    0
read_io_errs     0
flush_io_errs    0
corruption_errs  0
generation_errs  0

I then ran a fresh Btrfs filesystem with the same workload continuously for exactly:



26,226 seconds
= 7 hours 17 minutes 6 seconds

Final result:



write_io_errs    0
read_io_errs     0
flush_io_errs    0
corruption_errs  0
generation_errs  0

So real non-NCQ also survived the complete long qualification run without a single detected corruption event.

Performance impact

I also ran direct 15-minute A/B tests using the same workload.

Results:

configuration READ WRITE result
64/64 DMA + queue_depth=1 3897 KiB/s 3900 KiB/s PASS
31/31 DMA + queue_depth=1 3856 KiB/s 3859 KiB/s PASS
31/31 DMA + real noncq 3760 KiB/s 3763 KiB/s PASS

The direct 64/64 versus 31/31 difference at depth 1 was only about:



1.1%

so I found no practical performance reason to remove the 32-bit DMA restriction.

Real noncq was approximately:



2.5%

slower than 31/31 + queue_depth=1 in this particular SQLite-like stress workload.

For me that is a very small price for the more conservative configuration.

I cannot provide an exact 7-hour depth1-versus-noncq throughput comparison because the overnight depth1 test script terminated fio at its time limit before fio wrote its normal final bandwidth summary. The integrity result of that run is unaffected: the test duration and Btrfs counters were recorded separately and the complete run was clean.

What I think the tests now show

The strongest result is no longer simply:



"NCQ on = bad, NCQ off = good"

It is more specifically:



more than one outstanding command allowed
    -> corruption repeatedly reproducible

maximum one outstanding command
    -> >10 hours of qualification without corruption

real non-NCQ
    -> 7h17 continuous qualification without corruption

Because queue_depth=1 remained clean while NCQ/FPDMA could technically still be in use, the mere presence of the NCQ protocol does not appear sufficient by itself.

The strongest trigger I have isolated is therefore multiple concurrent/outstanding SATA commands.

I do not think the testing identifies exactly where the underlying race/bug resides. Possibilities still include:



JMB585 controller logic/configuration
JMB585 <-> Linux libata/AHCI interaction
ordering/coherency behaviour
Raspberry Pi PCIe/JMB585 interaction
Penta-specific board behaviour
or some combination of these

I therefore do not claim that I have proven an internal JMB585 hardware defect.

Relation to the earlier 64-bit DMA issue

There is another useful conclusion here.

The new corruption reproducer repeatedly fails while the Raspberry Pi DMA overlay is already active and runtime masks are:



31 / 31

Therefore the previously discussed >4 GiB / 64-bit-DMA mechanism is not sufficient by itself to explain this reproducer.

That does not mean the DMA restriction is useless or that the earlier DMA reports are invalid. It means I appear to have found an additional queueing-sensitive failure mode, or an interaction that is still present while DMA is restricted.

The fact that the huge sequential canary workload remained clean while this small-random-I/O + fsync workload fails rapidly also shows that workload shape matters much more than total I/O volume in reproducing this problem.

Final production configuration

For production I have therefore chosen the most conservative configuration that I actually qualified:

config.txt:



dtoverlay=pcie-32bit-dma-pi5

kernel command line:



libata.force=noncq

Runtime this gives:



dma_mask_bits             31
consistent_dma_mask_bits  31
queue_depth               1
NCQ                       not used

queue_depth=1 by itself also passed more than 10 hours of testing, but I chose real noncq because it removes the suspect queued/NCQ command path completely and the measured performance penalty was only about 2.5% in the reproducer.

I am not claiming this mathematically proves that corruption can never occur again. No finite stress test can provide that guarantee.

What I do have now is a workload that repeatedly corrupts within seconds/minutes when multiple commands are allowed, while the selected production configuration survived more than seven hours continuously with zero corruption.

For practical purposes, I consider that sufficient qualification to return this system to production.

I will keep normal production integrity monitoring enabled, including Btrfs device-error counters, periodic scrub, SMART monitoring and storage-related kernel logging.

Remaining root-cause work

For now I am stopping the stress-test branch because I have a usable production mitigation.

I also investigated the JMB585 firmware/configuration question. The Penta board contains an external Winbond SPI flash next to the JMB585, but I have not found a sufficiently well-established safe Linux method to dump that configuration flash read-only. I therefore did not write to or flash the controller.

A second Penta/JMB585 or a controller with a different SATA chip could further separate hardware-specific causes, but I am not buying additional hardware solely for this investigation at this point.

If Radxa, JMicron or the Linux ATA developers want to investigate the underlying cause further, I still have the reproducer, scripts and detailed logs, including the 60-minute 443-corruption run and the queue-depth sweep, and I am happy to provide them.

Thanks especially to @radrocks and @nks for the suggestions that helped narrow this down.

11

@SMV_Mobiel

Which kernel versions have you managed to reproduce this on so far?

The kernel community will only care about your problem if you can reproduce it on the latest kernel version, which right now is v7.3-rc2.

A quick look at arch/arm64/boot/dts/broadcom/bcm2712-rpi-5-b-base.dtsi and arch/arm64/boot/dts/broadcom/bcm2712.dtsi shows pcie1 as status = okay.

So I would recommend you to try to reproduce the problem there first.

Usually, NCQ problems are related to specific drives and not to a specific AHCI controller.

See e.g.:

Considering how common JMB585 is, I would be quite surprised if NCQ was broken in the controller itself (and thus broken when using any NCQ capable drive).

There are only two AHCI controllers that are marked as having broken NCQ implementations (AHCI_HFLAG_NO_NCQ):

If you do have any other NCQ capable drive to test with, that would be nice.

If only the Kingston drive is affected, it could be quirked with ATA_QUIRK_NONCQ.

It could also be worthwhile to see if there is newer FW for the Kingston drive.

12

I first reproduced the issue on the kernel that came with Raspberry Pi OS, which was 6.18.x at the time. I then downgraded to 6.12.x, where the same issue was reproducible, and later tested 7.2.x at your request, where it was reproducible again. So far I have been able to reproduce the same problem on every kernel version I have tested.

Both Kingston SEDC500R960G drives are running firmware SCEKJ2.8.

Now that I know what the problem is, I rebuilt my setup around it and restored the system into its normal working configuration. Because of that, I no longer have all of the temporary fio test files and logs from the earlier testing.

Everything is currently running again in the rebuilt configuration, so I would prefer not to keep tearing it down and repeating the same tests for every new kernel release unless there is a specific change or hypothesis that needs to be tested.