Repeated EXT4 metadata corruption on Raspberry Pi 5 with Penta SATA HAT (JMB585) despite healthy SSDs and RAID1

Hardware and peripherals SATA HAT
10

Final update: reproducible trigger isolated and practical workaround qualified

Thanks everyone for the suggestions in this thread. I have continued testing this quite extensively, and I now have enough controlled results to document the final practical outcome.

I do not think I have proven the exact low-level root cause, but I have been able to:

  • reproduce the silent corruption reliably on a single SSD without md RAID;

  • isolate queueing / multiple outstanding commands as the trigger in my reproducer;

  • test queue_depth=1 for more than 10 hours without corruption;

  • test real libata.force=noncq for a continuous 7+ hours without corruption;

  • measure the performance cost of the mitigations;

  • choose a conservative production configuration.

Raspberry Pi EEPROM

To answer the earlier EEPROM question, the board is using:



BOOTLOADER:
CURRENT: Tue May 26 15:01:25 UTC 2026
build epoch: 1779807685
firmware git hash: 086b83e3
release: default

So the subsequent fresh reproducer and queueing tests were not performed on an unknown newer EEPROM revision.

Important change: I found a reliable reproducer

The earlier 256 GiB sequential CRC32C canary tests did not reproduce the corruption, despite several TiB of verified reads.

I therefore changed the workload shape rather than simply increasing the amount of sequential I/O.

I used a dedicated Kingston SEDC500R960G behind the same Penta/JMB585, with a fresh single-device Btrfs filesystem and no md RAID:



fio --name=sqlite_sim --directory=/mnt/jmbtest \
    --rw=randrw --rwmixread=50 \
    --bs=4k --iodepth=8 \
    --ioengine=io_uring \
    --direct=1 --fsync=8 \
    --size=8G --time_based \
    --verify=crc32c \
    --numjobs=2

The Raspberry Pi DMA restriction was active:



dma_mask_bits:             31
consistent_dma_mask_bits:  31

With the normal SATA device queue depth:



queue_depth = 32

the workload reproduced Btrfs checksum corruption very quickly.

Early repetitions failed after approximately:



~80 seconds
~41 seconds

The important point is that the fio EIO was generated because Btrfs detected incorrect data via its checksum. There was still no corresponding low-level device failure.

During the corruption:



write_io_errs   = 0
read_io_errs    = 0
flush_io_errs   = 0
generation_errs = 0

SMART remained clean and I still found no corresponding:



ATA error
AHCI reset/error
SATA CRC error
PCIe/AER error

So this reproduces essentially the same “silent below the filesystem” behaviour that originally appeared as EXT4 metadata corruption.

60-minute confirmation at queue_depth=32

I then allowed the same workload to continue despite checksum errors.

After 30 minutes:



Btrfs corruption_errs delta: 102

After the full 60-minute run:



Btrfs corruption_errs delta: 443

write_io_errs:    0
read_io_errs:     0
flush_io_errs:    0
generation_errs:  0

ATA/AHCI/AER error matches: 0
SMART: clean

The kernel printed fewer literal csum failed messages because Btrfs/kernel logging was rate-limited. The suppressed callbacks exactly accounted for the difference, so I used the Btrfs corruption_errs delta as the primary PASS/FAIL counter.

This also established something important:

md RAID1 is not required to reproduce the problem.

A single SSD through the JMB585 is sufficient.

Queue-depth sweep

I then tested the same reproducer while limiting the SATA device queue depth.

The most important results were:

queue_depth result
32 FAIL after 44 s
1 PASS
2 FAIL after 103 s
3 FAIL after 57 s
4 FAIL after 58 s
7 FAIL after 48 s
8 FAIL after 63 s
9 FAIL after 66 s
10 FAIL after 47 s
12 FAIL after 76 s
13 FAIL after 77 s
14 FAIL after 44 s
15 FAIL after 66 s
16 FAIL after 58 s

Depths 6 and 11 happened to survive their short 90-second screening runs, but I do not interpret those as safe values. The failure is stochastic and several neighbouring/higher depths fail quickly.

The important boundary is that queue_depth=2 already reproduced corruption, while queue_depth=1 survived all extended qualification runs.

For queue_depth=1 I recorded:



300 s      PASS
1800 s     PASS
7200 s     PASS
26226 s    PASS

Including the earlier 5- and 30-minute tests, the accumulated clean test time at queue_depth=1 is:



37,626 seconds
= 10 hours 27 minutes 6 seconds

with:



corruption_errs = 0

throughout those clean runs.

The continuous overnight portion alone ran for approximately:



7 hours 17 minutes

without corruption.

Important terminology correction: queue_depth=1 is not the same as NCQ disabled

Earlier in the investigation I sometimes described queue_depth=1 as “NCQ off”.

That is technically imprecise.

Setting:



queue_depth=1

limits Linux to one outstanding command, but the ATA command can still use the NCQ/FPDMA command path.

Actual NCQ disablement requires something such as:



libata.force=noncq

This turned out to be useful because it allowed me to distinguish:



A) multiple outstanding commands are required to trigger the problem

versus

B) merely using the NCQ/FPDMA protocol is enough to trigger it

Real libata.force=noncq test

I therefore enabled:



libata.force=noncq

on the actual active kernel command line.

After reboot the kernel explicitly confirmed:



ata1.00: FORCE: modified (noncq)
ata1.00: ... NCQ (not used)

and:



dma_mask_bits:             31
consistent_dma_mask_bits:  31
queue_depth:                1

I first ran the same reproducer for 15 minutes.

Result:



PASS
write_io_errs    0
read_io_errs     0
flush_io_errs    0
corruption_errs  0
generation_errs  0

I then ran a fresh Btrfs filesystem with the same workload continuously for exactly:



26,226 seconds
= 7 hours 17 minutes 6 seconds

Final result:



write_io_errs    0
read_io_errs     0
flush_io_errs    0
corruption_errs  0
generation_errs  0

So real non-NCQ also survived the complete long qualification run without a single detected corruption event.

Performance impact

I also ran direct 15-minute A/B tests using the same workload.

Results:

configuration READ WRITE result
64/64 DMA + queue_depth=1 3897 KiB/s 3900 KiB/s PASS
31/31 DMA + queue_depth=1 3856 KiB/s 3859 KiB/s PASS
31/31 DMA + real noncq 3760 KiB/s 3763 KiB/s PASS

The direct 64/64 versus 31/31 difference at depth 1 was only about:



1.1%

so I found no practical performance reason to remove the 32-bit DMA restriction.

Real noncq was approximately:



2.5%

slower than 31/31 + queue_depth=1 in this particular SQLite-like stress workload.

For me that is a very small price for the more conservative configuration.

I cannot provide an exact 7-hour depth1-versus-noncq throughput comparison because the overnight depth1 test script terminated fio at its time limit before fio wrote its normal final bandwidth summary. The integrity result of that run is unaffected: the test duration and Btrfs counters were recorded separately and the complete run was clean.

What I think the tests now show

The strongest result is no longer simply:



"NCQ on = bad, NCQ off = good"

It is more specifically:



more than one outstanding command allowed
    -> corruption repeatedly reproducible

maximum one outstanding command
    -> >10 hours of qualification without corruption

real non-NCQ
    -> 7h17 continuous qualification without corruption

Because queue_depth=1 remained clean while NCQ/FPDMA could technically still be in use, the mere presence of the NCQ protocol does not appear sufficient by itself.

The strongest trigger I have isolated is therefore multiple concurrent/outstanding SATA commands.

I do not think the testing identifies exactly where the underlying race/bug resides. Possibilities still include:



JMB585 controller logic/configuration
JMB585 <-> Linux libata/AHCI interaction
ordering/coherency behaviour
Raspberry Pi PCIe/JMB585 interaction
Penta-specific board behaviour
or some combination of these

I therefore do not claim that I have proven an internal JMB585 hardware defect.

Relation to the earlier 64-bit DMA issue

There is another useful conclusion here.

The new corruption reproducer repeatedly fails while the Raspberry Pi DMA overlay is already active and runtime masks are:



31 / 31

Therefore the previously discussed >4 GiB / 64-bit-DMA mechanism is not sufficient by itself to explain this reproducer.

That does not mean the DMA restriction is useless or that the earlier DMA reports are invalid. It means I appear to have found an additional queueing-sensitive failure mode, or an interaction that is still present while DMA is restricted.

The fact that the huge sequential canary workload remained clean while this small-random-I/O + fsync workload fails rapidly also shows that workload shape matters much more than total I/O volume in reproducing this problem.

Final production configuration

For production I have therefore chosen the most conservative configuration that I actually qualified:

config.txt:



dtoverlay=pcie-32bit-dma-pi5

kernel command line:



libata.force=noncq

Runtime this gives:



dma_mask_bits             31
consistent_dma_mask_bits  31
queue_depth               1
NCQ                       not used

queue_depth=1 by itself also passed more than 10 hours of testing, but I chose real noncq because it removes the suspect queued/NCQ command path completely and the measured performance penalty was only about 2.5% in the reproducer.

I am not claiming this mathematically proves that corruption can never occur again. No finite stress test can provide that guarantee.

What I do have now is a workload that repeatedly corrupts within seconds/minutes when multiple commands are allowed, while the selected production configuration survived more than seven hours continuously with zero corruption.

For practical purposes, I consider that sufficient qualification to return this system to production.

I will keep normal production integrity monitoring enabled, including Btrfs device-error counters, periodic scrub, SMART monitoring and storage-related kernel logging.

Remaining root-cause work

For now I am stopping the stress-test branch because I have a usable production mitigation.

I also investigated the JMB585 firmware/configuration question. The Penta board contains an external Winbond SPI flash next to the JMB585, but I have not found a sufficiently well-established safe Linux method to dump that configuration flash read-only. I therefore did not write to or flash the controller.

A second Penta/JMB585 or a controller with a different SATA chip could further separate hardware-specific causes, but I am not buying additional hardware solely for this investigation at this point.

If Radxa, JMicron or the Linux ATA developers want to investigate the underlying cause further, I still have the reproducer, scripts and detailed logs, including the 60-minute 443-corruption run and the queue-depth sweep, and I am happy to provide them.

Thanks especially to @radrocks and @nks for the suggestions that helped narrow this down.