Penta SATA HAT errors with 4x 3.5" 8TB HDD

[Jebula999]

Hey peeps,

So it’s been a rough ride.
I am trying to build a 4 bay, RaidZ1 NAS using the Penta SATA HAT.

I am using:
4x Western Digital WD Red Plus WD80EFPX 8 TB Hard Drive - 3.5’’
1x Mean Well (LRS-150-12) AC/DC Enclosed Power Supply (PSU) 150W, 12V, 12.5A

Don’t judge the layout too harshly, it’s still in test bench form -_-
Post attempt “create topic” press, turns out I’m limited to images/videos… hopefully the message still comes accross.

In the beginning, I was having a lot of scary noises, little did I know they were faults/errors/failures:
Initial build (powered by Molex)
Getting errors like this (Sorry, this image was before I knew there were more logs):

I then decided to beef it up, with the power supply mentioned above.
I plugged it in using the DC Barrel jack.

It did not help…

PSU was outputting 12.05V
DC Barrel jack soldering on top of the Penta SATA HAT was reading ~11.6V

I was also being greeted by these errors:

So what was next, well I figured out that the molex and DC plug share a rail.
So I doubled it up and plugged both in (from the same power supply to be clear)
This seemed to have solved my problems
Day 2 - Double Power Works… for now

However, this was only for writing to the pool…
Little did I know, that when reading from the pool, the same errors would show up.

This was when trying to copy a few GB’s from the pool:

I noticed the voltage on the Pi HAT was still dropping to ~11.6-11.8 at the molex, so I decided to up the voltage to ~12.4 on the PSU.
This gave the molex a reading of ~12.2V, still way within spec and this solved most of the problems.

I later realized I needed to shuffle some files around within the pool… this is when things got really bad.

My LAN speed is limited to 1Gbps.
Now that the drives could read/write at 200+Mbps between themselves, rather than my line speed of 100Mbps, the errors came back, this time even worse…

So bad in fact one drive even died on me, no longer registered to the pool. (It’s back now, don’t worry)
Another drive (sata4) appeared to be a repeat offender.
So I moved it to sata3 to test, and now sata3 is a big offender (still wondering if drive failure or SATA HAT)
Apologies in advance for the language, this was sent to my friend with knowledge:
Day 2 - Drive Failure and repeated faults

Below is the snippet where the one drive completely died and got removed from the pool:

[ 318.872597] ata2.00: exception Emask 0x10 SAct 0x18000 SErr 0xb80100 action 0x6
[ 318.872623] ata2.00: irq_stat 0x08000000
[ 318.872627] ata2: SError: { UnrecovData 10B8B Dispar BadCRC LinkSeq }
[ 318.872641] ata2.00: failed command: READ FPDMA QUEUED
[ 318.872645] ata2.00: cmd 60/b0:78:38:ff:1c/02:00:0e:00:00/40 tag 15 ncq dma 352256 in
res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 318.872655] ata2.00: status: { DRDY }
[ 318.872659] ata2.00: failed command: WRITE FPDMA QUEUED
[ 318.872661] ata2.00: cmd 61/58:80:d0:bc:80/00:00:ef:00:00/40 tag 16 ncq dma 45056 out
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 318.872668] ata2.00: status: { DRDY }
[ 318.872677] ata2: hard resetting link
[ 319.360599] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 319.424594] ata2.00: configured for UDMA/133
[ 319.432608] ata2: EH complete
[ 326.108710] ata2.00: exception Emask 0x10 SAct 0x180000 SErr 0xb00100 action 0x6
[ 326.108749] ata2.00: irq_stat 0x08000000
[ 326.108754] ata2: SError: { UnrecovData Dispar BadCRC LinkSeq }
[ 326.108766] ata2.00: failed command: READ FPDMA QUEUED
[ 326.108772] ata2.00: cmd 60/00:98:30:84:29/08:00:0e:00:00/40 tag 19 ncq dma 1048576 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 326.108782] ata2.00: status: { DRDY }
[ 326.108789] ata2.00: failed command: READ FPDMA QUEUED
[ 326.108793] ata2.00: cmd 60/a8:a0:e0:8e:29/02:00:0e:00:00/40 tag 20 ncq dma 348160 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 326.108798] ata2.00: status: { DRDY }
[ 326.108808] ata2: hard resetting link
[ 331.460634] ata2: link is slow to respond, please be patient (ready=-19)
[ 336.164671] ata2: found unknown device (class 0)
[ 336.320665] ata2: softreset failed (device not ready)
[ 336.320669] ata2: hard resetting link
[ 341.672704] ata2: link is slow to respond, please be patient (ready=-19)
[ 346.376738] ata2: found unknown device (class 0)
[ 346.532734] ata2: softreset failed (device not ready)
[ 346.532739] ata2: hard resetting link
[ 351.888773] ata2: link is slow to respond, please be patient (ready=-19)
[ 356.884804] ata2: found unknown device (class 0)
[ 362.080836] ata2: link is slow to respond, please be patient (ready=0)
[ 381.568964] ata2: softreset failed (device not ready)
[ 381.568974] ata2: limiting SATA link speed to 3.0 Gbps
[ 381.568977] ata2: hard resetting link
[ 386.589000] ata2: found unknown device (class 0)
[ 386.745001] ata2: softreset failed (device not ready)
[ 386.745013] ata2: softreset failed
[ 386.745016] ata2: reset failed, giving up
[ 386.745019] ata2.00: disable device
[ 386.745067] ata2: EH complete
[ 386.745093] sd 1:0:0:0: [sdb] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=60s
[ 386.745094] sd 1:0:0:0: [sdb] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=60s
[ 386.745100] sd 1:0:0:0: [sdb] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 84 30 00 00 08 00 00 00
[ 386.745100] sd 1:0:0:0: [sdb] tag#21 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 0a d0 00 00 00 58 00 00
[ 386.745103] I/O error, dev sdb, sector 4018997968 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 386.745103] I/O error, dev sdb, sector 237601840 op 0x0:(READ) flags 0x0 phys_seg 16 prio class 0
[ 386.745109] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121651093504 size=1048576 flags=2148533376
[ 386.745110] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057725911040 size=45056 flags=3145856
[ 386.745938] sd 1:0:0:0: [sdb] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745951] sd 1:0:0:0: [sdb] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745956] sd 1:0:0:0: [sdb] tag#24 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 94 38 00 00 08 00 00 00
[ 386.745956] sd 1:0:0:0: [sdb] tag#23 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 0b 28 00 00 08 40 00 00
[ 386.745959] I/O error, dev sdb, sector 237605944 op 0x0:(READ) flags 0x0 phys_seg 18 prio class 0
[ 386.745960] I/O error, dev sdb, sector 4018998056 op 0x1:(WRITE) flags 0x0 phys_seg 17 prio class 0
[ 386.745963] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121653194752 size=348160 flags=3146112
[ 386.745966] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057725956096 size=45056 flags=3145856
[ 386.745971] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121653542912 size=700416 flags=2148533376
[ 386.745974] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057726001152 size=1036288 flags=2148533376
[ 386.745994] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745998] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 00 00 00 0a 10 00 00 00 10 00 00
[ 386.746000] I/O error, dev sdb, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.746004] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=270336 size=8192 flags=1245377
[ 386.746736] sd 1:0:0:0: [sdb] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.746750] sd 1:0:0:0: [sdb] tag#25 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 9e e8 00 00 02 a8 00 00
[ 386.746754] I/O error, dev sdb, sector 237608680 op 0x0:(READ) flags 0x0 phys_seg 6 prio class 0
[ 386.746760] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121654595584 size=348160 flags=3146112
[ 386.746786] sd 1:0:0:0: [sdb] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.746793] sd 1:0:0:0: [sdb] tag#26 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 a4 40 00 00 08 00 00 00
[ 386.746796] I/O error, dev sdb, sector 237610048 op 0x0:(READ) flags 0x0 phys_seg 16 prio class 0
[ 386.746801] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121655296000 size=1048576 flags=2148533376
[ 386.747016] sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747040] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 13 68 00 00 07 e8 00 00
[ 386.747045] I/O error, dev sdb, sector 4019000168 op 0x1:(WRITE) flags 0x0 phys_seg 16 prio class 0
[ 386.747054] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057727037440 size=1036288 flags=2148533376
[ 386.747285] sd 1:0:0:0: [sdb] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747299] sd 1:0:0:0: [sdb] tag#7 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e4 10 00 00 00 10 00 00
[ 386.747304] I/O error, dev sdb, sector 15628035088 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.747310] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=8001552916480 size=8192 flags=1245377
[ 386.747335] sd 1:0:0:0: [sdb] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747339] sd 1:0:0:0: [sdb] tag#8 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e6 10 00 00 00 10 00 00
[ 386.747342] I/O error, dev sdb, sector 15628035600 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.747346] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=8001553178624 size=8192 flags=1245377
[ 386.747439] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057728073728 size=720896 flags=2148533376
[ 386.747702] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121656696832 size=348160 flags=3146112
[ 386.747713] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121657397248 size=1048576 flags=2148533376
[ 386.747941] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121652494336 size=348160 flags=3146112
[ 386.754701] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 386.758408] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 386.758678] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525043] scsi_io_completion_action: 9 callbacks suppressed
[ 395.525085] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525093] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525095] blk_print_req_error: 9 callbacks suppressed
[ 395.525098] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525141] sd 1:0:0:0: [sdb] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525145] sd 1:0:0:0: [sdb] tag#1 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525148] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525153] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.525288] sd 1:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525296] sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525299] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525303] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.525325] sd 1:0:0:0: [sdb] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525329] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525330] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525343] sd 1:0:0:0: [sdb] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525345] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525347] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525349] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525508] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525513] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525517] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525543] sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525546] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525547] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525550] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525924] sd 1:0:0:0: [sdb] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525932] sd 1:0:0:0: [sdb] tag#9 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525934] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.526020] sd 1:0:0:0: [sdb] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.526024] sd 1:0:0:0: [sdb] tag#10 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.526026] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.526029] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.526309] sd 1:0:0:0: [sdb] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.526315] sd 1:0:0:0: [sdb] tag#11 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.526317] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.526336] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.531274] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.533186] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.533670] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.533798] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 397.079837] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 397.079915] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 397.079968] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.486317] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.486903] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.495726] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 407.515601] scsi_io_completion_action: 43 callbacks suppressed
[ 407.515611] sd 1:0:0:0: [sdb] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515619] sd 1:0:0:0: [sdb] tag#15 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 407.515622] blk_print_req_error: 43 callbacks suppressed
[ 407.515624] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 407.515648] sd 1:0:0:0: [sdb] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515652] sd 1:0:0:0: [sdb] tag#16 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 407.515655] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 407.515659] buffer_io_error: 24 callbacks suppressed
[ 407.515661] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 407.515836] sd 1:0:0:0: [sdb] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515842] sd 1:0:0:0: [sdb] tag#17 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 407.515845] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 407.515865] sd 1:0:0:0: [sdb] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515869] sd 1:0:0:0: [sdb] tag#18 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 407.515871] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 407.515874] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 408.261173] ata3.00: exception Emask 0x10 SAct 0x10018008 SErr 0x49f0000 action 0xe frozen
[ 408.261203] ata3.00: irq_stat 0x00400000, PHY RDY changed
[ 408.261209] ata3: SError: { PHYRdyChg PHYInt CommWake 10B8B Dispar LinkSeq DevExch }
[ 408.261216] ata3.00: failed command: READ FPDMA QUEUED
[ 408.261220] ata3.00: cmd 60/20:18:00:18:00/00:00:00:00:00/40 tag 3 ncq dma 16384 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261230] ata3.00: status: { DRDY }
[ 408.261235] ata3.00: failed command: READ FPDMA QUEUED
[ 408.261237] ata3.00: cmd 60/b0:78:f8:fe:4d/02:00:0e:00:00/40 tag 15 ncq dma 352256 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261244] ata3.00: status: { DRDY }
[ 408.261247] ata3.00: failed command: WRITE FPDMA QUEUED
[ 408.261249] ata3.00: cmd 61/58:80:98:f3:b3/00:00:ef:00:00/40 tag 16 ncq dma 45056 out
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261257] ata3.00: status: { DRDY }
[ 408.261261] ata3.00: failed command: WRITE FPDMA QUEUED
[ 408.261263] ata3.00: cmd 61/58:e0:f0:f3:b3/00:00:ef:00:00/40 tag 28 ncq dma 45056 out
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261271] ata3.00: status: { DRDY }
[ 408.261281] ata3: hard resetting link
[ 415.133170] ata3: link is slow to respond, please be patient (ready=-19)
[ 416.353183] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 416.385427] ata3.00: configured for UDMA/133
[ 416.385458] sd 2:0:0:0: [sdc] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=8s
[ 416.385464] sd 2:0:0:0: [sdc] tag#3 Sense Key : 0x5 [current]
[ 416.385468] sd 2:0:0:0: [sdc] tag#3 ASC=0x21 ASCQ=0x4
[ 416.385471] sd 2:0:0:0: [sdc] tag#3 CDB: opcode=0x88 88 00 00 00 00 00 00 00 18 00 00 00 00 20 00 00
[ 416.385474] I/O error, dev sdc, sector 6144 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 416.385519] ata3: EH complete

I’m now at a point where I can write as much as I want to the pool (so far), but I cannot read from the pool at all, otherwise drives start resetting/dyeing/erroring.

I’m really not sure what else to try at this point, and looking for some input from the community.

[radrocks]

My Linux computer I am working at has 24 inch 16:9 1080p60. I won’t rotate your YT vid 90 degrees, maybe others do. So next time, post a diagram of your power wiring instead in LANDSCAPE mode. Or at least turn that phone 90 degrees. Or a few clear pictures at least where you also yourself would be able to draw a diagram from in one year from now for example.

Anyway, you seem to route all the currents through that HAT, both from 5V and 12V. That is the last thing I would do, but that is what that HAT invites you to do. Have you ever build a PC with 4x 3.5inch HDD?

I have 1 8T Seagate SMR HDD ‘connected’ to a ROCK5B via its native on-chip SATA, but at least 12V is directly from the PSU. The 5V I get from the ROCK5B, but that is tricky, not an advice. Best to make sure the 5V and 12V for the HDDs are external and then also feed your SBC from there. In my case, ROCK5B can accept 12V fixed on the USB-C power connector. I bought an M.2 M-key ASM1166 for tests, that only has the SATA signals connector, power is DIY, which is fine as that is how PCs are for decades (I have enough unused PC PSUs).

You might have ground problems, loop like, that the SATA signal will be stressed/corrupted. Or check your 8T HDD spec. If SMR, it is all familiar what I see in the kernel log.

The OS kernel DTB etc is unclear. Also ZFS complicates it, is not native Linux, maybe the integration with Linux has issues on your installation. It a common topic on the internet.

[Jebula999]

My Linux computer I am working at has 24 inch 16:9 1080p60. I won’t rotate your YT vid 90 degrees, maybe others do. So next time, post a diagram of your power wiring instead in LANDSCAPE mode. Or at least turn that phone 90 degrees. Or a few clear pictures at least where you also yourself would be able to draw a diagram from in one year from now for example.

Point noted.
I was limited by the number of images / videos I could add at the time. And was sharing historical videos of my issue I was sending to friends for advice who were viewing them on mobile phones.

Kinda hard to go back and re-record old footage, and not keen on constantly letting the errors persist in case of data corruption, or in the worst case, complete hardware failure.

But the power supply us straight forward:

PSU -> 12.4V -> DC Barrel Jack
PSU -> 12.4V -> Molex Connector (12V and Ground)

Anyway, you seem to route all the currents through that HAT, both from 5V and 12V. That is the last thing I would do, but that is what that HAT invites you to do.

Correct, but, as per the official documentation, they say to NOT use the Pi’s own power, as the HAT will power the Pi.

From testing, the DC Barrel Jack and Molex share the same power rail, so it’s safe to supply input power from both ends (from the same power supply)
Less strain on both connectors.

ave you ever build a PC with 4x 3.5inch HDD?

I actually have one running at the moment with 4x 3.5inch HDD’s and 1x SSD.
Although set up in a 2x mirror pair.
Been running for more than a year with 0 problems.

You might have ground problems, loop like, that the SATA signal will be stressed/corrupted. Or check your 8T HDD spec. If SMR, it is all familiar what I see in the kernel log.

The OS kernel DTB etc is unclear. Also ZFS complicates it, is not native Linux, maybe the integration with Linux has issues on your installation. It a common topic on the internet.

Unfortunately not 100% sure what you meaning here.
But I have seen many a video do what I am trying to do, with 0 issues like I am having.
I think the HAT is also advertised to support 100TB (I could be mistaken here)

[radrocks]

PSU -> 12.4V -> DC Barrel Jack
PSU -> 12.4V -> Molex Connector (12V and Ground)

In theory you have a loop(s) here; There is 3 GND wires, in barrel jack, in Molex and shielding in SATA signal cable. I would only use the Molex. Or as said, power HDDs directly from PSU.

But of more importance is; What is OS, kernel, U-Boot. Al that is missing. Versions etc are needed else people will have no-way to compare or even reproduce. Also I forgot which SBC it is. That also might matter, there is that PCI-E flatcable that always scares me (fragile, can challenge signal integrity).

[Jebula999]

But of more importance is; What is OS, kernel, U-Boot. Al that is missing. Versions etc are needed else people will have no-way to compare or even reproduce. Also I forgot which SBC it is.

jeb@nas:~ $ lsb_release -a 2>/dev/null || cat /etc/os-release
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

jeb@nas:~ $ uname -a
Linux nas 6.12.34+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux

jeb@nas:~ $ cat /proc/cpuinfo | grep Model
Model           : Raspberry Pi 5 Model B Rev 1.1

I did try run
dpkg -l | grep u-boot || strings /dev/mmcblk0 | grep U-Boot | head
But nothing showed up…

That also might matter, there is that PCI-E flatcable that always scares me (fragile, can challenge signal integrity).

I believe I am using the one I received with the Radxa Penta SATA HAT (It came with 2)

Since I last posted, I did realize that if I pushed sideways (don’t shout at me) on the SATA cable (ata2) on the HAT itself, the error stopped on that drive, but then it moved to another drive (randomly).

When I stopped applying pressure… it went back to (ata2) as before.

I’m wondering if I have just terrible SATA cables here or something, as it appears for now at least it’s the case of the weakest link, or rather weakest connection.

[Jebula999]

I ain’t much of an electrical line diagram maker, never mind trying to actually remember which cables are which.
But hopefully the below piece of art helps… (I might have swapped the barrel cables around in the diagram, but I ensured they were correct when wired up)

[radrocks]

I often use pen and paper and take a picture or scan it, but your info and picture is OK now.

About the GND loops, it won’t be an issue here as short wires and relatively low currents. But if you follow the black wires, you pass the PSU and the HAT, and can go round endlessly. That is the loop.

But you already pinpoint the issue I think. I guess some SATA cable or connector or the PCI-E flatcable might have some bad contact, so that is fatal more or less. I had similar issue when my 8T HDD was connected to Asrock J1900 based board. That has older SATA2 ports. I still don’t know if the SATA cable was the issue, the J1900 or ATX PSU (PicoPSU) or Debian12 or maybe a combination. At least 2 times failure in checksums and many times SATA link failure. As only 1 HDD, you don’t at least trash a whole RAID array, but that is what you risk.

It is not only SATA cables, but also that additional PCI-E flatcable. I thought it was a Radxa SBC, but RPi5 has only 1 PCI-E lane, spec does not allow PCI-E v3 although it mostly works it seems. Maybe that is your issue.
To be honest, building RAID with ZFS and 16k pages RPi5 kernel is like a F1 car with bicycle wheels. YT guys do it for the clicks eventually, so I never watch those vids. As indicated, ROCK5B but also ROCK3A and many other RK35xx boards have on-board 1-lane or multi-lane PCI-E M.2 M-key slot, you can stick a 15 dollar 6-port ASM1166 in it, then all fixed direct connections, very high speed as well for SSD SATA. My ROCK5B uses EDK-UEFI v1.1 BIOS, so can run generic Opensuse Tumbleweed out-of-the-box or Ubuntu or else (FreeBSD I haven’t tried). It can also boot from SATA, no SD-card or so needed, so just like an Intel/AMD PC.
My RPi4 with USB3-SATA is the last RPi if I want storage, it works, but Raspberry is not good for storage servers.

[Jebula999]

Thank you for the info and the input.
I will keep this post updated when the new SATA cables arrive.

The reason for going with the Pi5 was that I already have a few for other projects.
I have a PC running Proxmox with a few VM’s, one of which was TrueNAS running.

But it was virtualized, with 4 mirrored 3TB drives (so only 6TB storage)
But due to it being on old 2nd hand drives, and in a virtualized VM I wasn’t sure how to fix if it broke, I decided to make a dedicated NAS.

I wanted it to be small and functional (I don’t need high speeds etc for video editing/workflows), just a media server and storage.

Naturally, SSD’s are very expensive and hard to find, especially in this country.
To make the same size NAS with SSD’s as opposed to HDD’s would cost > 5 times as much.
And one dead drive would be extremely expensive to replace. More than the cost of my entire 32TB Pi NAS combined.

This was done more for the “hobby” and tinkering aspect, than production/work heavy use case.

[radrocks]

I use VM’s on RPI4 (my router for example), on ROCK5B and ROCK3A. The ROCK3A is my main NAS, it is only Armbian/Debian. Both ROCK5B and ROCK3A can use a Radxa SATA M.2 E-key break-out board, cost 3 USD. And they then still have extra multi-lane M.2 M-key PCI-E for the NVME, on ROCK5B more than 2500 GByte/s so, great for HDD-caching and OS and many VMs. Whole ROCK3A, so my NAS, is about 4.5Watt idle.
I avoid Proxmox, just standard Debian libvirtd/KVM/QEMU. Also no RAID, just Btrfs send-receive between Linux boxes to have redundancy. Debian x86-64 and Debian aarch64 is the same for all configs and principles, only different binaries of course.