Penta SATA HAT errors with 4x 3.5" 8TB HDD

Hey peeps,

So it’s been a rough ride.
I am trying to build a 4 bay, RaidZ1 NAS using the Penta SATA HAT.

I am using:
4x Western Digital WD Red Plus WD80EFPX 8 TB Hard Drive - 3.5’’
1x Mean Well (LRS-150-12) AC/DC Enclosed Power Supply (PSU) 150W, 12V, 12.5A

Don’t judge the layout too harshly, it’s still in test bench form -_-
Post attempt “create topic” press, turns out I’m limited to images/videos… hopefully the message still comes accross.

In the beginning, I was having a lot of scary noises, little did I know they were faults/errors/failures:
Initial build (powered by Molex)
Getting errors like this (Sorry, this image was before I knew there were more logs):

I then decided to beef it up, with the power supply mentioned above.
I plugged it in using the DC Barrel jack.

It did not help…

PSU was outputting 12.05V
DC Barrel jack soldering on top of the Penta SATA HAT was reading ~11.6V

I was also being greeted by these errors:

So what was next, well I figured out that the molex and DC plug share a rail.
So I doubled it up and plugged both in (from the same power supply to be clear)
This seemed to have solved my problems
Day 2 - Double Power Works… for now

However, this was only for writing to the pool…
Little did I know, that when reading from the pool, the same errors would show up.

This was when trying to copy a few GB’s from the pool:

I noticed the voltage on the Pi HAT was still dropping to ~11.6-11.8 at the molex, so I decided to up the voltage to ~12.4 on the PSU.
This gave the molex a reading of ~12.2V, still way within spec and this solved most of the problems.

I later realized I needed to shuffle some files around within the pool… this is when things got really bad.

My LAN speed is limited to 1Gbps.
Now that the drives could read/write at 200+Mbps between themselves, rather than my line speed of 100Mbps, the errors came back, this time even worse…

So bad in fact one drive even died on me, no longer registered to the pool. (It’s back now, don’t worry)
Another drive (sata4) appeared to be a repeat offender.
So I moved it to sata3 to test, and now sata3 is a big offender (still wondering if drive failure or SATA HAT)
Apologies in advance for the language, this was sent to my friend with knowledge:
Day 2 - Drive Failure and repeated faults

Below is the snippet where the one drive completely died and got removed from the pool:

[ 318.872597] ata2.00: exception Emask 0x10 SAct 0x18000 SErr 0xb80100 action 0x6
[ 318.872623] ata2.00: irq_stat 0x08000000
[ 318.872627] ata2: SError: { UnrecovData 10B8B Dispar BadCRC LinkSeq }
[ 318.872641] ata2.00: failed command: READ FPDMA QUEUED
[ 318.872645] ata2.00: cmd 60/b0:78:38:ff:1c/02:00:0e:00:00/40 tag 15 ncq dma 352256 in
res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 318.872655] ata2.00: status: { DRDY }
[ 318.872659] ata2.00: failed command: WRITE FPDMA QUEUED
[ 318.872661] ata2.00: cmd 61/58:80:d0:bc:80/00:00:ef:00:00/40 tag 16 ncq dma 45056 out
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 318.872668] ata2.00: status: { DRDY }
[ 318.872677] ata2: hard resetting link
[ 319.360599] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 319.424594] ata2.00: configured for UDMA/133
[ 319.432608] ata2: EH complete
[ 326.108710] ata2.00: exception Emask 0x10 SAct 0x180000 SErr 0xb00100 action 0x6
[ 326.108749] ata2.00: irq_stat 0x08000000
[ 326.108754] ata2: SError: { UnrecovData Dispar BadCRC LinkSeq }
[ 326.108766] ata2.00: failed command: READ FPDMA QUEUED
[ 326.108772] ata2.00: cmd 60/00:98:30:84:29/08:00:0e:00:00/40 tag 19 ncq dma 1048576 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 326.108782] ata2.00: status: { DRDY }
[ 326.108789] ata2.00: failed command: READ FPDMA QUEUED
[ 326.108793] ata2.00: cmd 60/a8:a0:e0:8e:29/02:00:0e:00:00/40 tag 20 ncq dma 348160 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 326.108798] ata2.00: status: { DRDY }
[ 326.108808] ata2: hard resetting link
[ 331.460634] ata2: link is slow to respond, please be patient (ready=-19)
[ 336.164671] ata2: found unknown device (class 0)
[ 336.320665] ata2: softreset failed (device not ready)
[ 336.320669] ata2: hard resetting link
[ 341.672704] ata2: link is slow to respond, please be patient (ready=-19)
[ 346.376738] ata2: found unknown device (class 0)
[ 346.532734] ata2: softreset failed (device not ready)
[ 346.532739] ata2: hard resetting link
[ 351.888773] ata2: link is slow to respond, please be patient (ready=-19)
[ 356.884804] ata2: found unknown device (class 0)
[ 362.080836] ata2: link is slow to respond, please be patient (ready=0)
[ 381.568964] ata2: softreset failed (device not ready)
[ 381.568974] ata2: limiting SATA link speed to 3.0 Gbps
[ 381.568977] ata2: hard resetting link
[ 386.589000] ata2: found unknown device (class 0)
[ 386.745001] ata2: softreset failed (device not ready)
[ 386.745013] ata2: softreset failed
[ 386.745016] ata2: reset failed, giving up
[ 386.745019] ata2.00: disable device
[ 386.745067] ata2: EH complete
[ 386.745093] sd 1:0:0:0: [sdb] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=60s
[ 386.745094] sd 1:0:0:0: [sdb] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=60s
[ 386.745100] sd 1:0:0:0: [sdb] tag#22 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 84 30 00 00 08 00 00 00
[ 386.745100] sd 1:0:0:0: [sdb] tag#21 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 0a d0 00 00 00 58 00 00
[ 386.745103] I/O error, dev sdb, sector 4018997968 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[ 386.745103] I/O error, dev sdb, sector 237601840 op 0x0:(READ) flags 0x0 phys_seg 16 prio class 0
[ 386.745109] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121651093504 size=1048576 flags=2148533376
[ 386.745110] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057725911040 size=45056 flags=3145856
[ 386.745938] sd 1:0:0:0: [sdb] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745951] sd 1:0:0:0: [sdb] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745956] sd 1:0:0:0: [sdb] tag#24 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 94 38 00 00 08 00 00 00
[ 386.745956] sd 1:0:0:0: [sdb] tag#23 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 0b 28 00 00 08 40 00 00
[ 386.745959] I/O error, dev sdb, sector 237605944 op 0x0:(READ) flags 0x0 phys_seg 18 prio class 0
[ 386.745960] I/O error, dev sdb, sector 4018998056 op 0x1:(WRITE) flags 0x0 phys_seg 17 prio class 0
[ 386.745963] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121653194752 size=348160 flags=3146112
[ 386.745966] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057725956096 size=45056 flags=3145856
[ 386.745971] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121653542912 size=700416 flags=2148533376
[ 386.745974] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057726001152 size=1036288 flags=2148533376
[ 386.745994] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.745998] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 00 00 00 0a 10 00 00 00 10 00 00
[ 386.746000] I/O error, dev sdb, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.746004] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=270336 size=8192 flags=1245377
[ 386.746736] sd 1:0:0:0: [sdb] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.746750] sd 1:0:0:0: [sdb] tag#25 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 9e e8 00 00 02 a8 00 00
[ 386.746754] I/O error, dev sdb, sector 237608680 op 0x0:(READ) flags 0x0 phys_seg 6 prio class 0
[ 386.746760] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121654595584 size=348160 flags=3146112
[ 386.746786] sd 1:0:0:0: [sdb] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.746793] sd 1:0:0:0: [sdb] tag#26 CDB: opcode=0x88 88 00 00 00 00 00 0e 29 a4 40 00 00 08 00 00 00
[ 386.746796] I/O error, dev sdb, sector 237610048 op 0x0:(READ) flags 0x0 phys_seg 16 prio class 0
[ 386.746801] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121655296000 size=1048576 flags=2148533376
[ 386.747016] sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747040] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x8a 8a 00 00 00 00 00 ef 8d 13 68 00 00 07 e8 00 00
[ 386.747045] I/O error, dev sdb, sector 4019000168 op 0x1:(WRITE) flags 0x0 phys_seg 16 prio class 0
[ 386.747054] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057727037440 size=1036288 flags=2148533376
[ 386.747285] sd 1:0:0:0: [sdb] tag#7 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747299] sd 1:0:0:0: [sdb] tag#7 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e4 10 00 00 00 10 00 00
[ 386.747304] I/O error, dev sdb, sector 15628035088 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.747310] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=8001552916480 size=8192 flags=1245377
[ 386.747335] sd 1:0:0:0: [sdb] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 386.747339] sd 1:0:0:0: [sdb] tag#8 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e6 10 00 00 00 10 00 00
[ 386.747342] I/O error, dev sdb, sector 15628035600 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 386.747346] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=8001553178624 size=8192 flags=1245377
[ 386.747439] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=2 offset=2057728073728 size=720896 flags=2148533376
[ 386.747702] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121656696832 size=348160 flags=3146112
[ 386.747713] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121657397248 size=1048576 flags=2148533376
[ 386.747941] zio pool=tank vdev=/dev/disk/by-id/ata-WDC_WD80EFPX-68C4ZN0_WD-RD2D7X0H-part1 error=5 type=1 offset=121652494336 size=348160 flags=3146112
[ 386.754701] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 386.758408] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 386.758678] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525043] scsi_io_completion_action: 9 callbacks suppressed
[ 395.525085] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525093] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525095] blk_print_req_error: 9 callbacks suppressed
[ 395.525098] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525141] sd 1:0:0:0: [sdb] tag#1 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525145] sd 1:0:0:0: [sdb] tag#1 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525148] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525153] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.525288] sd 1:0:0:0: [sdb] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525296] sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525299] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525303] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.525325] sd 1:0:0:0: [sdb] tag#2 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525329] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525330] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525343] sd 1:0:0:0: [sdb] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525345] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525347] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525349] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525508] sd 1:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525513] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525517] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.525543] sd 1:0:0:0: [sdb] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525546] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.525547] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.525550] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.525924] sd 1:0:0:0: [sdb] tag#9 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.525932] sd 1:0:0:0: [sdb] tag#9 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.525934] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.526020] sd 1:0:0:0: [sdb] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.526024] sd 1:0:0:0: [sdb] tag#10 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 395.526026] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 395.526029] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.526309] sd 1:0:0:0: [sdb] tag#11 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 395.526315] sd 1:0:0:0: [sdb] tag#11 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 395.526317] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 395.526336] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.531274] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.533186] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 395.533670] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 395.533798] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 397.079837] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 397.079915] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 397.079968] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.486317] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.486903] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 398.495726] program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
[ 407.515601] scsi_io_completion_action: 43 callbacks suppressed
[ 407.515611] sd 1:0:0:0: [sdb] tag#15 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515619] sd 1:0:0:0: [sdb] tag#15 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 407.515622] blk_print_req_error: 43 callbacks suppressed
[ 407.515624] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 407.515648] sd 1:0:0:0: [sdb] tag#16 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515652] sd 1:0:0:0: [sdb] tag#16 CDB: opcode=0x88 88 00 00 00 00 03 a3 81 27 80 00 00 00 20 00 00
[ 407.515655] I/O error, dev sdb, sector 15628052352 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 407.515659] buffer_io_error: 24 callbacks suppressed
[ 407.515661] Buffer I/O error on dev sdb9, logical block 508, async page read
[ 407.515836] sd 1:0:0:0: [sdb] tag#17 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515842] sd 1:0:0:0: [sdb] tag#17 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 407.515845] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 407.515865] sd 1:0:0:0: [sdb] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=DRIVER_OK cmd_age=0s
[ 407.515869] sd 1:0:0:0: [sdb] tag#18 CDB: opcode=0x88 88 00 00 00 00 03 a3 80 e7 80 00 00 00 20 00 00
[ 407.515871] I/O error, dev sdb, sector 15628035968 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 407.515874] Buffer I/O error on dev sdb1, logical block 488376060, async page read
[ 408.261173] ata3.00: exception Emask 0x10 SAct 0x10018008 SErr 0x49f0000 action 0xe frozen
[ 408.261203] ata3.00: irq_stat 0x00400000, PHY RDY changed
[ 408.261209] ata3: SError: { PHYRdyChg PHYInt CommWake 10B8B Dispar LinkSeq DevExch }
[ 408.261216] ata3.00: failed command: READ FPDMA QUEUED
[ 408.261220] ata3.00: cmd 60/20:18:00:18:00/00:00:00:00:00/40 tag 3 ncq dma 16384 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261230] ata3.00: status: { DRDY }
[ 408.261235] ata3.00: failed command: READ FPDMA QUEUED
[ 408.261237] ata3.00: cmd 60/b0:78:f8:fe:4d/02:00:0e:00:00/40 tag 15 ncq dma 352256 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261244] ata3.00: status: { DRDY }
[ 408.261247] ata3.00: failed command: WRITE FPDMA QUEUED
[ 408.261249] ata3.00: cmd 61/58:80:98:f3:b3/00:00:ef:00:00/40 tag 16 ncq dma 45056 out
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261257] ata3.00: status: { DRDY }
[ 408.261261] ata3.00: failed command: WRITE FPDMA QUEUED
[ 408.261263] ata3.00: cmd 61/58:e0:f0:f3:b3/00:00:ef:00:00/40 tag 28 ncq dma 45056 out
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
[ 408.261271] ata3.00: status: { DRDY }
[ 408.261281] ata3: hard resetting link
[ 415.133170] ata3: link is slow to respond, please be patient (ready=-19)
[ 416.353183] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 416.385427] ata3.00: configured for UDMA/133
[ 416.385458] sd 2:0:0:0: [sdc] tag#3 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=8s
[ 416.385464] sd 2:0:0:0: [sdc] tag#3 Sense Key : 0x5 [current]
[ 416.385468] sd 2:0:0:0: [sdc] tag#3 ASC=0x21 ASCQ=0x4
[ 416.385471] sd 2:0:0:0: [sdc] tag#3 CDB: opcode=0x88 88 00 00 00 00 00 00 00 18 00 00 00 00 20 00 00
[ 416.385474] I/O error, dev sdc, sector 6144 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 416.385519] ata3: EH complete

I’m now at a point where I can write as much as I want to the pool (so far), but I cannot read from the pool at all, otherwise drives start resetting/dyeing/erroring.

I’m really not sure what else to try at this point, and looking for some input from the community.

My Linux computer I am working at has 24 inch 16:9 1080p60. I won’t rotate your YT vid 90 degrees, maybe others do. So next time, post a diagram of your power wiring instead in LANDSCAPE mode. Or at least turn that phone 90 degrees. Or a few clear pictures at least where you also yourself would be able to draw a diagram from in one year from now for example.

Anyway, you seem to route all the currents through that HAT, both from 5V and 12V. That is the last thing I would do, but that is what that HAT invites you to do. Have you ever build a PC with 4x 3.5inch HDD?

I have 1 8T Seagate SMR HDD ‘connected’ to a ROCK5B via its native on-chip SATA, but at least 12V is directly from the PSU. The 5V I get from the ROCK5B, but that is tricky, not an advice. Best to make sure the 5V and 12V for the HDDs are external and then also feed your SBC from there. In my case, ROCK5B can accept 12V fixed on the USB-C power connector. I bought an M.2 M-key ASM1166 for tests, that only has the SATA signals connector, power is DIY, which is fine as that is how PCs are for decades (I have enough unused PC PSUs).

You might have ground problems, loop like, that the SATA signal will be stressed/corrupted. Or check your 8T HDD spec. If SMR, it is all familiar what I see in the kernel log.

The OS kernel DTB etc is unclear. Also ZFS complicates it, is not native Linux, maybe the integration with Linux has issues on your installation. It a common topic on the internet.