RAID continue to resync

#1

Hi all,
I created a RAID 5 with 4 new hard disks, Western Digital 2TB each,
The RAID has been created through command line, because OMV can’t create RAID 5 from graphic interface with USB disks.
Anyway it works, but I noticed it started resync many times (each resync takes more or less 8-10 hours), so if I reboot the device for any reason very often it starts to resync, because the state of RAID is marked as “not clean”.
I checked some syslog files, and I found these errors related to “md0”:

Mar 9 05:16:32 raspberrypi kernel: [34583.664084] md/raid:md0: Disk failure on sdb, disabling device.
Mar 9 05:16:32 raspberrypi kernel: [34583.664084] md/raid:md0: Operation continuing on 3 devices.
Mar 9 05:16:32 raspberrypi kernel: [34583.666176] md/raid:md0: read error not correctable (sector 58016224 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666195] md/raid:md0: read error not correctable (sector 58016232 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666209] md/raid:md0: read error not correctable (sector 58016240 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666222] md/raid:md0: read error not correctable (sector 58016248 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666234] md/raid:md0: read error not correctable (sector 58016256 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666247] md/raid:md0: read error not correctable (sector 58016264 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666260] md/raid:md0: read error not correctable (sector 58016272 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666273] md/raid:md0: read error not correctable (sector 58016280 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666285] md/raid:md0: read error not correctable (sector 58016288 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.666300] md/raid:md0: read error not correctable (sector 58016296 on sda).
Mar 9 05:16:32 raspberrypi kernel: [34583.827990] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777794 starting block 22208661)
Mar 9 05:16:32 raspberrypi kernel: [34583.828330] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777794 starting block 22223584)
Mar 9 05:16:33 raspberrypi kernel: [34583.829634] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777794 starting block 22225286)
Mar 9 05:16:33 raspberrypi kernel: [34583.830529] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777792 starting block 21926224)
Mar 9 05:16:33 raspberrypi kernel: [34583.830542] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777792 starting block 21925904)
Mar 9 05:16:33 raspberrypi kernel: [34583.831103] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777792 starting block 21963088)
Mar 9 05:16:33 raspberrypi kernel: [34583.832462] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777774 starting block 21299841)
Mar 9 05:16:33 raspberrypi kernel: [34583.832961] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777788 starting block 21763925)
Mar 9 05:16:33 raspberrypi kernel: [34583.834097] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777788 starting block 21642812)
Mar 9 05:16:33 raspberrypi kernel: [34583.834420] EXT4-fs warning (device md0): ext4_end_bio:349: I/O error 10 writing to inode 101777788 starting block 21764024)
Mar 9 05:16:33 raspberrypi kernel: [34584.136336] Buffer I/O error on device md0, logical block 22208661
Mar 9 05:16:33 raspberrypi kernel: [34584.136398] Buffer I/O error on device md0, logical block 22208662
Mar 9 05:16:33 raspberrypi kernel: [34584.136405] Buffer I/O error on device md0, logical block 22208663
Mar 9 05:16:33 raspberrypi kernel: [34584.136411] Buffer I/O error on device md0, logical block 22208664
Mar 9 05:16:33 raspberrypi kernel: [34584.136418] Buffer I/O error on device md0, logical block 22208665
Mar 9 05:16:33 raspberrypi kernel: [34584.136425] Buffer I/O error on device md0, logical block 22208666
Mar 9 05:16:33 raspberrypi kernel: [34584.136431] Buffer I/O error on device md0, logical block 22208667
Mar 9 05:16:33 raspberrypi kernel: [34584.136437] Buffer I/O error on device md0, logical block 22208668
Mar 9 05:16:33 raspberrypi kernel: [34584.136443] Buffer I/O error on device md0, logical block 22208669
Mar 9 05:16:33 raspberrypi kernel: [34584.136449] Buffer I/O error on device md0, logical block 22208670
Mar 9 05:16:33 raspberrypi kernel: [34584.140700] Aborting journal on device md0-8.
Mar 9 05:16:33 raspberrypi kernel: [34584.149737] Buffer I/O error on dev md0, logical block 732463104, lost sync page write
Mar 9 05:16:33 raspberrypi kernel: [34584.149821] JBD2: Error -5 detected when updating journal superblock for md0-8.
Mar 9 05:16:33 raspberrypi kernel: [34584.150284] Buffer I/O error on dev md0, logical block 0, lost sync page write
Mar 9 05:16:33 raspberrypi kernel: [34584.150351] EXT4-fs (md0): I/O error while writing superblock
Mar 9 05:16:33 raspberrypi kernel: [34584.150359] EXT4-fs error (device md0): ext4_journal_check_start:83: Detected aborted journal
Mar 9 05:16:33 raspberrypi kernel: [34584.150381] EXT4-fs (md0): Remounting filesystem read-only
Mar 9 05:16:33 raspberrypi kernel: [34584.150397] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777794, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.157018] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777794, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.163415] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777792, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.174015] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777792, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.183482] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777792, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.186846] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777792, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.187315] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777788, error -30)
Mar 9 05:16:33 raspberrypi kernel: [34584.190653] EXT4-fs (md0): failed to convert unwritten extents to written extents – potential data loss! (inode 101777788, error -30)
Mar 9 05:16:36 raspberrypi systemd[1]: Unmounting /mnt/md0…
Mar 9 05:16:36 raspberrypi systemd[1]: mnt-md0.mount: Succeeded.
Mar 9 05:16:36 raspberrypi systemd[1]: Unmounted /mnt/md0.
Mar 9 07:17:56 raspberrypi kernel: [ 18.401286] md/raid:md0: not clean – starting background reconstruction
Mar 9 07:17:56 raspberrypi kernel: [ 18.401399] md/raid:md0: device sdd operational as raid disk 1
Mar 9 07:17:56 raspberrypi kernel: [ 18.401413] md/raid:md0: device sdc operational as raid disk 0
Mar 9 07:17:56 raspberrypi kernel: [ 18.401425] md/raid:md0: device sda operational as raid disk 2
Mar 9 07:17:56 raspberrypi kernel: [ 18.401437] md/raid:md0: device sdb operational as raid disk 3
Mar 9 07:17:56 raspberrypi kernel: [ 18.406352] md/raid:md0: raid level 5 active with 4 out of 4 devices, algorithm 2
Mar 9 07:17:56 raspberrypi kernel: [ 18.528014] md0: detected capacity change from 0 to 6000790732800
Mar 9 07:17:57 raspberrypi systemd[1]: Found device /dev/md0.
Mar 9 07:17:57 raspberrypi systemd[1]: Mounting /mnt/md0…
Mar 9 07:17:57 raspberrypi systemd-fsck[1065]: /dev/md0 is in use.
Mar 9 07:18:03 raspberrypi kernel: [ 24.894656] md: resync of RAID array md0
Mar 9 07:18:08 raspberrypi kernel: [ 30.465621] EXT4-fs (md0): recovery complete
Mar 9 07:18:09 raspberrypi kernel: [ 30.948954] EXT4-fs (md0): mounting with “discard” option, but the device does not support discard
Mar 9 07:18:09 raspberrypi kernel: [ 30.948983] EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: discard
Mar 9 07:18:09 raspberrypi systemd[1]: Mounted /mnt/md0.
Mar 9 07:19:10 raspberrypi kernel: [ 74.219700] md: md0: resync done.
Mar 9 23:06:00 raspberrypi systemd[1]: Unmounting /mnt/md0…
Mar 9 23:06:00 raspberrypi systemd[25667]: mnt-md0.mount: Succeeded.
Mar 9 23:06:00 raspberrypi systemd[1]: mnt-md0.mount: Succeeded.
Mar 9 23:06:00 raspberrypi systemd[1]: Unmounted /mnt/md0.
Mar 9 23:06:00 raspberrypi blkdeactivate[25743]: [MD]: deactivating raid5 device md0… skipping

But if I search “md0” and “Disk failure” in the others syslog files I found the same errors, many times, related to 3 of 4 hard disks:

Mar 7 04:20:06 raspberrypi kernel: [49665.238834] md/raid:md0: Disk failure on sdc, disabling device.

Mar 2 20:50:41 raspberrypi kernel: [ 8465.058719] md/raid:md0: Disk failure on sdb, disabling device.

Mar 7 21:40:25 raspberrypi kernel: [ 2839.161000] md/raid:md0: Disk failure on sdd, disabling device.

What do you think?
All disks are new, I think they are not broken.

Thank you

#2

2tb huh… What is model and what is your power supply?

#3

The model is:

WesternDigital WD20SPZX 2TB Sata III 2,5" 5400rpm 128MB 7mm

The power supply is WL120060CB-R
12V 5.0A

Do you think that power supply doesn’t support 4 hard disks?

Another detail, I didn’t update the firmware of interface because the seller suggested to don’t do it if not necessary

#4

Well, 2.5’’ not using 12v line at all, but i guess power supply is connected to RasPi?
4K%20RPM%20SATA%20Hard%20Drive%20and%2012%20mor

Also for power supply, google tells me this


If it’s it - ugh, please just buy USB Type C with PD, if raspi support it
Also what do you use for USB --> SATA adapter?

#5

Sorry I didn’t tell you I have this kit:

The power supply has been sent by the same seller.
I think the sata/usb interface takes power directly and turns power to Raspberry and to all hard disks.
Do you think that the power supply is not enough to power everything? The specification says that it must be at least 40W, mine should be 60W, correct?

#6

Ah, yeah, that’s important aspect, have you run script? Check wiki

#7

Yes of course I run the script, and everything works good, except the error shown above that cause the continue resync. I don’t understand if should be a problem related with the interface sata/usb or power supply or whatever

#8

Well, 4 drives is ~25 watts, so power supply should be fine. Now as for resync, show me output of

smartctl -a /dev/sda
and
badblocks -v /dev/sda

#9

I’ll check as soon as resync will be completed:

cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc[0] sdd[1] sdb[4] sda[2]
5860147200 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[==>…] resync = 14.6% (286375608/1953382400) finish=1500.9min speed=18509K/sec
bitmap: 13/15 pages [52KB], 65536KB chunk

It will takes more or less 25 hours, Probably it’s the 10th times!!

#10

You can also speed up the process…
https://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

1 Like
#11

Thank you, I just applied the tips, I really hope the resync will be shorter than usual
I’ll inform you soon.

#12

Resync completed this evening.

the output of smartctl -a /dev/sda

smartctl 6.6 2017-11-05 r4594 [aarch64-linux-5.10.11-v8+] (local build)
Copyright © 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Blue Mobile (SMR)
Device Model: WDC WD20SPZX-75UA7T1
Serial Number: WXN2E7000SJY
Firmware Version: 80003040
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA/ATAPI-7 (minor revision not indicated)
Local Time is: Mon Mar 15 21:10:15 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x00) Offline data collection not supported.
SMART capabilities: (0x0000) Automatic saving of SMART data is not implemented.
Error logging capability: (0x00) Error logging NOT supported.
No General Purpose Logging support.

SMART Error Log not supported

SMART Self-test Log not supported

Selective Self-tests/Logging not supported

I’ve lauched also command badblocks -v /dev/sda
but it seem very slow. On google I’ve found that fot 1TB it takes 70hours,
my hdd is 2TB, probably it will take 140hours (almost 6days).

Is it correct?

#13

Hm, i guess we will restrain from checking whole drive…

hdparm --read-sector 58016224 /dev/sda

As for smartctl - do

smartctl --smart=on /dev/sda
#14

pi@raspberrypi:~ $ sudo hdparm --read-sector 58016224 /dev/sda

/dev/sda:

reading sector 58016224: succeeded

5302 e8b4 8cc3 607d d4e2 494f b8f3 d091

ec12 8460 8243 45bc 28e8 f349 b862 3c3a

eeb9 4946 075d 1964 5a26 b54d 741a b834

b0a3 6766 b1e0 f800 00a8 4135 ad76 fd87

012b f470 b373 3b76 2087 36aa 5351 a5a5

12f2 d08a 8df0 df21 9120 dad6 b20a 27e9

186a 05e0 b365 e828 c0ab 8244 08ea 291b

7997 a511 38c5 2c8a 51dc d604 0a94 c6b5

4201 0d6b 3b5e 1bea 831e 5aee c5db 22a1

f043 8247 bcc2 e565 7b6d 76e8 fa67 235b

01a9 ab31 8970 8f1c e52a 8545 0423 b3f2

abe1 d91a ce1b 9768 3515 2265 3cd4 ad4d

d701 4ceb 2246 c9ad 6b47 2712 cf7f bebf

ec38 1736 24f7 2683 7100 0ca3 2fe2 64c7

7940 ed72 f6ea 58ec 31a4 0024 3a25 4e92

5b01 b550 6b4b 66b5 9a6a 81f0 0002 9101

ced5 3e5d cddd 1303 43a1 3c9f 5a48 4619

9566 418b ad1d 2095 602b f424 b8c6 b75e

cc7b 1c41 0535 f480 621f ec61 a081 1777

1519 4b11 a725 5937 0d69 c26c 46f8 ac49

6796 a864 6b46 1615 2cf7 def8 b3ff 45ae

b374 9de4 e5b5 b5e3 4445 3781 351c da5c

012a 86d3 3616 685e 4b3f 4335 fb02 092c

2ba7 40c3 6892 9b06 759b 76b4 4cbf 2b12

b9a6 4534 cb3a 57ab df3a f652 9c4b 35a3

8257 8561 1d14 8c86 8036 d489 2093 9f54

c997 e064 cef1 5267 524c 5d2d 26d9 5997

ab1d 1d1a 9205 a56b acd6 b98e 7360 d703

e000 0174 a982 17fd e759 b929 d1e7 cf00

77c5 8e42 2d36 e1b9 31f1 7d25 de65 8127

be90 a5b6 0b5a 0180 6572 f128 8604 4988

2260 795d bbc5 d26b 0cb9 096d 7544 93ca

pi@raspberrypi:~ $ sudo smartctl --smart=on /dev/sda

smartctl 6.6 2017-11-05 r4594 [aarch64-linux-5.10.11-v8+] (local build)

Copyright © 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF ENABLE/DISABLE COMMANDS SECTION ===

SMART Enabled.

But the output is the same:

pi@raspberrypi:~ $ sudo smartctl -a /dev/sda
smartctl 6.6 2017-11-05 r4594 [aarch64-linux-5.10.11-v8+] (local build)
Copyright © 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Blue Mobile (SMR)
Device Model: WDC WD20SPZX-75UA7T1
Serial Number: WXN2E7000SJY
Firmware Version: 80003040
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA/ATAPI-7 (minor revision not indicated)
Local Time is: Tue Mar 16 07:06:21 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x00) Offline data collection not supported.
SMART capabilities: (0x0000) Automatic saving of SMART data is not implemented.
Error logging capability: (0x00) Error logging NOT supported.
No General Purpose Logging support.

SMART Error Log not supported

SMART Self-test Log not supported

Selective Self-tests/Logging not supported

#15

Hm, unsupported seems not good. Ah yes, it’s raspi.
Then smart idea can be scratched. Hmm, block seems fine

#16

Do you have any idea why it resyncs so often?

If you need more info let me know.

Thank you for your help