USB SSD Benchmarks

stuartiannaylor · July 17, 2019, 4:40pm

I am doing a series of benchmarks with the same set of 4x 120gb Integral P series 5
https://integralmemory.com/product/p-series-5-performance-sata-iii-2.5-inch-solid-state-drive-ssd

I have also 4x Jmicron 578 usb adapters the powered 12v ones that can also run 3.5" disk.

Image used is the standard Debian https://dl.radxa.com/rockpi/images/debian/rockpi4_debian_stretch_lxde_armhf_20181105_2120-gpt.img.gz
Linux linaro-alip 4.4.154

So with 2 identical ssd in USB3.0-0 & USB3.0-1 we will check single disk speed.

/dev/sda

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    18252    27259    31814    32002    20574    26928
          102400      16    60376    87245    97004    98580    64215    88406
          102400     512   261755   232193   242331   252794   211577   279430
          102400    1024   304668   311887   289360   296418   250199   273380
          102400   16384   390597   393552   346084   354387   345981   393214

/dev/sdb

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    18323    26205    30070    30425    18320    26103
          102400      16    57640    79354    94251    92999    60041    85012
          102400     512   262524   269200   246613   254026   209976   241371
          102400    1024   299684   313775   300159   309836   254030   285209
          102400   16384   390050   396653   343984   355940   348794   394684

Slight difference but prob if test was run multiple times the average would be close enough not to matter and above doesn’t really.

Now start with raid
sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sda /dev/sdb
/dev/md0

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    22511    27894    32447    31611    20538    27974
          102400      16    67143    82704    90162   102588    64829    80332
          102400     512   273903   286319   267301   269415   222084   298235
          102400    1024   484016   498544   452781   464062   381425   418365
          102400   16384   757070   533962   659360   690695   621724   773455

sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

	Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     8999    10704    19223    16672     9380    10044                                                          
          102400      16    35224    33219    26717    93407    32162    38301                                                          
          102400     512   236433   149504   261918   206158   143571   224128                                                          
          102400    1024   187899   248715   178599   187869   159296   263155                                                          
          102400   16384   296295   267946   131671   333164   329056   290577

This time on a 4 port USB3.0 hub using the top USB3.0 port
sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sda /dev/sdb

	Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    13234    15227    18615    18588    13082    15241                                                          
          102400      16    31071    34921    38321    38415    31401    35065                                                          
          102400     512    58148    57295    56534    56256    55781    58886                                                          
          102400    1024    64100    64696    63539    63775    64067    62311                                                          
          102400   16384    68061    68293    66934    66678    66957    68150

sudo mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

	Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     8900     9830    31376    31014    17521     9536                                                          
          102400      16    18217    19297    94578    86954    61725    19474                                                          
          102400     512    32041    32341   248433   250812   199070    30996                                                          
          102400    1024    31819    32555   271239   271200   237609    32522                                                          
          102400   16384    33706    33294   327036   336360   329408    33838

I added this on the end because at times either via hub or 2 ports occasionally you get a stutter like a disk has gone on holiday.

	Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    13199    15578      800    18734    13116    15784                                                          
          102400      16    31451    35213    38820    38832    31419    34123                                                          
          102400     512    56975    56546    54976    55229    55360    57779                                                          
          102400    1024    62347    62775    61629    62024    62171    62006                                                          
          102400   16384    68176    68358    67332    66765    67234    68297

Really not sure what happened on that 4k read
journalctl

Jul 17 16:28:07 linaro-alip sudo[1202]:   linaro : TTY=pts/0 ; PWD=/home/linaro/ssda ; USER=root ; COMMAND=/usr/bin/iozone -e -I -a -s 100M -r 4k -r 16k -r 512
Jul 17 16:28:07 linaro-alip sudo[1202]: pam_unix(sudo:session): session opened for user root by linaro(uid=0)
Jul 17 16:28:53 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
Jul 17 16:28:53 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 04 3f 58 00 00 08 00
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x2a 2a 00 00 02 00 00 00 00 10 00
Jul 17 16:29:27 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler start
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 uas_zap_pending 0 uas-tag 2 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 uas_zap_pending 0 uas-tag 4 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x2a 2a 00 00 02 00 30 00 00 08 00
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#4 uas_zap_pending 0 uas-tag 5 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x2a 2a 00 00 02 10 f0 00 00 08 00
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#5 uas_zap_pending 0 uas-tag 6 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x2a 2a 00 00 02 11 68 00 00 08 00
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#6 uas_zap_pending 0 uas-tag 7 inflight: CMD 
Jul 17 16:29:27 linaro-alip kernel: sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x2a 2a 00 00 02 91 e8 00 00 08 00
Jul 17 16:29:27 linaro-alip kernel: usb 7-1.1: reset high-speed USB device number 3 using xhci-hcd
Jul 17 16:29:27 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler success
Jul 17 16:30:02 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
Jul 17 16:30:02 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 05 71 08 00 00 08 00
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD 
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x2a 2a 00 06 e4 04 38 00 00 10 00
Jul 17 16:30:28 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler start
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 uas_zap_pending 0 uas-tag 3 inflight: CMD 
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x2a 2a 00 00 02 00 38 00 00 18 00
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 uas_zap_pending 0 uas-tag 4 inflight: CMD 
Jul 17 16:30:28 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x2a 2a 00 00 02 11 e8 00 00 08 00
Jul 17 16:30:28 linaro-alip kernel: usb 7-1.1: reset high-speed USB device number 3 using xhci-hcd
Jul 17 16:30:28 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler success
Jul 17 16:31:37 linaro-alip sudo[1202]: pam_unix(sudo:session): session closed for user root

stuartiannaylor · July 19, 2019, 6:28pm

I ran out of barrel ports on my 4 way splitter and was using x3 with one on usb power.
Didn’t seem to go well so ordered an 8 way splitter as by 60 watt PSU has enough juice for SSD.

Still getting the same strange stutter on the read with 4x USB SSD on a single hub and port

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    16325    19503      771    22304    16456    19794
          102400      16    43018    48799    53117    53277    42447    48473
          102400     512    93399    94971    88913    89187    88105    92296
          102400    1024   106670   108065   102675   102685   102719   107363
          102400   16384   134801   134859   131228   132904   132588   135335

I run again and this time splutters on random read

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    16057    19286     1471    22397    16411    19659
          102400      16    43114    48048    52657    53525     1897    47722
          102400     512    93206    97257    89189    89084    88837    95354
          102400    1024   106621   108482   102109   103676   102243   107457
          102400   16384   133775   135737   132313   132760   132220   135196

I feel the problem with USB UAS is that it likes large rec lengths in big burst packets and as soon as you start switching quickly between small packets you get that network like flood where above a threshold the whole thing starts to flood and grinds to a relative halt.
In journalctl we get similar complaints

Jul 19 11:45:19 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler success
Jul 19 11:46:19 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
Jul 19 11:46:19 linaro-alip kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 03 5e 80 00 00 20 00
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x2a 2a 00 00 02 08 00 00 00 10 00
Jul 19 11:46:40 linaro-alip kernel: scsi host0: uas_eh_bus_reset_handler start
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 uas_zap_pending 0 uas-tag 3 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x2a 2a 00 00 02 11 d0 00 00 08 00
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 uas_zap_pending 0 uas-tag 4 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x2a 2a 00 00 02 12 48 00 00 08 00
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#4 uas_zap_pending 0 uas-tag 5 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x2a 2a 00 00 02 12 c8 00 00 08 00
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#5 uas_zap_pending 0 uas-tag 6 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#5 CDB: opcode=0x2a 2a 00 00 02 52 c8 00 00 08 00
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#6 uas_zap_pending 0 uas-tag 7 inflight: CMD
Jul 19 11:46:40 linaro-alip kernel: sd 0:0:0:0: [sda] tag#6 CDB: opcode=0x2a 2a 00 00 03 11 c8 00 00 08 00
Jul 19 11:46:40 linaro-alip kernel: usb 5-1.1: reset high-speed USB device number 3 using xhci-hcd

I am beginning to think don’t use the Radxa images for anything to be honest or build.
I downloaded the stretch version of Armbian and things run much smoother

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    18963    22287    25035    25049    18369    22151
          102400      16    47231    53357    56771    56815    45804    53052
          102400     512   101564   102198    93697    94003    93984    97632
          102400    1024   107587   112674   106697   106926   107523   109719
          102400   16384   137496   137575   134878   135269   135421   137749

Still it does the same jitter as expected just keep running and sooner or later it will happen but Armbian seems far more stable.
Prob the best thing would be disable UAS but the main rationale for this was to show how USB RAID is actually not a good idea.
The more disks or concurrent access that cause more swapping of small rec length files has a tendency to flood and performance drops substantially.

USB disks are great standalone and why packages like SnapRaid are great for media servers as predominantly we have large singular files with low levels of concurrency.

But just to continue lets try raid 1
Currently syncing using armbian which still give the below but far less than the Radxa images provided.

Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 08 77 e9 80 00 04 00 00
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#3 CDB: opcode=0x28 28 00 08 77 e6 00 00 03 80 00
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x28 28 00 08 77 e2 00 00 04 00 00
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD
Jul 19 13:44:56 rockpi kernel: sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x28 28 00 08 77 de 00 00 04 00 00
Jul 19 13:45:01 rockpi CRON[5015]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 19 13:45:01 rockpi CRON[5016]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 19 13:45:01 rockpi CRON[5017]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 19 13:45:01 rockpi CRON[5018]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs)
Jul 19 13:45:01 rockpi CRON[5016]: pam_unix(cron:session): session closed for user root
Jul 19 13:45:01 rockpi CRON[5015]: pam_unix(cron:session): session closed for user root
Jul 19 13:45:32 rockpi kernel: sd 0:0:0:0: [sda] tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD
Jul 19 13:45:32 rockpi kernel: sd 0:0:0:0: [sda] tag#4 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
Jul 19 13:45:32 rockpi kernel: scsi host0: uas_eh_bus_reset_handler start
Jul 19 13:45:32 rockpi kernel: usb 5-1.1: reset high-speed USB device number 3 using xhci-hcd
Jul 19 13:45:32 rockpi kernel: scsi host0: uas_eh_bus_reset_handler success

Still turns up whatever the above means, but much less on Armbian

rock@rockpi:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdd[3] sdc[2] sdb[1] sda[0]
      117155264 blocks super 1.2 [4/4] [UUUU]
      [===============>.....]  resync = 76.9% (90208512/117155264) finish=12.6min speed=35534K/sec
      bitmap: 1/1 pages [4KB], 65536KB chunk

So not that its a likely setup but just for bench marking here is 4 disk raid1

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                                               
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread                                     freread
          102400       4     8385     8623      535      789    11072     9089
          102400      16    18268    19453    23237    23231     1572    19035
          102400     512    30692    29779    31035    31053    53175    30762
          102400    1024    30422    31014    31332    31870    47433    31437
          102400   16384    32689    33451    39741    33947    40271    32539

Raid10 4 disk

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    12641    13752     1547    20502     1500    13279
          102400      16    28579    30729    39581    42442    35324    28938
          102400     512    51593    50513    74024    73150    67411    52353
          102400    1024    61118    62792    87334    86112    85567    61772
          102400   16384    66794    66896   278649   114893   124265    64606

RAID 5 4x USB SSD

        Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     6316    11337    23773    23700    17004    10041
          102400      16    22133    23548    53911     1612    43280    22876
          102400     512    34178    33239    90429    90910    90591    33428
          102400    1024    44931    45342   104407   104294   104489    45659
          102400   16384    69448    70525   131301   130093   131025    70567