Could you please confirm if PCI express indeed works at 3.0 version speeds?
ROCK 5B Debug Party Invitation
Those devices with black metal case must be nanopi’s R5S or R6S.
R5S as mentioned above when comparing performance/consumption between RK3568 and RK3588.
I’m about to setup a nice 2.5GbE ‘LAN party’ with those 3 ARM boards and 2 MacBooks with RTL8156 dongles 
BTW: never heard of R6S but there’s a little room for speculation about a soon to be announced NanoPi R3S (also based on RK3568).
OK, @willy (who can’t post here since registered just yesterday) pointed me in this direction:
R6S based on RK3588S
Sadly, the contained kernel 4.4 is already EOL:
[2022-07-05T18:44:44+0200] ueberall_l@desktop01:/tmp% wget "https://github.com/radxa/debos-radxa/releases/download/20220701-0727/rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img.xz"
[…]
[2022-07-05T18:45:47+0200] ueberall_l@desktop01:/tmp% xz -dv rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img.xz
rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img.xz (1/1)
100 % 140,6 MiB / 906,0 MiB = 0,155 92 MiB/s 0:09
[2022-07-05T18:45:59+0200] ueberall_l@desktop01:/tmp% fdisk -l ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img
Disk ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img: 905,101 MiB, 950000128 bytes,
1855469 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 34D6A485-96AB-4EAE-B781-BEA9FCBC918D
Device Start End Sectors Size Type
./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img1 32768 262143 229376 112M EFI System
./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img2 262144 1855435 1593292 778M Linux filesystem
[2022-07-05T18:46:07+0200] ueberall_l@desktop01:/tmp% mkdir ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt
[2022-07-05T18:46:46+0200] ueberall_l@desktop01:/tmp% sudo mount -t auto -o loop,offset=$((262144*512)) ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.img ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt
[sudo] password for ueberall_l:
[…]
[2022-07-05T18:49:16+0200] ueberall_l@desktop01:/tmp% dpkg-query -W --admindir=./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt/var/lib/dpkg >./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.dpkg-query.txt
[2022-07-05T18:50:11+0200] ueberall_l@desktop01:/tmp% grep ^linux- ./rockpi-s-ubuntu-focal-server-arm64-20220701-0851-gpt.dpkg-query.txt
linux-4.4-rock-pi-s-latest:arm64 4.4.143-69-rockchip
linux-base 4.5ubuntu3
linux-firmware-image-4.4.143-69-rockchip-g8ccef796d27d:arm64 4.4.143-69-rockchip
linux-headers-4.4.143-69-rockchip-g8ccef796d27d:arm64 4.4.143-69-rockchip
linux-image-4.4.143-69-rockchip-g8ccef796d27d:arm64 4.4.143-69-rockchip
[2022-07-05T18:51:35+0200] ueberall_l@desktop01:/tmp%
Given that competing SBC have patches for the 5.15 kernel (and newer ones) which also slowly find their way upstream (if they have not been included already), I’m afraid that this is simply way too old. The system might run, but kernel 4.4 was published 6.5 years ago and isn’t tested against current software packages/applications (because it’s EOL) unless you manage to pay for it.
Please at least consider including a kernel that is not already EOL or won’t be EOL by the time the hardware will be available (IMHO, even kernel 4.9 will not really fit the bill, it’ll require kernel 4.14 or newer).
EDIT: As pointed out below by tkaiser, I downloaded the wrong image, so the rant is misplaced and does not apply to the ROCK5B (only to other/older models). 
But we’re talking about Rock 5B / RK3588 here. And deal with a forward ported version 5.10 from Rockchip (not to be confused with official 5.10 LTS kernel which is a different thing).
1 Like
everyone does what they want, however when you get targets for free you can test it with the ammunition that come with it.
My bad! Using the correct image, you’re presented with
[2022-07-05T19:47:05+0200] ueberall_l@desktop01:/tmp% grep ^linux- ./rock-5b-ubuntu-focal-server-arm64-20220701-0826-gpt.dpkg-query.txt
linux-base 4.5ubuntu3
linux-headers-5.10.66-11-rockchip-geee2b32138fd:arm64 5.10.66-11-rockchip
linux-image-5.10.66-11-rockchip-geee2b32138fd:arm64 5.10.66-11-rockchip
[2022-07-05T19:47:32+0200] ueberall_l@desktop01:/tmp%
That’s an acceptable starting point. If you can trust the version number, that is. (EDIT: tkaiser is simply too fast to reply 
)
Well, the information about which kernel version we’re dealing with Rock 5B is here multiple times in this thread (part of sbc-bench output for example). But it’s important to understand how this kernel got to this version since as already said: it’s forward ported from 2.6.32 on and not the official 5.10 LTS kernel.
1 Like
1st quick check for networking. It’s a RTL8125BG connected via PCIe Gen2 x1:
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
In TX direction everything is fine even when letting iperf3 run on a little core:
root@rock-5b:/home/rock# taskset -c 5 iperf3 -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
[ 5] local 192.168.83.113 port 47964 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 272 MBytes 2.28 Gbits/sec 41 573 KBytes
[ 5] 1.00-2.00 sec 281 MBytes 2.36 Gbits/sec 0 701 KBytes
[ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec 0 728 KBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 735 KBytes
[ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 738 KBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 741 KBytes
[ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 745 KBytes
[ 5] 7.00-8.00 sec 281 MBytes 2.36 Gbits/sec 0 748 KBytes
[ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0 749 KBytes
[ 5] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec 0 749 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 41 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver
iperf Done.
root@rock-5b:/home/rock# taskset -c 1 iperf3 -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
[ 5] local 192.168.83.113 port 47968 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 227 MBytes 1.90 Gbits/sec 244 479 KBytes
[ 5] 1.00-2.00 sec 282 MBytes 2.37 Gbits/sec 0 689 KBytes
[ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 730 KBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 744 KBytes
[ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 749 KBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 755 KBytes
[ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 0 759 KBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 762 KBytes
[ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 765 KBytes
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 768 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec 244 sender
[ 5] 0.00-10.00 sec 2.69 GBytes 2.31 Gbits/sec receiver
iperf Done.
But in RX direction performance sucks with defaults. IRQs 133 and 149 are on little cores. And even when moving them to big cores I’m stuck at 510 Mbits/sec. Also the weird cpufreq behaviour (even with performance cpufreq governor clocking the cores as low as 408 MHz) doesn’t help.
This is some stuff for deeper investigations. But let’s check PCIe powermanagement first:
root@rock-5b:/home/rock# cat /sys/module/pcie_aspm/parameters/policy
default performance [powersave] powersupersave
Well, that doesn’t look right. Let’s switch to default et voilà:
Even on a little core 2.5GbE link saturated:
root@rock-5b:/home/rock# taskset -c 1 iperf3 -R -c 192.168.83.63
Connecting to host 192.168.83.63, port 5201
Reverse mode, remote host 192.168.83.63 is sending
[ 5] local 192.168.83.113 port 47980 connected to 192.168.83.63 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 272 MBytes 2.29 Gbits/sec
[ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec
[ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 8.00-9.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver
iperf Done.
1 Like
Some experiments with USB. Connected two SSDs in UAS capable USB3 enclosures:
root@rock-5b:/home/rock# lsusb -t
/: Bus 08.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
root@rock-5b:/home/rock# lsusb
Bus 006 Device 002: ID 174c:55aa ASMedia Technology Inc. Name: ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge
Bus 008 Device 002: ID 152d:3562 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
The try to create an mdraid0 (to test for bottlenecks with the USB3 ports) seems to fail: dmesg output
I know, most readers will now recommend to ‘blacklist UAS’ but the symptoms are maybe just related to some underpowering. Don’t know yet. At least there’s room for improvements (appending coherent_pool=2M to extlinux.conf)
I think in newer kernels coherent_pool auto expands as needed and that setting is merely a start point but shouldn’t hurt though. Anyway I remember from somewhere it autoexpands but can not remember which version that was introduced.
Well, adjusting this (and providing one of the two USB3 enclosures with own power) fixed the issues.
But it seems every tweak I added over the years to Armbian (like coherent pool size or IRQ affinity and ondemand governor tweaks) needs to be applied to Radxa’s OS images too.
Testing the raid0 with a simple iozone -e -I -a -s 1000M -r 1024k -r 16384k -i 0 -i 1:
kB reclen write rewrite read reread
1024000 1024 258830 270261 341979 344249
1024000 16384 270022 271088 667757 679947
That was running on a little core and is total crap. Now on a big core (by prefixing the iozone call with taskset -c 7):
kB reclen write rewrite read reread
1024000 1024 475382 457597 374120 374863
1024000 16384 777913 771802 736511 734899
Still crappy but when checking with sbc-bench -m it’s obvious that CPU clockspeeds don’t ramp up quickly and to the max:
Time big.LITTLE load %cpu %sys %usr %nice %io %irq Temp
19:43:03: 408/1008MHz 0.35 3% 1% 0% 0% 1% 0% 32.4°C
19:43:08: 408/1008MHz 0.33 0% 0% 0% 0% 0% 0% 32.4°C
19:43:13: 816/1200MHz 0.38 13% 3% 0% 0% 8% 1% 32.4°C
19:43:18: 1416/1200MHz 0.43 13% 2% 0% 0% 10% 0% 32.4°C
19:43:23: 600/1008MHz 0.56 14% 4% 0% 0% 7% 1% 33.3°C
19:43:28: 408/1008MHz 0.51 1% 0% 0% 0% 0% 0% 32.4°C
One echo 1 > /sys/devices/system/cpu/cpufreq/policy6/ondemand/io_is_busy later we’re at
kB reclen write rewrite read reread
1024000 1024 604081 603160 505333 506815
1024000 16384 791865 789173 775160 778706
Still not perfect (less than 800 MB/sec) but at least the big core immediately switches to highest clockspeed:
Time big.LITTLE load %cpu %sys %usr %nice %io %irq Temp
19:50:44: 408/1008MHz 0.00 2% 0% 0% 0% 0% 0% 32.4°C
19:50:49: 2400/1800MHz 0.00 5% 1% 0% 0% 3% 0% 32.4°C
19:50:54: 2400/1800MHz 0.08 12% 1% 0% 0% 11% 0% 32.4°C
19:50:59: 2400/1800MHz 0.15 13% 2% 0% 0% 9% 0% 34.2°C
19:51:04: 408/1008MHz 0.14 2% 0% 0% 0% 1% 0% 32.4°C
But 790 MB/s are close to what can be achieved with a RAID0 over two USB3 SuperSpeed buses since each bus maxes out at slightly above 400 MB/s.
Moral of the story: if you’re shipping with ondemand cpufreq governor you need to take care about io_is_busy and friends.
After adding the following to /etc/rc.local even on a little core storage performance is ok-ish:
echo default >/sys/module/pcie_aspm/parameters/policy
for cpufreqpolicy in 0 4 6 ; do
echo 1 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/io_is_busy
echo 25 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_down_factor
echo 200000 > /sys/devices/system/cpu/cpufreq/policy${cpufreqpolicy}/ondemand/sampling_rate
done
This is the result tested with taskset -c 1 iozone -e -I -a -s 1000M -r 1024k -r 16384k -i 0 -i 1:
kB reclen write rewrite read reread
1024000 1024 524857 526483 458726 459194
1024000 16384 780470 774856 733638 734297One reboot later with the JMS567 enclosure solely powered by the board we’re back in trouble land:
[ 126.480465] sd 1:0:0:0: [sdb] tag#17 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT
[ 126.480476] sd 1:0:0:0: [sdb] tag#17 CDB: opcode=0x2a 2a 00 00 f6 30 00 00 04 00 00
[ 162.321748] sd 1:0:0:0: [sdb] tag#5 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD
[ 162.321758] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
[ 162.338423] scsi host1: uas_eh_device_reset_handler start
[ 167.642039] xhci-hcd xhci-hcd.10.auto: Timeout while waiting for setup device command
[ 167.848680] usb 8-1: reset SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[ 167.867319] scsi host1: uas_eh_device_reset_handler success
[ 198.162935] sd 1:0:0:0: [sdb] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT
[ 198.162945] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x2a 2a 00 00 f6 3c 00 00 04 00 00
This is not an UAS problem but underpowering. At least the board is stuck in I/O now and no benchmark is able to run. Well, since USB3 storage sucks anyway time to move on to something else…
2 Likes
The GPU I guess would be of interest on Linux is this a Rockchip BSP kernel with Rockchip blobs or is it absent of drivers currently?
Well, without adding coherent_pool=2M to extlinux.conf the board does not even boot when the USB3 attached SSDs are connected. This is with an mdraid0 (so zero practical relevance for future Rock 5B users):
root@rock-5b:/mnt/md0# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid0 sdb[0] sda[1]
234307584 blocks super 1.2 512k chunks
unused devices: <none>Its only recent last year sometime just can not remember which revision it went in.
I can not even remember why I was reading about it, but prob due to being on 5.10
Purely mentioned that probably they should not be included in the Radxa image but definitely documented as tweaks and fixes.
Some consumption data…
I’m measuring with a NetIO PowerBox 4KF in a rather time consuming process that means ‘at the wall’ with charger included with averaged idle values over 4 minutes. I tested with 2 chargers so far:
- RPi 15W USB-C power brick
- Apple 94 USB-C charger
When checking with sensors tcpm_source_psy_4_0022-i2c-4-22 this looks like this with the RPi part
in0: 5.00 V (min = +5.00 V, max = +5.00 V)
curr1: 3.00 A (max = +3.00 A)
and like this with the Apple part:
in0: 9.00 V (min = +9.00 V, max = +9.00 V)
curr1: 3.00 A (max = +3.00 A)
27W could become a problem with lots of peripherals but @hipboi said the whole USB PD negotiation thing should be configurable via DT or maybe even at runtime.
Measured with RPi power brick and 2.5GbE network connection:
- No fan: idle consumption: 2390mW
- With fan: idle consumption: 3100mW
That means the little fan on my dev sample consumes 700mW that can be substracted from numbers (given Radxa will provide a quality metal case that dissipates heat out of the enclosure so no fan needed)
Now with the Apple Charger:
- fan / 2.5GbE: idle consumption: 3270mW
- fan and only GbE: idle consumption: 2960mW
The Apple charger obviously needs a little more juice for itself (+ ~150mW compared to the RPi power brick) and powering the 2.5GbE PHY vs. only Gigabit Ethernet speeds results in 300mW consumption difference.
-
echo default >/sys/module/pcie_aspm/parameters/policy--> idle consumption: 3380mW
Switching from powersave to default with PCIe powermanagement (needed to get advertised network speeds) adds ~100mW to idle consumption.
As such I propose the following changes in all of Radxa’s OS images (preferably ASAP but at least before review samples are sent out to those YT clowns like ExplainingComputers, ETA Prime and so on):
- setting
/sys/module/pcie_aspm/parameters/policyto default (w/o network RX performance is ruined) - appending
coherent_pool=2Mtoextlinux.conf(w/o most probably ‘UAS hassles’) - configuring
ondemandcpufreq governor withio_is_busyand friends (w/o storage performance sucks)
Would be great if you put it in the wiki.
In my tests on the USB cable, I measured 1.6W in idle with no fan, and 2.6W with it. At higher loads under 5V, the difference was a bit less but the fan wouldn’t spin as fast. When powered under 12V however, the fan speed remains stable but I can’t monitor the current. This seems consistent with your measurements, and you can count on 500-800mW for your PSU