ROCK 5 in ITX form factor

[tkaiser]

Are you talking about power[super]save? You changed that in 2022 directly after reporting my findings back then. And adjusting the policy doesn’t change anything:

root@rock-5-itx:/home/radxa# echo performance > /sys/module/pcie_aspm/parameters/policy
root@rock-5-itx:/home/radxa# lspci -vvPPDq | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:./]+|:\sASPM (\w+)?( \w+)? ?((En|Dis)abled)?)';
0001:10:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3588 (rev 01) (prog-if 00 [Normal decode])
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
pcilib: sysfs_read_vpd: read failed: Input/output error
pcilib: sysfs_read_vpd: read failed: Input/output error
0001:10:00.0/11:00.0 SATA controller: ASMedia Technology Inc. ASM1164 Serial ATA AHCI Controller (rev 02) (prog-if 01 [AHCI 1.0])
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0003:30:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3588 (rev 01) (prog-if 00 [Normal decode])
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0003:30:00.0/31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0004:40:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3588 (rev 01) (prog-if 00 [Normal decode])
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
0004:40:00.0/41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

In the thread mentioned ASPM status of all PCIe devices reads ASPM L1 Enabled.

Edit: adding pcie_aspm=force to kernel cmdline doesn’t change anything.

[RadxaYuntian]

Yeah I vaguely remember we got that info from you but I just couldn’t find the link. However, the link I posted claims only a 100 mW power saving with different configs, so I’m not sure if we can reduce much of the idle power.

[tkaiser]

the link I posted claims only a 100 mW power saving with different configs, so I’m not sure if we can reduce much of the idle power.

The 100mW was also measured by me back then but this was the result of switching ASPM policy between default and powersave on a system with a single RTL8125BG attached and otherwise no other PCIe lane occupied. ASPM was active but only policy changed.

Now we’re talking about ASPM being completely disabled and Rock 5 ITX having 4 PCIe lanes occupied by default (Gen3 x2 to the ASM1164 and 2 x Gen2 x1 to the RTL8125BG). And when adding a NVMe SSD and/or a Wi-Fi/BT card we’re talking about all PCIe 7 lanes being occupied. ASPM or not in such a situation might make a difference of several watt. And what I’m trying to do now is measuring exactly that

But I can’t figure out where ASPM got disabled if even pcie_aspm=force doesn’t bring it back. That’s all I’m asking/searching for, the actual numbers which difference this makes are to be measured afterwards.

[willy]

I’m seeing in your test results that the latency is indeed slightly worse than with LPDDR4X but that the read BW is overall 30-40% faster. I think that for most use cases the BW gains will offset the small losses in latency. It’s just those facing workloads made of lots of small reads that will be affected (e.g. linked lists and/or hash tables in memory), but even then the effect will be really limited. The smallest one can read from memory is a cache line (64B) and the tinymembench shows that it could be achieved in 140 ns previously and that it’s 150 ns now. It’s not exceptional but it’s not bad either. For example my odroid-h3 gives me 130ns where my rock5b shows 139ns. And at 4 parallel accesses, I’m seeing 165ns on both thanks to the 4 channels on RK3588. So we’re on par with low-end x86 here, definitely nothing to be ashamed of.

[tkaiser]

For example my odroid-h3 gives me 130ns where my rock5b shows 139ns. And at 4 parallel accesses, I’m seeing 165ns on both thanks to the 4 channels on RK3588. So we’re on par with low-end x86 here, definitely nothing to be ashamed of.

Sure, but expectations should be adjusted accordingly and advertising ‘10% faster memory’ should be stopped or at least put into perspective (I could imagine various use cases where the higher bandwidth really matters eg. video processing, GPU and maybe NPU).

Talking about ‘on par with low-end x86’ right now idle consumption is concerning since who wants to deal with all the ARM hassles when you can get a x86_64 board like an ODROID-H3 or H4+ with lower idle consumption?

And yes, I’m fully aware of the (way) higher consumption in load scenarios but with certain use cases this doesn’t matter much or at all.

[tkaiser]

ASPM or not in such a situation might make a difference of several watt. And what I’m trying to do now is measuring exactly that

Whatever happened the problem with ‘ASPM Disabled’ was clearly me since it works.

root@rock-5-itx:/home/radxa# . /usr/local/bin/sbc-bench.sh

root@rock-5-itx:/home/radxa# echo performance > /sys/module/pcie_aspm/parameters/policy

root@rock-5-itx:/home/radxa# CheckPCIe
  * ASMedia ASM1164 Serial ATA AHCI: Speed 8GT/s (ok), Width x2 (ok), driver in use: ahci, ASPM Disabled
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM Disabled
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM Disabled

root@rock-5-itx:/home/radxa# echo powersave > /sys/module/pcie_aspm/parameters/policy

root@rock-5-itx:/home/radxa# CheckPCIe
  * ASMedia ASM1164 Serial ATA AHCI: Speed 8GT/s (ok), Width x2 (ok), driver in use: ahci, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us  PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns  T_PwrOn=10us 
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us  PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns  T_PwrOn=10us 

root@rock-5-itx:/home/radxa# echo powersupersave > /sys/module/pcie_aspm/parameters/policy

root@rock-5-itx:/home/radxa# CheckPCIe
  * ASMedia ASM1164 Serial ATA AHCI: Speed 8GT/s (ok), Width x2 (ok), driver in use: ahci, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us  PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns  T_PwrOn=10us 
  * Realtek RTL8125 2.5GbE: Speed 5GT/s (ok), Width x1 (ok), driver in use: r8125, ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us  PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns  T_PwrOn=10us 

default and performance ASPM policy show no difference in consumption (ASPM disabled in both cases) and the same is true for powersave vs. powersupersave. W/o having a look at L1 substates we’re talking about a consumption difference of only 700mW with onboard PCIe peripherals.

[willy]

That’s exactly why I suggested a few months ago that high-end ARM-based SBCs are really challenging, because they can still be inferior in performance to some low-to-mid end x86 devices that come with more I/O capability, a very roughly similar termal envelope and equivalent prices. I really wanted to use an RK3588 1.5 year ago for my server, but the mainline status of this beast compared to a boring x86 chip was just apples to oranges.

I anticipate that in the forthcoming years we’ll see more ARM in the datacenter and more Intel on low-power devices because in the end the ones willing to make efforts are those saving money from it at a large scale, while end users just want something that runs out of the box, and when performance, consumption and price are comparable, what remains is ease of use.

[ydeletrain]

Prices are now known and are competitive. I was about to publish an article about my Rock5B-based NAS, but this new board costs less as it can use the 20 pin ATX plug and includes the SATA controller… I guess I will buy it and compare both to make sure the ITX flavor confirms its rank

So, the onboard SATA controller seems to be an ASM1164. I guess 2 of the PCIe 3.0 lanes are used for the purpose and explain why the M.2M slot only offers 2 lanes… now I could still compare to my ASM1166 and JMB585 modules, but since only software RAID is possible, performance on regular HDDs should be very close.

[ydeletrain]

Any advice on a cooler? Finding a simple low profile passive cooler is not so easy, most are proposed with a fan and are not suitable for transversal air flow when the fan is removed.
Maybe a 1U model? Then, it would be nice to cool both the SoC and the memory chips, so a flat base would be better.

[nyanmisaka]

I also received a sample not long ago. IMO any common 115x 1U low profile passive cooler should be overkill for the low power consumption of RK3588. Also, since SoC and RAM chips have different thicknesses, you may need to use thermal paste and thermal pads for them respectively. But considering that the RAM chip is LPDDR5, it doesn’t really matter even if it doesn’t touch the heat sink.

edit: This is the model I’m using. It will not conflict with the onboard RTC battery holder.

[ydeletrain]

@nyanmisaka Thank You for your feedback
Indeed, SoC, eMMc and memory chips only represent 10~15 W of heat to dissipate.
I think cutting a large and light aluminium heatsink like this one is a good and cheaper alternative. Then, using adhesive thermal pads of different heights might allow a sufficient sticky contact. Otherwise, I will have to find a refurbished LGA 115x 1U heatsink or simply wait for delivery of this heatsink

[tkaiser]

Final review update now that the device is ready to be bought and it’s confirmed that eMMC won’t be user-accessible by default: https://github.com/ThomasKaiser/Knowledge/commit/2489492d03db2961d6ac249cc6eefca4687ffa32

[dominik]

thanks for details,
hopefully we will get sooner or later more juice out of ddr5 as we all expect. For now it’s just the number and some chance that higher capacities will be more affordable

[Bret]

Interesting! I’m now trying my 4th cooler on my board as the first 3 just wouldn’t work and they were marked as being LGA 115x compatible. The mounting posts for the backplate were just too wide to go through the holes on the motherboard so here’s hoping 4th time’s the charm

[willy]

Hi all!

I’m also a lucky recipient of this really great board (thanks Radxa!). I’m sharing a few first-contact comments before I forget them:

• the board looks amazingly clean and well arranged. I’m normally not sensitive to PCB colors but this black varnish that hides the copper tracks combined with all the golden pads makes me think of some high-end audio gear Those who love to have a window on their enclosures to exhibit their boards to friends will probably be proud to have something new to show:
  
  
• the connectors and pinouts are well marked. For example you clearly see RJ45-1 and RJ45-2 etc, as well as the front panel’s pinout. That’s one of the advantage of the black paint, the white marking is perfect and uniform so you don’t have any doubt when reading anything:
  
  
• I think that a reset button close to the SPDIF connector (top right above) would be nice during testing or setup. For now I’m making shorts with a screwdriver on the f-panel connector.
• I was surprised to find that the 12V in is on a 2.5mm connector. I searched among all my power blocks (20+) and couldn’t find even a single one in 2.5, all voltages included:
  
  
  
  I found two salvaged male connectors coming from previous PSUs, so I connected one of them to a male 2.1mm connector to make an adapter:
  
  
  
  
  

As an alternative I could have used a 12V->ATX adapter since I have one but I preferred to test the board without extra components first.

• I thought it would be easier to find a heat sink, but I figured I didn’t seem to have the fixing back plate! I started the board without any heat sink and figured that it remains totally cool when idle. Regardless I managed to find one salvaged heat sink + fan + plate kit coming from two different devices that I could mount. I wanted to test it to see if the fan speed could be adjusted:
  

  
  
  
  
  
  I think it could possibly be useful to maybe provide a default plastic plate + long pins that allow to attach “something” with a pair of elastics or serflexes for use during early tests, since it’s likely not going to heat much anyway. I noticed that the fan starts at full speed and its speed significantly drops at the end of the boot, likely when properly set up to only adapt to temperature maybe.

• The UART connector is ordered GND-TXD-RXD as more and more boards these days it seems, so I could directly connect a CH340-based adapter I’m using with some other boards and that supports the default 1.5 Mbauds of Rockchip chips:
  

  

So from a hardware perspective, that’s quite awesome.

However, for now I’m stuck. The system boots on a “roobi” system that I found on the site was a new pre-installer (good idea, that could definitely help). However, I end up with a login prompt and tried many login/password combinations (root, rock, roobi, radxa, even nothing and I don’t remember what), and nothing is accepted. I failed to find the info. I know I’m not good at finding such possibly obvious info but I searched an hour or so, which is already a bit too much for a login/password pair. (At least during this search the SoC remained totally cool without its heat sink). Thus any hint to log in would be more than welcome!

I looked at the boot loader and found that you have approx 1s to choose between 1 and 2 (both end up with this login prompt, the second saying that root account is locked, maybe I did too many attempts?):

Cannot open access to console, the root account is locked.
See sulogin(8) man page for more details.

Press Enter to continue.
...
Debian GNU/Linux 11 roobi ttyFIQ0

roobi login:

It’s also possible to interrupt the boot by pressing Ctrl-C instead of choosing 1 or 2 and end up in u-boot. Partitions 1 and 2 seem to be FAT, part 3 is ext2/3/4 and u-boot by default accesses its kernel there from /boot. But for now I didn’t manage to boot it by hand to pass args and bypass the login.

There are some bluetooth hci0 timeout errors that pollute the console for one minute or so and after that they’re gone.

That’s all for this evening. Once I manage to log into it, I’ll be interested in giving it a try with an M.2 10GbE NIC I bought not long ago, and with 4 SSDs.

[tkaiser]

The system boots on a “roobi” system that

…is most likely meant to be operated with a connected display and keyboard/mouse? Maybe even accessing http://roobi.local/ from another machine on your network works if you use mDNS/ZeroConf?

I guess all you can do the console way is to somehow login, then wipe the eMMC to get the board booting from TF card or something similarly destructive?

[willy]

is most likely meant to be operated with a connected display and keyboard/mouse?

You think so ? I’m not against offering a possibility to install a board via a keyboard and screen (I even encourage it), but I would never expect that one to be the only solution. Yes I came over that page as well and I just considered it irrelevant to my use case, and it spoke about other vendor’s hardware so I didn’t insist.

Hmm I have a HDMI-to-USB capture adapter that I bought for the rare cases I need to connect to HDMI without having to carry a heavy display, I can try. But if that’s it, it would be gross, because the board presents a valid boot prompt, nothing that lets you imagine you’re not supposed to use that. At minimum there should be a usable root/root or so access on the console for those who want a simple install method.

BTW I have not found the board on the network, but I only connected the cable long after I exhausted all my imagination of unusable logins, so maybe it didn’t request an IP address.

But I’ll retry with the HDMI adapter just in case it shows anything, thanks for the suggestion.

[tkaiser]

I would never expect that one to be the only solution

Well, this is stuff for end users. It seems they simply made an Electron app to interact with the user and to setup things which is accessible via display and since being Electron might simply work over a network connection too.

And in case you still want to try local access maybe give ps:ps a try? But how to continue then? Accessing http://127.0.0.1 with a text browser?

Edit: Maybe executing roobi might then work (see here)

[willy]

Well, this is stuff for end users

I understand this but it doesn’t mean that those who know what they want to setup should go via the complicated path. It’s like when some distros didn’t let you run fdisk and/or would always reorder your partitions. On spinning rust you’d definitely want to place the swap closest to the most commonly used data and you’d end up with swap at the end, making the system super slow due to large seeks. I remember finding myself having to install in two steps using a secondary disk to later move the data.
I’ll give the ps:ps a try BTW, thanks for spotting this.

Last point regarding end users, end users install their PC by booting using a USB stick plugged into a USB port, not by using whatever proprietary install system. For example I had no trouble installing my LX2K that boots any regular distro from UEFI. There’s an EDK2 port for RK3588 that also covers Rock5B, I never tested it but it could be a more universal and conventional boot and setup method that targets end users.

[tkaiser]

end users install their PC by booting using a USB stick plugged into a USB port

Correct. And ‘their PC’ is x86.

In this stupid ARM world where each and every SoC needs its own bootloader (at least proprietary DRAM initialization) this simply doesn’t work which is the reason various SBC vendors came up with the idea of preflashing stuff like OOWOW, Roobi or whatever to present the user a list of OS images that contain all the proprietary crap necessary to boot on this specific device.

And this won’t change unless all ARM SoC and SBC vendors start to adopt Arm SystemReady (or whatever the current spec is called that usually has zero relevance for end users since nobody adopts anything of this) and preflash a SPI NOR with a bootloader (already containing all the proprietary crap needed for this specific device) being able to boot generic aarch64 OS images.