I can’t seem to be able to find a combination where nvme works with a newer 5.x kernel. Does anyone know of a way to make this work? If not - what is needed to make this work? Is it that newer kernels don’t support rk3399’s pcie controller?
Thanks,
Darin
Kernel 5.x and nvme
1 Like
It seems that Armbian 5.3.0-rc4 doesn’t have a problem booting from NVME (w/o MicroSD card that is). It seems however that in the FDT for RockPi 4, pcie is disabled. I used the u-boot that Radxa has (which supports nvme) and the slightly modified boot script that Armbian has. Here is what I had to do after getting into u-boot (and pressing Ctrl-C to interrupt). This is just an adaptation of the Armbian script that is on their image. The main difference is the patching of the FDT to enable the pcie.
Booting with a newer kernel made a lot of a difference in the performance as well. I added the result from running sbc-bench here https://github.com/ThomasKaiser/sbc-bench/blob/master/Results.md
As expected - the memory access is much faster.
So not sure why Radxa is not doing anything to get to a newer kernel. The only problem seems to be a lack of effort to just add these few lines into Armbian and have a nice working NVME boot.
setenv scriptaddr 0x00500000
setenv load_addr "0x39000000"
setenv overlay_error "false"
# default values
setenv rootdev "/dev/nvme0n1p1"
setenv verbosity "1"
setenv console "both"
setenv rootfstype "ext4"
setenv docker_optimizations "on"
echo "Boot script loaded from ${devtype} ${devnum}"
setenv prefix /boot/
if test -e ${devtype} ${devnum} ${prefix}armbianEnv.txt; then
load ${devtype} ${devnum} ${load_addr} ${prefix}armbianEnv.txt
env import -t ${load_addr} ${filesize}
fi
if test "${console}" = "display" || test "${console}" = "both"; then setenv consoleargs "console=tty1"; fi
if test "${console}" = "serial" || test "${console}" = "both"; then setenv consoleargs "console=ttyS2,1500000 ${consoleargs}"; fi
if test "${devtype}" = "nvme"; then part uuid nvme ${devnum}:1 partuuid; fi
setenv bootargs "root=${rootdev} rootwait rootfstype=${rootfstype} ${consoleargs} panic=10 consoleblank=0 loglevel=${verbosity} ubootpart=${partuuid}"
if test "${docker_optimizations}" = "on"; then setenv bootargs "${bootargs} cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1"; fi
setenv ramdisk_addr_r 0x04000000
setenv kernel_addr_r 0x02000000
load ${devtype} ${devnum} ${ramdisk_addr_r} ${prefix}uInitrd
load ${devtype} ${devnum} ${kernel_addr_r} ${prefix}Image
setenv fdt_addr_r 0x01f00000
setenv fdtcontroladdr f5f06228
setenv fdtfile rockchip/rk3399-rock-pi-4.dtb
load ${devtype} ${devnum} ${fdt_addr_r} ${prefix}dtb/${fdtfile}
fdt addr ${fdt_addr_r}
fdt resize 65536
load ${devtype} ${devnum} ${load_addr} ${prefix}dtb/rockchip/overlay/${overlay_prefix}-fixup.scr
echo "Applying kernel provided DT fixup script (${overlay_prefix}-fixup.scr)"
source ${load_addr}
setenv bootargs "earlycon=uart8250,mmio32,0xff1a0000,1500000n8 root=${rootdev} rootwait rootfstype=${rootfstype} ubootpart=${partuuid}"
fdt set /pcie@f8000000 status okay
fdt set /pcie@f8000000 num-lanes <4>
fdt set /pcie@f8000000 max-link-speed <2>
fdt set /syscon@ff770000/pcie-phy status okay
fdt set /pcie@f8000000 ep-gpios <0xb7 27 0>
fdt set /pcie@f8000000 pinctrl-names default
fdt set /pcie@f8000000 pinctrl-0 <0x145>
booti ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
2 Likes
Was your nvme showing up via lsblk or any other way prior to the fix you did? I ask because I’m having an issue myself where no matter what I do, I can not confirm that the Rock Pi even sees the nvme SSD I have. I’ve been bashing my head for the last 3 days trying to get this to see the nvme. I’m using the XPG GAMMIX 512GB S11 Pro 3D NAND PCIe NVMe Gen3x4 M.2 2280 SSD (AGAMMIXS11P-512GT-C)
and I ordered a nvme to usb 3.1 type- A adapter and was finally able to confirm that the nvme stick actually works. I can also get the Rock Pi 4b to see the nvme via usb.
1 Like
I don’t think it was. I had to rebuild the kernel in addition to the other stuff to make it see the the nvme. I think in the default config for rockpi-4b something disables the pcie. I’ll try to figure out what is the minimum set of things that needs flipped to make it work. I changed too many options so I can’t say which one made it work.
3 Likes
I finalized my setup so I can now boot Armbian with 5.3 kernel from SPI + nvme unattended (no mmc). Here is what is needed:
- Get Radxa’s u-boot flashed to SPI (see here how to do it https://wiki.radxa.com/Rockpi4/Linux_system_runs_on_M.2_NVME_SSD)
- Build Armbian image with PCI/NVME enabled. By default the
Device Drivers/PCI Support/PCI Controller Drivers/Rockchip PCIe host controlleris set to be compiled into module. It needs to be built into the kernel. So the default<M>needs to be changed to<*>.
git clone --depth 1 https://github.com/armbian/build
cd build
./compile.sh EXPERT=yes
or a faster way to get most options correct for 18.04 server image
./compile.sh BOARD=rockpi-4b BRANCH=dev RELEASE=bionic BUILD_MINIMAL=no BUILD_DESKTOP=no KERNEL_ONLY=no KERNEL_CONFIGURE=yes
- Write the image to NVME drive. I used an USB to NVME adapter (https://www.amazon.com/QNINE-Adapter-Performance-Portable-Support/dp/B07JKWHFRC) and balenaEtcher to do this.
- Mount the image somewhere so you can modify it. The image has only one ext4 partition. It is easy to modify on a PC running linux. For mac - https://www.paragon-software.com/us/home/extfs-mac/ - will probably work. It is also possible to modify the downloaded image and then write to the NVME.
- Append to
/boot/armbianEnv.txtthis linefdtfile=rockchip/rk3399-rock-pi-4.dtb. You can do it like thisecho "fdtfile=rockchip/rk3399-rock-pi-4.dtb" >> /boot/armbianEnv.txtThis is needed as Radxa’s u-boot name for the rock pi 4 dtb is different than what Armbian has. - Create a file
/boot/fixup.cmdwith the folowing:
fdt set /pcie@f8000000 status okay
fdt set /pcie@f8000000 num-lanes <4>
fdt set /pcie@f8000000 max-link-speed <2>
fdt set /syscon@ff770000/pcie-phy status okay
fdt set /pcie@f8000000 ep-gpios <0xb7 27 0>
fdt set /pcie@f8000000 pinctrl-names default
fdt set /pcie@f8000000 pinctrl-0 <0x145>
After creating the file you need to use mkimage to generate image for U-boot like this mkimage -C none -A arm -T script -d fixup.cmd fixup.scr
I used the mkimage from Armbian. It may be available elsewhere too. I also found this http://macappstore.org/u-boot-tools/ for OS X - it has mkimage
- You should be able to boot fine at this point. The serial console works too but doesn’t show the boot messages from the kernel. For me it shows the login prompt when the boot is finished. It takes around 7 sec from start to see
Starting Kernel ...in U-boot
7 Likes
I changed the default for the Rockchip PCIe driver and it will be builtin (vs module) for the future Armbian builds (https://github.com/armbian/build/pull/1623) so step 2 won’t be needed anymore.
2 Likes
@darp Can you try it with Armbian built from this branch: https://github.com/armbian/build/tree/rock-pi-4-enable-pcie-in-dts ?
Changes in the branch should eliminate the need for step 6.
I configured the PCIe for Gen1 speed as that’s what is officially supported by Rockchip but also created an overlay to let every brave soul use Gen2 speeds.
You activate it with overlays=pcie-gen2 in /boot/armbianEnv.txt.
I will address step 5. for rockchip64/rk3399 boards globally which should make it even easier.
1 Like
Yes, it works fine. I’ll update the steps when you merge.
You may as well do for rock pi 4 the same as in xxx-add-symlink-to-old-dtb-names-for-nanopi-and-nanopc.patch. Radxa’s u-boot expects the name to be rockpi-4b-linux.dtb. Good job!
1 Like
Hi guys,
Thanks a lot for your contributions! 
I didnt build myself an image or compiled the kernel, but downloaded the latest build from Buster server (kernel 5.3) as it seems your PR was merged into master.
I tried it out of the box with the latest radxa spi, then including the changes to /boot/armbianEnv.txt, then including the fixup.[cmd,scr] files, but without success.
The nvme light flashes quickly once and then i get blue and green fixed led.
The screen stays blank all the time
I don’t have a way to look at the console output at the moment.
Any tips to make this work out of an official armbian build?
Thanks!
1 Like
Hi @raleonardo
Similar situation as yours. Latest Armbian Buster current works fine on SD card. If I insert Radxa eMMC card it fails to boot.
However modifying /boot/armbianEnv.txt as specified in Step 5 worked and was able to boot!
The step 6 fixup.[cmd,scr] step should not be needed anymore after work done by Darp and Piter. Thanks guys for your work and clear instructions. 
I dunno where the issue is, could be related to the SPI or the NVME drive that you use. Which drive are you using btw?
Edit-
One more thing, it takes some time to boot initially! Wait for atleast a minute to see if it boots.
Step 6 is not needed anymore. I can’t seem to be able to edit my original post.
Thanks,
I didnt try armbian buster on the sd card yet.
I’m using an HP EX900 250GB m.2 nvme, which boots fine with the debian stretch gpt image from radxa.
I’ll try again with:
- uSD
- nvme + step 5
@darp Thanks for the info
1 Like
XPG GAMMIX 512GB S5 works and boot SPI without eMMC/SDcard
Armbian_19.11.3_Rockpi-4b_buster_current_5.3.11.img
official radxa 4b-u-boot
tested RockPi4a v1.4
1 Like
out of the box or with step 5?
sorry my bad english, what you mean “out of the box or with step 5”
what is out the box?
Out of the box means something that is in its default state, ie, untouched or unchanged.
He means if you changed /boot/armbianEnv.txt as specified in step 5 of guide by Darp, or it worked without changing anything.
Armbian_19.11.3_Rockpi-4b_buster_current_5.3.11.img , i DL it from armbian site and do nothing.
- flash to sdcard and from sdcard same …img to NVME
- boot with sdcard, (need connect pin 23 and 25), when “Starting kernel …” remove sdcard, it use NVME
- add to armbianEnv.txt “fdtfile=rockchip/rk3399-rock-pi-4.dtb”
-SPI flash with prommer direct to rockpi pins,19, 21, 23, 24
or radxa linaro sdcard xxxx rockpi4b_upgrade_bootloader.sh
or rkdeveloptool
all of them works
sorry
image is virgin Ambian, not my build
5. only to armbianEnv.txt “fdtfile=rockchip/rk3399-rock-pi-4.dtb”
Thanks for the info.
Again why do you need to bypass SPI (shorting 23 and 25) to be able to boot from the sdcard? Seems odd that its not possible to boot directly from SPI into the SD card
So, I finally made it to boot from the nve with the latest buster server armbian (changed with step 5).
Issues:
- It takes ages to boot (someone already reported here I guess)
- when I issue “hdparm -t --direct”,
- I get an error “HDIO_DRIVE_CMD (identify) failed: Inappropriate ioctl for device”
- the reported speed is too slow (around 190MBps (I remember having smth around 500 MBps in both radxa and arbian stretch ) )
might there be an issue with the driver in the 5.3 kernel?
@darp, would also step 5 at some point integrated into the build for the rockpi4b?
Any reason for this not to be in place?
Do you know where the armbianEnv.txt is created?
If SPI is empty, both, radxa and armbian boot, not need to connect pins, otherwise only official boot from the sdcard