SPI + NVMe booting(beta)

jack · July 18, 2020, 4:29am

update 2020-07-18:

Refer wiki page for the updated and maintained version.

https://wiki.radxa.com/Rockpi4/Linux_system_runs_on_M.2_NVME_SSD

update 2019-07-31:

Since many user have issue with SPI + NVMe booting, we have updated a new Debian stretch firmware with everything ready to use. Here is the steps to use it.

1. Boot with uSD card or eMMC module

Download the image from:

https://dl.radxa.com/rockpi4/images/debian/rockpi4-debian-stretch-desktop-arm64-20190730_2022-gpt.img.gz

Write it to uSD card or eMMC module with dd or ecther, attach your NVMe disk to ROCK Pi 4, and boot from uSD card or eMMC module.

2. Write the SPI flash

After boot from eMMC or uSD card, you should have /dev/mtd0 device if your ROCK Pi 4 has SPI flash soldered, run the script to write the bootloader to SPI flash.

    sudo /usr/local/sbin/rockpi4b_upgrade_bootloader.sh

Output

 You are currently running on board:
 ROCK PI 4B 
 There will be two steps that you have to confirm. One is upgrading bootloader on SPI Flash.
 The other is upgrading bootloader on eMMC or uSD card.
 
 Step one: upgrade bootloader on SPI Flash
 One boot device, SPI Flash, is found, would you like to upgrade bootloader on it?
 The installation would cost several minutes.
 If yes, type Y/y. If no, type N/n.

Type Y to confirm. It takes several minutes to erase and write the SPI flash. Have some tea and wait.

3. Write the NVMe disk

Copy the same image you downloaded on your ROCK Pi 4 and dd it to NVMe.

sudo apt-get install pigz
unpigz -p 6 -c  /path/to/rockpi4-debian-stretch-desktop-arm64-20190730_2022-gpt.img.gz  | sudo  dd of=/dev/nvme0n1 bs=4M conv=fsync status=progress

It should be pretty quick.

4. Boot from SPI + NVMe

Now power off the ROCK Pi 4, remove the eMMC module or uSD card. With NVMe attached and boot.
You should see the green led on first, then red/blue led on, then red/blue led start blink. When blinking, it means it boots the kernel in NVMe.

Tested NVMe SSD

For some reason, some NVMe SSD can be recognized under Linux but failed on bootloader. Here is some models we tested.

Known booting:

MaxMemory 128G: boots fine
HP EX900: boots fine
Kingbank 240G: boots fine
Lenovo SL700: boots fine

Known not booting:

Samsung EVO 960: it can detect gpt partition in NVMe but failed to load the kernel, no problem formatting or making fs under Linux. Maybe this SSD in our office is somehow defective.

/* Below is old content */

Dear community,

After fixing the bug of u-boot in SPI with NVMe driver, we have the working bootchain from SPI to NVMe now. The new u-boot package is pushed to the testing apt repository. Below is how to enable the SPI to NVMe boot chain.
deb http://apt.radxa.com/stretch-testing/ stretch main
or
deb http://apt.radxa.com/bionic-testing/ bionic main
To update to the latest testing repo, change your apt repo address to the testing one, and
sudo apt-get install rockpi4b-rk-u-boot-latest

/usr/local/sbin/rockpi4b_upgrade_bootloader.sh
Edit /boot/hw_intfc.conf and enabled the following:
intfc:uart4=off
intfc:spi1=on
intfc:dtoverlay=spi1-flash
Reboot, the SPI flash will be shown as /dev/mtd0 under Linux.

Use the script to flash the uboot image to SPI flash and type YES to perform the flash.
sudo /usr/local/sbin/rockpi4b_write_spi_flash.sh
Now we have the uboot image installed to SPI flash. It’s time to move the image to NVMe.

For new installation and simpacity, you can just download a fresh ROCK Pi 4 image(Android is not supported at the moment, other images should be all supported) dd the whole image to /dev/nvme0n1 on ROCK Pi 4.

For moving existing image from uSD or eMMC to NVMe disk, just dd /dev/mmcblk0 or /dev/mmcblk1 to /dev/nvme0n1.
sudo dd if=/dev/mmcblk0 conv=sync,noerror of=/dev/nvme0n1 bs=16M
or
sudo dd if=/dev/mmcblk1 conv=sync,noerror of=/dev/nvme0n1 bs=16M

keichwa · June 27, 2019, 8:56am

On https://wiki.radxa.com/Rockpi4/Linux_system_runs_on_M.2_NVME_SSD, you are saying that SPI Flash IC on ROCK Pi 4 is required as a pre-requisite. But later on, it is no longer mentioned.

Is it also needed if I have a running RockPi4 v1.3? Reading https://wiki.radxa.com/Rockpi4/dev/spi-install it looks to me that this SPI Flash IC is already soldered on the board. I could be wrog, though. I’m rather new to these SBCs. Or is it only available on v1.4?

Edit: I just found https://wiki.radxa.com/Rockpi4/hardware/spi_flash and it is available since v1.4, unfortunately.

stuartiannaylor · June 27, 2019, 11:27am

I you have 1.3 you will have to purchase and solder on.

stuartiannaylor · June 27, 2019, 11:29am

I presume Radxa or Allchina could provide ready flashed?

g.gabi · June 27, 2019, 6:48pm

I tried to activate Spi flash, but it is not working:

kernel: rockchip-spi ff1d0000.spi: Failed to request TX DMA channel
kernel: rockchip-spi ff1d0000.spi: Failed to request RX DMA channel
kernel: rockchip-spi ff1d0000.spi: no high_speed pinctrl state
kernel: m25p80 spi32766.0: unrecognized JEDEC id bytes: ff, ff, ff
kernel: m25p80: probe of spi32766.0 failed with error -2

Is deb http://apt.radxa.com/stretch-testing/bionic-testing correct?
Or deb http://apt.radxa.com/bionic-testing?

I have a Rock Pi 4 A rev.1.4, thanks in advance.

stuartiannaylor · June 27, 2019, 8:30pm

dunno @g.gabi think your the first on here I have seen with v1.4

@jack
sudo -i

For ubuntu

export DISTRO=bionic
echo "deb http://apt.radxa.com/$DISTRO/ bionic main" | sudo tee /etc/apt/sources.list.d/apt-radxa-com.list
wget -O -  apt.radxa.com/$DISTRO/public.key | sudo apt-key add -
sudo apt-get update && sudo apt-get upgrade

for debian

export DISTRO=stretch
echo "deb http://apt.radxa.com/$DISTRO/ stretch main" | sudo tee /etc/apt/sources.list.d/apt-radxa-com.list
wget -O -  apt.radxa.com/$DISTRO/public.key | sudo apt-key add -
sudo apt-get update && sudo apt-get upgrade

$DISTRO is the variable export DISTRO= sets

/etc/apt/sources.list and same in sources.list.d as you prob need to delete the bad lines added

g.gabi · June 27, 2019, 8:53pm

Thx, I did that for ubuntu minimal. So deb http://apt.radxa.com/stretch-testing/bionic-testing is a typo?
U-boot and kernel should be la(s)test .
Received my Rock Pi 8 weeks ago, so rev1.4 should be quit common?

stuartiannaylor · June 27, 2019, 8:40pm

Weird as got mine later and 1.3 and just sent one back to reichelt as didn’t want another 1.3
Must be still in the wild but depends where you source I guess

deb http://apt.radxa.com/stretch-testing/bionic-

dunno where you got that but yeah totally wrong stretch is debian and testing is the unstable repo
bionic is ubuntu and somehow you got a hybrid

g.gabi · June 27, 2019, 9:24pm

First post …

Wondered about rev1.4 when I got mine from reichelt, thought it’s because of model A.
Btw no more radxa on reichelt? And no ropis model A in stock in other european stores?

stuartiannaylor · June 27, 2019, 9:46pm

Yeah mine was a B and v1.3 the RockPiA-2gb is the best buy I think for server usage.
I got my m.2 4 port sata and just did a quick test with a cheap integral ssd.

Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    40425    48005    58438    57793    23966    46915
          102400      16   111826   135299   156540   153694    82364   123766
          102400     512   339600   352329   363463   366823   269568   357378
          102400    1024   292547   356682   363840   367742   296867   360861
          102400   16384   441573   357018   422514   429050   417921   449490

Need to get 3 more and just see what it bottle necks with with a stripe array

g.gabi · June 27, 2019, 10:03pm

can you boot from the m.2 4 port sata card? Pcie 1 or 2?

stuartiannaylor · June 27, 2019, 10:09pm

Have not tried but with the spi nor presume its the same.
Prob will have the system on eMMC as the RAID10 will just be network storage
sudo dd if=/dev/mmcblk0 conv=sync,noerror of=/dev/sda1 bs=16M

I guess wait and see but the nvme0n1 might be hardcoded but presume its an easy one to provide

@jack

jack · June 28, 2019, 6:44am

Sorry for the typo. I edited the original post. It should be right now.

jack · June 28, 2019, 6:47am

PCIE to SATA is not tested in u-boot yet. Currently the boot device is added as nvme in u-boot together with mmc0, mmc1, usb, network. We can have a try but I am afraid it will not be supported.

stuartiannaylor · June 28, 2019, 7:16am

Doesn’t really matter to be honest as likely the main purpose of slower multiple drives is to provide storage space rather than system.
The 4port Sata is perfect for a micro server and the system can either be SD or eMMC as it is preferable to be have the system separate.

Saying that if you can boot via pcie nvme then surely its just a find and replace of /dev/sda that probably could be implemented via overlay.

As said its likely its not a big deal for many, certainly isn’t for me.

g.gabi · June 28, 2019, 2:21pm

Thx for the correction, but still no luck activating SPI flash:

Jun 28 16:10:31 ropi-ra kernel: rockchip-spi ff1d0000.spi: Failed to request TX DMA channel
Jun 28 16:10:31 ropi-ra kernel: rockchip-spi ff1d0000.spi: Failed to request RX DMA channel
Jun 28 16:10:31 ropi-ra kernel: rockchip-spi ff1d0000.spi: no high_speed pinctrl state
Jun 28 16:10:31 ropi-ra kernel: m25p80 spi32766.0: unrecognized JEDEC id bytes: ff, ff, ff
Jun 28 16:10:31 ropi-ra kernel: m25p80: probe of spi32766.0 failed with error -2

any ideas?

One more question: No more radxa products on reichelt? And no RockPi model A in stock in other european stores? Or should I better make a new thread about that topic?

stuartiannaylor · June 28, 2019, 6:28pm

I would wait for @jack but think you might be alone for a while as not many of of have v1.4 or purchased flash.
You are blazing a trial prob pointless creating another and its just timezone to the radxa team now.

natecarlson · June 28, 2019, 7:26pm

If I purchase one from allnet today, is it guaranteed to be 1.4?

Just looking into the Rock Pi’s today, would really prefer to start with bootloader in SPI and full OS on NVMe.

Stephen · July 2, 2019, 1:26am

We have uploaded a new version of u-boot with red(or blue) led on, which would help to see if the image is installed to SPI Flash successfully.
See https://wiki.radxa.com/Rockpi4/Linux_system_runs_on_M.2_NVME_SSD

ac8dg · July 5, 2019, 1:51pm

Just ordered the model from allnet today
Rock pi 4B 4GB v1.3+ Limited Edition Bundle (incl. mounted 16MB SPI Flash)

plus Lenovo NVME SSD 256GB

Now I am not sure how to proceed to get the debian image to boot.

do I proceed with install as described for mico sd ?
or do I flash a emmc and follow that route ?