Concerns with ROCK Pi X soldered eMMC

Revision 1.4 of the board now has soldered eMMC instead of a detachable module. I have several concerns with this:

  1. Unable to offline dd a rootfs image to the storage for simple OS installation
  2. More complicated OS maintainance/upgrades for units deployed in the field
  3. Failed eMMC modules mean a dead board
  4. Stuck with a fixed amount of storage on a board

All these points make the board less viable for us to deploy at scale (100s of units per site, sites around the world).

Consider this an unofficial community resposse - Radxa might have something better for qts in the 100s…

  • If your target image will be based off a common Linux distribution, all of the Linux distributions out there typically have a way to fully automate the install process. On Debian GNU/Linux and Ubuntu, this is done via the “preseed” functionality of the installer. You can basically craft a USB image that will do a soup-to-nuts automated install of the OS and any customizations you want. You just boot a USB image and off it goes.

  • If this image is not based off a common Linux distribution, or you really just want to flash a flat image to the eMMC, you can always build a USB-based Linux-based “recovery” image that just dd’s the image to the eMMC once it is loaded. How you do this is to basically run a bash script that calls dd and such instead of starting getty (the user terminal emulator that presents the login prompt).

The advantages of both of the above solutions (USB based) are that you can always point a customer to an image that they can write to a USB drive and insert into a new Rock Pi X (or their current one) and have it flash the eMMC in the field. No special tooling or anything required.

I second this, the preseed is really the way to go in this circumstance if using Ubuntu, aside anykind of deal you can strike with Radxa… you could also docker everything outside the base OS…

This doesn’t solve the issue of a failing eMMC, which is the primary advantage of having it not soldered on. Hopefully Radxa will change it back in a future revision.

An installer preseed is definitely not a good alternative to being able to flash a prepared rootfs image.

  1. The preseed file needs to be created and maintained, minimal rootfs for many distros can be downloaded from official channels
  2. Time to a working system is significantly higher with a preseed, takes 10-15 minutes for the install process to complete
  3. You can chroot into a rootfs to perform setup and testing

you can always point a customer to an image that they can write to a USB drive and insert into a new Rock Pi X (or their current one) and have it flash the eMMC in the field

Might be useful for some specific cases, but not for ours or indeed many commercial applications.

It’s worth noting that a very large number of SBCs have the ability to install an OS with the rootfs way:

  • Raspberry Pi and similar including Rock Pi 4 - flashing an SD card
  • Nvidia Jetson boards - script in the driver package

This doesn’t solve the issue of a failing eMMC

Exactly this, or any other issues with units would now require a more complicated debugging solution/requiring additional units on hand to replace any problem ones.

From our understanding, we thought soldered eMMC is better for commercial customers since it will reduce the issue of eMMC module installation.

For installation, ROCK Pi X is always USB to eMMC. If you want to deploy a lot of devices, you can always generate a custom OS image on another PC and dd it to eMMC when booting from USB. Since it’s UEFI boot, basically you only need two partitions, one boot partition for .efi utilities, the other for rootfs. You can always chroot and pre-configure the “master image” before deployment.

So if I understand, with the soldered eMMC, the customer who will buy one Rock Pi X, will have to choose between the model A or B, the size of the memory (1,2,4 GB), and also the size of the eMMC module he want (16,32,64,128 GB?)

With the unsoldered eMMC there are (2x3=) 6 differents possibilities of Rock Pi X.
But with a soldered eMMC, there are (2x3x4=) 24 differents combinations of Rock Pi X

And 24 is certainly too much, so what size of eMMC will you select for each model of Rock Pi X ? Or maybe i’m wrong, maybe there is something i don’t understand when you talk about soldered eMMC ?

We don’t think a user who buy 1GB ram ROCK Pi X will need 128GB eMMC, in theory it’s possible but not practical. The lesson we learnt from ROCK Pi S/E which we provide many SKU but actually mostly users only want some certain ones.

The reason we solder the eMMC instead of the removable one is, current bios is not removable eMMC friendly. If we attach an empty eMMC to a board, install the OS, then some data is saved in the SPI flash(maybe the eMMC training info, we don’t know it yet). If we remove the eMMC module then power on the board without eMMC or we change to another eMMC module. The bios will detect the eMMC but not find the bootable image on the eMMC. We think this will cause some trouble for many users.

In summary, removable eMMC is not an issue as long as we install it and keep it is. Switching eMMC module will cause OS boot issue which we can not solve for now.

On the other hand, because we can not install the OS to uSD card, so eMMC is a must, if people are not aware of buying the eMMC as extra accessory is a must and found that they can not install the OS, they will just blame the vendor. (Installing OS to USB disk is possible)

At last, we think the user will be happy for the same price they get the additional eMMC storage(just because of the technical issue we can not solve), isn’t?

6 Likes

Updated the OP to include other concerns raised in the thread.

Aside from reliability/device failure issues, if the ROCK Pi X will support something like usb-install Like the ROCK Pi 4, then that will at least partially reduce our issues with soldered eMMC for OS installation.

What sized eMMC modules will you be soldering to each version of the board, and how will this impact pricing you shared in this comment?

soldered eMMC is the right choice. Similar to RPi COM3, the commercial version has same options.

Normally, in IoT scenarios, the use case that would require unknown storage requirements are rare.

In the case of UEFI, it’s possible (and even likely) that when the OS is installed, instead of storing the EFI binary at ESP:/EFI/BOOT/BOOTX64.EFI, it stores it at ESP:/EFI/OS-NAME/name.efi and adds an entry to the NVRAM with the location of the binary. If the eMMC gets removed, and the board gets turned back on, the bios will fail to locate this storage location and promptly remove it. Hence, it’s no longer able to find the OS.

I would test this theory, but my board is out of commission for some reason, so I can’t actually confirm this is indeed the case. If someone would like to test this (and isn’t sure exactly how), please let me know, and I’ll try to guide you through the exact process.

1 Like

Thanks for this info, it’s new to me :slight_smile:

I think we can test it with erase/reflashing the bios. The question is, is the adding entry in NVRAM(here is the SPI flash I guess) process is during the OS installation?

Anyway, we will dig more.

Remember, ROCK Pi X is PC. It can install OS without the help of another PC during the installation process(you need to prepare the bootable USB disk on another PC).

For the price, it remains the same, “for the same price they get the additional eMMC storage”.

What @chabad360 said is exactly correct.

In such cases, you can recover from the EFI shell by just specifying the path to the EFI image, e.g.:

fs0:\EFI\boot\grubx64.efi

It will then load GRUB, boot the users’ OS. Then NVRAM can be restored with:

sudo grub-install

Not just, in Window’s case, the Windows Boot Manager is known to be very aggressive in setting itself in the highest priority boot slot. So much so infact, that people who dual-boot are thankful when their laptop doesn’t follow the spec and allow the NVRAM modification from the OS.

But I digress, the UEFI spec specifies ESP:/EFI/BOOT/BOOTX64.EFI as the default place to look for an efi binary if there is no entry in the NVRAM. I know grub-install has an option for installing to the spec default location (using the --removable option), and so does refind-install (using the --usedefault option), I use that location on my Arch install because I didn’t have to deal with adding an NVRAM entry. Regardless, there are various methods of checking and modifying the NVRAM, for example efivars or efibootmgr.

Point is, that this is almost certainly the issue, and it’s also very easy to test. It’s -perhaps- a bit of a shame that the community wasn’t at all asked for troubleshooting help (especially considering that we were able to test this on the board itself). But then again it does seem on the surface to be a rather deep issue, so perhaps @jack and team didn’t think we’d be able to determine the source (again, a very fair assumption considering the issue appears complex), however, it would have been nice to have been given the chance…

Regardless, having the option of the eMMC be removable was nice as it provided added modularity and flexibility to the Rock Pi X.

Really? If that’s indeed the case, then why not (provided the eMMC was still removable) just bundle it with the device (i.e. a bonus choice of a free bundled eMMC)?

I have the Rock Pi X with the 32 gig emmc card installed with a copy of Win 10 on it.(not soldered on) After 30 minutes of installing it automatically, the pc ran in slow motion. It never started again. It will not respond to any other operating systems either linux based or win 10 that are flashed to USB3 drives or any other media. The bios does see these boot options but always goes blank when chosen. I am using Etcher and Rufus to flash the drives. The emmc card checks out alright when using Eseus to mange it. I have never had so much of a hard time installing an os on any type of computers. I have at least 5 scb’s ( two Radxa) that I swap out systems all the time with no problems, Any help or advise you can give is much appreciated. I am about ready to put the RockPi x on the anvil. I am more of a hardware guy not a software expert.

@Garrock Does the board post?

Also, can you describe what you’re doing in a bit more detail (maybe in a new post)?

I am trying to load an operating system on the Rock Pi X. I formatted the emmc and tried both fat32 and ntfs on the usb drives on both of them… I loaded the Radxa recommended systems using Etcher on a usb3 flash drive and used the bios settings in "boot over ride "section to choose the usb3 drive to boot to. The system then goes blank with no activity. I even waited about 40 minutes to see if anything happens but nothing. I tried several distros to no avail using both the fat32 and ntfs formats. I tried the same thing with Win 10 (from Microsoft dl) using Rufus–same thing. I also flashed Win10 directly to the emmc like it was when I got it. When it ran it was like molasses—slow. Now nothing. I have always used etcher or Raspberry Imager for all my Rock Pi’s and Raspberries with no problems. Having Windows on this x86 board would be nice but I don’t think it will run it. My other Rock Pi’s run great. This was given to me as a test subject for new and unusual cases. I would sure love to use it. Any ideas on what I’m missing? It is probably something simple!

Did you try the official guide?
https://wiki.radxa.com/RockpiX/install/Windows.iso

A couple suggestions:

  • Make sure to use the installer image for the x64 versions of the OS you’re trying to use.
  • You can try to use BIOS/Legacy boot mode (my X is out of commission currently so I cant give exact instructions).