Investigating smallest possible Debian Rootfs for RockPi-S

stulluk · September 20, 2020, 3:52pm

Hi, I have three RockPi-S (v13) , and two of them can boot via SDcard with prebuilt images from here:

https://wiki.radxa.com/RockpiS/downloads

(One of them even doesn’t boot via SDcard, I don’t know why, may be it is damaged during transport? )

Succesful SDcard bootup logs are here: https://pastebin.pl/view/cb55f626

Now, I am trying to bootup my v13 Rockpi-S’es ( I have three of them ) over builtin storage( i.e SDnand). Unfortunately, I get following error:

https://pastebin.pl/view/6f5bf6d5

I built boot0.img and system.img in a docker container as explained here:

https://wiki.radxa.com/RockpiS/dev/Debian

My Dockerfile content is:
FROM ubuntu:18.04

RUN apt -y update && apt -y upgrade

RUN apt install -y git gcc-aarch64-linux-gnu device-tree-compiler libncurses5 libncurses5-dev build-essential libssl-dev mtools 

RUN apt -y install bc python dosfstools u-boot-tools

For anyone interested, you can directly use it:
https://hub.docker.com/repository/docker/stulluk/rockpis-builder

My build script is ( buildall.sh):
#!/bin/bash

#Build u-boot
./build/mk-uboot.sh rockpis

printf "\n*************************\n"

#Build Kernel
./build/mk-kernel.sh rockpis

printf "\n*************************\n"

#Build rootfs part1
export ARCH=arm64
cd rootfs
apt -y install binfmt-support qemu-user-static cpio gdisk
dpkg -i ubuntu-build-service/packages/*        # ignore the broken dependencies, we will fix it next step
apt -y install -f
RELEASE=stretch TARGET=desktop ARCH=$ARCH ./mk-base-debian.sh

printf "\n*************************\n"
#Build rootfs part2

VERSION=debug ARCH=$ARCH ./mk-rootfs-stretch.sh  && ./mk-image.sh

printf "\n*************************\n"
#Combine everything
cd ..

build/mk-image.sh -c rk3308 -t system -r rootfs/linaro-rootfs.img

My docker build command is:

And its output is:

https://pastebin.pl/view/4d4dbe32

And then, burning these to my RockPi-S:

stulluk /F250/ROCKPI-S/rockchip-bsp (stable-4.4-rockpis) $  sudo rkdeveloptool ld
DevNo=1	Vid=0x2207,Pid=0x330e,LocationID=202	Maskrom
stulluk /F250/ROCKPI-S/rockchip-bsp (stable-4.4-rockpis) $  sudo rkdeveloptool db out/u-boot/rk3308_loader_uart0_m0_emmc_port_support_sd_20190717.bin 
Downloading bootloader succeeded.
stulluk /F250/ROCKPI-S/rockchip-bsp (stable-4.4-rockpis) $  sudo rkdeveloptool wl 0 out/system.img 
Write LBA from file (100%)
stulluk /F250/ROCKPI-S/rockchip-bsp (stable-4.4-rockpis) $  sudo rkdeveloptool rd
Reset Device OK.
stulluk /F250/ROCKPI-S/rockchip-bsp (stable-4.4-rockpis) $

And finally, the image can’t boot:

https://pastebin.pl/view/d0de46b7

Can you please guide me, where am I doing a mistake? Or, is there a bug in the BSP ?

jack · September 19, 2020, 2:21pm

How big is your image, make sure it’s smaller than the SD Nand size.

stulluk · September 19, 2020, 2:22pm

You are right, image seems much larger than Nand size !!

But I followed your wiki, how to make it smaller ???

stulluk · September 19, 2020, 2:29pm

Image size after above compilation is around 4Gbyte size, but SDnand is only 1Gbyte.

I need to find a way to build smaller image.

Do you have any suggestions ?

igorp · September 19, 2020, 4:21pm

Stock minimal Armbian image will fit perfectly on 1Gb.

Whatever you need extra:
https://docs.armbian.com/Developer-Guide_User-Configurations/#user-provided-image-customization-script

nish · September 19, 2020, 4:35pm

You can build a custom OS using yocto. The base minimal gpt image will be less than 300mb and console image of 600mb. You could follow the instructions given here to build your own image: https://github.com/nishantpoorswani/meta-radxa

Or

https://wiki.radxa.com/Yocto-layer-for-radxa-boards

You can customize and add whatever you need

igorp · September 19, 2020, 4:59pm

If you need that and if time input in learning and dealing with Yocto is justified. Yocto is nice but mastering it takes time - Yocto and buildroot compared with Armbian build system.

nish · September 19, 2020, 5:13pm

We’ve kept the readme simple and it is easy to build now currently. I honestly like Armbian btw

stulluk · September 20, 2020, 9:08am

@nish Thank you very much. I considered Yocto for a long time, and unfortunately I need a debian based rootfs with APT. Can I have APT with these instructions ?

Update/edit: Installed official yocto image:

/dev/root 125.1M 97.7M 18.1M 84% /
No rootfs resize?

No APT ?

No NetworkManager ?

No SSH ?

Therefore I am considering ISAR ( https://github.com/ilbers/isar ) for many SBC’s in the market, which uses bitbake + debian based rootfs.

Unfortunately, as Igor suggested, initial porting + learning curve to have a Isar based build system is too expensive for hobby projects.

These days, I also considered DEBOS ( https://github.com/go-debos/debos ) which also lacks documentation for porting to a new device.

@igorp Nice to see you here Igor. And thanks for your suggestion for armbian.

I also considered learning armbian, and I have some criticisms for it:
- No Incremental build: I couldn’t find a clear documentation about how to achieve incremental ( delta ) builds with armbian. For example, if I change only one line in u-boot or kernel or userpatches, how can I rebuild whole image in less than 2 mins ?

- Not enough documentation for porting armbian to a new board: I know you don’t want to maintain lots of custom boards which maintainers can not access. This is fair. But this avoids for someone like me who works on multiple Allwinner & RockChip & IMX8 based boards, and to have a generic build system for all of it.

- Not enough documentation for customisation considerations: How to create a extremely minimal rootfs, just to have Ethernet + Debian APT with Armbian ?

Finally, an answer to both of you. I have 3 projects which I would like to achieve with RockPi-S. One of them is to run docker-pihole on it. And this requires around 500Mbyte of storage space.

Even minimal image that I downloaded from radxa website is around 900+Mbyte when installed to SDnand. If I am not mistaken, it is almost not possible to install docker with apt ?

Am I missing anything ?

Need your suggestions please…

igorp · September 20, 2020, 11:37am

That very important feature is supported since early days, but not enabled by default - check documentation CLEAN parameter in the default config. One can easily recompile kernel & u-boot image in less then 1 minute (on a modern desktop CPU), but for the whole image you need several minutes. Everything that is possible is cached but there is a small room for improvements - if you remove one apt update that is done each time, you can squeeze some more but its not recommended. And if you develop multi-core packages deployment, a bit more.

It depends. For the purpose you are describing more then enough.

Armbin provides pretty generic system - just do some more research and check support term in case you want more help / shortcuts https://github.com/armbian/build#support Adding a new board? Relatively simple: https://github.com/armbian/build/commit/23604e8a0dcdf81ec6c28ccd4b2a64b90816d8e7#diff-7ec8d78f3fb79b669285f5a5c42069e3 … killing BSP complexity is ofc your cost. The same as with Yocto or buildroot.

RockpiS is fully supported for several months. You don’t need to waste your own time for that, so you just (read the docs) and make a custom image.

Ofc its possible.

stulluk · September 20, 2020, 12:29pm

@igorp thank you.

I will try to find out how to create an extremely minimal image ( only Network + APT + Docker ) for RockPi-S via Armbian. I will give a next try with CLEAN parameter, and share my build duration results, and try to document it.

@nish I am trying to follow your readme ( building via yocto ), and on my ancient build machine ( FX6300 + 12GB DRAM + multiple 250GByte Samsung Evo SSDs ), it is still building after 2 hours.

I really can’t understand why modern build systems are so heavyweight

But I will keep trying, lets see what will happen.

Many thanks.

nish · September 20, 2020, 12:56pm

You can build the console image. That has way more features like SSH, networkmanager, htop, dialog etc. You can get get APT but you will need to compile other packages using yocto and host them. Better to just add packages you want using IMAGE_INSTALL_append in the local.conf. We will be adding more instructions in the readme which will help new users to add packages and configure kernel according to their need in the coming days.

That’s because everything is fetched and compiled from scratch(including your toolchain) so hence the pain. The first build takes time but then subsequent builds are just incremental and don’t take much time.

I agree it is time consuming but I really like it from a deployment perspective.

stulluk · September 20, 2020, 2:37pm

Edit: apt is not there, but apt-get is there. Unfortunately, I can’t install any package because perl version in the yocto layer is higher than the one used in apt, and trying to find a way to fix this

Ok, first of all, for anyone interested, I will provide some amateur information about the system (HW/OS) below.

May be it helps somebody else who is experimenting on this nice board.

SDnand on the RockPi-S has following approximate sequential write speed:

root@rockpi-s-rk3308:~# dd if=/dev/zero of=/dev/mmcblk1 bs=16M status=progress
1006632960 bytes (1.0 GB, 960 MiB) copied, 92 s, 10.9 MB/s
dd: error writing '/dev/mmcblk1': No space left on device
61+0 records in
60+0 records out
1010827264 bytes (1.0 GB, 964 MiB) copied, 99.1463 s, 10.2 MB/s
root@rockpi-s-rk3308:~#

@nish thank you, I finally completed building console image:

stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $  `bitbake -k radxa-console-image`
WARNING: Host distribution "linuxmint-19.3" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |##################################################################################################| Time: 0:00:00
Loaded 3580 entries from dependency cache.
Parsing recipes: 100% |################################################################################################| Time: 0:00:00
Parsing of 2453 .bb files complete (2452 cached, 1 parsed). 3581 targets, 115 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "rockpi-s-rk3308"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0.4"
TUNE_FEATURES        = "aarch64 cortexa35 crc crypto"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "zeus:d88d62c20d7d8da85f02edb170dae0280624ad7e"
meta-oe              
meta-networking      
meta-python          
meta-multimedia      
meta-gnome           
meta-xfce            = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-radxa           = "zeus:d489b2909b45112434d843fba1aa18f8229949f2"

Initialising tasks: 100% |#############################################################################################| Time: 0:00:03
Sstate summary: Wanted 410 Found 0 Missed 410 Current 966 (0% match, 70% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Tasks Summary: Attempted 3884 tasks of which 3117 didn't need to be rerun and all succeeded.

Summary: There was 1 WARNING message shown.
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $

The image created is ( radxa-console-image-rockpi-s-rk3308-gpt.img ) around 400Mbyte:

stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $  ls -lah tmp/deploy/images/rockpi-s-rk3308/
total 776M
drwxr-xr-x 4 stulluk stulluk 4,0K Sep 20 15:40 .
drwxrwxr-x 3 stulluk stulluk 4,0K Sep 20 14:22 ..
-rw-r--r-- 2 stulluk stulluk 2,2K Sep 20 15:36 boot.cmd
-rw-r--r-- 2 stulluk stulluk 2,3K Sep 20 15:36 boot.scr
-rw-r--r-- 1 stulluk stulluk 127K Sep 20 15:40 idbloader.img
lrwxrwxrwx 2 stulluk stulluk   52 Sep 20 14:18 Image -> Image--4.4.143-r1-rockpi-s-rk3308-20200920092156.bin
-rw-r--r-- 2 stulluk stulluk  12M Sep 20 14:18 Image--4.4.143-r1-rockpi-s-rk3308-20200920092156.bin
lrwxrwxrwx 2 stulluk stulluk   52 Sep 20 14:18 Image-rockpi-s-rk3308.bin -> Image--4.4.143-r1-rockpi-s-rk3308-20200920092156.bin
-rwxr-xr-x 1 stulluk stulluk 190K Sep 20 15:35 mkimage
-rw-rw-r-- 2 stulluk stulluk 178M Sep 20 14:21 modules--4.4.143-r1-rockpi-s-rk3308-20200920092156.tgz
lrwxrwxrwx 2 stulluk stulluk   54 Sep 20 14:21 modules-rockpi-s-rk3308.tgz -> modules--4.4.143-r1-rockpi-s-rk3308-20200920092156.tgz
drwxr-xr-x 2 stulluk stulluk 4,0K Sep 20 14:21 overlays
drwxr-xr-x 2 stulluk stulluk 4,0K Sep 20 15:36 radxa-binary
-rw-r--r-- 2 stulluk stulluk 267M Sep 20 15:40 radxa-console-image-rockpi-s-rk3308-20200920123221.rootfs.ext4
-rw-r--r-- 2 stulluk stulluk  59K Sep 20 15:39 radxa-console-image-rockpi-s-rk3308-20200920123221.rootfs.manifest
-rw-r--r-- 2 stulluk stulluk 266K Sep 20 15:39 radxa-console-image-rockpi-s-rk3308-20200920123221.testdata.json
-rw-r--r-- 1 stulluk stulluk 112M Sep 20 15:40 radxa-console-image-rockpi-s-rk3308-boot.img
lrwxrwxrwx 2 stulluk stulluk   62 Sep 20 15:40 radxa-console-image-rockpi-s-rk3308.ext4 -> radxa-console-image-rockpi-s-rk3308-20200920123221.rootfs.ext4
-rw-r--r-- 1 stulluk stulluk 396M Sep 20 15:40 radxa-console-image-rockpi-s-rk3308-gpt.img
lrwxrwxrwx 2 stulluk stulluk   66 Sep 20 15:39 radxa-console-image-rockpi-s-rk3308.manifest -> radxa-console-image-rockpi-s-rk3308-20200920123221.rootfs.manifest
lrwxrwxrwx 2 stulluk stulluk   64 Sep 20 15:39 radxa-console-image-rockpi-s-rk3308.testdata.json -> radxa-console-image-rockpi-s-rk3308-20200920123221.testdata.json
-rw-r--r-- 2 stulluk stulluk  63K Sep 20 14:21 rk3308-rock-pi-s--4.4.143-r1-rockpi-s-rk3308-20200920092156.dtb
lrwxrwxrwx 2 stulluk stulluk   63 Sep 20 14:21 rk3308-rock-pi-s.dtb -> rk3308-rock-pi-s--4.4.143-r1-rockpi-s-rk3308-20200920092156.dtb
lrwxrwxrwx 2 stulluk stulluk   63 Sep 20 14:21 rk3308-rock-pi-s-rockpi-s-rk3308.dtb -> rk3308-rock-pi-s--4.4.143-r1-rockpi-s-rk3308-20200920092156.dtb
-rw-r--r-- 1 stulluk stulluk 1,0M Sep 20 15:40 trust.img
-rw-r--r-- 1 stulluk stulluk  173 Sep 20 15:40 trust.ini
lrwxrwxrwx 2 stulluk stulluk   33 Sep 20 15:36 u-boot.bin -> u-boot-rockpi-s-rk3308-1.0-r0.bin
-rw-r--r-- 1 stulluk stulluk 1,0M Sep 20 15:40 u-boot.img
-rw-r--r-- 2 stulluk stulluk 618K Sep 20 15:36 u-boot-rockpi-s-rk3308-1.0-r0.bin
lrwxrwxrwx 2 stulluk stulluk   33 Sep 20 15:36 u-boot-rockpi-s-rk3308.bin -> u-boot-rockpi-s-rk3308-1.0-r0.bin
-rw-r--r-- 2 stulluk stulluk  149 Sep 20 15:36 uEnv.txt
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $

I don’t know which of these files correspond to bootloader, like the file:

rk3308_loader_uart0_m0_emmc_port_support_sd_20190717.bin

Building took around 2+ hours. Since Yocto is quite incremental, I can modify something, and trigger :

bitbake -k radxa-console-image

I burned this file to a fast SDcard via etcher, and the RockPi-S boots (drop to shell prompt) in around 10 seconds.

https://pastebin.pl/view/ce63d072

Quite impressive.
And it resizes rootfs during the 1st bootup.
In order to login, username is “root” and password is “rock”.

After bootup, storage consumption is:

root@rockpi-s-rk3308:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        14G  185M   14G   2% /
devtmpfs        212M     0  212M   0% /dev
tmpfs           213M     0  213M   0% /dev/shm
tmpfs           213M  8.7M  204M   5% /run
tmpfs           213M     0  213M   0% /sys/fs/cgroup
tmpfs           213M     0  213M   0% /tmp
tmpfs           213M   16K  213M   1% /var/volatile
tmpfs            43M     0   43M   0% /run/user/0
root@rockpi-s-rk3308:~#

And ssh is OK:

root@rockpi-s-rk3308:~# systemctl | grep ssh
  sshd@1-192.168.1.111:22-192.168.1.247:35218.service                                        loaded active running   OpenSSH Per-Conne 
  sshdgenkeys.service                                                                        loaded active exited    OpenSSH Key Gener 
  system-sshd.slice                                                                          loaded active active    system-sshd.slice 
  sshd.socket                                                                                loaded active listening sshd.socket       
root@rockpi-s-rk3308:~#

And NetworkManager is also OK:

root@rockpi-s-rk3308:~# systemctl | grep NetworkManager
  NetworkManager.service                                                                     loaded active running   Network Manager                                                                     
root@rockpi-s-rk3308:~#

Unfortunately, there is no APT (yet):

root@rockpi-s-rk3308:~# apt
-sh: apt: command not found
root@rockpi-s-rk3308:~#

Then, I decided to test to burn this image to SDnand:

stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $  sudo rkdeveloptool db ../../../IMAGES/rk3308_loader_uart0_m0_emmc_port_support_sd_20190717.bin 
Downloading bootloader succeeded.
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $  sudo rkdeveloptool wl 0 tmp/deploy/images/rockpi-s-rk3308/radxa-console-image-rockpi-s-rk3308-gpt.img 
Write LBA from file (100%)
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $

After bootup, storage consumption is same as SDcard:

root@rockpi-s-rk3308:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       802M  185M  578M  25% /
devtmpfs        212M     0  212M   0% /dev
tmpfs           213M     0  213M   0% /dev/shm
tmpfs           213M  8.7M  204M   5% /run
tmpfs           213M     0  213M   0% /sys/fs/cgroup
tmpfs           213M     0  213M   0% /tmp
tmpfs           213M   16K  213M   1% /var/volatile
tmpfs            43M     0   43M   0% /run/user/0
root@rockpi-s-rk3308:~#

@nish : Thank you.

You can get get APT but you will need to compile other packages using yocto and host them. Better to just add packages you want using IMAGE_INSTALL_append in the local.conf.

How to do that? To be more precise, I want to run docker-pihole ( https://www.smarthomebeginner.com/run-pihole-in-docker-on-ubuntu-with-reverse-proxy/ ) on RockPi-S.

I did this on my DS167 board easily ( Allwinner A20 + 1GBRAM + 8Gbyte eMMC).

If I had APT + DOCKER in this yocto image, I thought that I could do it on RockPi-S too.

What is your suggestion to achieve this (docker-pihole on RockPi-S) via IMAGE_INSTALL_append ?

nish · September 20, 2020, 5:37pm

It is the loader.bin file in build/tmp/deploy/images/rockpi-s-rk3308.conf/radxa-binary folder.

You can search for list of packages that you build here: https://layers.openembedded.org/layerindex/branch/zeus/recipes/

As you will see the docker-ce recipe is given in meta-virtualization layer (https://layers.openembedded.org/layerindex/recipe/112720/)

These would be the steps to add docker to the image:

Fetch the layer using git in the poky folder where other layers like meta,meta-openembedded are present:

~/yocto/poky $ git clone git://git.yoctoproject.org/meta-virtualization -b zeus

Add meta-virtualization layer to the build/conf/bblayers.conf along with the dependencies (https://layers.openembedded.org/layerindex/branch/zeus/layer/meta-virtualization/)

Your build/conf/bblayers.conf should contain these:

  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-openembedded/meta-oe \
  ${TOPDIR}/../meta-openembedded/meta-networking \
  ${TOPDIR}/../meta-openembedded/meta-python \
  ${TOPDIR}/../meta-openembedded/meta-multimedia \ 
  ${TOPDIR}/../meta-openembedded/meta-gnome \
  ${TOPDIR}/../meta-openembedded/meta-xfce \
  ${TOPDIR}/../meta-radxa \
  ${TOPDIR}/../meta-openembedded/meta-filesystems \
  ${TOPDIR}/../meta-virtualization \

Note: meta-virtualization and meta-filesystem are added here. meta-filesystems was a dependency which wasn’t present before in the bblayers.conf

Now to instruct the build system to add the docker-ce package in our image we need to add it using the IMAGE_INSTALL_append variable in the build/conf/local.conf file:

IMAGE_INSTALL_append = " docker-ce"

You can add it to the end of the file as well.

Setup the environment using

~/yocto/poky $ source oe-init-build-env

Run the command to build the console image:

~/yocto/poky/build $ bitbake -k radxa-console-image

At the end of the build you should have docker-ce in your console-image. I hope you find this useful.

stulluk · September 20, 2020, 7:56pm

@nish amazing explanation, thank you very much.

stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  cat build/conf/bblayers.conf 
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-openembedded/meta-oe \
  ${TOPDIR}/../meta-openembedded/meta-networking \
  ${TOPDIR}/../meta-openembedded/meta-python \
  ${TOPDIR}/../meta-openembedded/meta-multimedia \ 
  ${TOPDIR}/../meta-openembedded/meta-gnome \
  ${TOPDIR}/../meta-openembedded/meta-xfce \
  ${TOPDIR}/../meta-radxa \
  ${TOPDIR}/../meta-openembedded/meta-filesystems \
  ${TOPDIR}/../meta-virtualization \
  "
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $

And:

stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  cat build/conf/local.conf 
#
# This file is your local configuration file and is where all local user settings
..........
DISTRO_FEATURES_append = " pam systemd x11"
VIRTUAL-RUNTIME_init_manager = "systemd"
PACKAGECONFIG_append_pn-systemd = " resolved networkd"
DISTRO_FEATURES_remove = "wayland"
IMAGE_INSTALL_append = "docker-ce"
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $

Unfortunately, here is the status:

stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  vi build/conf/bblayers.conf 
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  vi build/conf/local.conf 
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  source oe-init-build-env

### Shell environment set up for builds. ###

You can now run 'bitbake <target>'

Common targets are:
    core-image-minimal
    core-image-sato
    meta-toolchain
    meta-ide-support

You can also run generated qemu images with a command like 'runqemu qemux86'

Other commonly useful commands are:
 - 'devtool' and 'recipetool' handle common recipe tasks
 - 'bitbake-layers' handles common layer tasks
 - 'oe-pkgdata-util' handles common target package tasks
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $  bitbake -k radxa-console-image
WARNING: You have included the meta-virtualization layer, but 'virtualization' has not been enabled in your DISTRO_FEATURES. Some bbappend files may not take effect. See the meta-virtualization README for details on enabling virtualization support.
WARNING: Host distribution "linuxmint-19.3" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
WARNING: /F250/ROCKPI-S/yocto/poky/build/../meta-radxa/recipes-connectivity/bluetooth/bluetooth-rockpi-e.bb: Unable to get checksum for bluetooth-rockpi-e SRC_URI entry realtek-btfw-load.service: file could not be found                                      | ETA:  0:0WARNING: /F250/ROCKPI-S/yocto/poky/build/../meta-radxa/recipes-connectivity/bluetooth/bluetooth-rockpi-e.bb: Unable to get checksum for bluetooth-rockpi-e SRC_URI entry realtek-btfw-load.sh: file could not be found
Parsing recipes: 100% |################################################################################################| Time: 0:01:15
Parsing of 2594 .bb files complete (0 cached, 2594 parsed). 3735 targets, 128 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "rockpi-s-rk3308"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0.4"
TUNE_FEATURES        = "aarch64 cortexa35 crc crypto"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "zeus:d88d62c20d7d8da85f02edb170dae0280624ad7e"
meta-oe              
meta-networking      
meta-python          
meta-multimedia      
meta-gnome           
meta-xfce            = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-radxa           = "zeus:d489b2909b45112434d843fba1aa18f8229949f2"
meta-filesystems     = "zeus:2b5dd1eb81cd08bc065bc76125f2856e9383e98b"
meta-virtualization  = "zeus:48ec1b49871fef7d331ac1d0680a48d362c7482b"

Initialising tasks: 100% |#############################################################################################| Time: 0:00:03
Sstate summary: Wanted 163 Found 0 Missed 163 Current 1369 (0% match, 89% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
WARNING: radxa-console-image-1.0-r0 do_rootfs: Postinstall for package docker-ce failed with 1:
update-rc.d: /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/rootfs/etc/init.d/docker.init: file does not exist

ERROR: radxa-console-image-1.0-r0 do_rootfs: Postinstall scriptlets of ['docker-ce'] have failed. If the intention is to defer them to first boot,
then please place them into pkg_postinst_ontarget_${PN} ().
Deferring to first boot via 'exit 1' is no longer supported.
Details of the failure are in /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/temp/log.do_rootfs.
ERROR: Logfile of failure stored in: /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/temp/log.do_rootfs.12748
ERROR: Task (/F250/ROCKPI-S/yocto/poky/build/../meta-radxa/recipes-core/images/radxa-console-image.bb:do_rootfs) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4258 tasks of which 3869 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /F250/ROCKPI-S/yocto/poky/build/../meta-radxa/recipes-core/images/radxa-console-image.bb:do_rootfs
Summary: There were 5 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $

Here is the content of the error log file:

https://pastebin.pl/view/0d2cf8e8

See the end of it:

NOTE: Executing preinstall for package: docker-ce ...
NOTE: Running groupadd commands...
NOTE: docker-ce: Performing groupadd with [--root /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/rootfs -r docker]
NOTE: docker-ce: group docker already exists, not re-creating it

NOTE: Executing postinstall for package: docker-ce ...
WARNING: Postinstall for package docker-ce failed with 1:
update-rc.d: /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/rootfs/etc/init.d/docker.init: file does not exist

ERROR: Postinstall scriptlets of ['docker-ce'] have failed. If the intention is to defer them to first boot,
then please place them into pkg_postinst_ontarget_${PN} ().
Deferring to first boot via 'exit 1' is no longer supported.
Details of the failure are in /F250/ROCKPI-S/yocto/poky/build/tmp/work/rockpi_s_rk3308-poky-linux/radxa-console-image/1.0-r0/temp/log.do_rootfs.
DEBUG: Python function do_rootfs finished
stulluk /F250/ROCKPI-S/yocto/poky/build (zeus) $

Is it a bug, or did I miss any step ?

stulluk · September 20, 2020, 8:06pm

@igorp I made a small investigation on Armbian minimal Ubuntu image for RockPi-S.

Below log is just after burning this image to SDnand via rkdeveloptool, without installing anything:

stulluk /F250/ROCKPI-S/IMAGES $  ssh rockpi 
Warning: Permanently added '192.168.1.111' (ECDSA) to the list of known hosts.
root@192.168.1.111's password: 
 ____            _          _   ____  
|  _ \ ___   ___| | ___ __ (_) / ___| 
| |_) / _ \ / __| |/ / '_ \| | \___ \ 
|  _ < (_) | (__|   <| |_) | |  ___) |
|_| \_\___/ \___|_|\_\ .__/|_| |____/ 
                     |_|              
Welcome to Armbian 20.08.1 Focal with Linux 4.4.228-rockpis

System load:   0.65 0.28 0.10  	Up time:       1 min		
Memory usage:  11 % of 487MB  	IP:            192.168.1.111
CPU temp:      43°C           	
Usage of /:    68% of 870M   	

Last login: Sun Sep 20 22:58:56 2020
root@rockpi-s:~# systemctl | grep -i network
  networking.service                                                                         loaded active exited    Raise network interfaces                                                            
  NetworkManager.service                                                                     loaded active running   Network Manager                                                                     
  systemd-resolved.service                                                                   loaded active running   Network Name Resolution                                                             
  network-online.target                                                                      loaded active active    Network is Online                                                                   
  network-pre.target                                                                         loaded active active    Network (Pre)                                                                       
  network.target                                                                             loaded active active    Network                                                                             
  nss-lookup.target                                                                          loaded active active    Host and Network Name Lookups                                                       
root@rockpi-s:~# systemctl | grep -i ssh
  ssh.service                                                                                loaded active running   OpenBSD Secure Shell server                                                         
root@rockpi-s:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            240M     0  240M   0% /dev
tmpfs            49M  1.9M   47M   4% /run
/dev/mmcblk1p1  870M  553M  265M  68% /
tmpfs           244M     0  244M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           244M     0  244M   0% /sys/fs/cgroup
tmpfs           244M  4.0K  244M   1% /tmp
/dev/zram0       49M  748K   45M   2% /var/log
tmpfs            49M     0   49M   0% /run/user/0
root@rockpi-s:~# apt update
Hit:1 http://ports.ubuntu.com focal InRelease
Get:2 http://ports.ubuntu.com focal-security InRelease [107 kB]
Get:4 http://ports.ubuntu.com focal-updates InRelease [111 kB]
Get:5 http://ports.ubuntu.com focal-backports InRelease [98.3 kB]
Get:6 http://ports.ubuntu.com focal-security/main armhf Packages [172 kB]
Get:7 http://ports.ubuntu.com focal-security/universe arm64 Packages [67.0 kB]       
Get:8 http://ports.ubuntu.com focal-security/main arm64 Packages [181 kB]
Get:9 http://ports.ubuntu.com focal-security/universe armhf Packages [66.4 kB]
Get:10 http://ports.ubuntu.com focal-updates/main arm64 Packages [407 kB]       
Get:11 http://ports.ubuntu.com focal-updates/universe arm64 Packages [204 kB]
Get:12 http://ports.ubuntu.com focal-updates/main armhf Packages [397 kB]      
Get:13 http://ports.ubuntu.com focal-updates/universe armhf Packages [201 kB]
Get:3 https://us.mirrors.fossho.st/armbian/apt focal InRelease [18.2 kB]
Get:14 http://ports.ubuntu.com focal-backports/universe armhf Packages [3205 B]    
Get:15 http://ports.ubuntu.com focal-backports/universe arm64 Packages [3196 B]
Get:16 https://us.mirrors.fossho.st/armbian/apt focal/focal-utils arm64 Packages [3994 B]
Get:17 https://us.mirrors.fossho.st/armbian/apt focal/main armhf Packages [161 kB]
Get:18 https://us.mirrors.fossho.st/armbian/apt focal/main arm64 Packages [179 kB]
Get:19 https://us.mirrors.fossho.st/armbian/apt focal/focal-utils armhf Packages [3998 B]                                                                                                                                                                                      
Fetched 2385 kB in 8s (306 kB/s)                                                                                                                                                                                                                                               
Reading package lists... Done
Building dependency tree       
Reading state information... Done
25 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@rockpi-s:~# apt install docker.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  containerd iptables libip6tc2 libnetfilter-conntrack3 libnfnetlink0 libnftnl11 runc
Suggested packages:
  aufs-tools btrfs-progs debootstrap docker-doc rinse zfs-fuse | zfsutils firewalld nftables
Recommended packages:
  cgroupfs-mount | cgroup-lite git pigz ubuntu-fan apparmor
The following NEW packages will be installed:
  containerd docker.io iptables libip6tc2 libnetfilter-conntrack3 libnfnetlink0 libnftnl11 runc
0 upgraded, 8 newly installed, 0 to remove and 25 not upgraded.
Need to get 48.4 MB of archives.
After this operation, 272 MB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
root@rockpi-s:~#

So it is not possible to install docker, (Available 265Mbyte, only docker engine needs 272Mbyte) am I wrong?

If I am not mistaken, even if I build armbian for Rockpi-S by myself, it will not be possible to install docker, because it will not fit into total 870Mbyte of space.

Am I wrong ? Is there a way to reduce armbian rootfs size even more by customising build options? How much space could be gained ?

Many thanks for your answers.

igorp · September 20, 2020, 8:34pm

You need to install Docker, find a way.

Correct. Perhaps remove things to get there?

nish · September 20, 2020, 9:25pm

There seems to a be a conflict in systemd and sysvinit. Could you try this out.

In the local.conf could you add sysvinit to DISTRO_FEATURES_remove. So your variable should look like:

DISTRO_FEATURES_remove = “wayland sysvinit”

Do this. Source and fire the build again.

If this doesn’t work, revert back to original and try this out: https://github.com/Xilinx/meta-virtualization/issues/4

You can create a file named docker-ce_%.bbappend in meta-radxa/recipes-connectivity with the following content:

do_install_append() {
	install -d ${D}${sysconfdir}/init.d
	install -m 0755 ${WORKDIR}/docker.init ${D}${sysconfdir}/init.d/docker.init
}

Ideally this should be addressed by the maintainer of the meta-virtualization layer

stulluk · September 21, 2020, 8:56pm

Confirmed that this has worked, and I have docker in my image. Thank you up to this point.

Unfortunately, I still couldn’t be able to :
docker pull pihole/pihole:latest

Original yocto image was setting up boot partition size around 128MByte, and remaining as rootfs. See file meta-radxa/classes/rockchip-radxa-gpt-img.bbclass :

# More info at http://rockchip.wikidot.com/partitions
LOADER1_SIZE = "8000"
RESERVED1_SIZE = "128"
RESERVED2_SIZE = "8192"
LOADER2_SIZE = "8192"
ATF_SIZE = "8192"
#BOOT_SIZE = "229376"
BOOT_SIZE = "32768"

As seen above, I reduced boot partition size to 16Mbyte, and it didn’t affect bootup, and I got following new size:

root@rockpi-s-rk3308:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       863M  402M  412M  50% /
devtmpfs        212M     0  212M   0% /dev
tmpfs           213M     0  213M   0% /dev/shm
tmpfs           213M  8.7M  204M   5% /run
tmpfs           213M     0  213M   0% /sys/fs/cgroup
tmpfs           213M     0  213M   0% /tmp
tmpfs           213M   24K  213M   1% /var/volatile
tmpfs            43M     0   43M   0% /run/user/0
root@rockpi-s-rk3308:/#

At this point, I am a little confused (see http://rockchip.wikidot.com/partitions) :

16MByte reserved
16Mbyte boot partition as I set above
864Mbyte in Rootfs

Then where is the remaining: 960MByte total ( see above message of my dd speed test) - 864 -16 -16 = 60Mbyte is missing somewhere.

I checked reserved block percentage :

stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  tune2fs -l /F250/ROCKPI-S/yocto/poky/build/tmp/deploy/images/rockpi-s-rk3308/radxa-console-image-rockpi-s-rk3308.ext4 | grep "Reserved block count"
Reserved block count:     6836
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $

So, 6836*4096/1024/1024 = 26Mbytes ( am I calculating correctly ? )…

Original pihole docker image on my host seems less than 300Mbyte:

stulluk /F250/ROCKPI-S/yocto/poky (zeus) $  docker images
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
stulluk/rockpis-builder   latest              3467f6915865        2 days ago          502MB
stulluk/rockpis-builder   <none>              5cf255b1a0f2        2 days ago          484MB
pihole/pihole             latest              4642d275ab73        6 weeks ago         296MB
stulluk /F250/ROCKPI-S/yocto/poky (zeus) $

But unfortunately, again, docker-pihole doesn’t fit to remaining space on rockpi-S:

 root@rockpi-s-rk3308:~# docker pull pihole/pihole:latest
latest: Pulling from pihole/pihole
c408adf5aab3: Pull complete 
e1760851e187: Pull complete 
0f00de0e80bf: Pull complete 
a42b41aa464a: Extracting [==================================================>]  115.3MB/115.3MB
018c74957032: Download complete 
0e261c69a668: Download complete 
fdc2ebbe8553: Download complete 
11921923a10e: Download complete 
failed to register layer: Error processing tar file(exit status 1): write /var/www/html/admin/.git/objects/pack/pack-69011510806132c1126562871583280a974cf494.pack: no space left on device
root@rockpi-s-rk3308:~#

Here is latest directory sizes on yocto image on rockpi-S:

root@rockpi-s-rk3308:/# du -sh * | sort -nr 2>/dev/null
du: cannot access 'proc/761/task/761/fd/4': No such file or directory
du: cannot access 'proc/761/task/761/fdinfo/4': No such file or directory
du: cannot access 'proc/761/fd/4': No such file or directory
du: cannot access 'proc/761/fdinfo/4': No such file or directory
291M	usr
62M	lib
28K	home
20M	var
16K	opt
12M	boot
11M	etc
8.7M	run
5.7M	bin
4.0K	mnt
4.0K	media
2.3M	sbin
0	tmp
0	sys
0	proc
0	dev
root@rockpi-s-rk3308:/#

Most of the space is consumed by docker & kernel modules…etc

I also tried to remove some apps:

# MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "\
#    kernel-modules \
#    rtl-tools \
#    rkwifibt-firmware-rtl8723ds-bt \
#    bluetooth-rockpi-s \
#    resize-helper \
# "

MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS += "\
   kernel-modules \
   resize-helper \
"

It seems I have to give-up my dream 1Gbyte SDnand will not be enough for my use-case.

Actually, I like the general idea of SDnand, since lifetime of such storage device could be much more durable than commodity eMMC’s, and it could be easier to replace.

May be in future, there would be 2Gbyte or 4Gbyte versions of RockPi-S, and they would definitely be much more useful for my use-cases.

Again, many thanks to @nish and @igorp for your valuable recommendations.

I hope this investigation could be more useful for somebody else.

Still thinking: Could there be a way to strip down docker-pihole image?

igorp · September 21, 2020, 10:31pm

Stop looking for shortcuts and you will perhaps get somewhere.

We wasted weeks to come out with Armbian minimal - similar goal, but not that extreme since we still need to have regular functionality, certain tools, etc., Docker must be supported kernel wise but engine not present. You want to get more and better in much less time … also with less experiences.

getting minimal possible kernel config that run docker (few days, perhaps shortcut exits but I don’t know it)
optimising boot process to get some space - removing ramdisk (1h - few days)
remove python and other non-essential packages (few days)

Then you will perhaps have the needed space.