Options for running CM3 on current upstream kernel?

#1

Hi,

I am considering to base an audio-oriented project on CM3, due to its support for multiple I2S interfaces, compared to RPi CM4.

But the product requires support for USB gadgets in the recent kernels (5.19+). That code has the general part (all the USB gadget functions) and the HW-specific part - DWC3 in this case. I assume the DWC3 code in radxa 5.10 kernel (heavily patched 4.4 android kernel, IIUC) is OK to run on CM3. But there is no code for the general part as that has not been merged to the radxa kernel yet.

Also, when hitting problems (unavoidably, there will always be some), I discuss details with the gadget and DWC3 specialists on usb-linux mailing list, perform tests and submit the resultant patches to the upstream kernel, logically.

Radxa CM3 is not supported by armbian which seems to keep reasonably close the upstream, with a limited number of patches only (their patches on top of reasonable amount of patches in https://github.com/ayufan-rock64/linux-mainline-kernel/ , IIUC) .

Radxa Rock3A is supported in Armbian, with kernel 6.1. IMO the patch difference between Rock3A patches in https://github.com/armbian/build/tree/master/patch/kernel/archive/rockchip64-6.1 and patches for CM3 should be minimal. IMO mostly “just” DTS changes. That makes a very reasonable number of patches against the upstream. Unlike the radxa kernel which does not derive from the upstream kernel at all (“Branch linux-5.10-gen-rkr3.4 is 457612 commits ahead, 6732 commits behind rockchip-linux:develop-4.4.”) - impossible to discuss anything with the upstream developers.

Has anyone done such project, is this path recommendable? IMO keeping just patches for CM3 against the latest armbian build tree should be viable.

Thanks a lot for any suggestions.

Best regards,

Pavel.

#2

ROCK 3A is supported in mainline linux kernel since 5.19. CM3I (Radxa E25) will be supported in 6.3. CM3 and CM3S will be supported in future release.

our new BSP supports mainline linux kernel (currently v6.1.x) for Radxa CM3S and CM3I. we will add support for CM3.

#3

Naoki, thanks a lot for the great news. HW-specific patches over the latest upstream are just what I was hoping for :slight_smile: May I ask about your time estimate of adding CM3 support to your BSP? The CM3 + IO board have already arrived and I would like to start testing with the latest kernel.

IIUC the support should be very similar to the patches for CM3S. I could do it myself but I do not know the exact differences between the SODIMM and CM boards. The wiki says there are very few…

When the support is added to BSP, can I expect the board to be accessible upon booting (ethernet ssh link-local or display + USB keyboard)?

Thanks a lot for the support.

With regards,

Pavel.

#4

If device tree already exists, following this should not be hard. It just has to be done by someone holding hardware in their hands, to do at least basic tests. Minimal requirements for official Armbian support is if at least one person agrees to stick around and look after the device - major kernel upgrades almost always kills some functions and knowing which is already a valuable information.

#5

Igor, thanks a lot for your pointer. You are doing a great job with armbian, hats off (yes I know about the link to subscriptions :slight_smile: )

I do not think there is a device tree applicable to newer kernels for this board published yet, IIUC. After the radxa BSP gets the update (i.e. mostly the device tree files), I could add it to Armbian. But to be honest - if the official BSP (also debian) turns out OK, I will most likely stick with it and do my work first. I will/would try to push the patches (should any result from my work) to upstream kernel if possible, after Radxa/Naoki submit their support upstream.

#6

our new BSP supports mainline linux kernel (currently v6.1.x) for Radxa CM3S and CM3I . we will add support for CM3.

Hi Naoki, does commit https://github.com/radxa-repo/bsp/commit/7391fbbfb21991c194ffb834cb2f9a73d2a21b0e establish full support for CM3 + Radxa IO board in the latest branch? Can I start using it, without having to spend time on some hacking to make it work on this combo?

Also, is the OTG mode (USB gadgets) already supported?

Thanks a lot for the info,

Pavel.

#7

Hey Pavel,
Looks like CM3 DTS files have just been added to what will become mainline v6.3 ( https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/arch/arm64/boot/dts/rockchip/rk3566-radxa-cm3-io.dts) ! I can certainly build armbian against 6.1 with a patch to add the DTS+DTSI from there though have some issues with it (can’t see SD card / USB not working / had to edit to enable PCIe).
HTH
h-e-w

#8

Hey h-e-w, thanks for your important info. I am just a bit disappointed there has been no official radxa response for weeks as the vendor has the ultimate information and knows the answers.

Best regards,

Pavel.

#9

can you try our bsp?

./bsp linux latest
#10

Thanks for your suggestion.

This is my first experience with your bsp. The CM3 patch does not seem to fit. Maybe I am doing something wrong. I just started “./bsp linux latest” in the cloned bsp repo:

pavel@precision:~/personal-work/bsp$ ./bsp linux latest
main: Pulling from radxa-repo/bsp
a8ca11554fce: Pull complete 
a820af0dd4bc: Pull complete 
Digest: sha256:c3b164ecb70ca44514cdb7d6c3ef30f0dd0c194c436129e570d60cf0fccbf863
Status: Downloaded newer image for ghcr.io/radxa-repo/bsp:main
ghcr.io/radxa-repo/bsp:main
/home/pavel/personal-work/bsp/.src/linux /home/pavel/personal-work/bsp
Initialized empty Git repository in /home/pavel/personal-work/bsp/.src/linux/.git/
remote: Finding sources: 100% (83380/83380)
remote: Total 83380 (delta 3625), reused 16526 (delta 3625)
Receiving objects: 100% (83380/83380), 237.22 MiB | 4.53 MiB/s, done.
Resolving deltas: 100% (3625/3625), done.
From https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux
 * [new tag]             v6.1.12    -> v6.1.12
Updating files: 100% (78711/78711), done.
HEAD is now at 129c15b60 Linux 6.1.12
HEAD is now at 129c15b60 Linux 6.1.12
Running custom_source_action from /home/pavel/personal-work/bsp/linux/latest/0002-common/overlays.sh
/home/pavel/personal-work/bsp/linux/latest/0002-common /home/pavel/personal-work/bsp/.src/linux /home/pavel/personal-work/bsp
/home/pavel/personal-work/bsp/.src/overlays /home/pavel/personal-work/bsp/linux/latest/0002-common /home/pavel/personal-work/bsp/.src/linux /home/pavel/personal-work/bsp
Initialized empty Git repository in /home/pavel/personal-work/bsp/.src/overlays/.git/
fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
remote: Enumerating objects: 216, done.
remote: Counting objects: 100% (216/216), done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 216 (delta 162), reused 83 (delta 64), pack-reused 0
Receiving objects: 100% (216/216), 46.69 KiB | 1.61 MiB/s, done.
Resolving deltas: 100% (162/162), done.
From https://github.com/radxa/overlays
 * branch            23d0c4ae350f1ab94d72e7fe486f120826504401 -> FETCH_HEAD
HEAD is now at 23d0c4a Set overlays to be explicitly compatible with ROCK 3C
HEAD is now at 23d0c4a Set overlays to be explicitly compatible with ROCK 3C
/home/pavel/personal-work/bsp/linux/latest/0002-common /home/pavel/personal-work/bsp/.src/linux /home/pavel/personal-work/bsp
/home/pavel/personal-work/bsp/.src/linux /home/pavel/personal-work/bsp
Applying: arm64: dts: rockchip: enable tsadc for ROCK 4C+
Checking patch arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts...
Applied patch arch/arm64/boot/dts/rockchip/rk3399-rock-4c-plus.dts cleanly.
Applying: arm64: dts: rockchip: Add rk3566 based Radxa Compute Module 3
Checking patch arch/arm64/boot/dts/rockchip/rk3566-radxa-cm3.dtsi...
Applied patch arch/arm64/boot/dts/rockchip/rk3566-radxa-cm3.dtsi cleanly.
Applying: arm64: dts: rockchip: Add Radxa Compute Module 3 IO board
Checking patch arch/arm64/boot/dts/rockchip/Makefile...
error: while searching for:
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-pinenote-v1.2.dtb
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-quartz64-a.dtb
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-quartz64-b.dtb
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-roc-pc.dtb
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-soquartz-blade.dtb
dtb-$(CONFIG_ARCH_ROCKCHIP) += rk3566-soquartz-cm4.dtb

error: patch failed: arch/arm64/boot/dts/rockchip/Makefile:74
Checking patch arch/arm64/boot/dts/rockchip/rk3566-radxa-cm3-io.dts...
Applying patch arch/arm64/boot/dts/rockchip/Makefile with 1 reject...
Rejected hunk #1.
Applied patch arch/arm64/boot/dts/rockchip/rk3566-radxa-cm3-io.dts cleanly.
Patch failed at 0003 arm64: dts: rockchip: Add Radxa Compute Module 3 IO board
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
bsp
#11

sorry, please run

$ cd bsp
$ git submodule init
$ git submodule update
$ ./bsp linux latest
#12

Naoki, thanks a lot.

I moved past the patching OK, but got stuck at the first make. This is log with bash -x in the bsp script, and “make V=1” in the bsp_make function. The fixdep binary gets compiled to arm64 OK, but the subsequent /bin/sh complains about exec format - is the fixdep binary supposed to be for arm64?

...
Patchset 0200-upstream has been applied.
+ false
+ popd
/home/pavel/personal-work/bsp
+ bsp_prepare
+ return
+ [[ -n '' ]]
+ false
+ bsp_make defconfig
+ tee -a /home/pavel/personal-work/bsp/.src/build.log
++ bsp_version
++ make -C /home/pavel/personal-work/bsp/.src/linux -s kernelversion
+ local kernelversion=6.1.12
++ nproc
+ make V=1 -C /home/pavel/personal-work/bsp/.src/linux -j12 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- HOSTCC=aarch64-linux-gnu-gcc KDEB_COMPRESS=xz KDEB_CHANGELOG_DIST=unstable DPKG_FLAGS=-d LOCALVERSION=-latest KERNELRELEASE=6.1.12-1-latest KDEB_PKGVERSION=6.1.12-1 defconfig
make: Entering directory '/home/pavel/personal-work/bsp/.src/linux'
make -f ./scripts/Makefile.build obj=scripts/basic
  aarch64-linux-gnu-gcc -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu11 -Wdeclaration-after-statement         -o scripts/basic/fixdep scripts/basic/fixdep.c   
/bin/sh: 1: scripts/basic/fixdep: Exec format error
make[1]: *** [scripts/Makefile.host:111: scripts/basic/fixdep] Error 126
make[1]: *** Deleting file 'scripts/basic/fixdep'
make: *** [Makefile:633: scripts_basic] Error 2
make: Leaving directory '/home/pavel/personal-work/bsp/.src/linux'
++ docker inspect bsp '--format={{.State.ExitCode}}'
+ DOCKER_EXIT_CODE=2
+ docker container rm bsp
bsp
+ return 2

Thanks a lot,

Pavel.

#13

fixdep for arm64 is required to make working linux-headers package.

please run

$ sudo apt install qemu-user-static
#14

Naoki, thanks a lot, that seems to help, debs built, image with the 6.1.12 kernel rbuilt and mounted on loopback OK, will try to flash the CM3 now.

Please would you consider adding the package to the apt install list in https://radxa-doc.github.io/software/dev/index.html ?

Also the submodule update to https://github.com/radxa-repo/bsp/blob/main/README.md

Thanks a lot,

Pavel.

#15

Hi,

Can you share the CM3 image with kernel 6.1? (Trying to replace RPi CM4)

My camera drivers only support kernel 5.4 and later so current kernel 4.x image doesn’t work.

Thanks

#16

Hello KenSense, have you got some luck with CM3 image with kernel 6.1?

thanks

#17

I finally tested the “bsp latest + rbuild -k latest-image.deb radxa-cm3-io bullseye cli” image built in March, with kernel 6.1. CM3+ Radxa IO board boots OK, but there is no ethernet device present in /sys/class/net (the official radxa debian image with the old kernel 4.13 runs network OK).

Before trying to resolve this issue with Radxa I decided to use the currently latest versions. BSP compiled 6.4.11 OK.

I did ‘git pull’ in my rbuild directory with no problem, updated to

commit 37b820b75460a6e385a5cd753ca506d52ece5032 (HEAD -> main, origin/main, origin/HEAD)
Author: ZHANG Yuntian <95260730+RadxaYuntian@users.noreply.github.com>
Date:   Fri Aug 18 20:01:04 2023 +0800

    Add ca-certificates

Command ‘./rbuild -k linux-image-6.4.11-1-latest_6.4.11-1_arm64.deb radxa-cm3-io bullseye cli’ failed with swapon issues in docker:

Using debian_bullseye_cli.tar rootfs.
Running /debos --artifactdir /home/pavel/personal-work/radxa/rbuild --template-var soc:"rk3566" --template-var kernel_dbg:"" --template-var install_vendor_package:"true" --template-var dkms:"r8125-dkms" --template-var kernel_profile:"" --template-var suite:"bullseye" --template-var flavor:"cli" --template-var partition_type:"gpt" --template-var firmware:"" --template-var overlay:"" --template-var radxa_mirror:"" --template-var rbuild_cmd:"./rbuild -k linux-image-6.4.11-1-latest_6.4.11-1_arm64.deb radxa-cm3-io bullseye cli" --template-var soc_family:"rockchip" --template-var repo_prefix:"" --template-var efi_end:"332MiB" --template-var kernel:"linux-image-6.4.11-1-latest_6.4.11-1_arm64.deb" --template-var header:"linux-headers-6.4.11-1-latest_6.4.11-1_arm64.deb" --template-var distro_mirror:"" --template-var architecture:"arm64" --template-var board:"radxa-cm3-io" --template-var distro:"debian" --template-var image:"radxa-cm3-io_debian_bullseye_cli.img" --template-var rbuild_rev:"37b820b75460a6e385a5cd753ca506d52ece5032" /home/pavel/personal-work/radxa/rbuild/common/image.yaml --internal-image /dev/disk/by-fakemachine-label/fakedisk-0 --internal-image /dev/disk/by-fakemachine-label/fakedisk-1 using uml backend
2023/08/22 11:02:50 ==== Set up swap ====
2023/08/22 11:02:51 ==== Enable swap device ====
4153409536 bytes (4.2 GB, 3.9 GiB) copied, 2 s, 2.1 GB/s
2023/08/22 11:02:53 set -e... | 4096+0 records in
2023/08/22 11:02:53 set -e... | 4096+0 records out
2023/08/22 11:02:53 set -e... | 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.06615 s, 2.1 GB/s
2023/08/22 11:02:53 set -e... | Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
2023/08/22 11:02:53 set -e... | no label, UUID=4300a314-1987-4883-b949-86283aaaf50c
2023/08/22 11:02:53 set -e... | swapon: /swapfile_rbuild: swapon failed: Invalid argument
2023/08/22 11:02:53 Action `Enable swap device` failed at stage Run, error: exit status 255

That’s weird, because swapon should work like that. Nevertheless I found the June commit https://github.com/radxa-repo/rbuild/commit/63c6d96a149d3c893f6bd808eb636eb296228e3d + subsequent changes and removed the swap sections from image|intermediate|rootfs.yaml files - no swap would be provided during the build.

After this change the build continued, until it hit missing radxa package radxa-system-config-rockchip:

2023/08/22 11:06:27 ==== Install radxa-system-config-rockchip ====
Get:1 https://deb.debian.org/debian bullseye InRelease [116 kB]                
Get:2 https://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]     
Get:3 https://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:4 https://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:5 https://radxa-repo.github.io/apt bullseye InRelease [3569 B]
Get:6 https://deb.debian.org/debian bullseye/main arm64 Packages [8071 kB]     
Get:7 https://deb.debian.org/debian bullseye/main Translation-en [6240 kB]     
Get:8 https://deb.debian.org/debian bullseye/contrib arm64 Packages [41.0 kB]  
Get:9 https://deb.debian.org/debian bullseye/contrib Translation-en [46.9 kB]  
Get:10 https://deb.debian.org/debian bullseye/non-free arm64 Packages [73.0 kB]
Get:11 https://deb.debian.org/debian bullseye/non-free Translation-en [92.2 kB]
Get:12 https://deb.debian.org/debian bullseye-backports/main arm64 Packages [419 kB]
Get:13 https://deb.debian.org/debian bullseye-backports/main Translation-en [353 kB]
Get:14 https://deb.debian.org/debian bullseye-backports/contrib arm64 Packages [5180 B]
Get:15 https://deb.debian.org/debian bullseye-backports/contrib Translation-en [5984 B]
Get:16 https://deb.debian.org/debian bullseye-backports/non-free arm64 Packages [9876 B]
Get:17 https://deb.debian.org/debian bullseye-backports/non-free Translation-en [27.7 kB]
Get:18 https://deb.debian.org/debian-security bullseye-security/main arm64 Packages [247 kB]
Get:19 https://deb.debian.org/debian-security bullseye-security/main Translation-en [164 kB]
Get:20 https://deb.debian.org/debian-security bullseye-security/non-free Translation-en [464 B]
Get:21 https://deb.debian.org/debian bullseye-updates/main arm64 Packages [14.7 kB]
Get:22 https://deb.debian.org/debian bullseye-updates/main Translation-en [9964 B]
Get:23 https://radxa-repo.github.io/apt bullseye/main all Packages [38.3 kB]   
Get:24 https://radxa-repo.github.io/apt bullseye/main arm64 Packages [48.2 kB] 
Fetched 16.2 MB in 32s (509 kB/s)                                              
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
2023/08/22 11:07:05 apt | E: Unable to locate package radxa-system-config-rockchip
2023/08/22 11:07:05 Action `Install radxa-system-config-rockchip` failed at stage Run, error: exit status 100

I could not google out this package anywhere.

@RadxaYuntian & @RadxaNaoki : please can you help with these issues?

Thanks a lot,

Pavel.

#18

Also, since my 6.1 BSP kernel did not run properly on CM3 + IO, I wonder which linux profile should be chosen for BSP https://github.com/radxa-repo/bsp/blob/main/docs/getting_started.md#build-your-first-package . I do not understand the specific profiles (rk356x, rk3328) vs. generic (amlogic, rockchip) vs. version (stable, latest). I would like rk356x latest :slight_smile:
Thanks,

Pavel.

#19

As of the swapon issue - my ZFS does not support raw swapfiles due to the “file with holes” feature. I added a github issue for disabling swap creation with some rbuild parameter https://github.com/radxa-repo/rbuild/issues/26 .

#20

Clean rbuild cloned from github (not just git pull from the previous try in March) did not hit the missing package radxa-system-config-rockchip error, and (after removing the swapon code) built the image based on latest 6.4.11 OK.

This image recognized the network adapters OK, even configured dwc3 composite gadget for udc-ethernet (with issues but that will be sorted out later). For now everything is OK, I can share the build should anyone want to use it. But IMO a list of steps I did would be more useful than just a static image…