ROCK Pi X - Custom/tuned Linux kernel

I think udev might be a better fit than systemd for this based on your description - thoughts?

I am also curious if we can somehow avoid flapping of the WLAN interface, or how we express to systemd or some other system that it needs to wait until this check/unload/reload process is complete.

Do you have a specific error message about it? It should workā€¦ I can look into why it does notā€¦ maybe I will install Lubuntu to test.

I am using Debian GNU/Linux. It should work fine on that.

Yes, the firmware package will conflict with firmware-realtek, firmware-brcm80211, and another firmware package that is not coming to mind. I was thinking that all firmware packages should be removed if this one is in use. Let me see if I can formally declare a Conflicts: in the package to hopefully make this clear.

If you know how, add the following to your kernel command line:

efi=no_disable_early_pci_dma

Also, @tjcs90, perhaps you could remove early_pci_dma, as well as add support for EFI stub booting?

Depends, does udev have a way of only running it once? To my understanding, such a script would only need to be run once per device, so perhaps itā€™s best to just mention this step in the setup guide.

I moved to Debian, your kernel seems to be working fine.

I had to manually remove several firmware-* packages to avoid conflicts with your deb, but the process was smooth

I can see HDMI and headphone codec are up and running, but Iā€™m expecting I2S pins on 40-pin header to not be available. Is it possible to have them enabled somehow?

aha, so I looked into the I2S issue. I have a bad feeling that it is Intel vaporware.

See these threads:


From those threads, there are two pieces of information:

  • Some of this appears to require firmware (BIOS) support. Unknown if Rock Pi X has it.
  • Linux support was somewhat recently missing, unknown if it has been upstreamed.

I looked at the Linux git tree linked. I donā€™t see anything really pertaining to I2S except maybe the new SOF backend. I swapped out the HiFi ALSA backend for SOF and built a new kernel. You will need the firmware package Iā€™ve updated in the first post order to use it (firmware-rockpix-20200907):
https://www.cen64.com/uploads/rockpix/sof/linux-image-5.8.7_5.8.7-1_amd64.deb

However, when I boot the kernel, while the HDMI audio codecs continue to workā€¦

[Mon Sep  7 09:57:19 2020] sof-audio-acpi 808622A8:00: Firmware info: version 1:5:1-88707
[Mon Sep  7 09:57:19 2020] sof-audio-acpi 808622A8:00: Firmware: ABI 3:16:0 Kernel ABI 3:16:0
[Mon Sep  7 09:57:19 2020] sof-audio-acpi 808622A8:00: warning: unknown ext header type 3 size 0x1c
[Mon Sep  7 09:57:19 2020] sof-audio-acpi 808622A8:00: warning: unknown ext header type 4 size 0x10

Likely have to figure this out before getting much further here.

1 Like

Thanks for the update.

I think I2S should be supported from driver point-of-view, because the codec NAU88L24 works fine, and this codec gets the audio stream via I2S interface (I2S bus #0)

Unfortunately the I2S bus on 40-pin header is a different one (I2S bus #1)

Iā€™m not very familiar with X86, so I donā€™t know if I2S Bus#1 needs to be enabled somehow in the UEFI, or if only the driver part is missing.

Exactly - I think the ACPI tables have to ā€œdescribeā€ the SOF interface though, much like a device tree would. I donā€™t think that is being done, but my knowledge here is also more limited so I could be in the wrong.

Found an issue today with WiFi firmware patch - all users should redownload/upgrade to the 5.8.7 kernel package released just now.

Whatā€™s the new patch?

Same location as before - I update it as I go:
https://www.cen64.com/uploads/rockpix/brcmfmac-5.8.patch

The issue: In 5.8.6, one of the patches I had pulled from upstream got backported to the 5.8 series (I had originally rebased all the patches against 5.8.5 IIRC). Found an issue in the build process that was skipping over some unclean patchesā€¦ I need to be more careful when bumping minor series with how big this brcmfmac patch is. :slight_smile:

1 Like

One thing I have noticed is the power_save feature of the WLAN results in quite a large wake up time for the radio. You can actually see this if you have an ssh console open and strike a key after leaving it alone for 2-3 seconds. The first keypress will take a GOOD deal of time to register.

I added a udev rule to the latest firmware package to disable WLAN power_save for this reason.

In the current 5.8 series, if WiFi firmware crashes, itā€™s game over. You lose, reload the module (can be hard to do without WiFi)!

Broadcom engineers introduced a commit for 5.9 which will reset the firmware when it crashes automatically. However, I saw this commit caused an kernel BUG that made the WiFi just as, if not more, inaccessible:

[Mon Sep  7 13:20:57 2020] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
[Mon Sep  7 13:20:57 2020] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[Mon Sep  7 13:20:57 2020] ieee80211 phy0: brcmf_cfg80211_get_tx_power: error (-5)
[Mon Sep  7 13:20:58 2020] brcmfmac: brcmf_sdiod_probe: Failed to set F1 blocksize
[Mon Sep  7 13:20:58 2020] ------------[ cut here ]------------
[Mon Sep  7 13:20:58 2020] Kernel BUG at __slab_free.constprop.0+0x142/0x280 [verbose debug info unavailable]
[Mon Sep  7 13:20:58 2020] invalid opcode: 0000 [#1] SMP PTI
[Mon Sep  7 13:20:58 2020] CPU: 3 PID: 32 Comm: kworker/3:1 Tainted: G            E     5.8.7 #1
[Mon Sep  7 13:20:58 2020] Workqueue: events brcmf_core_bus_reset [brcmfmac]
[Mon Sep  7 13:20:58 2020] RIP: 0010:__slab_free.constprop.0+0x142/0x280
[Mon Sep  7 13:20:58 2020] Code: 18 e9 04 ff ff ff 9c 5f fa f0 49 0f ba 2c 24 00 72 71 4d 3b 6c 24 20 74 13 49 0f ba 34 24 00 57 9d eb aa 80 4c 24 4b 80 eb 8b <0f> 0b 49 3b 54 24 28 75 e6 49 89 5c 24 20 49 89 4c 24 28 49 0f ba
[Mon Sep  7 13:20:58 2020] RSP: 0000:ffffb92f8015bd80 EFLAGS: 00010246
[Mon Sep  7 13:20:58 2020] RAX: ffff8f55f7f5b930 RBX: ffff8f55f7f5b900 RCX: ffff8f55f7f5b900
[Mon Sep  7 13:20:58 2020] RDX: 00000000802a001e RSI: fffffcf781dfd6c0 RDI: ffff8f558008d500
[Mon Sep  7 13:20:58 2020] RBP: ffffb92f8015be10 R08: 0000000000000001 R09: ffff8f55f7f5b900
[Mon Sep  7 13:20:58 2020] R10: 0000000000005d34 R11: 0000000000000008 R12: fffffcf781dfd6c0
[Mon Sep  7 13:20:58 2020] R13: ffff8f55f7f5b900 R14: ffff8f558008d500 R15: 0000000000000000
[Mon Sep  7 13:20:58 2020] FS:  0000000000000000(0000) GS:ffff8f55fad80000(0000) knlGS:0000000000000000
[Mon Sep  7 13:20:58 2020] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Mon Sep  7 13:20:58 2020] CR2: 00007fd5680dec00 CR3: 0000000073e82000 CR4: 00000000001006e0
[Mon Sep  7 13:20:58 2020] Call Trace:
[Mon Sep  7 13:20:58 2020]  ? recalibrate_cpu_khz+0x10/0x10
[Mon Sep  7 13:20:58 2020]  ? ktime_get_mono_fast_ns+0x49/0x90
[Mon Sep  7 13:20:58 2020]  ? rpm_suspend+0x8e/0x540
[Mon Sep  7 13:20:58 2020]  ? __wake_up_common_lock+0x86/0xc0
[Mon Sep  7 13:20:58 2020]  kfree+0x1a8/0x1c0
[Mon Sep  7 13:20:58 2020]  brcmf_sdiod_remove+0x3b/0xa0 [brcmfmac]
[Mon Sep  7 13:20:58 2020]  brcmf_sdiod_probe+0x157/0x1e0 [brcmfmac]
[Mon Sep  7 13:20:58 2020]  brcmf_sdio_bus_reset+0x4e/0x90 [brcmfmac]
[Mon Sep  7 13:20:58 2020]  process_one_work+0x185/0x2e0
[Mon Sep  7 13:20:58 2020]  worker_thread+0x4c/0x3a0
[Mon Sep  7 13:20:58 2020]  ? rescuer_thread+0x370/0x370
[Mon Sep  7 13:20:58 2020]  kthread+0x113/0x130
[Mon Sep  7 13:20:58 2020]  ? __kthread_bind_mask+0x60/0x60
[Mon Sep  7 13:20:58 2020]  ret_from_fork+0x22/0x30

I narrowed down the kernel BUG during reset after firmware crash to a double free that can occur under certain conditions, and was able to fix it! Now when the WiFi firmware crashes, it successfully resets itself. I hope to get this commit upstream. For now, itā€™s included in the patch set for 5.8.

[Mon Sep  7 15:43:37 2020] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed
[Mon Sep  7 15:43:37 2020] ieee80211 phy0: brcmf_fil_cmd_data: bus is down. we have nothing to do.
[Mon Sep  7 15:43:37 2020] ieee80211 phy0: brcmf_cfg80211_get_tx_power: error (-5)
[Mon Sep  7 15:43:37 2020] ieee80211 phy0: brcmf_netdev_start_xmit: xmit rejected state=0
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_sdiod_probe: Failed to set F1 blocksize
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_sdio_bus_reset: Failed to probe after sdio device reset: ret -123
[Mon Sep  7 15:43:38 2020] mmc1: card 0001 removed
[Mon Sep  7 15:43:38 2020] mmc1: new ultra high speed SDR104 SDIO card at address 0001
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available
[Mon Sep  7 15:43:38 2020] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Jun 12 2020 12:11:45 version 7.45.96.66 (c7a32cb@shgit) (r745790) FWID 01-cffa7eb1
[Mon Sep  7 15:43:38 2020] ieee80211 phy1: brcmf_netdev_set_mac_address: Setting cur_etheraddr failed, -52
[Mon Sep  7 15:43:45 2020] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready

I will note that after crashes now, I do note quite a bit of noise in the logs after one firmware crash. The kernel/SBC had not reset and I came back to working WiFi, so it did recover, but might have taken ~15 mins to do so in one case. Remaining cases appear to have recovered instantly.

[Mon Sep  7 15:53:53 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:53:53 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:53:59 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:53:59 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:54:05 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:54:05 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52
[Mon Sep  7 15:54:11 2020] ieee80211 phy1: brcmf_cfg80211_get_station: GET STA INFO failed, -52

The Silvermont optimizations also appear to help a good deal in some instances.

Using the Volano chat server benchmark, I logged this metric with Silvermont optimizations for the kernel:

VolanoMark version = 2.9.0
Messages sent      = 100000
Messages received  = 1900000
Total messages     = 2000000
Elapsed time       = 55.024 seconds
Average throughput = 36348 messages per second

A bog standard -O2 -mtune=generic kernel was about 10% slower. This result is so good that I am almost skeptical and need to rerun.

VolanoMark version = 2.9.0
Messages sent      = 100000
Messages received  = 1900000
Total messages     = 2000000
Elapsed time       = 60.06 seconds
Average throughput = 33300 messages per second

I also noticed some instability with WiFi, connection drops after some time even if itā€™s not reported by the network manager.

I had no problems with the driver included in Lubuntu or Debian

Make sure you use the firmware here:

https://dl.radxa.com/rockpix/drivers/firmware/AP6255_BT_WIFI_Firmware.zip

1 Like

I manually replaced the files installed by the .deb with the ones suggested by @jack, WiFi is stable now! I could play a youtube video for more that one hour, before the change it was stopping after few minutes.

Iā€™m going to test bluetooth later on

They are the same filesā€¦ :stuck_out_tongue:

Maybe you installed new version of kernel? I have been updating itā€¦

$ md5sum BT_WIFI_Firmware/wifi/* fw/lib/firmware/brcm/brcmfmac43455-sdio.*
24348b4a7470a81032166f2601eb80e2  BT_WIFI_Firmware/wifi/brcmfmac43455c0-sdio.bin
d3ad92472ef030c3897062ff809a9083  BT_WIFI_Firmware/wifi/brcmfmac43455-sdio.ROCK Pi-ROCK Pi X.txt
24348b4a7470a81032166f2601eb80e2  fw/lib/firmware/brcm/brcmfmac43455-sdio.bin
d3ad92472ef030c3897062ff809a9083  fw/lib/firmware/brcm/brcmfmac43455-sdio.txt

I installed the latest packages you released, very weird

Sorry about that, 5.8.6 release and some 5.8.7 releases were not that great/stable. I should use patch revisions when I publish the package but have been keeping it at -1 and just re-releasing the package.

Should be smooth sailing now, no more breaking changes anticipated. It is quite stable and performant at this point.

Have a very big optimization (patch set to compile kernel with -flto) that I have been stress testing on my lab of ~6 SBCs + Rock Pi X. It seems very stable. Will publish soon along with 5.8.8 that was released just now.