Freezing Rock 5B with Rsync transfer

When I make rsync with files 1.2 TB
After 5-6 minutes of transfering with around 80MB/s with nvme Kingston NV1 2TB,
aluminium case: https://shop.allnetchina.cn/collections/rock5-model-b/products/rock-5b-metal-case
power adapter Baseus CCGAN65-1ACC (with power meter I saw that voltage was 20V and ~0.3 A

rock03:~:% free -m
total used free shared buff/cache available
Mem: 15718 247 4169 9 11301 15304
Swap: 0 0 0
rock03:~:% uptime
07:42:31 up 23:50, 2 users, load average: 1.31, 0.47, 0.17
rock03:~:% free -m
total used free shared buff/cache available
Mem: 15718 579 273 9 14865 14972
Swap: 0 0 0

rock 5B was freezing
Can anyone tell me what the problem - high cpu/disk/memory temperature or insufficient power supply or something else? Please give me ideas or logs that I can get more information?

OS release

rock03 :~: cat /etc/os-release
PRETTY_NAME=“Armbian 23.08.0-trunk jammy”

Kernel

uname -a
Linux rock03 5.10.160-rk35xx #1 SMP Sun Jul 30 15:08:45 EEST 2023 aarch64 aarch64 aarch64 GNU/Linu

I get from kernel logs:

Aug 15 07:52:20 rock03 kernel: [86433.486527] Code: 38401423 380014c3 36080064 78402423 (780024c3)
Aug 15 07:52:20 rock03 kernel: [86433.487071] —[ end trace 3e7271a3a6292c0e ]—
Aug 15 07:52:24 rock03 kernel: [86437.793545] Unable to handle kernel paging request at virtual address ffff0002eda18000
Aug 15 07:52:24 rock03 kernel: [86437.794259] Mem abort info:
Aug 15 07:52:24 rock03 kernel: [86437.794510] ESR = 0x96000047
Aug 15 07:52:24 rock03 kernel: [86437.794784] EC = 0x25: DABT (current EL), IL = 32 bits
Aug 15 07:52:24 rock03 kernel: [86437.795288] SET = 0, FnV = 0
client_loop: send disconnect: Broken pipe

another freezing

Aug 15 09:25:03 rock03 kernel: [ 2747.666245] Code: 12800081 97ffffba f9400260 37680040 (d4210000)
Aug 15 09:25:03 rock03 kernel: [ 2747.666780] —[ end trace cb7a41e3f7bbfb1f ]—
Aug 15 09:25:03 rock03 kernel: [ 2747.676390] ------------[ cut here ]------------
Aug 15 09:25:03 rock03 kernel: [ 2747.676825] WARNING: CPU: 7 PID: 0 at kernel/rcu/tree.c:624 rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.677591] Modules linked in: rpcsec_gss_krb5 binfmt_misc sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod panfrost gpu_sched pgdrv r8169
Aug 15 09:25:03 rock03 kernel: [ 2747.678868] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G D 5.10.160-rk35xx #1
Aug 15 09:25:03 rock03 kernel: [ 2747.679577] Hardware name: Radxa ROCK 5B (DT)
Aug 15 09:25:03 rock03 kernel: [ 2747.679963] pstate: 20c003c9 (nzCv DAIF +PAN +UAO -TCO BTYPE=–)
Aug 15 09:25:03 rock03 kernel: [ 2747.680487] pc : rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.680923] lr : rcu_eqs_enter.constprop.0+0x1c/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.681360] sp : ffff80000a463ea0
Aug 15 09:25:03 rock03 kernel: [ 2747.681655] x29: ffff80000a463ea0 x28: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.682124] x27: 0000000000000000 x26: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.682594] x25: 0000000000000000 x24: 0000027f2d16fd13
Aug 15 09:25:03 rock03 kernel: [ 2747.683063] x23: ffff0001060fb880 x22: ffff0001060fb880
Aug 15 09:25:03 rock03 kernel: [ 2747.683534] x21: ffff0001060fb898 x20: ffff80000983f7c0
Aug 15 09:25:03 rock03 kernel: [ 2747.684002] x19: ffff80000983f7c0 x18: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.684471] x17: 000000000000000b x16: 00000000000001d2
Aug 15 09:25:03 rock03 kernel: [ 2747.684939] x15: ffff000136108ea8 x14: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.685408] x13: 0000000200000001 x12: 0000000100000001
Aug 15 09:25:03 rock03 kernel: [ 2747.685878] x11: 0000000000000212 x10: 0000000000000ad0
Aug 15 09:25:03 rock03 kernel: [ 2747.686346] x9 : ffff800008c503c8 x8 : ffff0001008d38f0
Aug 15 09:25:03 rock03 kernel: [ 2747.686815] x7 : 0000000000000004 x6 : 0000000000000004
Aug 15 09:25:03 rock03 kernel: [ 2747.687284] x5 : 00000000037c8e39 x4 : 00ffffffffffffff
Aug 15 09:25:03 rock03 kernel: [ 2747.687753] x3 : 0000000000000000 x2 : 4000000000000002
Aug 15 09:25:03 rock03 kernel: [ 2747.688223] x1 : 4000000000000000 x0 : ffff0003fde2a7c0
Aug 15 09:25:03 rock03 kernel: [ 2747.688692] Call trace:
Aug 15 09:25:03 rock03 kernel: [ 2747.688913] rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.689321] rcu_idle_enter+0x18/0x24
Aug 15 09:25:03 rock03 kernel: [ 2747.689653] cpuidle_enter_state+0xe0/0x24c
Aug 15 09:25:03 rock03 kernel: [ 2747.690025] cpuidle_enter+0x40/0x58
Aug 15 09:25:03 rock03 kernel: [ 2747.690345] do_idle+0x23c/0x260
Aug 15 09:25:03 rock03 kernel: [ 2747.690630] cpu_startup_entry+0x2c/0x44
Aug 15 09:25:03 rock03 kernel: [ 2747.690982] secondary_start_kernel+0x1dc/0x1f0
Aug 15 09:25:03 rock03 kernel: [ 2747.691385]
Aug 15 09:25:03 rock03 kernel: [ 2747.691385] PC: 0xffff80000905fdd4:
Aug 15 09:25:03 rock03 kernel: [ 2747.691821] fbd4 aa0203f4 2a0303f3 52800080 97bf0583 12001c00 2a1303e3 2a1503e1 52800004

and another freezing

Aug 15 11:13:29 rock02 kernel: [ 1248.522685] ------------[ cut here ]------------
Aug 15 11:13:29 rock02 kernel: [ 1248.522703] kernel BUG at arch/arm64/kernel/traps.c:410!
Aug 15 11:13:29 rock02 kernel: [ 1248.522768] Internal error: Oops - BUG: 0 [#1] SMP
Aug 15 11:13:29 rock02 kernel: [ 1248.523654] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc rpcsec_gss_krb5 overlay sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod panfrost gpu_sched pgdrv r8169
Aug 15 11:13:29 rock02 kernel: [ 1248.526458] CPU: 3 PID: 4827 Comm: rsync Not tainted 5.10.160-rk35xx #1
Aug 15 11:13:29 rock02 kernel: [ 1248.527046] Hardware name: Radxa ROCK 5B (DT)
Aug 15 11:13:29 rock02 kernel: [ 1248.527438] pstate: 00400009 (nzcv daif +PAN -UAO -TCO BTYPE=–)
Aug 15 11:13:29 rock02 kernel: [ 1248.527978] pc : do_undefinstr+0x180/0x1e8
Aug 15 11:13:29 rock02 kernel: [ 1248.528345] lr : do_undefinstr+0x90/0x1e8
Aug 15 11:13:29 rock02 kernel: [ 1248.528702] sp : ffff80000b8bb9d0

and another freezing

Aug 15 11:35:18 rock03 kernel: [ 537.753649] Unable to handle kernel paging request at virtual address feff00010858c76c
Aug 15 11:35:18 rock03 kernel: [ 537.754382] Mem abort info:
Aug 15 11:35:18 rock03 kernel: [ 537.754640] ESR = 0x96000004
Aug 15 11:35:18 rock03 kernel: [ 537.754920] EC = 0x25: DABT (current EL), IL = 32 bits
Aug 15 11:35:18 rock03 kernel: [ 537.755395] SET = 0, FnV = 0
Aug 15 11:35:18 rock03 kernel: [ 537.755673] EA = 0, S1PTW = 0
Aug 15 11:35:18 rock03 kernel: [ 537.755963] Data abort info:
Aug 15 11:35:18 rock03 kernel: [ 537.756220] ISV = 0, ISS = 0x00000004
Aug 15 11:35:18 rock03 kernel: [ 537.756563] CM = 0, WnR = 0
Aug 15 11:35:18 rock03 kernel: [ 537.756832] [feff00010858c76c] address between user and kernel address ranges
Aug 15 11:35:18 rock03 kernel: [ 537.757463] Internal error: Oops: 96000004 [#1] SMP
Aug 15 11:35:18 rock03 kernel: [ 537.757895] Modules linked in: rpcsec_gss_krb5 binfmt_misc sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod pgdrv r8169 panfrost gpu_sched
Aug 15 11:35:18 rock03 kernel: [ 537.759207] CPU: 1 PID: 217 Comm: kworker/u16:3 Not tainted 5.10.160-rk35xx #1
Aug 15 11:35:18 rock03 kernel: [ 537.759848] Hardware name: Radxa ROCK 5B (DT)
Aug 15 11:35:18 rock03 kernel: [ 537.760251] Workqueue: ext4-rsv-conversion ext4_end_io_rsv_work
Aug 15 11:35:18 rock03 kernel: [ 537.760786] pstate: 60c00089 (nZCv daIf +PAN +UAO -TCO BTYPE=–)
Aug 15 11:35:18 rock03 kernel: [ 537.761321] pc : do_raw_spin_lock+0x28/0xdc
Aug 15 11:35:18 rock03 kernel: [ 537.761702] lr : _raw_spin_lock_irqsave+0x2c/0x40
Aug 15 11:35:18 rock03 kernel: [ 537.762121] sp : ffff80000bc8bc30
Aug 15 11:35:18 rock03 kernel: [ 537.762422] x29: ffff80000bc8bc30 x28: feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.762900] x27: 0000000000000000 x26: feff00010858c700
Aug 15 11:35:18 rock03 kernel: [ 537.763379] x25: feff00010858c700 x24: 0000000000000000
Aug 15 11:35:18 rock03 kernel: [ 537.763857] x23: ffff8000090c804e x22: 00000000000000ac
Aug 15 11:35:18 rock03 kernel: [ 537.764335] x21: ffff000102f07600 x20: feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.764814] x19: feff00010858c768 x18: ffff80000b925000
Aug 15 11:35:18 rock03 kernel: [ 537.765293] x17: 0000000000000000 x16: 0000000000000000
Aug 15 11:35:18 rock03 kernel: [ 537.765771] x15: ffff80000a0845c0 x14: 181f700000008800
Aug 15 11:35:18 rock03 kernel: [ 537.766250] x13: 000ce800181f6800 x12: 00008800000ce000
Aug 15 11:35:18 rock03 kernel: [ 537.766727] x11: 181f600000008800 x10: 000cd800181f5800
Aug 15 11:35:18 rock03 kernel: [ 537.767205] x9 : ffff800009065a20 x8 : 0000000000000001
Aug 15 11:35:18 rock03 kernel: [ 537.767683] x7 : ffff000122e9bd80 x6 : 0000000000400000
Aug 15 11:35:18 rock03 kernel: [ 537.768161] x5 : 0000000000000640 x4 : fffffe00061658f4
Aug 15 11:35:18 rock03 kernel: [ 537.768638] x3 : 0000000000000ac0 x2 : 00000000000000bf
Aug 15 11:35:18 rock03 kernel: [ 537.769116] x1 : 00000000dead4ead x0 : feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.769595] Call trace:
Aug 15 11:35:18 rock03 kernel: [ 537.769826] do_raw_spin_lock+0x28/0xdc
Aug 15 11:35:18 rock03 kernel: [ 537.770173] _raw_spin_lock_irqsave+0x2c/0x40
Aug 15 11:35:18 rock03 kernel: [ 537.770565] ext4_finish_bio+0xe8/0x1e0
Aug 15 11:35:18 rock03 kernel: [ 537.770912] ext4_release_io_end+0xa4/0xfc
Aug 15 11:35:18 rock03 kernel: [ 537.771280] ext4_end_io_rsv_work+0x114/0x130
Aug 15 11:35:18 rock03 kernel: [ 537.771671] process_one_work+0x1e0/0x298
Aug 15 11:35:18 rock03 kernel: [ 537.772038] worker_thread+0x1e0/0x278
Aug 15 11:35:18 rock03 kernel: [ 537.772374] kthread+0xf4/0x104
Aug 15 11:35:18 rock03 kernel: [ 537.772666] ret_from_fork+0x10/0x30
Aug 15 11:35:18 rock03 kernel: [ 537.772990]
Aug 15 11:35:18 rock03 kernel: [ 537.772990] PC: 0xffff8000080d6a38:

Appreciate posting this. I had a similar problem using FTP protocol where it would simply shut down network after a few minutes of transfer.

what’s the iamge you used, is also the armbian ?

we will check this issue

Issue on debian, ubuntu server and armbian. Was a while ago so I should retest latest images. Appriciate taking a look…

First, for the god sake, use spoilers for logs

Second, retest it with Radxa images

Third, does it happens if you do microSD/flash drive transfer?

Fourth, does it happens if you rsync from nvme to nvme?

Does it happens if you dd file from /dev/urandom to nvme? Something like
sudo dd if=/dev/urandom of=/test.img bs=1m count=65536

Fifth, does it happens with cp?

Just out of curiosity.

ouch, never mind, 1.5A is the input. Ignore.

But if this 65W is like the one i tested 33W, it could negotiate 20v, 1.35A or something…

See here: USB-C PD power charger behavior

Try images with kernel 6.5.y at the bottom of the page. You need to boot from SD card and there is no video out. Use serial console or SSH access …

Allright thanks :slight_smile:

Rsync transferring fix was :frowning:

Limiting RAM frequency improves stability for me and stops the random hangs/reboots:
echo 1560000000 | sudo tee /sys/ class / devfreq / dmc / max_freq

1 Like

Known problem on some boards: https://github.com/ThomasKaiser/sbc-bench/issues/55

But that this happens with rsync is somewhat strange? Can you test through sbc-bench -s walking through the DMC frequencies from bottom to top? At which RAM clock is your board freezing?

Will this also be remembered if rebooting the device? do we need to set governor userspace and how?

I also had to reduce cpu frequency sigh. Was this hardware ready when it launched?? (yes i bought mine as some of the early models) I mean happy it seems to be working so far but kinda sad most of my HW must be run outside of specs to be able to function.

1 Like

Hmm I said that too early even lowering to 2ghz it crashed when using WinSCP same time I was trying to browse a plex server.

update: with latest debian image it is crashing still but it seems its able to reconnect every time some seconds later at least so not a full network shutdown anymore. This is using the official raddax power supply so not sure if that has helped too.

Also if it is helpfull I transfered a few books I have bought from humblebundle on a another rock pi 5b using calibre web and this did not fail a single transfer. I am wondering if it is related to larger files and that it is then related to some memory problem? But why would it then shut down the network ?

hey there,
this problem still persists !
I’m using a 65Watt power supply and also added a heatsink around the nvme …
network ping is still answering, but the rock5b is not accessable anymore.
this is a hard problem, I bought this to replace my raspi 4.
but with this performance issues … no way .

1 Like

Have you tired this method

"
Limiting RAM frequency improves stability for me and stops the random hangs/reboots:
echo 1560000000 | sudo tee /sys/ class / devfreq / dmc / max_freq
"

Known problem on some boards: https://github.com/ThomasKaiser/sbc-bench/issues/55

Today I was down to 528MHz, the upload with winscp survives a little longer, but after about an hour the rock5b died again….
could someone try it on their machine to reproduce it and maybe find a solution?

just to add, I have a rock5b with 32GB ram and a samsung 980 1TB NVMe. also tried different OS with different kernel versions.
looks like, if I limit the upload to 1MBit/s it works ….
But the rock5b has what, 2.5GBit ethernet …

It might be the SSD. 980s are quite power hungry and Rock 5b was reported to hang when using them. Try using a less top of the line SSD and it will never hang any more.

Hi There, you may want to try this: Radxa Rock 5b Hangs multiple times a week

BTW: Depends on which win version you are using but in my win 11 i can use scp from terminal without needing putty / winscp.
Best