Freezing Rock 5B with Rsync transfer

#1

When I make rsync with files 1.2 TB
After 5-6 minutes of transfering with around 80MB/s with nvme Kingston NV1 2TB,
aluminium case: https://shop.allnetchina.cn/collections/rock5-model-b/products/rock-5b-metal-case
power adapter Baseus CCGAN65-1ACC (with power meter I saw that voltage was 20V and ~0.3 A

rock03:~:% free -m
total used free shared buff/cache available
Mem: 15718 247 4169 9 11301 15304
Swap: 0 0 0
rock03:~:% uptime
07:42:31 up 23:50, 2 users, load average: 1.31, 0.47, 0.17
rock03:~:% free -m
total used free shared buff/cache available
Mem: 15718 579 273 9 14865 14972
Swap: 0 0 0

rock 5B was freezing
Can anyone tell me what the problem - high cpu/disk/memory temperature or insufficient power supply or something else? Please give me ideas or logs that I can get more information?

OS release

rock03 :~: cat /etc/os-release
PRETTY_NAME=“Armbian 23.08.0-trunk jammy”

Kernel

uname -a
Linux rock03 5.10.160-rk35xx #1 SMP Sun Jul 30 15:08:45 EEST 2023 aarch64 aarch64 aarch64 GNU/Linu

I get from kernel logs:

Aug 15 07:52:20 rock03 kernel: [86433.486527] Code: 38401423 380014c3 36080064 78402423 (780024c3)
Aug 15 07:52:20 rock03 kernel: [86433.487071] —[ end trace 3e7271a3a6292c0e ]—
Aug 15 07:52:24 rock03 kernel: [86437.793545] Unable to handle kernel paging request at virtual address ffff0002eda18000
Aug 15 07:52:24 rock03 kernel: [86437.794259] Mem abort info:
Aug 15 07:52:24 rock03 kernel: [86437.794510] ESR = 0x96000047
Aug 15 07:52:24 rock03 kernel: [86437.794784] EC = 0x25: DABT (current EL), IL = 32 bits
Aug 15 07:52:24 rock03 kernel: [86437.795288] SET = 0, FnV = 0
client_loop: send disconnect: Broken pipe

another freezing

Aug 15 09:25:03 rock03 kernel: [ 2747.666245] Code: 12800081 97ffffba f9400260 37680040 (d4210000)
Aug 15 09:25:03 rock03 kernel: [ 2747.666780] —[ end trace cb7a41e3f7bbfb1f ]—
Aug 15 09:25:03 rock03 kernel: [ 2747.676390] ------------[ cut here ]------------
Aug 15 09:25:03 rock03 kernel: [ 2747.676825] WARNING: CPU: 7 PID: 0 at kernel/rcu/tree.c:624 rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.677591] Modules linked in: rpcsec_gss_krb5 binfmt_misc sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod panfrost gpu_sched pgdrv r8169
Aug 15 09:25:03 rock03 kernel: [ 2747.678868] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G D 5.10.160-rk35xx #1
Aug 15 09:25:03 rock03 kernel: [ 2747.679577] Hardware name: Radxa ROCK 5B (DT)
Aug 15 09:25:03 rock03 kernel: [ 2747.679963] pstate: 20c003c9 (nzCv DAIF +PAN +UAO -TCO BTYPE=–)
Aug 15 09:25:03 rock03 kernel: [ 2747.680487] pc : rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.680923] lr : rcu_eqs_enter.constprop.0+0x1c/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.681360] sp : ffff80000a463ea0
Aug 15 09:25:03 rock03 kernel: [ 2747.681655] x29: ffff80000a463ea0 x28: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.682124] x27: 0000000000000000 x26: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.682594] x25: 0000000000000000 x24: 0000027f2d16fd13
Aug 15 09:25:03 rock03 kernel: [ 2747.683063] x23: ffff0001060fb880 x22: ffff0001060fb880
Aug 15 09:25:03 rock03 kernel: [ 2747.683534] x21: ffff0001060fb898 x20: ffff80000983f7c0
Aug 15 09:25:03 rock03 kernel: [ 2747.684002] x19: ffff80000983f7c0 x18: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.684471] x17: 000000000000000b x16: 00000000000001d2
Aug 15 09:25:03 rock03 kernel: [ 2747.684939] x15: ffff000136108ea8 x14: 0000000000000000
Aug 15 09:25:03 rock03 kernel: [ 2747.685408] x13: 0000000200000001 x12: 0000000100000001
Aug 15 09:25:03 rock03 kernel: [ 2747.685878] x11: 0000000000000212 x10: 0000000000000ad0
Aug 15 09:25:03 rock03 kernel: [ 2747.686346] x9 : ffff800008c503c8 x8 : ffff0001008d38f0
Aug 15 09:25:03 rock03 kernel: [ 2747.686815] x7 : 0000000000000004 x6 : 0000000000000004
Aug 15 09:25:03 rock03 kernel: [ 2747.687284] x5 : 00000000037c8e39 x4 : 00ffffffffffffff
Aug 15 09:25:03 rock03 kernel: [ 2747.687753] x3 : 0000000000000000 x2 : 4000000000000002
Aug 15 09:25:03 rock03 kernel: [ 2747.688223] x1 : 4000000000000000 x0 : ffff0003fde2a7c0
Aug 15 09:25:03 rock03 kernel: [ 2747.688692] Call trace:
Aug 15 09:25:03 rock03 kernel: [ 2747.688913] rcu_eqs_enter.constprop.0+0x34/0x88
Aug 15 09:25:03 rock03 kernel: [ 2747.689321] rcu_idle_enter+0x18/0x24
Aug 15 09:25:03 rock03 kernel: [ 2747.689653] cpuidle_enter_state+0xe0/0x24c
Aug 15 09:25:03 rock03 kernel: [ 2747.690025] cpuidle_enter+0x40/0x58
Aug 15 09:25:03 rock03 kernel: [ 2747.690345] do_idle+0x23c/0x260
Aug 15 09:25:03 rock03 kernel: [ 2747.690630] cpu_startup_entry+0x2c/0x44
Aug 15 09:25:03 rock03 kernel: [ 2747.690982] secondary_start_kernel+0x1dc/0x1f0
Aug 15 09:25:03 rock03 kernel: [ 2747.691385]
Aug 15 09:25:03 rock03 kernel: [ 2747.691385] PC: 0xffff80000905fdd4:
Aug 15 09:25:03 rock03 kernel: [ 2747.691821] fbd4 aa0203f4 2a0303f3 52800080 97bf0583 12001c00 2a1303e3 2a1503e1 52800004

and another freezing

Aug 15 11:13:29 rock02 kernel: [ 1248.522685] ------------[ cut here ]------------
Aug 15 11:13:29 rock02 kernel: [ 1248.522703] kernel BUG at arch/arm64/kernel/traps.c:410!
Aug 15 11:13:29 rock02 kernel: [ 1248.522768] Internal error: Oops - BUG: 0 [#1] SMP
Aug 15 11:13:29 rock02 kernel: [ 1248.523654] Modules linked in: xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc rpcsec_gss_krb5 overlay sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod panfrost gpu_sched pgdrv r8169
Aug 15 11:13:29 rock02 kernel: [ 1248.526458] CPU: 3 PID: 4827 Comm: rsync Not tainted 5.10.160-rk35xx #1
Aug 15 11:13:29 rock02 kernel: [ 1248.527046] Hardware name: Radxa ROCK 5B (DT)
Aug 15 11:13:29 rock02 kernel: [ 1248.527438] pstate: 00400009 (nzcv daif +PAN -UAO -TCO BTYPE=–)
Aug 15 11:13:29 rock02 kernel: [ 1248.527978] pc : do_undefinstr+0x180/0x1e8
Aug 15 11:13:29 rock02 kernel: [ 1248.528345] lr : do_undefinstr+0x90/0x1e8
Aug 15 11:13:29 rock02 kernel: [ 1248.528702] sp : ffff80000b8bb9d0

and another freezing

Aug 15 11:35:18 rock03 kernel: [ 537.753649] Unable to handle kernel paging request at virtual address feff00010858c76c
Aug 15 11:35:18 rock03 kernel: [ 537.754382] Mem abort info:
Aug 15 11:35:18 rock03 kernel: [ 537.754640] ESR = 0x96000004
Aug 15 11:35:18 rock03 kernel: [ 537.754920] EC = 0x25: DABT (current EL), IL = 32 bits
Aug 15 11:35:18 rock03 kernel: [ 537.755395] SET = 0, FnV = 0
Aug 15 11:35:18 rock03 kernel: [ 537.755673] EA = 0, S1PTW = 0
Aug 15 11:35:18 rock03 kernel: [ 537.755963] Data abort info:
Aug 15 11:35:18 rock03 kernel: [ 537.756220] ISV = 0, ISS = 0x00000004
Aug 15 11:35:18 rock03 kernel: [ 537.756563] CM = 0, WnR = 0
Aug 15 11:35:18 rock03 kernel: [ 537.756832] [feff00010858c76c] address between user and kernel address ranges
Aug 15 11:35:18 rock03 kernel: [ 537.757463] Internal error: Oops: 96000004 [#1] SMP
Aug 15 11:35:18 rock03 kernel: [ 537.757895] Modules linked in: rpcsec_gss_krb5 binfmt_misc sch_fq_codel fuse ip_tables ipv6 dm_crypt encrypted_keys trusted dm_mod pgdrv r8169 panfrost gpu_sched
Aug 15 11:35:18 rock03 kernel: [ 537.759207] CPU: 1 PID: 217 Comm: kworker/u16:3 Not tainted 5.10.160-rk35xx #1
Aug 15 11:35:18 rock03 kernel: [ 537.759848] Hardware name: Radxa ROCK 5B (DT)
Aug 15 11:35:18 rock03 kernel: [ 537.760251] Workqueue: ext4-rsv-conversion ext4_end_io_rsv_work
Aug 15 11:35:18 rock03 kernel: [ 537.760786] pstate: 60c00089 (nZCv daIf +PAN +UAO -TCO BTYPE=–)
Aug 15 11:35:18 rock03 kernel: [ 537.761321] pc : do_raw_spin_lock+0x28/0xdc
Aug 15 11:35:18 rock03 kernel: [ 537.761702] lr : _raw_spin_lock_irqsave+0x2c/0x40
Aug 15 11:35:18 rock03 kernel: [ 537.762121] sp : ffff80000bc8bc30
Aug 15 11:35:18 rock03 kernel: [ 537.762422] x29: ffff80000bc8bc30 x28: feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.762900] x27: 0000000000000000 x26: feff00010858c700
Aug 15 11:35:18 rock03 kernel: [ 537.763379] x25: feff00010858c700 x24: 0000000000000000
Aug 15 11:35:18 rock03 kernel: [ 537.763857] x23: ffff8000090c804e x22: 00000000000000ac
Aug 15 11:35:18 rock03 kernel: [ 537.764335] x21: ffff000102f07600 x20: feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.764814] x19: feff00010858c768 x18: ffff80000b925000
Aug 15 11:35:18 rock03 kernel: [ 537.765293] x17: 0000000000000000 x16: 0000000000000000
Aug 15 11:35:18 rock03 kernel: [ 537.765771] x15: ffff80000a0845c0 x14: 181f700000008800
Aug 15 11:35:18 rock03 kernel: [ 537.766250] x13: 000ce800181f6800 x12: 00008800000ce000
Aug 15 11:35:18 rock03 kernel: [ 537.766727] x11: 181f600000008800 x10: 000cd800181f5800
Aug 15 11:35:18 rock03 kernel: [ 537.767205] x9 : ffff800009065a20 x8 : 0000000000000001
Aug 15 11:35:18 rock03 kernel: [ 537.767683] x7 : ffff000122e9bd80 x6 : 0000000000400000
Aug 15 11:35:18 rock03 kernel: [ 537.768161] x5 : 0000000000000640 x4 : fffffe00061658f4
Aug 15 11:35:18 rock03 kernel: [ 537.768638] x3 : 0000000000000ac0 x2 : 00000000000000bf
Aug 15 11:35:18 rock03 kernel: [ 537.769116] x1 : 00000000dead4ead x0 : feff00010858c768
Aug 15 11:35:18 rock03 kernel: [ 537.769595] Call trace:
Aug 15 11:35:18 rock03 kernel: [ 537.769826] do_raw_spin_lock+0x28/0xdc
Aug 15 11:35:18 rock03 kernel: [ 537.770173] _raw_spin_lock_irqsave+0x2c/0x40
Aug 15 11:35:18 rock03 kernel: [ 537.770565] ext4_finish_bio+0xe8/0x1e0
Aug 15 11:35:18 rock03 kernel: [ 537.770912] ext4_release_io_end+0xa4/0xfc
Aug 15 11:35:18 rock03 kernel: [ 537.771280] ext4_end_io_rsv_work+0x114/0x130
Aug 15 11:35:18 rock03 kernel: [ 537.771671] process_one_work+0x1e0/0x298
Aug 15 11:35:18 rock03 kernel: [ 537.772038] worker_thread+0x1e0/0x278
Aug 15 11:35:18 rock03 kernel: [ 537.772374] kthread+0xf4/0x104
Aug 15 11:35:18 rock03 kernel: [ 537.772666] ret_from_fork+0x10/0x30
Aug 15 11:35:18 rock03 kernel: [ 537.772990]
Aug 15 11:35:18 rock03 kernel: [ 537.772990] PC: 0xffff8000080d6a38:

#2

Appreciate posting this. I had a similar problem using FTP protocol where it would simply shut down network after a few minutes of transfer.

#3

what’s the iamge you used, is also the armbian ?

#4

we will check this issue

#5

Issue on debian, ubuntu server and armbian. Was a while ago so I should retest latest images. Appriciate taking a look…

#6

First, for the god sake, use spoilers for logs

Second, retest it with Radxa images

Third, does it happens if you do microSD/flash drive transfer?

Fourth, does it happens if you rsync from nvme to nvme?

Does it happens if you dd file from /dev/urandom to nvme? Something like
sudo dd if=/dev/urandom of=/test.img bs=1m count=65536

Fifth, does it happens with cp?

#7

Just out of curiosity.

ouch, never mind, 1.5A is the input. Ignore.

But if this 65W is like the one i tested 33W, it could negotiate 20v, 1.35A or something…

See here: USB-C PD power charger behavior

#8

Try images with kernel 6.5.y at the bottom of the page. You need to boot from SD card and there is no video out. Use serial console or SSH access …

#9

Allright thanks :slight_smile:

#10

Rsync transferring fix was :frowning:

Limiting RAM frequency improves stability for me and stops the random hangs/reboots:
echo 1560000000 | sudo tee /sys/ class / devfreq / dmc / max_freq

1 Like
#11

Known problem on some boards: https://github.com/ThomasKaiser/sbc-bench/issues/55

But that this happens with rsync is somewhat strange? Can you test through sbc-bench -s walking through the DMC frequencies from bottom to top? At which RAM clock is your board freezing?

#12

Will this also be remembered if rebooting the device? do we need to set governor userspace and how?

#13

I also had to reduce cpu frequency sigh. Was this hardware ready when it launched?? (yes i bought mine as some of the early models) I mean happy it seems to be working so far but kinda sad most of my HW must be run outside of specs to be able to function.

1 Like
#14

Hmm I said that too early even lowering to 2ghz it crashed when using WinSCP same time I was trying to browse a plex server.

update: with latest debian image it is crashing still but it seems its able to reconnect every time some seconds later at least so not a full network shutdown anymore. This is using the official raddax power supply so not sure if that has helped too.

Also if it is helpfull I transfered a few books I have bought from humblebundle on a another rock pi 5b using calibre web and this did not fail a single transfer. I am wondering if it is related to larger files and that it is then related to some memory problem? But why would it then shut down the network ?

#15

hey there,
this problem still persists !
I’m using a 65Watt power supply and also added a heatsink around the nvme …
network ping is still answering, but the rock5b is not accessable anymore.
this is a hard problem, I bought this to replace my raspi 4.
but with this performance issues … no way .

1 Like
#16

Have you tired this method

"
Limiting RAM frequency improves stability for me and stops the random hangs/reboots:
echo 1560000000 | sudo tee /sys/ class / devfreq / dmc / max_freq
"

Known problem on some boards: https://github.com/ThomasKaiser/sbc-bench/issues/55

#17

Today I was down to 528MHz, the upload with winscp survives a little longer, but after about an hour the rock5b died again….
could someone try it on their machine to reproduce it and maybe find a solution?

just to add, I have a rock5b with 32GB ram and a samsung 980 1TB NVMe. also tried different OS with different kernel versions.
looks like, if I limit the upload to 1MBit/s it works ….
But the rock5b has what, 2.5GBit ethernet …

#18

It might be the SSD. 980s are quite power hungry and Rock 5b was reported to hang when using them. Try using a less top of the line SSD and it will never hang any more.

#19

Hi There, you may want to try this: Radxa Rock 5b Hangs multiple times a week

BTW: Depends on which win version you are using but in my win 11 i can use scp from terminal without needing putty / winscp.
Best

#20

Now after testing and try and and …
I monitored the system all the time an now I found this.
if the RAM Cache + Buffer metrics reaches 10.3 GiB it freezes. It’s reproducable. Now I need an idea how to overcome this border

puffer-cache-full
puffer-cache-full.jpg1117×262 30.7 KB

Radxa Rock 5b Hangs multiple times a week