MTE on 64 GB: failures, explanation, CIX please help!

[summa]

I have an Orion O6 with 64 GB RAM. I’m using firmware 1.0.0-3, which offers MTE support, and a MTE-capable kernel without kASAN. I can prove that MTE is not happy on this platform.

I wrote a program that allocates a block of memory with MTE (using Linux mmap system call). Then it fills the allocation tag area for the block with a pattern. Then it checks the pattern for correctness. And when the block gets large enough to not fit in CPU caches, I get failures (mte-failures.txt in zip file). You can build and run the test yourself, just unpack the zip file and run make.

I believe the reason the test fails is that the Arm MTSX block used to provide MTE tag storage in LPDDR5 does not support CD8180’s memory map shape. The MTSX block is configured by the Cortex-M SE core during boot, while setting up the CI-700 interconnect.

The setup, even for 64 GB RAM (which splits the RAM range into [ 0x0080000000, 0x07FFFFFFFF ] low 30 GB range and [ 0x8000000000, 0x887FFFFFFF ] high 34 GB range), configures the MTSX for contiguous memory map (mtu_tag_addr_ctl.memory_map_mode = 000b). What’s worse, the MTSX tag base address is set as follows for different amounts of RAM:

  RAM     mtu_tag_addr_base   (+ tags + 64MB)

 4096 MB:  0000000174000000  0000000180000000
 8192 MB:  000000026c000000  0000000280000000
16384 MB:  000000045c000000  0000000480000000
32768 MB:  000000083c000000  0000000880000000
65536 MB:  0000000ffc000000  0000001080000000

For 32 GB and 64 GB RAM, the base falls outside of defined RAM regions. I wonder if this is the source of MTE trouble. Please ask CIX for a comment on this. If nothing else helps, you could restrict RAM size with MTE enabled to 30 GB, then everything will fit in the contiguous, low RAM range:

RAM mtu_tag_addr_base (+ tags + 64MB)

30720 MB: 00000007c0000000 0000000800000000

For a 32 GB board this is not much of a sacrifice, you get 29 GB with MTE instead of the 31 GB you could theoretically get. (For 64 GB it is a pity, but better than not having MTE that actually works.)

You could also use 30 GB of low range for data, and put MTSX tags at the start of the high range.

I can’t attach a ZIP file because my account is new. But I have uploaded it for you: Filebin | 0dsgzt1ltpau8nmj

Thank you for your attention, I know this is very technical, but I hope I can help CIX debug this quickly.

[volyrique]

Thank you for the detailed analysis! Actually I noticed the same issue, with an even simpler reproducer - set the allocation tag on a granule and then load from it repeatedly in an infinite loop (using the sample code from the Linux kernel documentation, for example). After a while (i.e. less than an hour) the program is going to crash with a tag check failure, which is nonsensical, of course. The time until the failure depends on the system load probably - often it happens quickly, after a couple of minutes at most.

I believe that MTE was working properly with a firmware version between 1.0.0-1 and 1.0.0-2, but unfortunately I didn’t keep a record of the exact revisions of the various submodules and there wasn’t a tagged release in between.

@summa

For a 32 GB board this is not much of a sacrifice, you get 29 GB with MTE instead of the 31 GB you could theoretically get.

You are onto something because, if I remember correctly, the total system memory seen by the kernel with the functional firmware version was between 29000 and 30000 MB (I am on a 32 GB board), while with 1.0.0-2 it is 30302 MB - surprisingly high the first time I saw it.

[summa]

I downgraded my board to 1.0.0-2 (public 1.0.0-1 doesn’t give me the Memory Tagging Extensions toggle in UEFI setup). Fails exactly the same way.

If my theory about why this happens is true, the test you ran could sometimes work for an extended period of time (until a cache eviction happens - that’s why I tried to make a test that scales its memory footprint until eviction is forced). Or maybe 32 GB was squeaking by as you said. Also, the MTSX setup logic is present even in older SE binaries (including ones that do not advertise it), and looks more or less the same, except MTE flag is taken from “dfd value” instead of explicit setting.

It’d be so nice to have a board with working MTE. kASAN is a big time saver for kernel work and the board has PCIe, so one can test Linux GPU and NPU drivers with it.

[jimhamiru]

I doubt this is related, but just in case (as it pertains, partially, to memory issues):

Llama.cpp - Benchmarks

Hi all, while running llama-bench on larger models (e.g. 32B+ Q4_KM), I often kept getting hard crashes on my Orion O6 (fan remains at full pelt and it became inaccessible, requiring hard reset). Turns out this was just my system config that needed tweaking to make sure I wasn’t overcommitting to memory that I didn’t have available. With the follow tweaks, larger models can be benched without issue (they’re obviously slow though). sudo sh -c 'echo 2 > /proc/sys/vm/overcommit_memory' sudo sh -…

If relevant, I’m using the ARM Ubuntu Server 25.04 server image (but had same symptom when benching on Radxa’s stock Debian image). No idea if MTE is enabled on either of those kernels.

[RadxaYuntian]

I believe you have to enable both MTE in BIOS and KASAN in kernel to have it working correctly. Those 2 settings need to be on and off at the same time. Otherwise you will get random crash/reboot. This is something we have encountered during bring up.

[summa]

It doesn’t really crash without kASAN, just keeps working OK (on newer versions that correctly trim RAM). Only MTE itself has issues, which I’m pretty sure are related to the settings of the hardware. Can you try the test program on a 16 GB unit? I only have two 64 GB boards so I can’t test on 16 GB, and I think it might have different results. You will need MTE enabled in kernel (kASAN optional, either on or off is OK) as well as in BIOS.

The options in kernel do this: CONFIG_ARM64_MTE enables MTE support for applications (so PROT_MTE will work in userland programs). It doesn’t make Linux use MTE for kernel.

CONFIG_KASAN_HW_TAGS makes Linux kernel use MTE for its own purposes. Both would require MTE enabled in BIOS.

This is a Linux kernel that has a ramdisk with the test program built in. You can just run it on your Orion O6 as any other ACPI mode kernel. It outputs to serial console port.

Filebin | svxodnd8mm8a8vib (updated for RDR)

[summa]

Filebin | s1dwr56lao7g6spf

Here’s a version like above but with MTE + kASAN enabled in kernel. You can see it has kASAN when it boots, but it still fails the test program

[viosu]

Quote: “The MTSX block is configured by the Cortex-M SE core during boot, while setting up the CI-700 interconnect.”

I was looking for the related code, do you know if it’s publicly available ?

[clingfei]

Thanks for your analysis! Has this problem been resolved? Or are there other ways to make MTE work correctly in the kernel?

[clingfei]

I updated my BIOS from 1.0.0-2 to 1.2.1, it seems that the failures has been fixed.
However, the available RAM becomes 28GB, while the real RAM is 64GB.

[volyrique]

I am observing something similar with the 32 GB variant of the Orion O6 board and the same firmware version - the available memory drops from 30798 MB down to 28292 MB, which is significantly smaller than the theoretical value of 29805 MB, but at least MTE seems to function correctly. I left the test program that I mentioned before running for an hour and there were no failures.