Hi all, while running llama-bench on larger models (e.g. 32B+ Q4_KM), I often kept getting hard crashes on my Orion O6 (fan remains at full pelt and it became inaccessible, requiring hard reset).
Turns out this was just my system config that needed tweaking to make sure I wasn’t overcommitting to memory that I didn’t have available.
With the follow tweaks, larger models can be benched without issue (they’re obviously slow though).
sudo sh -c 'echo 2 > /proc/sys/vm/overcommit_memory'
sudo sh -c 'echo 80 > /proc/sys/vm/overcommit_ratio'