[BUG] Gemma 4 E2B infinite generation loop on AX-M1 via AXCL PCIe (x86 host)

Hardware and peripherals
1

Hi,

I’m running the Radxa AICore AX-M1 on an x86 host and experiencing an infinite generation loop with the Gemma 4 E2B model.

Hardware:

  • Host: Intel i7-10700K, x86_64, Debian 13 (Trixie)
  • AI accelerator: Radxa AICore AX-M1 (AX8850, PCIe M.2)
  • Second card: M5Stack LLM-8850 (AX8850, PCIe M.2)
  • AXCL Driver: V3.6.5_P1_20251018122826

Software:

  • axllm binary: e75d3022642d, built Jun 8 2026, backend: AXCL x86
  • Model: AXERA-TECH/gemma-4-E2B-it-GPTQ-INT4 (downloaded from HuggingFace)
  • Running inside Proxmox LXC with AXCL device passthrough

Symptom:
Model loads correctly, remain_cmm ~5005 MB, TTFT ~373ms, but instead of valid text generates an infinite loop of repeated symbols (I!I!I! or I I I I). Decode speed ~9.83 tok/s — tokens are being generated but content is garbage.

What works:

  • DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int4 works correctly on the same hardware at ~4.5 tok/s
  • axcl-smi correctly detects both AX8850 cards

Related:

  • AXERA-TECH/ax-llm issue #39 (gemma4 infinite generation loop)
  • AXERA-TECH/ax-llm issue #21 (similar issue with Qwen3-1.7B on AX8850 AXCL)
  • AXERA confirmed in issue #39 that Gemma4 currently does not support inference on AXCL compute cards

Is there any update on AXCL PCIe support for Gemma4 on AX-M1? Is this something Radxa is working on with AXERA, or should this be tracked purely upstream?

Thanks

2

Confirmed still present on AXCL 3.16.0 (2026-07-31 driver)

Reproduced the same infinite-repetition symptom on fresh setup, same hardware class as OP (M5Stack LLM-8850 / AX-M1 via AXCL PCIe):

  • Host: x86_64, AXCL-SMI/Driver V3.16.0_20260731020156

  • axllm rebuilt from axllm branch, commit 34221cea26d9 (built 2026-08-26), linked against system AXCL 3.16.0 (not the bundled 3.6.2 from build_axcl_x86.sh)

  • Model: AXERA-TECH/gemma-4-E2B-it-GPTQ-INT4, freshly downloaded same day

05:26:43.676 INF run:30 | AXCLWorker start with devid 0
...
05:28:20.786 INF Run:666 | ttft: 388.25 ms
CertainlyCertainlyCertainlyCertainlyCertainlyCertainly... (infinite loop after first token)

Same failure mode as OP’s original report (IIMIMIMIM... there vs CertainlyCertainly... here) — first token generates fine (ttft normal), then generation collapses into repeating the same token indefinitely.

This confirms the issue survives the AXCL driver update from whatever version was current in May to 3.16.0 in July/August. Given commit 6d39a9b (Jul 3) already documents this as a known AXCL-runtime-side shared-KV numerical divergence with no ax-llm-side fix — is there any tracking on the AXCL SDK side (closed-source), or should users just wait indefinitely? Would help to know if this is actively being worked on upstream.