Wifi firmware crashes when running in AP

I’m running Armbian 21.02.3 Buster on a RockPi 4B.
I setup an access point with the onboard wifi AP6256. Everything is working fine with good performance but only for few days.
Every 3-4 days, the wifi stops working with this log in dmesg

[221057.075187] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[221057.075202] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[221124.657840] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[221127.217786] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[221127.217806] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110

By enabling debug mode on the brcmfmac driver, I got more logs :

[85658.544429] brcmfmac: CONSOLE: 085641.821 wl0: PHYTX error
[85660.512477] brcmfmac: CONSOLE: 085643.786 wl0: PHYTX error
[85660.512502] brcmfmac: CONSOLE: 085643.788 wl0: PHYTX error

A reboot is needed to get it back.
That really look like a firmware crash. I tried different firmware versions, some are quicker to fail but they all crash in the end.

Is there a way to fix that ?