I tried bringing up a Radeon R7 250 card which does highlight a number of limitations with the pcie interface on the RK3588. Anyway some things do work and a write up on my blog.
Adventures with external GPU card + Gen 3 pcie
What is the status now?
Also, I guess main thread for this topic is currently https://forum.radxa.com/t/idea-about-plug-amd-gpu-via-nvme/13132/23.
Adding some of my own experiences here so they are more easily findable than on discord, this was done with an Nvidia RTX 3090 (under Linux 6.8.2):
dmesg | grep 0000:01:00.0
[ 2.819204] pci 0000:01:00.0: [10de:2204] type 00 class 0x030000 PCIe Legacy Endpoint
[ 2.819246] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x00ffffff]
[ 2.819280] pci 0000:01:00.0: BAR 1 [mem 0x00000000-0x0fffffff 64bit pref]
[ 2.819313] pci 0000:01:00.0: BAR 3 [mem 0x00000000-0x01ffffff 64bit pref]
[ 2.819334] pci 0000:01:00.0: BAR 5 [io 0x0000-0x007f]
[ 2.819355] pci 0000:01:00.0: ROM [mem 0x00000000-0x0007ffff pref]
[ 2.819610] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 2.819960] pci 0000:01:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[ 2.820325] pci 0000:01:00.0: vgaarb: setting as boot VGA device
[ 2.820330] pci 0000:01:00.0: vgaarb: bridge control possible
[ 2.820334] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 2.831115] pci 0000:01:00.0: BAR 1 [mem size 0x10000000 64bit pref]: can't assign; no space
[ 2.831120] pci 0000:01:00.0: BAR 1 [mem size 0x10000000 64bit pref]: failed to assign
[ 2.831126] pci 0000:01:00.0: BAR 3 [mem size 0x02000000 64bit pref]: can't assign; no space
[ 2.831131] pci 0000:01:00.0: BAR 3 [mem size 0x02000000 64bit pref]: failed to assign
[ 2.831136] pci 0000:01:00.0: BAR 0 [mem size 0x01000000]: can't assign; no space
[ 2.831141] pci 0000:01:00.0: BAR 0 [mem size 0x01000000]: failed to assign
[ 2.831145] pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: can't assign; no space
[ 2.831150] pci 0000:01:00.0: ROM [mem size 0x00080000 pref]: failed to assign
[ 2.831165] pci 0000:01:00.0: BAR 5 [io 0x100000-0x10007f]: assigned
[ 2.833528] pci 0000:01:00.1: D0 power state depends on 0000:01:00.0
@Mario on discord said:
Ah, Linux ignores the 64-bit pref window for some reason. I remember seeing this too, not sure why the driver does this.
It also wants 16 MB of 32-bit memory for BARO.
RK3588 supports precisely 16 MB of such memory, but the DTB splits it in several sub-regions (one for 1/0 space iirc).
1 Like
I have also tried serveral differnent graphics cards. They all get recognized but as in your post something is blocking it from working atm
I‘m actually way further now thanks to some patches on the armbian discord. So far as in everything gets addressed correctly and I get some info from the GPU
You can see the new and Nouveau driver results here while I‘m trying Nvidia‘s drivers rn.
1 Like
End result for today
Nvidia proprietary:
[ 8.600582] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0x65:1468)
[ 8.600674] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
[ 8.600817] [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NvKmsKapiDevice
[ 8.601012] [drm:nv_drm_probe_devices [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00000100] Failed to register device
Nouveau:
nouveau 0000:01:00.0: DRM: core notifier timeout
Got it so far as to have the PCIe device be accessible but the driver seems to have (the same as RPI) issues with the implementation of PCIe.
2 Likes
I only see one patch in the GitHub repo, here: https://github.com/HeyMeco/Rockchip-pcie-devices/blob/main/RK3588/_cards_gpu/Nvidia_RTX-3090-Founders-Edition.md#the-pcie-patch-for-mainline-that-brought-us-so-far
Are there more patches required as well, @meco?
From the kernel side not for now.
Interesting - I wonder if this would help as well @meco ? Looks like Mario is suggesting a change to the PCIe ranges?
Yes it came up when we both looked at it over in the Radxa Discord. He is the one who made the patch for the mainline pcie driver too.
Aha - gotcha. I wonder if applying this Mario patch, along with the one you documented in your GitHub, gets you any further?