Idea about plug AMD GPU via NVME

#8

How did you managed to do that?
I ended with this

[  124.474708] [drm] amdgpu kernel modesetting enabled.
[  124.476599] amdgpu 0002:23:00.0: enabling device (0000 -> 0003)
[  124.476635] [drm] initializing kernel modesetting (NAVI14 0x1002:0x7340 0x1DA2:0xE421 0xC5).
[  124.476650] amdgpu 0002:23:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
[  124.476695] [drm] register mmio base: 0xF2200000
[  124.476701] [drm] register mmio size: 524288
[  124.476731] [drm] PCIE atomic ops is not supported
[  124.478522] [drm] add ip block number 0 <nv_common>
[  124.478530] [drm] add ip block number 1 <gmc_v10_0>
[  124.478538] [drm] add ip block number 2 <navi10_ih>
[  124.478557] [drm] add ip block number 3 <psp>
[  124.478573] [drm] add ip block number 4 <smu>
[  124.478588] [drm] add ip block number 5 <gfx_v10_0>
[  124.478603] [drm] add ip block number 6 <sdma_v5_0>
[  124.478616] [drm] add ip block number 7 <vcn_v2_0>
[  124.478622] [drm] add ip block number 8 <jpeg_v2_0>
[  124.512534] amdgpu 0002:23:00.0: amdgpu: Fetched VBIOS from ROM BAR
[  124.512554] amdgpu: ATOM BIOS: 113-E4210SM-U03
[  124.512602] [drm] VCN decode is enabled in VM mode
[  124.512607] [drm] VCN encode is enabled in VM mode
[  124.512612] [drm] JPEG decode is enabled in VM mode
[  124.512639] [drm] GPU posting now...
[  124.512815] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[  124.512860] amdgpu 0002:23:00.0: BAR 2: releasing [mem 0x990000000-0x9901fffff 64bit pref]
[  124.512872] amdgpu 0002:23:00.0: BAR 0: releasing [mem 0x980000000-0x98fffffff 64bit pref]
[  124.512981] pcieport 0002:22:00.0: BAR 9: releasing [mem 0x980000000-0x997ffffff 64bit pref]
[  124.512994] pcieport 0002:21:00.0: BAR 9: releasing [mem 0x980000000-0x997ffffff 64bit pref]
[  124.513003] pcieport 0002:20:00.0: BAR 9: releasing [mem 0x980000000-0x997ffffff 64bit pref]
[  124.513044] pcieport 0002:20:00.0: BAR 9: no space for [mem size 0x300000000 64bit pref]
[  124.513054] pcieport 0002:20:00.0: BAR 9: failed to assign [mem size 0x300000000 64bit pref]
[  124.513069] pcieport 0002:21:00.0: BAR 9: no space for [mem size 0x300000000 64bit pref]
[  124.513078] pcieport 0002:21:00.0: BAR 9: failed to assign [mem size 0x300000000 64bit pref]
[  124.513093] pcieport 0002:22:00.0: BAR 9: no space for [mem size 0x300000000 64bit pref]
[  124.513102] pcieport 0002:22:00.0: BAR 9: failed to assign [mem size 0x300000000 64bit pref]
[  124.513117] amdgpu 0002:23:00.0: BAR 0: no space for [mem size 0x200000000 64bit pref]
[  124.513127] amdgpu 0002:23:00.0: BAR 0: failed to assign [mem size 0x200000000 64bit pref]
[  124.513143] amdgpu 0002:23:00.0: BAR 2: no space for [mem size 0x00200000 64bit pref]
[  124.513153] amdgpu 0002:23:00.0: BAR 2: failed to assign [mem size 0x00200000 64bit pref]
[  124.513166] pcieport 0002:20:00.0: PCI bridge to [bus 21-23]
[  124.513178] pcieport 0002:20:00.0:   bridge window [io  0x1000-0x1fff]
[  124.513191] pcieport 0002:20:00.0:   bridge window [mem 0xf2200000-0xf23fffff]
[  124.513208] pcieport 0002:20:00.0: PCI bridge to [bus 21-23]
[  124.513217] pcieport 0002:20:00.0:   bridge window [io  0x1000-0x1fff]
[  124.513226] pcieport 0002:20:00.0:   bridge window [mem 0xf2200000-0xf23fffff]
[  124.513235] pcieport 0002:20:00.0:   bridge window [mem 0x980000000-0x997ffffff 64bit pref]
[  124.513245] pcieport 0002:21:00.0: PCI bridge to [bus 22-23]
[  124.513259] pcieport 0002:21:00.0:   bridge window [io  0x1000-0x1fff]
[  124.513281] pcieport 0002:21:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[  124.513297] pcieport 0002:21:00.0:   bridge window [mem 0x980000000-0x997ffffff 64bit pref]
[  124.513324] pcieport 0002:22:00.0: PCI bridge to [bus 23]
[  124.513336] pcieport 0002:22:00.0:   bridge window [io  0x1000-0x1fff]
[  124.513357] pcieport 0002:22:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[  124.513374] pcieport 0002:22:00.0:   bridge window [mem 0x980000000-0x997ffffff 64bit pref]
[  124.513433] [drm] Not enough PCI address space for a large BAR.
[  124.513447] amdgpu 0002:23:00.0: BAR 0: assigned [mem 0x980000000-0x98fffffff 64bit pref]
[  124.513492] amdgpu 0002:23:00.0: BAR 2: assigned [mem 0x990000000-0x9901fffff 64bit pref]
[  124.513552] amdgpu 0002:23:00.0: amdgpu: VRAM: 8176M 0x0000008000000000 - 0x00000081FEFFFFFF (8176M used)
[  124.513564] amdgpu 0002:23:00.0: amdgpu: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
[  124.513575] [drm] Detected VRAM RAM=8176M, BAR=256M
[  124.513582] [drm] RAM width 128bits GDDR6
[  124.514182] [TTM] Zone  kernel: Available graphics memory: 1913096 KiB
[  124.514188] [TTM] Initializing pool allocator
[  124.514223] [TTM] Initializing DMA pool allocator
[  124.514358] [drm] amdgpu: 8176M of VRAM memory ready
[  124.514375] [drm] amdgpu: 2802M of GTT memory ready.
[  124.514391] [drm] GART: num cpu pages 131072, num gpu pages 131072
[  124.514595] [drm] PCIE GART of 512M enabled (table at 0x0000008000000000).
[  124.575879] [drm] Found VCN firmware Version ENC: 1.16 DEC: 5 VEP: 0 Revision: 4
[  124.575934] amdgpu 0002:23:00.0: amdgpu: Will use PSP to load VCN firmware
[  127.521038] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load kdb failed!
[  127.521880] [drm:psp_hw_init [amdgpu]] *ERROR* PSP firmware loading failed
[  127.522743] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* hw_init of IP block <psp> failed -22
[  127.522791] amdgpu 0002:23:00.0: amdgpu: amdgpu_device_ip_init failed
[  127.522812] amdgpu 0002:23:00.0: amdgpu: Fatal error during GPU init
[  127.523047] amdgpu 0002:23:00.0: amdgpu: amdgpu: finishing device.
#9

Probably related to the card model. GCN cards are known to be working but RDNA cards not so much. When I was playing with RISC-V development boards that does not have integrated GPUs, GCN cards boots just fine but I cannot use any RDNA cards on that yet.

#10

But this looks like bars problem.
It looks like pcie ranges are no properly set in dts

#11

Did you done anything special RADXA?
[https://twitter.com/theradxa/status/1592374915231289344/photo/1]

2 Likes
#12

Poor Jeff Geerling xD He has tried getting an external GPU to work on the Pi4 and failed miserably. Some unknown guy does it on the Rock5 in a few weeks

2 Likes
#13

@geerlingguy work maybe opened doors to this, I don’t think that all that effort was useless and sure was fun to watch.

#14

i think @geerlingguy had to struggle with the pi because of its wierd construction of its soc but the rock 5b is more standard

#15

Of course he did. I also found the idea very interesting and fun to watch. I also didn’t mean that the effort was useless.

#16

I’m trying to replicate this on my own Rock 5 model B … it would be nice to know how @radxa was able to do this. I’m still searching around for any way I can recompile the kernel with support for radeon or amdgpu drivers. Might try installing them from package too, to see if that just works. We’ll see.

5 Likes
#17

@geerlingguy i think with this beast of an sbc it will be not so much work like with the raspberry pi 4
(maybe its even faster if you compile the linux kernel on the rock 5b instead of your mac studio)

#18

You can build image with armbian and you can configure linux kernel build in armbian :slight_smile:

#19

I found instructions for recompiling the kernel here: https://wiki.radxa.com/Rock5/guide/build-kernel-on-5b

When I did that (enabling the ATI radeon graphics module), the kernel compiled okay, but when I tried switching to it, the Rock 5 would no longer boot (the blue LED would flash in a pattern with two blinks, and no display output would happen).

#20

@geerlingguy I know for your videos/reviews you’d probably prefer to work with Radxa’s image, but if you want to get it to work I’d suggest working it with the Armbian kernel and build platform.

3 Likes
#21

If it is flashing with two blinks the kernel is probably loaded but the GPU failed to initialize. You can use an uart serial to see what is happening.

#22

thats never a bad idea hooking up a serial cable

#23

Hi all, I see the discussion of using external GPU on the RK3588 / Rock 5 B in this thread and in this other thread https://forum.radxa.com/t/adventures-with-external-gpu-card-gen-3-pcie/16315.

Has there been any progress since, @mtx512rk, @sharelter, @RadxaYuntian / @jack , anyone?

Also, the post above from December 2022 and the post with a rotating cube, do they show the Radeon on an RK3588 - so someone actually got it to work - or are they showing something else?

Adventures with external GPU card + Gen 3 pcie
#24

I started experimenting with the RK3568 regarding external GPU (Nvidia RTX 3090) and quickly came to an end with the 32mb pcie memory.
Reading through the Technical Reference Manuals for RK3568/88 and checking with the collabora/mainline dtsi the additional 1GB pcie memory from RK3588 is now present and should in theory overcome the bar space issues. Nobody seems to have publicly experimented in that regard but I will post my results as soon as my Rock5B gets finally delivered in the next 2-3 weeks.

Until then keep an eye on https://github.com/HeyMeco/Rockchip-pcie-devices where I will post results as soon as my Rock5B arrives.

1 Like
#25

To save you time, there is little chance of the RTX 3090 working unless you can modify the Nvidia drivers. The RK3568/88 pcie is crippled as per this thread and most graphic cards rely on pcie snoop feature. Rockchip decided not to implement (purchase) the IP. Therefore the gpu driver would need some workarounds to get it working, Furthermore you may need additional hacks in userspace applications as the dma buffers needed be within the 32 bit range (ie below 4GB). So it can be a long and arduous job debugging why an application isn’t rendering correctly. If you still want to test then try with a 4GB board to avoid the 32 bit buffer issue although it negates having a 64bit cpu.

2 Likes
#26

@mtx512rk Bit late but I still wanted to actually try it. Ofc it didn’t work like you stated with the prior issues but at least I have it documented now: https://github.com/HeyMeco/Rockchip-pcie-devices/issues/2
@geerlingguy got it as far as with the RPI’s

#27

Radxa Yautian the fans dont forget… yous gotta deliver whats promised… Can or Will it run Crysis ? :pensive::crossed_fingers:t2: