ROCK 5B Debug Party Invitation

[avaf]

Running an old glmark2-es2-drm (1920x1080) is capped at 60 FPS, no idea how to override vsync, unfortunately. The new version could not run with drm.

glmark2-es2-drm 
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Using display 0x556d312c80 with EGL version 0.0
EGL Version "1.4 Valhall-"g6p0-01eac0""
EGL Vendor "ARM"
EGL Extensions "EGL_NV_context_priority_realtime EGL_KHR_partial_update EGL_KHR_config_attribs EGL_KHR_image EGL_KHR_image_base EGL_KHR_fence_sync EGL_KHR_wait_sync EGL_KHR_gl_colorspace EGL_KHR_get_all_proc_addresses EGL_IMG_context_priority EGL_KHR_no_config_context EGL_EXT_image_dma_buf_import EGL_EXT_image_dma_buf_import_modifiers EGL_EXT_yuv_surface EGL_EXT_pixel_format_float EGL_ARM_pixmap_multisample_discard EGL_KHR_gl_texture_2D_image EGL_KHR_gl_renderbuffer_image EGL_KHR_create_context EGL_KHR_surfaceless_context EGL_KHR_gl_texture_cubemap_image EGL_EXT_image_gl_colorspace EGL_EXT_create_context_robustness"
=======================================================
    glmark2 2021.02
=======================================================
    OpenGL Information
    GL_VENDOR:     ARM
    GL_RENDERER:   Mali-LODX
    GL_VERSION:    OpenGL ES 3.2 v1.g6p0-01eac0.3b670429c215b5ca848ad5d6b11ce5bc
=======================================================
[build] use-vbo=false: FPS: 60 FrameTime: 16.667 ms
[build] use-vbo=true: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=nearest: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=linear: FPS: 60 FrameTime: 16.667 ms
[texture] texture-filter=mipmap: FPS: 60 FrameTime: 16.667 ms
[shading] shading=gouraud: FPS: 60 FrameTime: 16.667 ms
[shading] shading=blinn-phong-inf: FPS: 60 FrameTime: 16.667 ms
[shading] shading=phong: FPS: 60 FrameTime: 16.667 ms
[shading] shading=cel: FPS: 60 FrameTime: 16.667 ms
[bump] bump-render=high-poly: FPS: 60 FrameTime: 16.667 ms
[bump] bump-render=normals: FPS: 60 FrameTime: 16.667 ms
[bump] bump-render=height: FPS: 60 FrameTime: 16.667 ms
[effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 60 FrameTime: 16.667 ms
[effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 60 FrameTime: 16.667 ms
[pulsar] light=false:quads=5:texture=false: FPS: 60 FrameTime: 16.667 ms
[desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 60 FrameTime: 16.667 ms
[desktop] effect=shadow:windows=4: FPS: 60 FrameTime: 16.667 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 60 FrameTime: 16.667 ms
[buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 60 FrameTime: 16.667 ms
[buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 60 FrameTime: 16.667 ms
[ideas] speed=duration: FPS: 60 FrameTime: 16.667 ms
[jellyfish] <default>: FPS: 60 FrameTime: 16.667 ms
[terrain] <default>: FPS: 59 FrameTime: 16.949 ms
[shadow] <default>: FPS: 60 FrameTime: 16.667 ms
[refract] <default>: FPS: 60 FrameTime: 16.667 ms
[conditionals] fragment-steps=0:vertex-steps=0: FPS: 60 FrameTime: 16.667 ms
[conditionals] fragment-steps=5:vertex-steps=0: FPS: 60 FrameTime: 16.667 ms
[conditionals] fragment-steps=0:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
[function] fragment-complexity=low:fragment-steps=5: FPS: 60 FrameTime: 16.667 ms
[function] fragment-complexity=medium:fragment-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
[loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 60 FrameTime: 16.667 ms
=======================================================
                                  glmark2 Score: 59 
=======================================================

regarding SDL2, the problem was that i was using HDMI2 and it had problems finding geometry, using HDMI1 is just fine.

[avaf]

Looks like everybody is on discord now? Anyway, here is the Radxa camera. IMX415.
I believe they changed the connector if you look at the last batch image. I am figuring out how things work, i will post here if i have any progress.

Connector:

Radxa Camera:

Detecting the camera:

[    4.984014] rockchip-csi2-dphy-hw fedc0000.csi2-dphy0-hw: csi2 dphy hw probe successfully!
[    4.984175] rockchip-csi2-dphy csi2-dphy0: csi2 dphy0 probe successfully!
[    5.874168] rkcif rkcif-mipi-lvds2: Entity type for entity rockchip-mipi-csi2 was not initialized!
[    5.874176] rockchip-mipi-csi2: Async registered subdev
[    5.874183] rockchip-mipi-csi2: probe success, v4l2_dev:rkcif-mipi-lvds2!
[    5.874546] rkisp_hw fdcb0000.rkisp: Adding to iommu group 13
[    5.874689] rkisp_hw fdcb0000.rkisp: is_thunderboot: 0
[    5.874694] rkisp_hw fdcb0000.rkisp: max input:0x0@0fps
[    5.874699] rkisp_hw fdcb0000.rkisp: Missing rockchip,grf property
[    5.874770] rkisp_hw fdcb0000.rkisp: no find phandle sram
[    5.875034] rkisp rkisp0-vir0: rkisp driver version: v01.09.00
[    5.875157] rkisp rkisp0-vir0: Entity type for entity rkisp-isp-subdev was not initialized!
[    5.951036] imx415 3-001a: driver version: 00.01.07
[    5.951058] imx415 3-001a:  Get hdr mode failed! no hdr default
[    5.951183] imx415 3-001a: Failed to get power-gpios
[    5.951195] imx415 3-001a: could not get default pinstate
[    5.951205] imx415 3-001a: could not get sleep pinstate
[    5.951217] imx415 3-001a: Looking up dvdd-supply from device tree
[    5.951233] imx415 3-001a: Looking up dvdd-supply property in node /i2c@feab0000/imx415@1a failed
[    5.951856] imx415 3-001a: supply dvdd not found, using dummy regulator
[    5.951905] imx415 3-001a: Looking up dovdd-supply from device tree
[    5.951911] imx415 3-001a: Looking up dovdd-supply property in node /i2c@feab0000/imx415@1a failed
[    5.951919] imx415 3-001a: supply dovdd not found, using dummy regulator
[    5.951935] imx415 3-001a: Looking up avdd-supply from device tree
[    5.951939] imx415 3-001a: Looking up avdd-supply property in node /i2c@feab0000/imx415@1a failed
[    5.951946] imx415 3-001a: supply avdd not found, using dummy regulator
[    6.011551] imx415 3-001a: Detected imx415 id 0000e0
[    6.011622] rockchip-csi2-dphy csi2-dphy0: dphy0 matches m00_b_imx415 3-001a:bus type 5
[    6.013247] rkisp0-vir0: Async subdev notifier completed
[    6.258507] rkisp rkisp0-vir0: clear unready subdev num: 0
[    9.388385] imx415 3-001a: set exposure(shr0) 2047 = cur_vts(2250) - val(203)

First attempt:

[  287.556278] rkcif-mipi-lvds2: stream[1] start streaming
[  287.556285] rkcifhw fdce0000.rkcif: Only support one master device, master device count 0
[  287.557065] rkcif-mipi-lvds2: Allocate dummy buffer, size: 0x007e9000
[  287.557087] rkcif-mipi-lvds2: can not find output format: 0x32315559
[  287.557103] rockchip-mipi-csi2 fdd30000.mipi2-csi2: stream on, src_sd: 00000000ca9c3ea3, sd_name:rockchip-csi2-dphy0
[  287.557115] rockchip-mipi-csi2 fdd30000.mipi2-csi2: stream ON
[  287.557130] rockchip-csi2-dphy0: dphy0, data_rate_mbps 892
[  287.557150] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:1, dphy0
[  287.557151] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:1, dphy0
[  287.557154] imx415 3-001a: s_stream: 1. 3864x2192, hdr: 0, bpp: 10
[  287.617600] imx415 3-001a: set vblank 0x3a vts 2250
[  287.618922] imx415 3-001a: set exposure(shr0) 2047 = cur_vts(2250) - val(203)
[  287.619810] imx415 3-001a: set analog gain 0x0
[  292.946868] rkcif-mipi-lvds2: stream[1] start stopping, total mode 0x1, cur 0x1
[  293.953300] rockchip-mipi-csi2 fdd30000.mipi2-csi2: stream off, src_sd: 00000000ca9c3ea3, sd_name:rockchip-csi2-dphy0
[  293.953336] rockchip-mipi-csi2 fdd30000.mipi2-csi2: stream OFF
[  293.954388] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream_stop stream stop, dphy0
[  293.954396] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:0, dphy0
[  293.954417] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:0, dphy0
[  293.954427] imx415 3-001a: s_stream: 0. 3864x2192, hdr: 0, bpp: 10
[  294.012515] rkcif-mipi-lvds2: stream[1] stopping finished

[avaf]

video dev: (incomplete due to space constraint)

/dev/video0
Driver Info:
	Driver name      : rkcif
	Card type        : rkcif
	Bus info         : platform:rkcif-mipi-lvds2
	Driver version   : 5.10.66
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkcif
	Model            : rkcif-mipi-lvds2
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000002
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : stream_cif_mipi_id0
	Function         : V4L2 I/O
	Pad 0x01000004   : 0: Sink
	  Link 0x02000043: from remote pad 0x100002f of entity 'rockchip-mipi-csi2': Data, Enabled
	  Link 0x02000059: from remote pad 0x1000030 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200006f: from remote pad 0x1000031 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000085: from remote pad 0x1000032 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200009b: from remote pad 0x1000033 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000b1: from remote pad 0x1000034 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000c7: from remote pad 0x1000035 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000dd: from remote pad 0x1000036 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000f3: from remote pad 0x1000037 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000109: from remote pad 0x1000038 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200011f: from remote pad 0x1000039 of entity 'rockchip-mipi-csi2': Data
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'NV16' (Y/CbCr 4:2:2)
	[1]: 'NV61' (Y/CrCb 4:2:2)
	[2]: 'NV12' (Y/CbCr 4:2:0)
	[3]: 'NV21' (Y/CrCb 4:2:0)
	[4]: 'YUYV' (YUYV 4:2:2)
	[5]: 'YVYU' (YVYU 4:2:2)
	[6]: 'UYVY' (UYVY 4:2:2)
	[7]: 'VYUY' (VYUY 4:2:2)
	[8]: 'RGB3' (24-bit RGB 8-8-8)
	[9]: 'RGBP' (16-bit RGB 5-6-5)
	[10]: 'BGRH' (18-bit BGRX 6-6-6-14)
	[11]: 'RGGB' (8-bit Bayer RGRG/GBGB)
	[12]: 'GRBG' (8-bit Bayer GRGR/BGBG)
	[13]: 'GBRG' (8-bit Bayer GBGB/RGRG)
	[14]: 'BA81' (8-bit Bayer BGBG/GRGR)
	[15]: 'RG10' (10-bit Bayer RGRG/GBGB)
	[16]: 'BA10' (10-bit Bayer GRGR/BGBG)
	[17]: 'GB10' (10-bit Bayer GBGB/RGRG)
	[18]: 'BG10' (10-bit Bayer BGBG/GRGR)
	[19]: 'RG12' (12-bit Bayer RGRG/GBGB)
	[20]: 'BA12' (12-bit Bayer GRGR/BGBG)
	[21]: 'GB12' (12-bit Bayer GBGB/RGRG)
	[22]: 'BG12' (12-bit Bayer BGBG/GRGR)
	[23]: 'BYR2' (16-bit Bayer BGBG/GRGR)
	[24]: 'GB16' (16-bit Bayer GBGB/RGRG)
	[25]: 'GR16' (16-bit Bayer GRGR/BGBG)
	[26]: 'RG16' (16-bit Bayer RGRG/GBGB)
	[27]: 'Y16 ' (16-bit Greyscale)
	[28]: 'GREY' (8-bit Greyscale)
	[29]: 'EBD8' (Embedded data 8-bit)
	[30]: 'SPD6' (Shield pix data 16-bit)
	[31]: 'Y12 ' (12-bit Greyscale)
	[32]: 'Y10 ' (10-bit Greyscale)
===============
/dev/video1
Driver Info:
	Driver name      : rkcif
	Card type        : rkcif
	Bus info         : platform:rkcif-mipi-lvds2
	Driver version   : 5.10.66
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkcif
	Model            : rkcif-mipi-lvds2
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000006
	Type             : V4L Video
Entity Info:
	ID               : 0x00000005 (5)
	Name             : stream_cif_mipi_id1
	Function         : V4L2 I/O
	Pad 0x01000008   : 0: Sink
	  Link 0x02000045: from remote pad 0x100002f of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200005b: from remote pad 0x1000030 of entity 'rockchip-mipi-csi2': Data, Enabled
	  Link 0x02000071: from remote pad 0x1000031 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000087: from remote pad 0x1000032 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200009d: from remote pad 0x1000033 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000b3: from remote pad 0x1000034 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000c9: from remote pad 0x1000035 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000df: from remote pad 0x1000036 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000f5: from remote pad 0x1000037 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200010b: from remote pad 0x1000038 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000121: from remote pad 0x1000039 of entity 'rockchip-mipi-csi2': Data
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'NV16' (Y/CbCr 4:2:2)
	[1]: 'NV61' (Y/CrCb 4:2:2)
	[2]: 'NV12' (Y/CbCr 4:2:0)
	[3]: 'NV21' (Y/CrCb 4:2:0)
	[4]: 'YUYV' (YUYV 4:2:2)
	[5]: 'YVYU' (YVYU 4:2:2)
	[6]: 'UYVY' (UYVY 4:2:2)
	[7]: 'VYUY' (VYUY 4:2:2)
	[8]: 'RGB3' (24-bit RGB 8-8-8)
	[9]: 'RGBP' (16-bit RGB 5-6-5)
	[10]: 'BGRH' (18-bit BGRX 6-6-6-14)
	[11]: 'RGGB' (8-bit Bayer RGRG/GBGB)
	[12]: 'GRBG' (8-bit Bayer GRGR/BGBG)
	[13]: 'GBRG' (8-bit Bayer GBGB/RGRG)
	[14]: 'BA81' (8-bit Bayer BGBG/GRGR)
	[15]: 'RG10' (10-bit Bayer RGRG/GBGB)
	[16]: 'BA10' (10-bit Bayer GRGR/BGBG)
	[17]: 'GB10' (10-bit Bayer GBGB/RGRG)
	[18]: 'BG10' (10-bit Bayer BGBG/GRGR)
	[19]: 'RG12' (12-bit Bayer RGRG/GBGB)
	[20]: 'BA12' (12-bit Bayer GRGR/BGBG)
	[21]: 'GB12' (12-bit Bayer GBGB/RGRG)
	[22]: 'BG12' (12-bit Bayer BGBG/GRGR)
	[23]: 'BYR2' (16-bit Bayer BGBG/GRGR)
	[24]: 'GB16' (16-bit Bayer GBGB/RGRG)
	[25]: 'GR16' (16-bit Bayer GRGR/BGBG)
	[26]: 'RG16' (16-bit Bayer RGRG/GBGB)
	[27]: 'Y16 ' (16-bit Greyscale)
	[28]: 'GREY' (8-bit Greyscale)
	[29]: 'EBD8' (Embedded data 8-bit)
	[30]: 'SPD6' (Shield pix data 16-bit)
	[31]: 'Y12 ' (12-bit Greyscale)
	[32]: 'Y10 ' (10-bit Greyscale)
===============
/dev/video10
Driver Info:
	Driver name      : rkcif
	Card type        : rkcif
	Bus info         : platform:rkcif-mipi-lvds2
	Driver version   : 5.10.66
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkcif
	Model            : rkcif-mipi-lvds2
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x0300002b
	Type             : V4L Video
Entity Info:
	ID               : 0x00000029 (41)
	Name             : rkcif_tools_id2
	Function         : V4L2 I/O
	Pad 0x0100002a   : 0: Sink
	  Link 0x02000057: from remote pad 0x100002f of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200006d: from remote pad 0x1000030 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000083: from remote pad 0x1000031 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000099: from remote pad 0x1000032 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000af: from remote pad 0x1000033 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000c5: from remote pad 0x1000034 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000db: from remote pad 0x1000035 of entity 'rockchip-mipi-csi2': Data
	  Link 0x020000f1: from remote pad 0x1000036 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000107: from remote pad 0x1000037 of entity 'rockchip-mipi-csi2': Data
	  Link 0x0200011d: from remote pad 0x1000038 of entity 'rockchip-mipi-csi2': Data
	  Link 0x02000133: from remote pad 0x1000039 of entity 'rockchip-mipi-csi2': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'RGGB' (8-bit Bayer RGRG/GBGB)
	[1]: 'GRBG' (8-bit Bayer GRGR/BGBG)
	[2]: 'GBRG' (8-bit Bayer GBGB/RGRG)
	[3]: 'BA81' (8-bit Bayer BGBG/GRGR)
	[4]: 'RG10' (10-bit Bayer RGRG/GBGB)
	[5]: 'BA10' (10-bit Bayer GRGR/BGBG)
	[6]: 'GB10' (10-bit Bayer GBGB/RGRG)
	[7]: 'BG10' (10-bit Bayer BGBG/GRGR)
	[8]: 'RG12' (12-bit Bayer RGRG/GBGB)
	[9]: 'BA12' (12-bit Bayer GRGR/BGBG)
	[10]: 'GB12' (12-bit Bayer GBGB/RGRG)
	[11]: 'BG12' (12-bit Bayer BGBG/GRGR)
===============
/dev/video11
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_mainpath
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000007
	Type             : V4L Video
Entity Info:
	ID               : 0x00000006 (6)
	Name             : rkisp_mainpath
	Function         : V4L2 I/O
	Pad 0x01000009   : 0: Sink
	  Link 0x0200000a: from remote pad 0x1000004 of entity 'rkisp-isp-subdev': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'UYVY' (UYVY 4:2:2)
	[1]: '422P' (Planar YUV 4:2:2)
	[2]: 'NV16' (Y/CbCr 4:2:2)
	[3]: 'NV61' (Y/CrCb 4:2:2)
	[4]: 'YM16' (Planar YUV 4:2:2 (N-C))
	[5]: 'NV21' (Y/CrCb 4:2:0)
	[6]: 'NV12' (Y/CbCr 4:2:0)
	[7]: 'NM21' (Y/CrCb 4:2:0 (N-C))
	[8]: 'NM12' (Y/CbCr 4:2:0 (N-C))
	[9]: 'YU12' (Planar YUV 4:2:0)
	[10]: 'YM24' (Planar YUV 4:4:4 (N-C))
	[11]: 'RGGB' (8-bit Bayer RGRG/GBGB)
	[12]: 'GRBG' (8-bit Bayer GRGR/BGBG)
	[13]: 'GBRG' (8-bit Bayer GBGB/RGRG)
	[14]: 'BA81' (8-bit Bayer BGBG/GRGR)
	[15]: 'RG10' (10-bit Bayer RGRG/GBGB)
	[16]: 'BA10' (10-bit Bayer GRGR/BGBG)
	[17]: 'GB10' (10-bit Bayer GBGB/RGRG)
	[18]: 'BG10' (10-bit Bayer BGBG/GRGR)
	[19]: 'RG12' (12-bit Bayer RGRG/GBGB)
	[20]: 'BA12' (12-bit Bayer GRGR/BGBG)
	[21]: 'GB12' (12-bit Bayer GBGB/RGRG)
	[22]: 'BG12' (12-bit Bayer BGBG/GRGR)
===============
/dev/video12
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_selfpath
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x0300000d
	Type             : V4L Video
Entity Info:
	ID               : 0x0000000c (12)
	Name             : rkisp_selfpath
	Function         : V4L2 I/O
	Pad 0x0100000f   : 0: Sink
	  Link 0x02000010: from remote pad 0x1000004 of entity 'rkisp-isp-subdev': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'UYVY' (UYVY 4:2:2)
	[1]: '422P' (Planar YUV 4:2:2)
	[2]: 'NV16' (Y/CbCr 4:2:2)
	[3]: 'NV61' (Y/CrCb 4:2:2)
	[4]: 'YM16' (Planar YUV 4:2:2 (N-C))
	[5]: 'NV21' (Y/CrCb 4:2:0)
	[6]: 'NV12' (Y/CbCr 4:2:0)
	[7]: 'NM21' (Y/CrCb 4:2:0 (N-C))
	[8]: 'NM12' (Y/CbCr 4:2:0 (N-C))
	[9]: 'YU12' (Planar YUV 4:2:0)
	[10]: 'YM24' (Planar YUV 4:4:4 (N-C))
	[11]: 'GREY' (8-bit Greyscale)
	[12]: 'XR24' (32-bit BGRX 8-8-8-8)
	[13]: 'RGBP' (16-bit RGB 5-6-5)
	[14]: 'FBCG' (Rockchip fbc gain)
===============
/dev/video13
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_fbcpath
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000013
	Type             : V4L Video
Entity Info:
	ID               : 0x00000012 (18)
	Name             : rkisp_fbcpath
	Function         : V4L2 I/O
	Pad 0x01000015   : 0: Sink
	  Link 0x02000016: from remote pad 0x1000004 of entity 'rkisp-isp-subdev': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'FBC0' (Rockchip yuv420sp fbc encoder)
	[1]: 'FBC2' (Rockchip yuv422sp fbc encoder)
===============
/dev/video14
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_iqtool
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04201000
		Video Capture Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000019
	Type             : V4L Video
Entity Info:
	ID               : 0x00000018 (24)
	Name             : rkisp_iqtool
	Function         : V4L2 I/O
	Pad 0x0100001b   : 0: Sink
	  Link 0x0200001c: from remote pad 0x1000004 of entity 'rkisp-isp-subdev': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

	[0]: 'UYVY' (UYVY 4:2:2)
	[1]: '422P' (Planar YUV 4:2:2)
	[2]: 'NV16' (Y/CbCr 4:2:2)
	[3]: 'NV61' (Y/CrCb 4:2:2)
	[4]: 'YM16' (Planar YUV 4:2:2 (N-C))
	[5]: 'NV21' (Y/CrCb 4:2:0)
	[6]: 'NV12' (Y/CbCr 4:2:0)
	[7]: 'NM21' (Y/CrCb 4:2:0 (N-C))
	[8]: 'NM12' (Y/CbCr 4:2:0 (N-C))
	[9]: 'YU12' (Planar YUV 4:2:0)
	[10]: 'YM24' (Planar YUV 4:4:4 (N-C))
	[11]: 'RGGB' (8-bit Bayer RGRG/GBGB)
	[12]: 'GRBG' (8-bit Bayer GRGR/BGBG)
	[13]: 'GBRG' (8-bit Bayer GBGB/RGRG)
	[14]: 'BA81' (8-bit Bayer BGBG/GRGR)
	[15]: 'RG10' (10-bit Bayer RGRG/GBGB)
	[16]: 'BA10' (10-bit Bayer GRGR/BGBG)
	[17]: 'GB10' (10-bit Bayer GBGB/RGRG)
	[18]: 'BG10' (10-bit Bayer BGBG/GRGR)
	[19]: 'RG12' (12-bit Bayer RGRG/GBGB)
	[20]: 'BA12' (12-bit Bayer GRGR/BGBG)
	[21]: 'GB12' (12-bit Bayer GBGB/RGRG)
	[22]: 'BG12' (12-bit Bayer BGBG/GRGR)
===============
/dev/video15
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_rawrd0_m
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x0300001f
	Type             : V4L Video
Entity Info:
	ID               : 0x0000001e (30)
	Name             : rkisp_rawrd0_m
	Function         : V4L2 I/O
	Pad 0x01000021   : 0: Source
	  Link 0x02000022: to remote pad 0x1000002 of entity 'rkisp-isp-subdev': Data
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

===============
/dev/video16
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_rawrd2_s
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000025
	Type             : V4L Video
Entity Info:
	ID               : 0x00000024 (36)
	Name             : rkisp_rawrd2_s
	Function         : V4L2 I/O
	Pad 0x01000027   : 0: Source
	  Link 0x02000028: to remote pad 0x1000002 of entity 'rkisp-isp-subdev': Data
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

===============
/dev/video17
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp_rawrd1_l
	Bus info         : platform:rkisp0-vir0
	Driver version   : 1.9.0
	Capabilities     : 0x84202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04202000
		Video Output Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x0300002b
	Type             : V4L Video
Entity Info:
	ID               : 0x0000002a (42)
	Name             : rkisp_rawrd1_l
	Function         : V4L2 I/O
	Pad 0x0100002d   : 0: Source
	  Link 0x0200002e: to remote pad 0x1000002 of entity 'rkisp-isp-subdev': Data
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture Multiplanar

===============
/dev/video18
Driver Info:
	Driver name      : rkisp_v6
	Card type        : rkisp-statistics
	Bus info         : platform: rkisp
	Driver version   : 1.9.0
	Capabilities     : 0x84a00000
		Metadata Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04a00000
		Metadata Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : rkisp0-vir0
	Model            : rkisp0
	Serial           : 
	Bus info         : 
	Media version    : 5.10.66
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.10.66
Interface Info:
	ID               : 0x03000032
	Type             : V4L Video
Entity Info:
	ID               : 0x00000030 (48)
	Name             : rkisp-statistics
	Function         : V4L2 I/O
	Pad 0x01000031   : 0: Sink
	  Link 0x02000034: from remote pad 0x1000005 of entity 'rkisp-isp-subdev': Data, Enabled
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

===============

[tkaiser]

Looks like everybody is on discord now?

“the debug party has essentially ended and some of us realise this”

[avaf]

ouch, so it is about to be released then?

The good thing is that i think i have hdmi in working, haha…

[tkaiser]

The good thing is that i think i have hdmi in working

Well, I would appreciate sharing your findings/proceedings here (and not in some hidden/closed Discord channel).

BTW: When I added IMX415 info to my review I claimed ‘8MP camera able to stream 4K @ 90 frames/sec’ based on SONY material. Is that correct?

[avaf]

I grabbed a frame from the camera, asked 1920x1080 nv12 and receive this:
-rw-rw-r-- 1 rock rock 12441600 Aug 7 16:03 frame_1920x1080.nv12

i need to be sure about what i am really getting. I expect 4k@30 with current driver, but the size is wrong.
v4l2-ctl --device /dev/video0 --stream-mmap=4 --stream-count=1 --stream-skip=150 --set-fmt-video=width=1920,height=1080,pixelformat=NV12
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps

[avaf]

3840x2160 * 3 / 2 = 12441600 = 4k@30

[avaf]

You can update you review, current driver can stream 4k@30fps.

Some additional info so far: i could not make camera engine works properly, and current driver delivers only ‘4k’ frames. Passing wrong frame size freeze the board.

I need to find out how to set up the engine correctly, so i can experiment with a few other things:

video device: /dev/video0
width: 3840 x height: 2160
INFO: libSDL: compiled with=2.0.20 linked against=2.0.20
INFO: Renderer Driver (default): (null)
INFO: Renderer Driver (set): opengles2
SDL information:
    SDL_FRAMEBUFFER_ACCELERATION: (null)
    SDL_RENDER_DRIVER: opengles2
    SDL_RENDER_OPENGL_SHADERS: (null)
    SDL_RENDER_LOGICAL_SIZE_MODE: (null)
    SDL_RENDER_SCALE_QUALITY: (null)
    SDL_RENDER_VSYNC: (null)
    SDL_VIDEO_HIGHDPI_DISABLED: (null)
    SDL_VIDEO_WIN_D3DCOMPILER: (null)
    SDL_VIDEO_WINDOW_SHARE_PIXEL_FORMAT: (null)
    SDL_VIDEO_DOUBLE_BUFFER: (null)
videodevice: /dev/video0 (final)
Device information:
  Device path:  /dev/video0
Stream settings:
  Frame format: NV12
  Frame size:   3840x2160
Unable to set frame rate: Inappropriate ioctl for device
Unable to read out current frame rate: Inappropriate ioctl for device
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
frame rate: 30.0601

Done, beautiful image, imx415 4k@30fps rendered on 1920x1080 with kmsdrm. led light bulb here.

Update:
Currently all frame sizes works at 30 fps on my tests.

Tested so far:

1280x768@30fps
160x120@30fps
1920x1080@30fps
320x240@30fps
3840x2160@30fps
640x480@30fps
800x600@30fps

Apparently, the stream is always at 3840x2160, and the engine scales the image and enhances it for the other sizes. getting a frame of size 3840x2160 takes 9% CPU usage, and the 1280x768 takes 13% CPU. There is a mode for 1920x1080 in hdr.

[icecream95]

Trying to rmmod bifrost_kbase when building it as a module causes a kernel panic:

[  104.686695][ T2260] Kernel panic - not syncing: panic_on_set_domain set ...
[  104.687322][ T2260] CPU: 2 PID: 2260 Comm: rmmod Not tainted 5.10.66-ixn-97906-gc78a1e33c719 #6
[  104.688089][ T2260] Hardware name: Radxa ROCK 5B (DT)
[  104.688534][ T2260] Call trace:
[  104.688829][ T2260]  dump_backtrace+0x0/0x1e0
[  104.689221][ T2260]  show_stack+0x1c/0x24
[  104.689581][ T2260]  dump_stack_lvl+0xc4/0xe8
[  104.689971][ T2260]  dump_stack+0x14/0x50
[  104.690328][ T2260]  panic+0x170/0x3a4
[  104.690665][ T2260]  rockchip_pd_power+0x4e4/0x5dc
[  104.691091][ T2260]  rockchip_pd_power_on+0x28/0x30
[  104.691526][ T2260]  _genpd_power_on+0xbc/0x15c
[  104.691928][ T2260]  genpd_power_on+0xac/0x1a0
[  104.692330][ T2260]  genpd_runtime_resume+0x9c/0x240
[  104.692776][ T2260]  __rpm_callback+0x90/0x154
[  104.693177][ T2260]  rpm_callback+0x28/0x8c
[  104.693556][ T2260]  rpm_resume+0x51c/0x780
[  104.693934][ T2260]  __pm_runtime_resume+0x40/0x90
[  104.694360][ T2260]  __device_release_driver+0x3c/0x230
[  104.694828][ T2260]  driver_detach+0xc4/0x150
[  104.695219][ T2260]  bus_remove_driver+0x60/0xe0
[  104.695631][ T2260]  driver_unregister+0x34/0x60
[  104.696045][ T2260]  platform_driver_unregister+0x18/0x20
[  104.696791][ T2260]  kbase_platform_driver_exit+0x18/0x217c [bifrost_kbase]
[  104.697410][ T2260]  __arm64_sys_delete_module+0x198/0x264
[  104.697903][ T2260]  el0_svc_common.constprop.0+0x80/0x230
[  104.698394][ T2260]  do_el0_svc+0x28/0x90
[  104.698751][ T2260]  el0_svc+0xc/0x14
[  104.699086][ T2260]  el0_sync_handler+0xe0/0x110
[  104.699498][ T2260]  el0_sync+0x158/0x180
[  104.699862][ T2260] SMP: stopping secondary CPUs
[  105.866945][ T2260] SMP: failed to stop secondary CPUs 0-7

This makes Panfrost development a bit tricker, though I probably could find workarounds to keep using kbase.

[icecream95]

It appears that GPU frequencies are a bit of a lie. Here are frequencies I measured, though it might be different for different chips:

devfreq	Actual MHz
300	330
400	423
500	536
600	637
700	703
800	780
900	889
1000	990

[avaf]

Update from my side, decoding is working with ffplay, i tested the following files:

bbb_sunflower_1080p_30fps_normal.mp4  
jellyfish-20-mbps-hd-hevc-10bit.mkv
jellyfish-120-mbps-4k-uhd-h264.mkv    
jellyfish-20-mbps-hd-hevc.mkv
jellyfish-20-mbps-hd-h264.mkv

OPS, i realized ffplay is using software decoding:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
   5823 rock      20   0 1861860 200304  63700 S 195.0   1.2   0:43.55 ffplay 

Encoding with gstreamer also works, but encoding h264 from a camera stream (3840x2160) results in 5 fps (playback on my old intel box).

H264:
gst-launch-1.0 v4l2src device=/dev/video11 io-mode=dmabuf ! 'video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1' ! mpph264enc ! filesink location=test_3840x2160_30fps_h264.mp4

H265:
gst-launch-1.0 v4l2src device=/dev/video11 io-mode=dmabuf ! 'video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1' ! mpph265enc ! filesink location=test_3840x2160_30fps_h265.mkv

For some reason, my intel box (ancient) displays it at 3 fps on 1080p. So maybe my intel box could not handle the 4k h264/h265 files properly.

Playing the h265 file in Rock5B with 1080p is just fine:

ffplay -i test_3840x2160_30fps_h265.mkv 
ffplay version git-2022-05-25-73d7bc2 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-librga --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-shared --enable-decoder='aac,ac3,flac' --extra-cflags=-I/usr/src/linux-headers-5.10.66-rk3588/include
  libavutil      57.  7.100 / 57.  7.100
  libavcodec     59. 12.100 / 59. 12.100
  libavformat    59.  8.100 / 59.  8.100
  libavdevice    59.  0.101 / 59.  0.101
  libavfilter     8. 16.100 /  8. 16.100
  libswscale      6.  1.100 /  6.  1.100
  libswresample   4.  0.100 /  4.  0.100
  libpostproc    56.  0.100 / 56.  0.100
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
[hevc @ 0x7f94000c10] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, hevc, from 'test_3840x2160_30fps_h265.mkv':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv), 3840x2160, 30 fps, 30 tbr, 1200k tbn
    nan M-V:    nan fd=   9 aq=    0KB vq=    0KB sq=    0B f=0/0

[willy]

I don’t understand why you’re speaking about the rock4, I don’t have it and never spoke about it. Confused.

[Dante4]

@willy, Nvm, I’m messed with your photo, just forget i said anything, sorry

[willy]

That’s interesting. @tkaiser and I noticed the same for the CPU frequencies. Lower ones measure slightly higher than advertised and higher ones measure a bit lower. Typically my CPU claims to run at 2304 MHz but measures 2267. It’s as if instead of cheating by making all upper frequencies equal, they would now progressively dampen them so that it’s harder to detect.

[Dante4]

Rock5b IS working with PCIe switch (in my case 8port U.2 nvme board) on radxa ubuntu focal!

[Allen.Smithee]

inspiring your M.2 > PCIe x16 > sff-8643 x8 > adapter nvme
each ssf-8643 can give 4 lane sata3.

I plan to try M.2 > sff-8643 > PCIe x16 > 4x1 bifurcation > JMB585 x2 > SSD Sata3

or 2x2 bifurcation > JMB585 x 2 > SSD Sata3

I’m a little afraid of the sff-8643 to the pciex16…

[Dante4]

Also working with 4 NVMe switch board ANM24PE16. So a quick test regarding it with fio (only 30s and 4k io=32 and 64k io=32). All NVMe formatted as 4k logical.

root@rock-5b:/home/rock# lsscsi
[N:0:1:1]    disk    KINGSTON SKC2500M81000G__1                 /dev/nvme0n1
[N:1:1:1]    disk    KINGSTON SKC2500M81000G__1                 /dev/nvme1n1
[N:2:4:1]    disk    Samsung SSD 983 DCT M.2 960GB__1           /dev/nvme2n1
[N:3:4:1]    disk    Samsung SSD 983 DCT M.2 960GB__1           /dev/nvme3n1

Single NVMe (n1 + n2) via switch 4k read

4k io=32 1 thread, random read:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=4k -iodepth=32 -rw=randread -filename=/dev/nvme1n1
read: IOPS=205k, BW=800MiB/s (838MB/s)(23.4GiB/30001msec)
nvme1n1: ios=6110887/0, merge=0/0, ticks=100603/0, in_queue=100602, util=99.68%

4k io=32 4 threads, random read:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=4k -iodepth=32 -rw=randread -filename=/dev/nvme1n1
Jobs (all jobs): 4 (f=4): [r(4)][100.0%][r=1624MiB/s][r=416k IOPS]
nvme1n1: ios=12407866/0, merge=0/0, ticks=3120570/0, in_queue=3120569, util=99.96%

4k io=32 1 threads, random read:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=4k -iodepth=32 -rw=randread -filename=/dev/nvme2n1
read: IOPS=204k, BW=796MiB/s (834MB/s)(23.3GiB/30001msec)
 nvme2n1: ios=6081630/0, merge=0/0, ticks=144942/0, in_queue=144942, util=99.69%

4k io=32 4 threads, random read:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=4k -iodepth=32 -rw=randread -filename=/dev/nvme2n1
Jobs (all jobs): 4 (f=4): [r(4)][100.0%][r=2225MiB/s][r=570k IOPS][eta 00m:00s]
nvme2n1: ios=16851241/0, merge=0/0, ticks=3599192/0, in_queue=3599192, util=99.89%

Single NVMe (n1 + n2) via switch 4k write

4k io=32 1 thread, random write:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=4k -iodepth=32 -rw=randwrite -filename=/dev/nvme1n1
 write: IOPS=171k, BW=667MiB/s (700MB/s)(19.6GiB/30001msec); 0 zone resets
nvme1n1: ios=17/5100863, merge=0/0, ticks=1/137856, in_queue=137857, util=99.82%

4k io=32 4 threads, random write:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=4k -iodepth=32 -rw=randwrite -filename=/dev/nvme1n1
Jobs (All jobs): 4 (f=4): [w(4)][100.0%][w=954MiB/s][w=244k IOPS][eta 00m:00s]
nvme1n1: ios=14/7506401, merge=0/0, ticks=0/3006631, in_queue=3006631, util=99.77%

4k io=32 1 threads, random write:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=4k -iodepth=32 -rw=randwrite -filename=/dev/nvme2n1
 write: IOPS=190k, BW=742MiB/s (778MB/s)(21.7GiB/30001msec); 0 zone resets
 nvme2n1: ios=13/5673605, merge=0/0, ticks=1/73403, in_queue=73404, util=99.66%

4k io=32 4 threads, random write:

fio -name=rndr4k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=4k -iodepth=32 -rw=randwrite -filename=/dev/nvme2n1
Jobs (All jobs): 4 (f=4): [w(4)][100.0%][w=1191MiB/s][w=305k IOPS][eta 00m:00s]
nvme2n1: ios=17/8963788, merge=0/0, ticks=1/3680680, in_queue=3680681, util=99.74%

Single NVMe (n1 + n2) via switch 64k read

64k io=32 1 threads, seq read:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=64k -iodepth=32 -rw=read -filename=/dev/nvme1n1
  read: IOPS=47.6k, BW=2974MiB/s (3118MB/s)(87.1GiB/30001msec)
nvme1n1: ios=1422056/0, merge=0/0, ticks=941038/0, in_queue=941038, util=99.78%

64k io=32 4 threads, seq read:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=4k -iodepth=32 -rw=read -filename=/dev/nvme1n1
 Jobs (all jobs): 4 (f=4): [R(4)][100.0%][r=2972MiB/s][r=47.6k IOPS][eta 00m:00s]
 nvme1n1: ios=1105233/0, merge=0/0, ticks=3568212/0, in_queue=3568212, util=99.93%

64k io=32 1 threads, seq read:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=64k -iodepth=32 -rw=read -filename=/dev/nvme2n1
read: IOPS=45.7k, BW=2857MiB/s (2996MB/s)(83.7GiB/30001msec)
nvme2n1: ios=1366948/0, merge=0/0, ticks=933056/0, in_queue=933057, util=99.73%

64k io=32 4 threads, seq read:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=64k -iodepth=32 -rw=read -filename=/dev/nvme2n1
Jobs (All jobs): 4 (f=4): [R(4)][100.0%][r=2293MiB/s][r=36.7k IOPS][eta 00m:00s]
nvme2n1: ios=1247837/0, merge=0/0, ticks=3762737/0, in_queue=3762738, util=99.80%

Single NVMe (n1 + n2) via switch 64k write

64k io=32 1 threads, seq write:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=64k -iodepth=32 -rw=write -filename=/dev/nvme1n1
write: IOPS=29.7k, BW=1856MiB/s (1946MB/s)(54.4GiB/30001msec); 0 zone resets
nvme1n1: ios=14/886685, merge=0/0, ticks=1/857459, in_queue=857460, util=99.76%

64k io=32 4 threads, seq write:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=64k -iodepth=32 -rw=write -filename=/dev/nvme1n1
Jobs (All jobs): 4 (f=4): [W(4)][100.0%][w=1912MiB/s][w=30.6k IOPS][eta 00m:00s]
 nvme1n1: ios=42/912157, merge=0/0, ticks=2/3250556, in_queue=3250559, util=99.88%

64k io=32 1 threads, seq write:

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=1 -bs=64k -iodepth=32 -rw=write -filename=/dev/nvme2n1
write: IOPS=19.5k, BW=1217MiB/s (1276MB/s)(35.6GiB/30002msec); 0 zone resets
nvme2n1: ios=45/581677, merge=0/0, ticks=4/924309, in_queue=924313, util=99.88%

64k io=32 4 threads, seq write (basically, Samsung 983 is 1.2GiB on write by Samsung design):

fio -name=seqw64k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=64k -iodepth=32 -rw=write -filename=/dev/nvme2n1
Jobs (all jobs): 4 (f=4): [W(4)][100.0%][w=1126MiB/s][w=18.0k IOPS][eta 00m:00s]
nvme2n1: ios=13/580539, merge=0/0, ticks=1/3760516, in_queue=3760517, util=99.78%

And a bit of max perfomance from pcie 3.0 x4 and CPU:

mdadm raid0 4 NVMe 128k seq read

128k io=32 4 threads, seq read:

mdadm --create /dev/md0 --verbose --level=0 --raid-devices=4 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
fio -name=seq128k32 -ioengine=libaio -direct=1 -buffered=0 -invalidate=1 -runtime=30 -numjobs=4 -bs=128k -iodepth=32 -rw=read -filename=/dev/md0
 Jobs: 4 (f=4): [R(4)][100.0%][r=3133MiB/s][r=25.1k IOPS][eta 00m:00s]
md0: ios=749024/0, merge=0/0, ticks=3745024/0, in_queue=3745024, util=99.94%, aggrios=187942/0, aggrmerge=0/0, aggrticks=940406/0, aggrin_queue=940406, aggrutil=99.49%
nvme0n1: ios=187951/0, merge=0/0, ticks=2691964/0, in_queue=2691965, util=99.49%
nvme3n1: ios=187936/0, merge=0/0, ticks=124958/0, in_queue=124958, util=99.46%
nvme2n1: ios=187937/0, merge=0/0, ticks=123346/0, in_queue=123346, util=99.46%
nvme1n1: ios=187947/0, merge=0/0, ticks=821358/0, in_queue=821358, util=99.48%

[Allen.Smithee]

did you note the values ​​of the Cpu line in the end of task log?

[Dante4]

Not really, that was just for quick tests.
I will redo them anyway after I recompile kernel to add support for my 10G nic