Rock 3A extremely low performance in GStreamer

#9

What is the bottleneck? Is it the CPU? Userspace or the kernel? Use a profiling tool, either a generic tool like perf or something gstreamer-specific to find what is preventing it from running faster. Even just looking at the CPU usage could be helpful.

But if you don’t mind compiling gstreamer (or at least the gst-plugins-base subproject), applying this patch might help. It at least made things work better for me.

diff --git a/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglmemorypbo.c b/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglmemorypbo.c
index 13fdba3..5d629c0 100644
--- a/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglmemorypbo.c
+++ b/subprojects/gst-plugins-base/gst-libs/gst/gl/gstglmemorypbo.c
@@ -187,6 +187,8 @@ _gl_mem_create (GstGLMemoryPBO * gl_mem, GError ** error)
   GstGLContext *context = gl_mem->mem.mem.context;
   GstGLBaseMemoryAllocatorClass *alloc_class;
 
+  return TRUE;
+
   alloc_class = GST_GL_BASE_MEMORY_ALLOCATOR_CLASS (parent_class);
   if (!alloc_class->create ((GstGLBaseMemory *) gl_mem, error))
     return FALSE;
#10

Since last time I had to reinstall the OS and now I can’t install mpp at all.
So I can’t check anything anymore.

The following packages have unmet dependencies:
 gstreamer1.0-rockchip1 : Depends: librockchip-mpp1 but it is not installable
E: Unable to correct problems, you have held broken packages.

Btw. what would be the pipeline for CSI? I can’t get OV5647 to change resolution and framerate.
Interfacing with the sensor itself works but it always defaults to 2592x1944 @15 FPS.
I don’t think v4l2 is meant to work here.

#11

Search for ov5647 on Rock 3, there you find how to change resolution and framerate using v4l2-ctl.

#12

Doesn’t work for me since I need 720p@30 in OpenCV.
I don’t see anything about it on this forum.

v4l2-ctl does seem to change the parameters but it doesn’t matter since they reset after opening VideoCapture.

#13

That’s why i provided this: Rock 3A Camera support

camera-mode = <0>;
or
camera-mode = <1>;

Edit:

maybe you should also use:

vid_capture.set(CAP_PROP_FRAME_WIDTH, capture_width);
vid_capture.set(CAP_PROP_FRAME_HEIGHT, capture_height);
vid_capture.set(CAP_PROP_FPS, framerate);

I think the modern way (v4l2) is:

cv::VideoCapture cap;
cap.open(index, cv::CAP_V4L2);
if (!cap.isOpened())
{
    std::cerr << "***Could not initialize capturing...***" << std::endl;
    return -1;
}
cap.set(cv::CAP_PROP_FRAME_WIDTH, capture_width);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, capture_height);
cap.set(cv::CAP_PROP_FPS, framerate);
#14

This certainly is not a simple topic.
But I don’t know what am I supposed to do with what you linked.

By default OpenCV opens CSI camera with GStreamer.
It doesn’t get opened if you specify V4L2 API (opening by index).
And if it does (/dev/video-camera0) then set method returns False so that doesn’t work. Wish it was that simple.

#15

I don’t have the board with me right now, but try this pipeline and see if you can get an output:

sudo gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,width=1280,height=720, framerate=30/1 ! xvimagesink

Note that you need to find the correct /dev/videoX node with v4l2-ctl.

#16

No, no. You misunderstood me.

What you’ve just written is the most basic pipeline which I had been trying to get it to work for weeks.
But I don’t have the knowledge to go deeper than installing packages or editing device tree overlays in accordance to instructions. And so there’s no packages or instructions for this.

What I meant by “not knowing what to do with what you linked” is that - ok, you’ve written some patch that allowed you to make v4l2 able to change resolution and framerate. I don’t understand what that patch is, what it’s for and where to put it (if I even have to). There was also something about editing drivers… I can’t really do that because I don’t know where and what to change and how to load the changes back.

Long story short - the pipeline has never worked.
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.

#17

I see.

I don’t know what skills you have. And i don’t know if gstreamer works or not, it should. You need to tell the sensor which mode (resolution/framerate) prior to calling any v4l2 function.
You do that with this set of function:

media-ctl -d /dev/media0 --set-v4l2 ‘“ov5647 5-0036”:0[fmt:SGBRG10_1X10/2592x1944]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[fmt:SGBRG10_1X10/2592x1944]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[crop:(0,0)/2592x1944]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:2[crop:(0,0)/2592x1944]’
v4l2-ctl -d /dev/video0 --set-selection=target=crop,top=0,left=0,width=2592,height=1944

media-ctl -d /dev/media0 --set-v4l2 ‘“ov5647 5-0036”:0[fmt:SGBRG10_1X10/1920x1080]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[fmt:SGBRG10_1X10/1920x1080]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[crop:(0,0)/1920x1080]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:2[crop:(0,0)/1920x1080]’
v4l2-ctl -d /dev/video0 --set-selection=target=crop,top=0,left=0,width=1920,height=1080

media-ctl -d /dev/media0 --set-v4l2 ‘“ov5647 5-0036”:0[fmt:SGBRG10_1X10/640x480]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[fmt:SGBRG10_1X10/640x480]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:0[crop:(0,0)/640x480]’
media-ctl -d /dev/media0 --set-v4l2 ‘“rkisp-isp-subdev”:2[crop:(0,0)/640x480]’
v4l2-ctl -d /dev/video0 --set-selection=target=crop,top=0,left=0,width=640,height=480

And then you call gstreamer for 480x640 or it will not work. The patch is supposed to free you from calling this set of functions every time it is reset. If you set it for 640x480 and call gstreamer with 1280x720, you get an error.

If this is basic for you, ignore everything i have written.

Edit:
I am not sure the driver support 1280x720, try 1920x1080 just to check what you get.

#18

Ok. So this is the current state of things.

Pipeline with the resolution of 1280x720 opens with framerate set at 15 FPS.
It just “corrects” the resolution to 1280x1080 from what I can see, which is not a big deal.
However v4l2-ctl reported the sensor can do half its original resolution so 1296x972 at 40-something FPS. But it’s impossible to set with any method known by me.

And I need the higher framerates.
P.S. Your commands don’t work (first two kinds: invalid argument, second two: syntax errors with ‘(’).

#19
#20

Ok. The commands got accepted, but they had no effects at all.
I must have 720p@30.

#21

Let’s check if you can have 720p in this mode.
what is the output of:

sudo v4l2-ctl --device /dev/video0 --stream-mmap=4 --stream-count=1 --stream-skip=150 --set-fmt-video=width=1280,height=720,pixelformat=NV12

/dev/video0 or any other /dev/videoX possible.

I looked at the driver and seems only these resolutions are available in all modes:

640x480@60fps
2560x1920@?fps
1920x1080@30fps
2592x1944@15fps
1296x972@45fps (in theory)

Maybe one way to get 720p is to resize the grabbed image (1080p) with gstreamer using hw acceleration.

#22

Yes. 1296x972 would be the best alternative. I could work with that.
Even though the datasheet of the sensor says it should be able to do 720p@60.

The output is:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.65 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.83 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.89 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.92 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.94 fps

resize the grabbed image (1080p) with gstreamer using hw acceleration.

Ah, yes, the near-mythical hardware acceleration on these boards…
Would’ve been cool if it were possible for a mere mortal to enable it.

#23

I had a good laugh here.

Not long ago, there was ‘rgaconvert’ property in gstreamer, i think it would do that. It was removed.
I am not a gstreamer advanced user, so i did not care much…

videoscale ! video/x-raw,width=1280,height=720 !
#24

I’ll just crop it in OpenCV.
But the problem still remains that 1296x972@45 doesn’t want to work for some reason.
And to be honest neither does any other resolution.
They all return “Internal data stream error” in GStreamer.

The only thing that works is 2592x1944@15 because that’s the default.
Correction: that resolution works by default but it also can’t be set explicitly in GStreamer.

#25

Whats is your OpenCV version? And distro?

#26

4.6.0 built from source. Distro is Radxa’s Ubuntu Focal. 4.19.193-65-rockchip-gf1279cb7a052.
What’s the importance here though? It’s GStreamer that doesn’t cooperate.

#27

You would get the frames using v4l2, with no gstreamer involved. But you would still need to convert NV12 to BGR i think.

If it was 4.5d i could try something like that, or if you disclose how you built 4.6.0 i could have a try.

#28

I’d just like it to work. Whatever the method.
Other problems are for later.

Btw. any chance libcamera would work here?
I tried compiling it here but there were permission errors during installation.