ROCK 5B Debug Party Invitation

#651

@avaf Wondering if you fancy trying the NPU
dmesg is complaining with

[    6.676382] RKNPU fdab0000.npu: Adding to iommu group 0
[    6.676613] RKNPU fdab0000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[    6.676744] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[    6.677595] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[    6.678270] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdab0000-0xfdabffff]
[    6.678294] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdac0000-0xfdacffff]
[    6.678312] RKNPU fdab0000.npu: can't request region for resource [mem 0xfdad0000-0xfdadffff]
[    6.680414] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[    6.680949] RKNPU fdab0000.npu: Looking up mem-supply from device tree
[    6.682047] RKNPU fdab0000.npu: leakage=8
[    6.682101] RKNPU fdab0000.npu: Looking up rknpu-supply from device tree
[    6.693838] RKNPU fdab0000.npu: pvtm=859
[    6.699208] RKNPU fdab0000.npu: pvtm-volt-sel=2
[    6.700393] RKNPU fdab0000.npu: avs=0
[    6.700643] RKNPU fdab0000.npu: l=10000 h=85000 hyst=5000 l_limit=0 h_limit=800000000 h_table=0
[    6.716728] RKNPU fdab0000.npu: failed to find power_model node
[    6.716743] RKNPU fdab0000.npu: RKNPU: failed to initialize power model
[    6.716756] RKNPU fdab0000.npu: RKNPU: failed to get dynamic-coefficient

Kernel config needs to enable CONFIG_ROCKCHIP_RKNPU_SRAM=y

#
# RKNPU
#
CONFIG_ROCKCHIP_RKNPU=y
CONFIG_ROCKCHIP_RKNPU_DEBUG_FS=y
# CONFIG_ROCKCHIP_RKNPU_PROC_FS is not set
# CONFIG_ROCKCHIP_RKNPU_FENCE is not set
CONFIG_ROCKCHIP_RKNPU_DRM_GEM=y
# end of RKNPU

I really don’t know about above if correct or legacy with rknpu & not rknpu2

In the dtb we have

		rknpu = "/npu@fdab0000";
		npu_opp_table = "/npu-opp-table";
		rknpu_mmu = "/iommu@fdab9000";

	npu@fdab0000 {
		compatible = "rockchip,rk3588-rknpu";
		reg = <0x00 0xfdab0000 0x00 0x10000 0x00 0xfdac0000 0x00 0x10000 0x00 0xfdad0000 0x00 0x10000>;
		interrupts = <0x00 0x6e 0x04 0x00 0x6f 0x04 0x00 0x70 0x04>;
		interrupt-names = "npu0_irq\0npu1_irq\0npu2_irq";
		clocks = <0x0e 0x06 0x02 0x12d 0x02 0x122 0x02 0x124 0x02 0x12e 0x02 0x123 0x02 0x125 0x02 0x131>;
		clock-names = "clk_npu\0aclk0\0aclk1\0aclk2\0hclk0\0hclk1\0hclk2\0pclk";
		assigned-clocks = <0x0e 0x06>;
		assigned-clock-rates = <0xbebc200>;
		resets = <0x02 0x1e6 0x02 0x1b0 0x02 0x1c0 0x02 0x1e8 0x02 0x1b2 0x02 0x1c2>;
		reset-names = "srst_a0\0srst_a1\0srst_a2\0srst_h0\0srst_h1\0srst_h2";
		power-domains = <0x4e 0x09 0x4e 0x0a 0x4e 0x0b>;
		power-domain-names = "npu0\0npu1\0npu2";
		operating-points-v2 = <0x97>;
		iommus = <0x98>;
		status = "okay";
		rknpu-supply = <0x99>;
		mem-supply = <0x99>;
		phandle = <0x218>;
	};

	npu-opp-table {
		compatible = "operating-points-v2";
		nvmem-cells = <0x9a>;
		nvmem-cell-names = "leakage";
		rockchip,pvtm-voltage-sel = <0x00 0x32f 0x00 0x330 0x343 0x01 0x344 0x35c 0x02 0x35d 0x375 0x03 0x376 0x38e 0x04 0x38f 0x270f 0x05>;
		rockchip,pvtm-pvtpll;
		rockchip,pvtm-offset = <0x50>;
		rockchip,pvtm-sample-time = <0x44c>;
		rockchip,pvtm-freq = <0xc3500>;
		rockchip,pvtm-volt = <0xb71b0>;
		rockchip,pvtm-ref-temp = <0x19>;
		rockchip,pvtm-temp-prop = <0xffffff8f 0xffffff8f>;
		rockchip,pvtm-thermal-zone = "npu-thermal";
		clocks = <0x02 0x12a>;
		clock-names = "pclk";
		rockchip,grf = <0x9b>;
		volt-mem-read-margin = <0xd0bd8 0x01 0xbac48 0x02 0xa4cb8 0x03 0x78d98 0x04>;
		low-volt-read-margin = <0x04>;
		intermediate-threshold-freq = <0x7a120>;
		rockchip,init-freq = <0xf4240>;
		rockchip,temp-hysteresis = <0x1388>;
		rockchip,low-temp = <0x2710>;
		rockchip,low-temp-min-volt = <0xb71b0>;
		rockchip,high-temp = <0x14c08>;
		rockchip,high-temp-max-freq = <0xc3500>;
		phandle = <0x97>;

		opp-300000000 {
			opp-hz = <0x00 0x11e1a300>;
			opp-microvolt = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
			opp-microvolt-L1 = <0xa7d8c 0xa7d8c 0xcf850 0xa7d8c 0xa7d8c 0xcf850>;
			opp-microvolt-L2 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L3 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L4 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L5 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
		};

		opp-400000000 {
			opp-hz = <0x00 0x17d78400>;
			opp-microvolt = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
			opp-microvolt-L1 = <0xa7d8c 0xa7d8c 0xcf850 0xa7d8c 0xa7d8c 0xcf850>;
			opp-microvolt-L2 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L3 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L4 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L5 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
		};

		opp-500000000 {
			opp-hz = <0x00 0x1dcd6500>;
			opp-microvolt = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
			opp-microvolt-L1 = <0xa7d8c 0xa7d8c 0xcf850 0xa7d8c 0xa7d8c 0xcf850>;
			opp-microvolt-L2 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L3 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L4 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L5 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
		};

		opp-600000000 {
			opp-hz = <0x00 0x23c34600>;
			opp-microvolt = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
			opp-microvolt-L1 = <0xa7d8c 0xa7d8c 0xcf850 0xa7d8c 0xa7d8c 0xcf850>;
			opp-microvolt-L2 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L3 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L4 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L5 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
		};

		opp-700000000 {
			opp-hz = <0x00 0x29b92700>;
			opp-microvolt = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
			opp-microvolt-L3 = <0xa7d8c 0xa7d8c 0xcf850 0xa7d8c 0xa7d8c 0xcf850>;
			opp-microvolt-L4 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
			opp-microvolt-L5 = <0xa4cb8 0xa4cb8 0xcf850 0xa4cb8 0xa4cb8 0xcf850>;
		};

		opp-800000000 {
			opp-hz = <0x00 0x2faf0800>;
			opp-microvolt = <0xb71b0 0xb71b0 0xcf850 0xb71b0 0xb71b0 0xcf850>;
			opp-microvolt-L2 = <0xb40dc 0xb40dc 0xcf850 0xb40dc 0xb40dc 0xcf850>;
			opp-microvolt-L3 = <0xb1008 0xb1008 0xcf850 0xb1008 0xb1008 0xcf850>;
			opp-microvolt-L4 = <0xadf34 0xadf34 0xcf850 0xadf34 0xadf34 0xcf850>;
			opp-microvolt-L5 = <0xaae60 0xaae60 0xcf850 0xaae60 0xaae60 0xcf850>;
		};

		opp-900000000 {
			opp-hz = <0x00 0x35a4e900>;
			opp-microvolt = <0xc3500 0xc3500 0xcf850 0xc3500 0xc3500 0xcf850>;
			opp-microvolt-L1 = <0xc042c 0xc042c 0xcf850 0xc042c 0xc042c 0xcf850>;
			opp-microvolt-L2 = <0xbd358 0xbd358 0xcf850 0xbd358 0xbd358 0xcf850>;
			opp-microvolt-L3 = <0xba284 0xba284 0xcf850 0xba284 0xba284 0xcf850>;
			opp-microvolt-L4 = <0xb71b0 0xb71b0 0xcf850 0xb71b0 0xb71b0 0xcf850>;
			opp-microvolt-L5 = <0xb40dc 0xb40dc 0xcf850 0xb40dc 0xb40dc 0xcf850>;
		};

		opp-1000000000 {
			opp-hz = <0x00 0x3b9aca00>;
			opp-microvolt = <0xcf850 0xcf850 0xcf850 0xcf850 0xcf850 0xcf850>;
			opp-microvolt-L1 = <0xcc77c 0xcc77c 0xcf850 0xcc77c 0xcc77c 0xcf850>;
			opp-microvolt-L2 = <0xc96a8 0xc96a8 0xcf850 0xc96a8 0xc96a8 0xcf850>;
			opp-microvolt-L3 = <0xc65d4 0xc65d4 0xcf850 0xc65d4 0xc65d4 0xcf850>;
			opp-microvolt-L4 = <0xc3500 0xc3500 0xcf850 0xc3500 0xc3500 0xcf850>;
			opp-microvolt-L5 = <0xc042c 0xc042c 0xcf850 0xc042c 0xc042c 0xcf850>;
		};
	};

		syssram = "/sram@ff001000";

	sram@ff001000 {
		compatible = "mmio-sram";
		reg = <0x00 0xff001000 0x00 0xef000>;
		#address-cells = <0x01>;
		#size-cells = <0x01>;
		ranges = <0x00 0x00 0xff001000 0xef000>;
		phandle = <0x2a8>;

		rkvdec-sram@0 {
			reg = <0x00 0x78000>;
			phandle = <0xac>;
		};

		rkvdec-sram@78000 {
			reg = <0x78000 0x77000>;
			phandle = <0xae>;
		};
	};

But this is not in the dtb

    rknpu_sram: rknpu_sram@0 {
        reg = <0x0 0xef000>; // 956KB
    };

Maybe with the info in rknpu2 this one is easier than some of the other problems?

#652

I don’t have CONFIG_ROCKCHIP_RKNPU_SRAM enabled. npu output is the same as mine.
I will try these NPU SRAM instructions this weekend…

1 Like
#653

The problem with ffmpeg is that it is trying to use a pixel format with RGA that isn’t supported by RGA3… actually that isn’t the problem, the problem is that RGA is being used at all.

The RGA is used to convert from YUV semi-planar to YUV planar, but that is not very optimal if the buffer is required for display… then the chain becomes:

[VDPU]→ YCbCr semi-planar —[RGA2]→ YCbCr planar —[memcpy]→ YCbCr planar, GPU buffer —[GPU]→ RGBA linear.

Much more efficient would be this:

[VDPU]→ AFBC YCbCr —[GPU]→ AFBC RGBA,

or even going from the video decoder directly to an overlay plane on the display. That does require a lot more plumbing though, and I don’t know how easily it fits into ffmpeg’s architecture.

However, even the inefficient pipeline involving the unnecessary blit and memcpy is fast enough for 4k 30fps, but perhaps not much more. I may attempt to get it working, but it might require some pretty hacky code.

#654

I get that too, and have had no problems with the NPU. (Well, apart from the mess that is Rockchip userspace, and insecure kernel code…)

Using SRAM might improve performance a bit, but it isn’t required.

#655

This does not explain why it works for the board with 4GB RAM and not on 16 GB RAM.
If you can have a look and fix it i don’t care if a hack code is added. Rockchip dropped support to ffmpeg anyway.

There is support for AFBC on gstreamer, i tried to use rkximagesink but got a black screen. I need to patch the stock gstreamer i have and see if i can get it right.

#656

I don’t know icecream as ‘test.py’ seems to run but I have been watching /sys/devices/platform/fdab0000.npu/devfreq/fdab0000.npu/load and it doesn’t shift?
Never seems to shift from 100@1000000000Hz

Is it using the npu?

#657

Wait, so did Rockchip use devfreq for rknpu but then never actually set the usage?

*sigh*

The NPU has three cores, and the load for each is calculated per core, but then that data is never used to pass a total load to the devfreq code.

To see if a program is at least trying to submit commands to the NPU, you can try something like this:

strace -e ioctl ./rknn_create_mem_demo model.rknn image.png |& grep '0x64, 0x41, 0x68'

#658

Okay, I’ve got a fix…

It should apply on top of the ffmpeg patches already used in @amazingfate’s PPA.

Playing 4k 24fps video takes about 10% of a single A76 core, which is not as good as it could be, but much better than >50% on all cores with software decode.

Note that permissions must be adjusted on some device files for this to work:

sudo chmod a+rw /dev/dma_heap /dev/dma_heap/system-dma32 /dev/dma_heap/system-uncached-dma32

I also had a shot at adding AV1 support, but it doesn’t work for some reason…

1 Like
#659

Yeah I should of said it is running but have doubts if correct should of said as if I molest the demo it gives some pretty curious results.
I did a rough hack to estimate fps but 1st thing is how much variance from 130 - 200 fps on a batch of a 1000
ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)

183.34
194.06
176.54

ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0)

133.09
150.23
157.08

Which has me scratching my head as surely 3 cores vs 1 would have a much better improvement. I guess maybe this is OK as a Pi4 from https://pytorch.org/tutorials/intermediate/realtime_rpi.html can get approx 10 fps but its a pretty useless comparison (I did misread at 1st and thought it 100fps)
I don’t do much with the smaller image models as my interest has been mainly speech recognition, but there seems to be more load than I expected and also the variance in FPS would also seem curious.
I am not sure if things are right to be honest even if I do run strace.

core 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x64, 0x41, 0x68), 0x7fc66978b0) = 0
all cores ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x64, 0x41, 0x68), 0x7fdc06d310) = 0

/home/rock/rknn-toolkit2/rknn_toolkit_lite2/examples/inference_with_lite/test.py

    # run on RK356x/RK3588 with Debian OS, do not need specify target.
    if host_name == 'RK3588':
        print(host_name)
        #ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0)
        ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
    else:
        ret = rknn_lite.init_runtime()
    if ret != 0:
        print('Init runtime environment failed')
        exit(ret)
    print('done')

    # Inference
    starttime = time()
    print('--> Running model')
    for i in range(10000):
      outputs = rknn_lite.inference(inputs=[img])
      #show_top5(outputs, i)

    print('done')
    print(10000 / (time() - starttime))

If anyone can enable a kernel just to test and the dts entries from above please do.
Reading a bit more maybe the results are representative of a 6tops npu as guess the cpu load is relatively light even if more than expected, still trying to grasp how the cores should work as didn’t even realise it had cores as such and ran just as a npu. :slight_smile:

I cobbled together what should be a cpu resnet18 which only manages 4fps and stuggling to get a comparison. Prob will have to work on it a bit more.

from time import time
from torchvision.io import read_image
from torchvision.models import resnet18, ResNet18_Weights
from torchvision import transforms
import cv2
from PIL import Image

# convert opencv output from BGR to RGB
ori_img = cv2.imread('./space_shuttle_224.jpg')
img = cv2.cvtColor(ori_img, cv2.COLOR_BGR2RGB)

# Step 1: Initialize model with the best available weights
weights = ResNet18_Weights.DEFAULT
model = resnet18(weights=weights)
model.eval()

# Step 2: Initialize the inference transforms
preprocess = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# Step 3: Apply inference preprocessing transforms
batch = preprocess(img).unsqueeze(0)

# Step 4: Use the model and print the predicted category
starttime = time()
print('--> Running model')
for i in range(100):
    prediction = model(batch).squeeze(0).softmax(0)
    
print('done')
print(100 / (time() - starttime))

class_id = prediction.argmax().item()
score = prediction[class_id].item()
category_name = weights.meta["categories"][class_id]
print(f"{category_name}: {100 * score:.1f}%")
#660

I applied your patch to my ffmpeg version and still breaks on rga, can you please share a link to @amazingfate ffmpeg code and patches?

arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Input #0, matroska,webm, from '/home/rock/Videos/jellyfish-5-mbps-hd-hevc.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 4729 kb/s
  Stream #0:0(und): Video: hevc (Main), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-04 22:13:45
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
[hevc_rkmpp @ 0x7f94009230] Decoder noticed an info change (1920x1080), format=0
[hevc_rkmpp @ 0x7f94009230] calling rkmpp_get_buffer2
[hevc_rkmpp @ 0x7f94009230] Allocated buffer of size 3133471 -> 0x7f84000e20 (CPU 0x7f8bd02000)
rga_api version 1.8.1_[4]
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[20, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 2304, 1080, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f8bd02000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[hevc_rkmpp @ 0x7f94009230] RGA blit failed
[hevc_rkmpp @ 0x7f94009230] Doing slow software conversion
[hevc_rkmpp @ 0x7f94009230] calling rkmpp_get_buffer2    0B f=0/0   
[hevc_rkmpp @ 0x7f94009230] Allocated buffer of size 3133471 -> 0x7f940246b0 (CPU 0x7f83501000)
#661

It is about the RGA driver, it checks if > 4G and bounces, so i wonder about the code below, in my ffmpeg version.

pic->buf[0] = av_buffer_create(buffer->cpu, size[0],
                               rkmpp_buffer_cache, buffer, 0);


static inline bool rga_mm_check_memory_limit(struct rga_scheduler_t *scheduler, int mm_flag)
{
	if (!scheduler)
		return false;

	if (scheduler->data->mmu == RGA_MMU &&
	    !(mm_flag & RGA_MEM_UNDER_4G)) {
		pr_err("%s unsupported Memory larger than 4G!\n",
		       rga_get_mmu_type_str(scheduler->data->mmu));
		return false;
	}

	return true;
}


/* If it is within 0~4G, return 1 (true). */
static int rga_mm_check_range_sgt(struct sg_table *sgt)
{
	int i;
	struct scatterlist *sg;
	phys_addr_t s_phys = 0;

	for_each_sg(sgt->sgl, sg, sgt->orig_nents, i) {
		s_phys = sg_phys(sg);
		if ((s_phys > 0xffffffff) || (s_phys + sg->length > 0xffffffff))
			return 0;
	}

	return 1;
}

static inline int rga_mm_check_range_phys_addr(phys_addr_t paddr, size_t size)
{
	return ((paddr + size) <= 0xffffffff);
}
#662

https://ppa.launchpadcontent.net/liujianfeng1994/rockchip-multimedia/ubuntu/pool/main/f/ffmpeg/

Under debian/patches. Or you can just use https://github.com/JeffyCN/FFmpeg

#663

@nyanmisaka, Thanks for the work, but still have issues. Same with Jeffy 5.1.git version.

I am not cross-compiling (Debian). Can you check if the configure is missing something?
Can you share your output?

rock@rock5b:~/rockchip/ffmpeg/ffmpeg-4.4.2$ sudo ./ffplay ~/Videos/jellyfish-5-mbps-hd-hevc.mkv -vcodec h264_rkmpp -loglevel debug
ffplay version 4.4.2 Copyright © 2003-2021 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-librga --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-decoder=‘aac,ac3,flac’ --extra-cflags=-I/usr/src/linux-headers-5.10.110-debugx/include
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘5’.
Initialized opengles2 renderer.
[NULL @ 0x7f70000c00] Opening ‘/home/rock/Videos/jellyfish-5-mbps-hd-hevc.mkv’ for reading
[file @ 0x7f70001650] Setting default whitelist ‘file,crypto,data’
[matroska,webm @ 0x7f70000c00] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
[matroska,webm @ 0x7f70000c00] Before avformat_find_stream_info() pos: 905 bytes read:32768 seeks:0 nb_streams:1
[hevc @ 0x7f70003810] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7f70003810] Decoding VPS
[hevc @ 0x7f70003810] Main profile bitstream
[hevc @ 0x7f70003810] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7f70003810] Decoding SPS
[hevc @ 0x7f70003810] Main profile bitstream
[hevc @ 0x7f70003810] Decoding VUI
[hevc @ 0x7f70003810] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7f70003810] Decoding PPS
[matroska,webm @ 0x7f70000c00] All info found
[matroska,webm @ 0x7f70000c00] After avformat_find_stream_info() pos: 121987 bytes read:131689 seeks:0 frames:3
Input #0, matroska,webm, from ‘/home/rock/Videos/jellyfish-5-mbps-hd-hevc.mkv’:
Metadata:
COMPATIBLE_BRANDS: iso4hvc1iso6
MAJOR_BRAND : iso4
MINOR_VERSION : 1
ENCODER : Lavf56.3.100
Duration: 00:00:30.10, start: 0.067000, bitrate: 4729 kb/s
Stream #0:0(und), 3, 1/1000: Video: hevc (Main), 1 reference frame, yuv420p(tv, left), 1920x1080 [SAR 1:1 DAR 16:9], 0/1, 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
Metadata:
CREATION_TIME : 2016-02-04 22:13:45
LANGUAGE : und
HANDLER_NAME : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
[h264_mp4toannexb @ 0x7f70003e20] Global extradata truncated, corrupted stream or invalid MP4/AVCC bitstream
Failed to open file ‘/home/rock/Videos/jellyfish-5-mbps-hd-hevc.mkv’ or configure filtergraph
[AVIOContext @ 0x7f70009760] Statistics: 131689 bytes read, 0 seeks

#664

rock@rock5b:~/rockchip/ffmpeg/ffmpeg-4.4.2$ sudo ./ffplay ~/Videos/motogp.mp4 -vcodec h264_rkmpp -loglevel debug
ffplay version 4.4.2 Copyright © 2003-2021 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-librga --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-decoder=‘aac,ac3,flac’ --extra-cflags=-I/usr/src/linux-headers-5.10.110-debugx/include
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘5’.
Initialized opengles2 renderer.
[NULL @ 0x7f80000c00] Opening ‘/home/rock/Videos/motogp.mp4’ for reading
[file @ 0x7f80001640] Setting default whitelist ‘file,crypto,data’
[mpegts @ 0x7f80000c00] Format mpegts probed with size=8192 and score=98
[mpegts @ 0x7f80000c00] stream=0 stream_type=1b pid=100 prog_reg_desc=
[mpegts @ 0x7f80000c00] stream=1 stream_type=f pid=101 prog_reg_desc=
[mpegts @ 0x7f80000c00] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:2
[mpegts @ 0x7f80000c00] Continuity check failed for pid 4191 expected 2 got 1
Last message repeated 1 times
[h264 @ 0x7f80006b00] non-existing SPS 0 referenced in buffering period
[extract_extradata @ 0x7f800eda30] nal_unit_type: 9(AUD), nal_ref_idc: 0
[extract_extradata @ 0x7f800eda30] nal_unit_type: 6(SEI), nal_ref_idc: 0
Last message repeated 1 times
[extract_extradata @ 0x7f800eda30] nal_unit_type: 7(SPS), nal_ref_idc: 3
[extract_extradata @ 0x7f800eda30] nal_unit_type: 8(PPS), nal_ref_idc: 3
[extract_extradata @ 0x7f800eda30] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7f80006b00] nal_unit_type: 9(AUD), nal_ref_idc: 0
[h264 @ 0x7f80006b00] nal_unit_type: 6(SEI), nal_ref_idc: 0
Last message repeated 1 times
[h264 @ 0x7f80006b00] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f80006b00] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f80006b00] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7f80006b00] non-existing SPS 0 referenced in buffering period
[h264 @ 0x7f80006b00] Format yuv420p chosen by get_format().
[h264 @ 0x7f80006b00] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x7f80006b00] no picture
[mpegts @ 0x7f80000c00] probing stream 1 pp:2500B sq= 0B f=0/0
[mpegts @ 0x7f80000c00] Probe with size=1495, packets=1 detected aac with score=51
[mpegts @ 0x7f80000c00] probed stream 1
[mpegts @ 0x7f80000c00] max_analyze_duration 5000000 reached at 5000000 microseconds st:0
[mpegts @ 0x7f80000c00] Packet had TEI flag set; marking as corrupt
Last message repeated 1 times
[mpegts @ 0x7f80000c00] stream 0 : no PTS found at end of file, duration not set
[mpegts @ 0x7f80000c00] stream 1 : no PTS found at end of file, duration not set
[mpegts @ 0x7f80000c00] After avformat_find_stream_info() pos: 0 bytes read:3199120 seeks:3 frames:489
Input #0, mpegts, from ‘/home/rock/Videos/motogp.mp4’:
Duration: N/A, start: 220.111333, bitrate: N/A
Program 1
Stream #0:0[0x100], 252, 1/90000: Video: h264 (Main), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, progressive, topleft), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 0/1, 50 fps, 50 tbr, 90k tbn, 100 tbc
Stream #0:1[0x101], 237, 1/90000: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 194 kb/s
detected 8 logical cores
[ffplay_abuffer @ 0x7f805c18e0] Setting ‘sample_rate’ to value ‘48000’
[ffplay_abuffer @ 0x7f805c18e0] Setting ‘sample_fmt’ to value ‘fltp’
[ffplay_abuffer @ 0x7f805c18e0] Setting ‘channels’ to value ‘2’
[ffplay_abuffer @ 0x7f805c18e0] Setting ‘time_base’ to value ‘1/48000’
[ffplay_abuffer @ 0x7f805c18e0] Setting ‘channel_layout’ to value ‘0x3’
[ffplay_abuffer @ 0x7f805c18e0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[ffplay_abuffersink @ 0x7f806a8cf0] auto-inserting filter ‘auto_resampler_0’ between the filter ‘ffplay_abuffer’ and the filter ‘ffplay_abuffersink’
[AVFilterGraph @ 0x7f805c69b0] query_formats: 2 queried, 0 merged, 3 already done, 0 delayed
[auto_resampler_0 @ 0x7f806a0810] [SWR @ 0x7f800c9920] Using fltp internally between filters
[auto_resampler_0 @ 0x7f806a0810] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz
[h264_mp4toannexb @ 0x7f8063e1c0] The input looks like it is Annex B already
[h264_rkmpp @ 0x7f8051e5b0] Format yuv420p chosen by get_format().
[h264_rkmpp @ 0x7f8051e5b0] Initializing RKMPP decoder.
[h264_rkmpp @ 0x7f8051e5b0] RKMPP decoder initialized successfully.
[mpegts @ 0x7f80000c00] Continuity check failed for pid 4191 expected 2 got 1
Last message repeated 1 times
[NULL @ 0x7f80006b00] nal_unit_type: 7(SPS), nal_ref_idc: 3
[NULL @ 0x7f80006b00] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264_rkmpp @ 0x7f8051e5b0] Flush.
Audio frame changed from rate:48000 ch:2 fmt:fltp layout:stereo serial:-1 to rate:48000 ch:2 fmt:fltp layout:stereo serial:1
[h264_rkmpp @ 0x7f8051e5b0] Wrote 147459 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 12053 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 4797 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 20177 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Decoder noticed an info change (1920x1080), format=0
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[ffplay_abuffer @ 0x7f780057e0] Setting ‘sample_fmt’ to value ‘fltp’
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[ffplay_abuffer @ 0x7f780057e0] Setting ‘channels’ to value ‘2’
[ffplay_abuffer @ 0x7f780057e0] Setting ‘time_base’ to value ‘1/48000’
[ffplay_abuffer @ 0x7f780057e0] Setting ‘channel_layout’ to value ‘0x3’
[ffplay_abuffer @ 0x7f780057e0] tb:1/48000 samplefmt:fltp samplerate:48000 chlayout:0x3
[ffplay_abuffersink @ 0x7f78005df0] auto-inserting filter ‘auto_resampler_0’ between the filter ‘ffplay_abuffer’ and the filter ‘ffplay_abuffersink’
[AVFilterGraph @ 0x7f805c69b0] query_formats: 2 queried, 0 merged, 3 already done, 0 delayed
[auto_resampler_0 @ 0x7f78005f20] [SWR @ 0x7f78005fd0] Using fltp internally between filters
[auto_resampler_0 @ 0x7f78005f20] ch:2 chl:stereo fmt:fltp r:48000Hz -> ch:2 chl:stereo fmt:s16 r:48000Hz
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Wrote 22627 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Wrote 4832 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f807e6770 (CPU 0x7f7c2f9000)
rga_api version 1.8.1_[4]
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[21, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f7c2f9000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
Video frame changed from size:0x0 format:none serial:-1 to size:1920x1080 format:yuv420p serial:1
[ffplay_buffer @ 0x7f74026570] Setting ‘video_size’ to value ‘1920x1080’
[ffplay_buffer @ 0x7f74026570] Setting ‘pix_fmt’ to value ‘0’
[ffplay_buffer @ 0x7f74026570] Setting ‘time_base’ to value ‘1/90000’
[ffplay_buffer @ 0x7f74026570] Setting ‘pixel_aspect’ to value ‘1/1’
[ffplay_buffer @ 0x7f74026570] Setting ‘frame_rate’ to value ‘50/1’
[ffplay_buffer @ 0x7f74026570] w:1920 h:1080 pixfmt:yuv420p tb:1/90000 fr:50/1 sar:1/1
[AVFilterGraph @ 0x7f805226d0] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f74026760 (CPU 0x7f6c0fe000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[30, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f6c0fe000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f74026760 (CPU 0x7f6c0fe000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[27, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f6c0fe000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
Created 1920x1080 texture with SDL_PIXELFORMAT_IYUV.
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f74026760 (CPU 0x7f6c0fe000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[36, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f6c0fe000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f74026760 (CPU 0x7f6c0fe000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[33, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f6c0fe000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.69KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f74026760 (CPU 0x7f6c0fe000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[39, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f6c0fe000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Wrote 21154 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 5111 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 18160 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Wrote 4972 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Wrote 21382 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Wrote 5217 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f8051e5b0] Buffer full
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f807e8110 (CPU 0x7f8766e000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[38, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f8766e000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[h264_rkmpp @ 0x7f8051e5b0] Wrote 20266 bytes to decoder
[h264_rkmpp @ 0x7f8051e5b0] Received a frame.
[h264_rkmpp @ 0x7f8051e5b0] Allocated buffer of size 3139231 -> 0x7f807e9650 (CPU 0x7f86c6d000)
RgaBlit(1465) RGA_BLIT fail: Invalid argument
RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[30, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f86c6d000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8051e5b0] Doing slow software conversion
[AVIOContext @ 0x7f80009740] Statistics: 4313232 bytes read, 3 seeks

rock@rock5b:~/rockchip/ffmpeg/ffmpeg-4.4.2$

#665

@amazingfate , @icecream95, @nyanmisaka

I decided to install liujianfeng1994 deb packages on my ubuntu jammy, same errors. I think the deb packages do not contain @icecream95 patches.

rock@rock5b:~/deb/ffmpeg$ ffplay ~/videos/Big_Buck_Bunny_1080_10s_10MB.mp4 -vcodec h264_rkmpp
ffplay version 4.4.2-0ubuntu0.22.04.1+rkmpp1 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1+rkmpp1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libdrm --enable-librga --enable-rkmpp --enable-version3 --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/rock/videos/Big_Buck_Bunny_1080_10s_10MB.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
  Duration: 00:00:10.00, start: 0.000000, bitrate: 8387 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 8382 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
[h264_rkmpp @ 0x7f8c003530] Decoder noticed an info change (1920x1080), format=0
rga_api version 1.8.1_[4] 0 aq=    0KB vq=  867KB sq=    0B f=0/0   
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[34, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f914c0010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8c003530] Doing slow software conversion
 RgaBlit(1465) RGA_BLIT fail: Invalid argument2KB sq=    0B f=0/0   
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[44, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f80ca1010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f8c003530] Doing slow software conversion
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[47, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f802ae010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[42, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f770f5010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
    Last message repeated 2 times

hevc 10-bit

rock@rock5b:~/deb/ffmpeg$ ffplay ~/videos/jellyfish-10-mbps-hd-hevc-10bit.mkv -vcodec h264_rkmpp
ffplay version 4.4.2-0ubuntu0.22.04.1+rkmpp1 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1+rkmpp1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libdrm --enable-librga --enable-rkmpp --enable-version3 --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Input #0, matroska,webm, from '/home/rock/videos/jellyfish-10-mbps-hd-hevc-10bit.mkv':
  Metadata:
    COMPATIBLE_BRANDS: iso4hvc1iso6
    MAJOR_BRAND     : iso4
    MINOR_VERSION   : 1
    ENCODER         : Lavf56.3.100
  Duration: 00:00:30.10, start: 0.067000, bitrate: 10095 kb/s
  Stream #0:0(und): Video: hevc (Main 10), yuv420p10le(tv), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 29.97 tbc (default)
    Metadata:
      CREATION_TIME   : 2016-02-06 00:36:35
      LANGUAGE        : und
      HANDLER_NAME    : hevc@GPAC0.5.2-DEV-rev565-g71748d7-ab-suite
[h264_mp4toannexb @ 0x7f88003e60] Global extradata truncated, corrupted stream or invalid MP4/AVCC bitstream
Failed to open file '/home/rock/videos/jellyfish-10-mbps-hd-hevc-10bit.mkv' or configure filtergraph

Just to make sure it is using liujianfeng1994 deb packages.

rock@rock5b:~/deb/ffmpeg$ ldd /usr/bin/ffplay

linux-vdso.so.1 (0x0000007f897bc000)
libavdevice.so.58 => /lib/aarch64-linux-gnu/libavdevice.so.58 (0x0000007f896f0000)
libavfilter.so.7 => /lib/aarch64-linux-gnu/libavfilter.so.7 (0x0000007f892a0000)
libavformat.so.58 => /lib/aarch64-linux-gnu/libavformat.so.58 (0x0000007f89000000)
libavcodec.so.58 => /lib/aarch64-linux-gnu/libavcodec.so.58 (0x0000007f87e70000)
libpostproc.so.55 => /lib/aarch64-linux-gnu/libpostproc.so.55 (0x0000007f87e50000)
libswresample.so.3 => /lib/aarch64-linux-gnu/libswresample.so.3 (0x0000007f87e20000)
libswscale.so.5 => /lib/aarch64-linux-gnu/libswscale.so.5 (0x0000007f87d90000)
libavutil.so.56 => /lib/aarch64-linux-gnu/libavutil.so.56 (0x0000007f87ad0000)

ls -la /lib/aarch64-linux-gnu/libavcodec.so.58

rock@rock5b:~/deb/ffmpeg$ ls -la /lib/aarch64-linux-gnu/libavcodec.so.58.134.100
-rw-r–r-- 1 root root 12193760 Nov 16 13:56 /lib/aarch64-linux-gnu/libavcodec.so.58.134.100

h264 liujianfeng1994 deb

ffplay ~/videos/jellyfish-20-mbps-hd-h264.mkv -vcodec h264_rkmpp -loglevel debug
ffplay version 4.4.2-0ubuntu0.22.04.1+rkmpp1 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1+rkmpp1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libdrm --enable-librga --enable-rkmpp --enable-version3 --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Initialized opengles2 renderer.
[NULL @ 0x7f64000c10] Opening '/home/rock/videos/jellyfish-20-mbps-hd-h264.mkv' for reading
[file @ 0x7f64001690] Setting default whitelist 'file,crypto,data'
[matroska,webm @ 0x7f64000c10] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
[matroska,webm @ 0x7f64000c10] Before avformat_find_stream_info() pos: 5527 bytes read:32768 seeks:0 nb_streams:1
[h264 @ 0x7f640033f0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f640033f0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f640033f0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f640033f0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f640033f0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7f640033f0] Format yuv420p chosen by get_format().
[h264 @ 0x7f640033f0] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x7f640033f0] no picture 
[matroska,webm @ 0x7f64000c10] All info found 0KB sq=    0B f=0/0   
[matroska,webm @ 0x7f64000c10] After avformat_find_stream_info() pos: 394227 bytes read:414855 seeks:0 frames:3
Input #0, matroska,webm, from '/home/rock/videos/jellyfish-20-mbps-hd-h264.mkv':
  Metadata:
    encoder         : libebml v1.2.0 + libmatroska v1.1.0
    creation_time   : 2016-02-06T04:00:52.000000Z
  Duration: 00:00:30.03, start: 0.000000, bitrate: 19914 kb/s
  Stream #0:0(eng), 3, 1/1000: Video: h264 (High), 1 reference frame, yuv420p(tv, bt709, progressive, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 0/1, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
[h264_rkmpp @ 0x7f64004d80] Format yuv420p chosen by get_format().
[h264_rkmpp @ 0x7f64004d80] Initializing RKMPP decoder.
[h264_rkmpp @ 0x7f64004d80] RKMPP decoder initialized successfully.
[h264_rkmpp @ 0x7f64004d80] Flush. 0KB vq=  424KB sq=    0B f=0/0   
[h264_rkmpp @ 0x7f64004d80] Wrote 250109 bytes to decoder
[h264_rkmpp @ 0x7f64004d80] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f64004d80] Wrote 93719 bytes to decoder
[h264_rkmpp @ 0x7f64004d80] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f64004d80] Wrote 44900 bytes to decoder
[h264_rkmpp @ 0x7f64004d80] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f64004d80] Wrote 45636 bytes to decoder
[h264_rkmpp @ 0x7f64004d80] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f64004d80] Buffer full
[h264_rkmpp @ 0x7f64004d80] Received a frame.
rga_api version 1.8.1_[4] 0 aq=    0KB vq= 3240KB sq=    0B f=0/0   
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[34, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f620d1010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f64004d80] Doing slow software conversion
Video frame changed from size:0x0 format:none serial:-1 to size:1920x1080 format:yuv420p serial:1
detected 8 logical cores
[ffplay_buffer @ 0x7f54014e30] Setting 'video_size' to value '1920x1080'
[ffplay_buffer @ 0x7f54014e30] Setting 'pix_fmt' to value '0'
[ffplay_buffer @ 0x7f54014e30] Setting 'time_base' to value '1/1000'
[ffplay_buffer @ 0x7f54014e30] Setting 'pixel_aspect' to value '1/1'
[ffplay_buffer @ 0x7f54014e30] Setting 'frame_rate' to value '19001/634'
[ffplay_buffer @ 0x7f54014e30] w:1920 h:1080 pixfmt:yuv420p tb:1/1000 fr:19001/634 sar:1/1
[AVFilterGraph @ 0x7f640032e0] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[h264_rkmpp @ 0x7f64004d80] Wrote 76359 bytes to decoder
[h264_rkmpp @ 0x7f64004d80] Received a frame.
Created 1920x1080 texture with SDL_PIXELFORMAT_IYUV.
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[42, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f514b1010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f64004d80] Doing slow software conversion
[h264_rkmpp @ 0x7f64004d80] Received a frame.
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[44, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f502ae010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f64004d80] Doing slow software conversion
[h264_rkmpp @ 0x7f64004d80] Received a frame.
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[40, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f4b905010, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f64004d80] Doing slow software conversion
[AVIOContext @ 0x7f640097a0] Statistics: 4174672 bytes read, 0 seeks

h264 with @icecream95 patch

rock@rock5b:~/deb/ffmpeg/ffmpeg-4.4.2$ ./ffplay ~/videos/jellyfish-20-mbps-hd-h264.mkv -vcodec h264_rkmpp -loglevel debug
ffplay version 4.4.2 Copyright (c) 2003-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.3.0-1ubuntu1~22.04)
  configuration: --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-librga --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --disable-shared --enable-decoder='aac,ac3,flac' --extra-cflags=-I/usr/src/linux-headers-5.10.110-debugx/include
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '5'.
Initialized opengles2 renderer.
[NULL @ 0x7f70000c10] Opening '/home/rock/videos/jellyfish-20-mbps-hd-h264.mkv' for reading
[file @ 0x7f70001660] Setting default whitelist 'file,crypto,data'
[matroska,webm @ 0x7f70000c10] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
[matroska,webm @ 0x7f70000c10] Before avformat_find_stream_info() pos: 5527 bytes read:32768 seeks:0 nb_streams:1
[h264 @ 0x7f700033c0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f700033c0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f700033c0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f700033c0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f700033c0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7f700033c0] Format yuv420p chosen by get_format().
[h264 @ 0x7f700033c0] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x7f700033c0] no picture 
[matroska,webm @ 0x7f70000c10] All info found 0KB sq=    0B f=0/0   
[matroska,webm @ 0x7f70000c10] After avformat_find_stream_info() pos: 394227 bytes read:414855 seeks:0 frames:3
Input #0, matroska,webm, from '/home/rock/videos/jellyfish-20-mbps-hd-h264.mkv':
  Metadata:
    encoder         : libebml v1.2.0 + libmatroska v1.1.0
    creation_time   : 2016-02-06T04:00:52.000000Z
  Duration: 00:00:30.03, start: 0.000000, bitrate: 19914 kb/s
  Stream #0:0(eng), 3, 1/1000: Video: h264 (High), 1 reference frame, yuv420p(tv, bt709, progressive, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 0/1, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
[h264_rkmpp @ 0x7f70004d50] Format yuv420p chosen by get_format().
[h264_rkmpp @ 0x7f70004d50] Initializing RKMPP decoder.
[h264_rkmpp @ 0x7f70004d50] RKMPP decoder initialized successfully.
[h264_rkmpp @ 0x7f70004d50] Flush.
[h264_rkmpp @ 0x7f70004d50] Wrote 250109 bytes to decoder
[h264_rkmpp @ 0x7f70004d50] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f70004d50] Wrote 93719 bytes to decoder
[h264_rkmpp @ 0x7f70004d50] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f70004d50] Wrote 44900 bytes to decoder
[h264_rkmpp @ 0x7f70004d50] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f70004d50] Wrote 45636 bytes to decoder
[h264_rkmpp @ 0x7f70004d50] Timeout getting decoded frame.
    Last message repeated 1 times
[h264_rkmpp @ 0x7f70004d50] Buffer full
[h264_rkmpp @ 0x7f70004d50] Wrote 76359 bytes to decoder
[h264_rkmpp @ 0x7f70004d50] Received a frame.
[h264_rkmpp @ 0x7f70004d50] Allocated buffer of size 3139231 -> 0x7f70005470 (CPU 0x7f740c5000)
rga_api version 1.8.1_[4]
544:src has not fd and address for render
Video frame changed from size:0x0 format:none serial:-1 to size:1920x1080 format:yuv420p serial:1
detected 8 logical cores
[ffplay_buffer @ 0x7f6c03aec0] Setting 'video_size' to value '1920x1080'
[ffplay_buffer @ 0x7f6c03aec0] Setting 'pix_fmt' to value '0'
[ffplay_buffer @ 0x7f6c03aec0] Setting 'time_base' to value '1/1000'
[ffplay_buffer @ 0x7f6c03aec0] Setting 'pixel_aspect' to value '1/1'
[ffplay_buffer @ 0x7f6c03aec0] Setting 'frame_rate' to value '19001/634'
[ffplay_buffer @ 0x7f6c03aec0] w:1920 h:1080 pixfmt:yuv420p tb:1/1000 fr:19001/634 sar:1/1
[AVFilterGraph @ 0x7f7002a7d0] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[h264_rkmpp @ 0x7f70004d50] Received a frame.
[h264_rkmpp @ 0x7f70004d50] Allocated buffer of size 3139231 -> 0x7f70098d80 (CPU 0x7f640bb000)
 RgaBlit(1465) RGA_BLIT fail: Invalid argument
 RgaBlit(1466) RGA_BLIT fail: Invalid argument
fd-vir-phy-hnd-format[2, (nil), (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2560, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
fd-vir-phy-hnd-format[0, 0x7f640bb000, (nil), 0, 0]
rect[0, 0, 1920, 1080, 1920, 1088, 2816, 0]
f-blend-size-rotation-col-log-mmu[0, 0, 0, 0, 0, 0, 1]
This output the user patamaters when rga call blit fail
[h264_rkmpp @ 0x7f70004d50] Doing slow software conversion
Created 1920x1080 texture with SDL_PIXELFORMAT_IYUV.
[h264_rkmpp @ 0x7f70004d50] Received a frame.
[h264_rkmpp @ 0x7f70004d50] Allocated buffer of size 3139231 -> 0x7f60000ff0 (CPU 0x7f5e1d9000)

Just for the record, summarizing my findings:

image
image.png803×324 58.9 KB

image
image.png803×120 25.4 KB

1 Like
#666

You might be using the wrong librga version?

I’ve pushed an update to my patch snippet so that the rga kernel driver is used directly rather than going through librga.

I also added a second patch to make MPP be used by default. It even works with Firefox for mp4 videos at least, if I set MOZ_DISABLE_RDD_SANDBOX=1. But I don’t know if that’s a wise thing to do… I’ve already come across videos that cause the VPU to fault, which means that the video ends up looking very corrupted. And disabling the sandbox exposes the media decoder to all the fun kernel bugs I’ve been discovering.

I’ve also worked out that RGA isn’t even necessary… if I change the codec to report the output format as NV12 rather than YUV420P, then no conversion is necessary. So perhaps v3 of the patch will simply take the dma-buf written to by MPP, import it into /dev/mali0 and map it there. (If I try to map the dma-buf directly, it will be uncached and very slow to read.)

#667

NPU curiosity once more.

I am not sure what the userspace govenor is for /sys/devices/platform/fdab0000.npu/devfreq/fdab0000.npu/governor but performance doesn’t garner any max increase but it does make multiple results far more consistant.
I am still doing a simple hack to get approx fps

    # Inference
    starttime = time()
    print('--> Running model')
    for i in range(10000):
      outputs = rknn_lite.inference(inputs=[img])
      #show_top5(outputs, i)

    print('done')
    print(10000 / (time() - starttime))

    rknn_lite.release()

If any of you guys do have a spare minute as wow @avaf that was some brilliant imperical testing going on there, but if there is a moment to maybe try something else do you fancy plumbing in the sram and setting here

Its a bit weird as the root seems to be to run the toolkit on x86 on ubuntu to use the runtime rknn_toolkit_lite2 subfolder on debian but the example is pretty easy to get running.

Its working and I have hacked all cores to be running ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) and just curious when enabled what and if any perf gain is made

1.4.0
Support more NPU operators, such as Reshape、Transpose、MatMul、 Max、Min、exGelu、exSoftmax13、Resize etc.
Add Weight Share function, reduce memory usage.
Add Weight Compression function, reduce memory and bandwidth usage.(RK3588/RV1103/RV1106)
RK3588 supports storing weights or feature maps on SRAM, reducing system bandwidth consumption.
RK3588 adds the function of running a single model on multiple cores at the same time.
Add new output layout NHWC (C has alignment restrictions) .
Improve support for non-4D input.
Add more examples such as rknn_yolov5_android_apk_demo and rknn_internal_mem_reuse_demo.
Bug fix.

Its working and just curiosity, but if anyone can either way info would be interesting.
PS the default is set to a userspace governor is that part of the toolkit that I have overridden as it does provide quite a lot of variance?

PS RK3588 adds the function of running a single model on multiple cores at the same time. can you just set this to auto and run up to 3 different models on different cores in paralell? You do get a perf gain sort of similar to tensorflow levels where 2x threads gives extra and 4 seems to give no more than 2 and seems to scale similar.
Is 6 tops actually x3 2 Tops cores?
There isn’t a huge amount of documentation but the NPU on the RK3588 for me is pretty exciting :slight_smile:

#668

Patch applied and tested. Unfortunately still has the same issue with rga. Maybe your v3 would work.
Would you care to push your kernel tree with all your patches already applied? I can build the kernel and test it again.

Output:

rock@rock5b:~/Videos$ sudo ~/rockchip/ffmpeg/ffmpeg-4.4.2/ffplay jellyfish-10-mbps-hd-h264.mkv -loglevel debug
ffplay version 4.4.2 Copyright © 2003-2021 the FFmpeg developers
built with gcc 10 (Debian 10.2.1-6)
configuration: --prefix=/usr --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-decoder=h264_v4l2m2m --disable-decoder=vp8_v4l2m2m --disable-decoder=mpeg2_v4l2m2m --disable-decoder=mpeg4_v4l2m2m --disable-libxvid --disable-libx264 --disable-libx265 --enable-rkmpp --enable-nonfree --enable-gpl --enable-version3 --enable-libmp3lame --enable-libpulse --enable-libv4l2 --enable-libdrm --enable-libxml2 --enable-librtmp --enable-libfreetype --enable-openssl --enable-opengl --enable-libopus --enable-libvorbis --enable-decoder=‘aac,ac3,flac’ --extra-cflags=-I/usr/src/linux-headers-5.10.110-debugx/include
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘5’.
Initialized opengles2 renderer.
[NULL @ 0x7f9c000c00] Opening ‘jellyfish-10-mbps-hd-h264.mkv’ for reading
[file @ 0x7f9c001640] Setting default whitelist ‘file,crypto,data’
[matroska,webm @ 0x7f9c000c00] Format matroska,webm probed with size=2048 and score=100
st:0 removing common factor 1000000 from timebase
[matroska,webm @ 0x7f9c000c00] Before avformat_find_stream_info() pos: 5527 bytes read:32768 seeks:0 nb_streams:1
[h264 @ 0x7f9c003350] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f9c003350] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f9c003350] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x7f9c003350] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x7f9c003350] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x7f9c003350] Format yuv420p chosen by get_format().
[h264 @ 0x7f9c003350] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x7f9c003350] no picture
[matroska,webm @ 0x7f9c000c00] All info found 0KB sq= 0B f=0/0
[matroska,webm @ 0x7f9c000c00] After avformat_find_stream_info() pos: 225468 bytes read:248705 seeks:0 frames:3
Input #0, matroska,webm, from ‘jellyfish-10-mbps-hd-h264.mkv’:
Metadata:
encoder : libebml v1.2.0 + libmatroska v1.1.0
creation_time : 2016-02-06T04:00:51.000000Z
Duration: 00:00:30.03, start: 0.000000, bitrate: 9955 kb/s
Stream #0:0(eng), 3, 1/1000: Video: h264 (High), 1 reference frame, yuv420p(tv, bt709, progressive, left), 1920x1080 (1920x1088) [SAR 1:1 DAR 16:9], 0/1, 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
[h264_rkmpp @ 0x7f9c004990] Format yuv420p chosen by get_format().
[h264_rkmpp @ 0x7f9c004990] Initializing RKMPP decoder.
[h264_rkmpp @ 0x7f9c004990] RKMPP decoder initialized successfully.
[h264_rkmpp @ 0x7f9c004990] Flush.
[h264_rkmpp @ 0x7f9c004990] Wrote 144918 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 51638 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 23413 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 24020 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000e20 (CPU 0x7fa102f000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
Video frame changed from size:0x0 format:none serial:-1 to size:1920x1080 format:yuv420p serial:1
detected 8 logical cores
[ffplay_buffer @ 0x7f98020cf0] Setting ‘video_size’ to value ‘1920x1080’
[ffplay_buffer @ 0x7f98020cf0] Setting ‘pix_fmt’ to value ‘0’
[ffplay_buffer @ 0x7f98020cf0] Setting ‘time_base’ to value ‘1/1000’
[ffplay_buffer @ 0x7f98020cf0] Setting ‘pixel_aspect’ to value ‘1/1’
[ffplay_buffer @ 0x7f98020cf0] Setting ‘frame_rate’ to value ‘19001/634’
[ffplay_buffer @ 0x7f98020cf0] w:1920 h:1080 pixfmt:yuv420p tb:1/1000 fr:19001/634 sar:1/1
[AVFilterGraph @ 0x7f9c078180] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[h264_rkmpp @ 0x7f9c004990] Wrote 36739 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f9c2e65f0 (CPU 0x7fa0133000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000fb0 (CPU 0x7f900fe000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
Created 1920x1080 texture with SDL_PIXELFORMAT_IYUV.
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c001110 (CPU 0x7f8b60d000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.13KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000e20 (CPU 0x7fa102f000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.13KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f9c2e65f0 (CPU 0x7fa0133000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.13KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000fb0 (CPU 0x7f900fe000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.B f=0/0
[h264_rkmpp @ 0x7f9c004990] Wrote 36501 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 83500 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 31003 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 88583 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Wrote 36025 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Wrote 36473 bytes to decoder
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
Last message repeated 1 times
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Timeout getting decoded frame.B f=0/0
[h264_rkmpp @ 0x7f9c004990] Buffer full
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c001110 (CPU 0x7f8b60d000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Wrote 91287 bytes to decoder 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Received a frame.
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000e20 (CPU 0x7fa102f000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.89KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f9c2e65f0 (CPU 0x7fa0133000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[h264_rkmpp @ 0x7f9c004990] Received a frame.89KB sq= 0B f=0/0
[h264_rkmpp @ 0x7f9c004990] Allocated buffer of size 3139231 -> 0x7f8c000fb0 (CPU 0x7f900fe000)
[h264_rkmpp @ 0x7f9c004990] Doing slow software conversion
[AVIOContext @ 0x7f9c009740] Statistics: 2629809 bytes read, 0 seeks

Kernel log:

[49074.212055] rga_mm: RGA_MMU unsupported Memory larger than 4G!
[49074.212093] rga_mm: scheduler core[4] unsupported mm_flag[0x0]!
[49074.212103] rga_mm: rga_mm_map_buffer map dma_buf error!
[49074.212110] rga_mm: job buffer map failed!
[49074.212118] rga_mm: src channel map job buffer failed!
[49074.212125] rga_mm: failed to map buffer
[49074.212136] rga_job: rga_job_commit: failed to map job info
[49074.212166] rga_job: request[1] task[0] job_commit failed.
[49074.212175] rga_job: rga request commit failed!
[49074.212182] rga: request[1] submit failed!

#669

Cross-compiling the kernel with SRAM and re-running the demo.
Can you write the commands you would like to test?

#670

Yeah you just need to copy https://github.com/rockchip-linux/rknpu2/blob/master/runtime/RK3588/Linux/librknn_api/aarch64/librknnrt.so to /usr/lib but on 1st run it will err and tell you anyway.

The runtime wheel is https://github.com/rockchip-linux/rknn-toolkit2/blob/master/rknn_toolkit_lite2/packages/rknn_toolkit_lite2-1.4.0-cp39-cp39-linux_aarch64.whl
So you need debian or work out how to pass the npu to docker.

That is the runtime installed and there is a ready example at

Where the normal test.py I have done a rough and ready hack to measure avg fps

    # init runtime environment
    print('--> Init runtime environment')
    # run on RK356x/RK3588 with Debian OS, do not need specify target.
    if host_name == 'RK3588':
        ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
    else:
        ret = rknn_lite.init_runtime()
    if ret != 0:
        print('Init runtime environment failed')
        exit(ret)
    print('done')

    # Inference
    starttime = time()
    print('--> Running model')
    for i in range(10000):
      outputs = rknn_lite.inference(inputs=[img])
      #show_top5(outputs, i)

    print('done')
    print(10000 / (time() - starttime))

    rknn_lite.release()

196.19549498656974
131.63509792655853

I really don’t understand why I get such a variance in fps and maybe just being my usual dumb :slight_smile:
The 1st run from boot was 50% faster !?

The variance just seems to happen when running all cores which I think are 3x 2 tops npu cores.
ret = rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0) provides much less variance.

164.40189807323299
164.30673703400316

Maybe sram needs to be inplace for multicore to work reliable but results seem to bounce around in terms of performance.
Yeah you can run on each core x3 instances with each core taking about 1 watt so if you partition a model you can still get 6 tops or 4 tops across 2 or x3 2 tops.
All very cool but wondering if the sram does help with weights but you might have to declare for use

1. Specify Internal to use SRAM:

Automatic size mode, will try to allocate enough remaining SRAM from the system for Internal use
export RKNN_INTERNAL_MEM_TYPE=sram
In the specified size mode, it will try to allocate the specified 256KB SRAM from the system for Internal use
export RKNN_INTERNAL_MEM_TYPE=sram#256
2. Specify Weight to use SRAM:

Automatic size mode, will try to allocate enough remaining SRAM from the system for Weight to use
export RKNN_SEPARATE_WEIGHT_MEM=1
export RKNN_WEIGHT_MEM_TYPE=sram
Specify the size method, it will try to allocate the specified 128KB SRAM from the system to Weight
export RKNN_SEPARATE_WEIGHT_MEM=1
export RKNN_WEIGHT_MEM_TYPE=sram#128
3. Mixed designation

The RKNPU driver supports SRAM memory management, and supports specifying SRAM for both Internal and Weight use at the same time, as follows:
export RKNN_INTERNAL_MEM_TYPE=sram#256
export RKNN_SEPARATE_WEIGHT_MEM=1
export RKNN_WEIGHT_MEM_TYPE=sram#128