mpv also requires below fixes to play drm_prime properly. I mainlined NV16 part, but p010 should be missing in the mainline mpv.
From 504b2de386bffc0746c243b5f937fb0c5b7c4447 Mon Sep 17 00:00:00 2001
From: hbiyik <boogiepop@gmx.com>
Date: Thu, 28 Sep 2023 22:25:25 +0200
Subject: [PATCH 1/2] hwdec_drmprime: add nv16 support
NV16 is the half subsampled version of NV12 format. Decoders which
support High 4:2:2 of h264 provide the frame in NV16 format to establish
richer colorspace. Similar profiles are also available in HEVC and other
popular codecs. This commit allows NV16 frames to be displayed over
drmprime layers.
Signed-off-by: hbiyik <boogiepop@gmx.com>
---
video/out/hwdec/dmabuf_interop_gl.c | 1 +
video/out/hwdec/hwdec_drmprime.c | 2 ++
2 files changed, 3 insertions(+)
diff --git a/video/out/hwdec/dmabuf_interop_gl.c b/video/out/hwdec/dmabuf_interop_gl.c
index bd33474289..e7fb1031a0 100644
--- a/video/out/hwdec/dmabuf_interop_gl.c
+++ b/video/out/hwdec/dmabuf_interop_gl.c
@@ -176,6 +176,7 @@ static bool vaapi_gl_map(struct ra_hwdec_mapper *mapper,
if (p_mapper->desc.layers[i].nb_planes > 1) {
switch (p_mapper->desc.layers[i].format) {
case DRM_FORMAT_NV12:
+ case DRM_FORMAT_NV16:
format[0] = DRM_FORMAT_R8;
format[1] = DRM_FORMAT_GR88;
break;
diff --git a/video/out/hwdec/hwdec_drmprime.c b/video/out/hwdec/hwdec_drmprime.c
index 5051207413..290f11c535 100644
--- a/video/out/hwdec/hwdec_drmprime.c
+++ b/video/out/hwdec/hwdec_drmprime.c
@@ -29,6 +29,7 @@
#include "libmpv/render_gl.h"
#include "options/m_config.h"
+#include "video/fmt-conversion.h"
#include "video/out/drm_common.h"
#include "video/out/gpu/hwdec.h"
#include "video/out/hwdec/dmabuf_interop.h"
@@ -117,6 +118,7 @@ static int init(struct ra_hwdec *hw)
int num_formats = 0;
MP_TARRAY_APPEND(p, p->formats, num_formats, IMGFMT_NV12);
MP_TARRAY_APPEND(p, p->formats, num_formats, IMGFMT_420P);
+ MP_TARRAY_APPEND(p, p->formats, num_formats, pixfmt2imgfmt(AV_PIX_FMT_NV16));
MP_TARRAY_APPEND(p, p->formats, num_formats, 0); // terminate it
p->hwctx.hw_imgfmt = IMGFMT_DRMPRIME;
--
2.42.0
From 76a34a8f31fd69094adef09c4b2146da7a842a10 Mon Sep 17 00:00:00 2001
From: boogie <boogiepop@gmx.com>
Date: Wed, 1 Nov 2023 20:35:19 +0100
Subject: [PATCH 2/2] hwdec_drmprime: add p010 support
Removes the limitation that P010 DRMPrime Avframes were filtered out
Signed-off-by: hbiyik <boogiepop@gmx.com>
---
video/out/hwdec/hwdec_drmprime.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/video/out/hwdec/hwdec_drmprime.c b/video/out/hwdec/hwdec_drmprime.c
index 290f11c535..9c63ab49ff 100644
--- a/video/out/hwdec/hwdec_drmprime.c
+++ b/video/out/hwdec/hwdec_drmprime.c
@@ -119,6 +119,7 @@ static int init(struct ra_hwdec *hw)
MP_TARRAY_APPEND(p, p->formats, num_formats, IMGFMT_NV12);
MP_TARRAY_APPEND(p, p->formats, num_formats, IMGFMT_420P);
MP_TARRAY_APPEND(p, p->formats, num_formats, pixfmt2imgfmt(AV_PIX_FMT_NV16));
+ MP_TARRAY_APPEND(p, p->formats, num_formats, pixfmt2imgfmt(AV_PIX_FMT_P010));
MP_TARRAY_APPEND(p, p->formats, num_formats, 0); // terminate it
p->hwctx.hw_imgfmt = IMGFMT_DRMPRIME;
--
2.42.0
