Rock 5C Lite H264 Streaming (Browser) Experiments

I received my Rock 5C lite a few days ago and it looks like I was lucky to unlock all the cores. So I think.

CPU开核结果:8核
GPU开核结果:YES
解码器1开核结果:YES
解码器2开核结果:YES
编码器1开核结果:YES
编码器2开核结果:YES

I tried a few things and was inspired by this piCamera blog (https://www.codeinsideout.com/blog/pi/stream-picamera-h264/) this is my first try to have something similar, without python.

The first idea was to use Nyanmisaka’s ffmpeg, but since there is no real-world encoder example, i decided to use pipe, named pipe to be exact. In theory, it should be fast enough, should be.

Here is the idea:

I created the named pipe and used ffmpeg to write to the write end of the pipe:

ffmpeg -f v4l2 -input_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v h264_rkmpp -qp_init 30 -b:v 2000K -vprofile baseline -vf format=yuv420p -r 25 stream.h264 -y

Somehow this POC did not work as expected, the first image is shown and the subsequent tiles are drawn but there is that annoying effect we see when there is insufficient bandwidth.

The good news is there is near zero latency, and i could push the h264 stream to more browsers without degrading much.

I am not sure if the ffmpeg pipe is correct or if using the named pipe is a bad idea.

I found another open source player called JMuxer (https://github.com/samirkumardas/jmuxer/?tab=readme-ov-file) which seems to display the stream correctly but complains about NALU , the drawback is the latency, is too high.

@nyanmisaka can you confirm if the ffmpeg pipe is correct to have NALU sent correctly?

I tested the concept on X86 (which worked with YUYV webcam) and the Zero 3W:

ffmpeg pipe on X86:

ffmpeg -f v4l2 -input_format yuyv422 -framerate 30 -video_size 1920x1080 -i /dev/video0 -c:v libx264 -profile:v baseline -vf format=yuv420p -r 30 stream.h264 -y

1 Like

The decoder expects an .mp4 file and does not support weighted prediction for P-frames and CABAC entropy encoding. To create such bitstreams use ffmpeg and x264 with the following command line options:

Those h264 decoders in JavaScript are not a fully compliant implementation. Therefore, it is not surprising that any artifacts appear without disabling some advanced encoding features.

1 Like

Just for the sake of information, and until i find out how to disable those advanced encoding features i tried a new player with better results, and good latency but still lots of artifacts.

BTW, these flags seem not available anymore:
-wpredp 0
-tune zerolatency

I want to try this one (https://videojs.com/) but i haven’t found a way to send/receive a websocket message that will start pushing the stream on my ws server.
If anyone has experience with the videojs and can point out how to use it with websocket i would appreciate it.

Here is the script i use for the WSAvcPlayer that works:

<!DOCTYPE html>
<html>

<head>
	<title>h264-live-player web client demo</title>
</head>

<body>
	<p id='frame_buffer'></p>
	<br />
	<canvas id='cam' style="width:300px; height:200px;">
		<!-- provide WSAvcPlayer -->
		<script type="text/javascript" src="WSAvcPlayer.js">
		;
		</script>
		<script type="text/javascript">
		var canvas = document.getElementById('cam')
		var fb = document.getElementById('frame_buffer')
		// Create h264 player
		var wsavc = new WSAvcPlayer(canvas, "webgl", 1, 35);
		//expose instance for button callbacks
		window.wsavc = wsavc;
		var uri = window.location.protocol.replace(/http/, 'ws') + '//' + window.location.hostname + ':8080'
		console.log('uri', uri);
		var result = wsavc.connect(uri);
		wsavc.on('disconnected', () => console.log('WS Disconnected'))
		wsavc.on('connected', () => wsavc.send('*play'))
		wsavc.on('frame_shift', (fbl) => {
			fb.innerText = 'fl: ' + fbl
		})
		wsavc.on('initalized', (payload) => {
			console.log('Initialized', payload)
		})
		wsavc.on('stream_active', active => console.log('Stream is ', active ? 'active' : 'offline'))
		wsavc.on('custom_event_from_server', event => console.log('got event from server', event))
		</script>
</body>

</html>

Here is the videojs script that needs to be fixed to decode and display the stream.

<!DOCTYPE html>
<html>

<head>
	<meta charset='utf-8'>
	<link href="video-js.min.css" rel="stylesheet">
	<script src="video.min.js"></script>
	<script src="sockjs.min.js"></script>
	<style>
	/* change player background color */
	#myVideo {
		background-color: #1a535c;
	}
	</style>
</head>

<body>
	<video id="myVideo" playsinline class="video-js vjs-default-skin"></video>
	<script>
	/* eslint-disable */
	var options = {
		controls: true,
		width: 320,
		height: 240,
		fluid: false,
		bigPlayButton: false,
		controlBar: {
			volumePanel: false
		}
	};
	// apply some workarounds for opera browser
	// applyVideoWorkaround();
	var player = videojs('myVideo', options, function() {
		// print version information at startup
		var msg = 'Using video.js ' + videojs.VERSION;
		videojs.log(msg);
		// connect to websocket server
		var wsUri = window.location.protocol.replace(/http/, 'ws') + '//' + window.location.hostname + ':8080'
		console.log('connecting to websocket: ' + wsUri);
		var ws = new WebSocket(wsUri)
		ws.binaryType = 'arraybuffer'
		ws.onopen = function(e) {
			ws.send('*play');
			ws.binaryType = 'arraybuffer'
			console.log('Client connected')
		}
		ws.onmessage = function(msg) {
			// decode stream
			console.log('msg received')
            // HOWTO
			// window.player.decode(new Uint8Array(msg.data));
            // 
		}
		ws.onclose = function(e) {
			console.log('Client disconnected')
		}
	});
	// error handling
	player.on('deviceError', function() {
		console.warn('device error:', player.deviceErrorCode);
	});
	player.on('error', function(element, error) {
		console.error(error);
	});
	// user clicked the record button and started recording
	player.on('startRecord', function() {
		console.log('started recording!');
	});
	// user completed recording and stream is available
	player.on('finishRecord', function() {
		// the blob object contains the recorded data that
		// can be downloaded by the user, stored on server etc.
		console.log('finished recording: ', player.recordedData);
	});
	</script>
</body>

</html>

I decided to give the HTML5 MediaSource a try, unfortunately, it does not now show the video stream. I don’t know which codec to use…

<!DOCTYPE html>
<html>

<head>
	<meta charset='utf-8'>
</head>

<body>
        <video id="myVideo" autoplay width="320" height="240"></video>
	<script>
	window.MediaSource = window.MediaSource || window.WebKitMediaSource;
	if(!!!window.MediaSource) {
		alert('MediaSource API is not available!');
	}
	var wsUri = window.location.protocol.replace(/http/, 'ws') + '//' + window.location.hostname + ':8080'
	console.log('connecting to websocket: ' + wsUri);
	var websocket = new WebSocket(wsUri);
	websocket.binaryType = 'arraybuffer';
	var mediaSource = new MediaSource();
	var buffer;
	var queue = [];
	var video = document.getElementById('myVideo');
	video.src = window.URL.createObjectURL(mediaSource);
	mediaSource.addEventListener('sourceopen', function(e) {
		// video.play();
		console.log('mediaSource sourceopen', mediaSource.readyState);
		var mimeCodec = 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"';
		buffer = mediaSource.addSourceBuffer(mimeCodec);
		buffer.addEventListener('updateend', () => {
			// mediaSource.endOfStream();
			console.log('mediaSource updateend', mediaSource.readyState); // ended
		});
		buffer.addEventListener('updatestart', function(e) {
			console.log('updatestart: ' + mediaSource.readyState);
		});
		buffer.addEventListener('error', function(e) {
			console.log('error: ' + mediaSource.readyState);
		});
		buffer.addEventListener('abort', function(e) {
			console.log('abort: ' + mediaSource.readyState);
		});
		buffer.addEventListener('update', function() { 
			console.log('mediaSource update', mediaSource.readyState); 
			if(queue.length > 0 && !buffer.updating) {
				buffer.appendBuffer(queue.shift());
			}
		});
	}, false);
	mediaSource.addEventListener('sourceopen', function(e) {
		console.log('sourceopen: ' + mediaSource.readyState);
	});
	mediaSource.addEventListener('sourceended', function(e) {
		console.log('sourceended: ' + mediaSource.readyState);
	});
	mediaSource.addEventListener('sourceclose', function(e) {
		console.log('sourceclose: ' + mediaSource.readyState);
	});
	websocket.addEventListener('message', function(e) {
             console.log('message: ' + e.data.byteLength);
		if(typeof e.data !== 'string') {
			if(buffer.updating || queue.length > 0) {
				queue.push(e.data);
			} else {
				buffer.appendBuffer(e.data);
			}
		}
	}, false);
	websocket.addEventListener('open', function(e) {
		websocket.send('*play');
	}, false);
	</script>
</body>

</html>

Giving up on HTML5 MediaSource , i can’t write mp4 to pipe and all codecs i tried failed:

// var mimeCodec = ‘video/mp4; codecs=“avc1.4d002a”’;
var mimeCodec = ‘video/mp4;codecs=“avc1.64001E”’;
// var mimeCodec = ‘video/mp4; codecs=“avc1.4D0033, mp4a.40.2”’;

I tested with an H264 USB camera module.
After some trying i found that JMuxer works best with:

JVT NAL sequence, H.264 video, main @ L 41
latency: 1 sec.

v4l2 command:

v4l2-ctl --verbose -d /dev/video23 --set-fmt-video=width=1920,height=1080 --stream-mmap=4 --set-selection=target=crop,flags=0,top=0,left=0,width=1920,height=1080 --stream-to=stream.h264

The closest i get with the ffmpeg pipe:

ffmpeg -f v4l2 -input_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v h264_rkmpp -qp_init 22 -movflags frag_keyframe+empty_moov+faststart -b:v 4000K -vprofile main -vf format=yuv420p -r 25 -bufsize 600k stream.h264 -y

JVT NAL sequence, H.264 video, main @ L 40
latency: 30 sec.

1 Like

I was trying almost the same a while ago. Didn’t find the perfect solution either. WebRTC looked promising but I wasn’t able to get it to work with hardware encoding…

WebRTC could be an alternative, but i think you add an extra layer of complexity.
There is something similar with very low latency, but it uses gstreamer and an external server, take a look at this post here (the rtmp part):

I found this interesting code (gstreamer —> WebRTC) that could be ported to C if you are a C++ wizard:

I added the 4.1 level which reduced the bandwidth, but still a high latency.

ffmpeg -f v4l2 -input_format nv12 -framerate 30 -video_size 1920x1080 -i /dev/video11 -c:v h264_rkmpp -qp_init 22 -movflags frag_keyframe+empty_moov+faststart -b:v 4000K -vprofile main -level:v 4.1 -vf format=yuv420p -r 25 -bufsize 600k -f segment stream.h264 -y

As far as I understood it’s more complex from an implementation perspective. But the workload/latency should be lower since the datastream is encoded and decoded directly and it’s working in the browser without vlc etc. I may have a second look in the future. Currently I’m experimenting with collabora u-boot and pxe. It’s working for rock5b and I’m trying to get 5+ to work (well it already is, but only with 6gb ram and after rkbin-update).

@nyanmisaka

Is there a way to force ffmpeg / mpp to emit NAL more often? I think the huge latency is due to this (despite the error in decoding). the H264 USB camera algorithm emits 4x NAL (in a shorter period) than mpp.