Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)
From SomLabs Wiki
Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)
Prerequisites
This tutorial based on Debian 9.2 image for VisionSOM-6ULL module. Please visit How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux or How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows for more detailed instruction how to create a bootable microSD card.
To enable support for LCD-TFT display, please download and apply enable-tft-lcd.patch for somlabs-visionsom-6ull.dts device tree file:
wget http://wiki.somlabs.com/images/c/cd/Enable-tft-lcd.zip unzip Enable-tft-lcd.zip patch /home/developer/source/somlabs-dts-1.0/somlabs-visionsom-6ull.dts ./enable-tft-lcd.patch
For more info how to customize, build and upload Device Tree file, please visit How to customize Debian 9.2 device tree.
Installing GStreamer multimedia framework
GStreamer is a pipeline-based multimedia framework which allows a programmer to create a variety of media-handling components, including simple audio playback, audio and video playback, recording and streaming. More info about gstreamer can be found here:
- https://en.wikipedia.org/wiki/GStreamer
- https://gstreamer.freedesktop.org/documentation/
- https://github.com/GStreamer
Install gstreamer1.x packages:
root@somlabs:~# apt-get update root@somlabs:~# apt-get install gstreamer1.0-x gstreamer1.0-tools root@somlabs:~# apt-get install gstreamer1.0-plugins-good root@somlabs:~# apt-get install gstreamer1.0-plugins-bad root@somlabs:~# apt-get install gstreamer1.0-libav
At this point you could use:
root@somlabs:~# gst-launch-1.0 videotestsrc ! fbdevsink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock <Ctrl+c> Interrupt: Stopping pipeline ... Execution ended after 0:00:04.732790131 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...
to display prebuild image/video pattern on the display, but it will not stretch to the display, and will not be hardware accelerated:
Adding GStreamer PxP engine support via gstreamer-imx
The multimedia performance of i.MX 6ULL processor is enhanced by a Pixel Processing Pipeline (PXP) to support 2D image processing, including color-space conversion, scaling, alpha-blending, and rotation. You can easily add PxP support via GStreamer and Gstreamer-imx plugins.
Install build dependencies for the gstreamer-imx package:
root@somlabs:~# apt-get install build-essential autoconf libtool root@somlabs:~# apt-get install wget python pkg-config root@somlabs:~# apt-get install libgstreamer1.0-dev root@somlabs:~# apt-get install libgstreamer-plugins-base1.0-dev
Clone gstreamer-imx repository (if you don't already have git installed, simply install it with sudo apt install git):
root@somlabs:~# git clone git://github.com/Freescale/gstreamer-imx.git Cloning into 'gstreamer-imx'... remote: Counting objects: 4349, done. remote: Total 4349 (delta 0), reused 0 (delta 0), pack-reused 4349 Receiving objects: 100% (4349/4349), 1.91 MiB | 1.35 MiB/s, done. Resolving deltas: 100% (3129/3129), done.
To compile gstreamer-imx we need a Linux kernel headers with i.MX additions for the PxP subsystems (kernel headers where at least linux/pxp_device.h can be found). Let's install .deb package with kernel headers (prebuilt .deb package is available as a part of somlabs-visionsom-6ull-debian-rootfs-qemu.tar.xz image):
root@somlabs:~# dpkg -i linux-headers-4.1.15_4.1.15-1_armhf.deb
Gstreamer-imx project uses the waf meta build system, to configure project run:
root@somlabs:~# cd gstreamer-imx root@somlabs:~/gstreamer-imx# ln -s /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ /usr/lib/gstreamer-1.0 root@somlabs:~/gstreamer-imx# ./waf configure --prefix=/usr --kernel-headers=/usr/src/linux-headers-4.1.15/include Setting top to : /root/gstreamer-imx Setting out to : /root/gstreamer-imx/build Checking for 'gcc' (C compiler) : /usr/bin/gcc Checking for compiler switch -O2 : yes ... checking for linux/pxp_device.h : yes PxP elements will be built checking for linux/fb.h and the IPU header linux/ipu.h : yes IPU elements will be built Checking for 'libimxvpuapi >= 0.10.3' : not found could not find installed imxvpuapi library - VPU elements will not be built ... 'configure' finished successfully (25.634s)
Now build and install gstreamer-imx:
root@somlabs:~/gstreamer-imx# ./waf Waf: Entering directory `/root/gstreamer-imx/build' [ 1/35] Compiling src/common/canvas.c [ 2/35] Compiling src/common/fd_object.c [ 3/35] Compiling src/common/phys_mem_allocator.c ... [27/35] Compiling src/compositor/gst-backport/gstimxbpvideoaggregator.c [28/35] Compiling src/compositor/compositor.c [29/35] Linking build/src/compositor/libgstimxcompositor.so [30/35] Linking build/src/blitter/libgstimxblitter.so [31/35] Linking build/src/ipu/libgstimxipu.so [32/35] Linking build/src/pxp/libgstimxpxp.so [33/35] Symlinking build/src/blitter/libgstimxblitter.so [34/35] Symlinking build/src/compositor/libgstimxcompositor.so [35/35] Symlinking build/src/common/libgstimxcommon.so Waf: Leaving directory `/root/gstreamer-imx/build' 'build' finished successfully (1m45.921s) root@somlabs:~/gstreamer-imx# ./waf install Waf: Entering directory `/root/gstreamer-imx/build' + install /usr/lib/libgstimxcommon.so.0.13.0 + symlink /usr/lib/libgstimxcommon.so (to libgstimxcommon.so.0.13.0) + symlink /usr/lib/libgstimxcommon.so.0 (to libgstimxcommon.so.0.13.0) + install /usr/lib/pkgconfig/gstimxcommon.pc ... Waf: Leaving directory `/root/gstreamer-imx/build' 'install' finished successfully (0.584s)
After this step you should be able to see newly installed imx plugins with gst-inspect-1.0 tool (which allows to get documentation on available elements and detailed information on a specific element):
root@somlabs:~/gstreamer-imx# gst-inspect-1.0 | grep imx imxpxp: imxpxpvideotransform: Freescale PxP video transform imxpxp: imxpxpvideosink: Freescale PxP video sink
Inspect imxpxpvideosink element:
root@somlabs:~/gstreamer-imx# gst-inspect-1.0 imxpxpvideosink
Factory Details:
Rank primary + 1 (257)
Long-name Freescale PxP video sink
Klass Sink/Video
Description Video output using the Freescale PxP API
Author Carlos Rafael Giani <dv@pseudoterminal.org>
…
Element Properties:
name : The name of the object
flags: readable, writable
String. Default: "imxpxpvideosink0"
parent : The parent of the object
flags: readable, writable
Object of type "GstObject"
Hardware-accelerated video-stream playback
Download sample video file:
wget http://craftymind.com/factory/html5video/BigBuckBunny_640x360.mp4
The easiest pipeline for decoding local file:
gst-launch-1.0 playbin uri=file://<file> video-sink=imxpxpvideosink
or
gst-launch-1.0 filesrc location=<file> ! decodebin ! imxpxpvideosink use-vsync=true
An example of more complex pipeline (picture-in-picture with videomixer component – logo.jpg file as a background, mixed with H264 video):
gst-launch-1.0 filesrc location=logo.jpg ! jpegdec ! imagefreeze ! videomixer name=mix sink_1::xpos=80 sink_1::ypos=100 ! imxpxpvideosink sync=false use-vsync=true filesrc location=video_001.mp4 ! decodebin ! mix.
‘Picture-in-picture’ example as a C application:
#include <stdlib.h>
#include <glib.h>
#include <gst/gst.h>
#define GST_VIDEOSINK "imxpxpvideosink"
#define SINK1_X_OFFSET 80
#define SINK1_Y_OFFSET 100
#define SYNC_FALSE 1
#define VSYNC_TRUE 1
static GMainLoop *loop = NULL;
static guint timeout = 1;
/*
* bus_call()
*/
static gboolean
bus_call (GstBus *bus,
GstMessage *msg,
gpointer data)
{
switch (GST_MESSAGE_TYPE (msg)) {
/* due to ImageFreeze component in pipeline - 'EOS' case
* should not be reached */
case GST_MESSAGE_EOS:
g_print (" [error ] unexpected 'EOS' message\n");
g_main_loop_quit (loop);
break;
case GST_MESSAGE_ERROR: {
gchar *debug;
GError *error;
gst_message_parse_error (msg, &error, &debug);
g_free (debug);
g_print (" [error ] %s\n", error->message);
g_error_free (error);
g_main_loop_quit (loop);
break;
}
default:
break;
}
return TRUE;
}
/*
* timeout_callback()
*
*/
static gboolean
timeout_callback (gpointer data)
{
g_print (" [status] timeout callback\n");
/* FIXME: send EOS to pipeline */
g_main_loop_quit (loop);
return FALSE;
}
static void
on_pad_added (GstElement *element,
GstPad *pad,
gpointer data)
{
GstElement *vidmixer = (GstElement *) data;
GstPad *sink_1;
gchar *name;
name = gst_pad_get_name (pad);
g_print (" [status] new pad %s was created\n", name);
/* set timeout */
if (g_strcmp0 (name, "src_0") == 0)
{
g_print (" [status] set timeout callback\n");
g_timeout_add (timeout, timeout_callback, loop);
}
g_free (name);
/* link decoder with videomixer */
g_print (" [status] on-pad-added callback\n");
gst_element_link (element, vidmixer);
/* set X/Y offsets for video sink_1 */
sink_1 = gst_element_get_static_pad (vidmixer, "sink_1");
g_object_set (sink_1, "ypos", SINK1_Y_OFFSET, NULL);
g_object_set (sink_1, "xpos", SINK1_X_OFFSET, NULL);
gst_object_unref (sink_1);
}
int
main (int argc,
char *argv[])
{
GstBus *bus;
GstElement *pipeline, *im_source, *vd_source, *jpegdec;
GstElement *imgfrez, *vidmixer, *decoder, *sink;
gchar *logo, *filename;
guint bus_watch_id;
/* init */
gst_init (&argc, &argv);
if (argc != 4) {
g_print (" [error ] invalid input args\n");
return -1;
}
logo = argv[1];
filename = argv[2];
timeout = (guint) atoi(argv[3]);
loop = g_main_loop_new (NULL, FALSE);
g_print (" [status] configuration: logo=%s\n", filename);
g_print (" [status] configuration: filename=%s\n", filename);
g_print (" [status] configuration: timeout=%d\n", timeout);
/* gstreamer elements */
pipeline = gst_pipeline_new ("somlabs-demo-player");
im_source = gst_element_factory_make ("filesrc", "image-source");
vd_source = gst_element_factory_make ("filesrc", "video-source");
jpegdec = gst_element_factory_make ("jpegdec", "jpeg-decoder");
imgfrez = gst_element_factory_make ("imagefreeze", "image-freeze");
vidmixer = gst_element_factory_make ("videomixer", "video-mixer");
decoder = gst_element_factory_make ("decodebin", "decode-bin");
sink = gst_element_factory_make (GST_VIDEOSINK, "video-sink");
if (!pipeline || !im_source || !vd_source || !jpegdec ||
!imgfrez || !vidmixer || !decoder || !sink) {
g_print (" [error ] cannot create element for pipeline\n");
return -1;
}
/* set up the pipeline */
g_object_set (G_OBJECT (im_source), "location", logo, NULL);
g_object_set (G_OBJECT (vd_source), "location", filename, NULL);
#if SYNC_FALSE
g_print (" [status] configuration: 'sync'=FALSE\n");
g_object_set (G_OBJECT (sink), "sync", FALSE, NULL);
#endif
#if VSYNC_TRUE
g_print (" [status] configuration: 'use-vsync'=TRUE\n");
g_object_set (G_OBJECT (sink), "use-vsync", TRUE, NULL);
#endif
/* add message handler */
bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
bus_watch_id = gst_bus_add_watch (bus, bus_call, NULL);
gst_object_unref (bus);
/* add all elements into the pipeline */
gst_bin_add_many (GST_BIN (pipeline), im_source, jpegdec, imgfrez,
vidmixer, sink, vd_source, decoder, NULL);
/* link elements together */
gst_element_link (im_source, jpegdec);
gst_element_link (jpegdec, imgfrez);
gst_element_link (vd_source, decoder);
gst_element_link (imgfrez, vidmixer);
gst_element_link (vidmixer,sink);
g_signal_connect (decoder, "pad-added", G_CALLBACK (on_pad_added), vidmixer);
/* set the pipeline to "playing" state */
g_print (" [status] pipeline: GST_STATE_PLAYING\n");
gst_element_set_state (pipeline, GST_STATE_PLAYING);
/* setup timeout */
g_timeout_add (timeout, timeout_callback, NULL);
/* enter to main loop */
g_print (" [status] enter to mainloop...\n");
g_main_loop_run (loop);
/* out of the main loop, clean up nicely */
g_print (" [status] pipeline: GST_STATE_NULL\n");
gst_element_set_state (pipeline, GST_STATE_NULL);
g_print (" [status] exiting...\n");
gst_object_unref (GST_OBJECT (pipeline));
g_source_remove (bus_watch_id);
g_main_loop_unref (loop);
return 0;
}
To compile and run app:
gcc som-player.c -o som-player `pkg-config --cflags --libs gstreamer-1.0` ./som-player
Ready to use image file
You can also use a prepared image file.
You need to download the image file: SoMLabs-VisionSOM-6ULL-Hardware-accelerated-video-playback.zip
Then you need to write the image file to microSD card. More information you can find in manuals: How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux or How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows.
