Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)
From SomLabs Wiki
Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)
Contents
Prerequisites
This tutorial based on Debian 9.2 image for VisionSOM-6ULL module. Please visit How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux or How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows for more detailed instruction how to create a bootable microSD card.
To enable support for LCD-TFT display, please download and apply enable-tft-lcd.patch for somlabs-visionsom-6ull.dts device tree file:
wget http://wiki.somlabs.com/images/c/cd/Enable-tft-lcd.zip unzip Enable-tft-lcd.zip patch /home/developer/source/somlabs-dts-1.0/somlabs-visionsom-6ull.dts ./enable-tft-lcd.patch
For more info how to customize, build and upload Device Tree file, please visit How to customize Debian 9.2 device tree.
Installing GStreamer multimedia framework
GStreamer is a pipeline-based multimedia framework which allows a programmer to create a variety of media-handling components, including simple audio playback, audio and video playback, recording and streaming. More info about gstreamer can be found here:
- https://en.wikipedia.org/wiki/GStreamer
- https://gstreamer.freedesktop.org/documentation/
- https://github.com/GStreamer
Install gstreamer1.x packages:
root@somlabs:~# apt-get update root@somlabs:~# apt-get install gstreamer1.0-x gstreamer1.0-tools root@somlabs:~# apt-get install gstreamer1.0-plugins-good root@somlabs:~# apt-get install gstreamer1.0-plugins-bad root@somlabs:~# apt-get install gstreamer1.0-libav
At this point you could use:
root@somlabs:~# gst-launch-1.0 videotestsrc ! fbdevsink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock <Ctrl+c> Interrupt: Stopping pipeline ... Execution ended after 0:00:04.732790131 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...
to display prebuild image/video pattern on the display, but it will not stretch to the display, and will not be hardware accelerated:
Adding GStreamer PxP engine support via gstreamer-imx
The multimedia performance of i.MX 6ULL processor is enhanced by a Pixel Processing Pipeline (PXP) to support 2D image processing, including color-space conversion, scaling, alpha-blending, and rotation. You can easily add PxP support via GStreamer and Gstreamer-imx plugins.
Install build dependencies for the gstreamer-imx package:
root@somlabs:~# apt-get install build-essential autoconf libtool root@somlabs:~# apt-get install wget python pkg-config root@somlabs:~# apt-get install libgstreamer1.0-dev root@somlabs:~# apt-get install libgstreamer-plugins-base1.0-dev
Clone gstreamer-imx repository (if you don't already have git installed, simply install it with sudo apt install git):
root@somlabs:~# git clone git://github.com/Freescale/gstreamer-imx.git Cloning into 'gstreamer-imx'... remote: Counting objects: 4349, done. remote: Total 4349 (delta 0), reused 0 (delta 0), pack-reused 4349 Receiving objects: 100% (4349/4349), 1.91 MiB | 1.35 MiB/s, done. Resolving deltas: 100% (3129/3129), done.
To compile gstreamer-imx we need a Linux kernel headers with i.MX additions for the PxP subsystems (kernel headers where at least linux/pxp_device.h can be found). Let's install .deb package with kernel headers (prebuilt .deb package is available as a part of somlabs-visionsom-6ull-debian-rootfs-qemu.tar.xz image):
root@somlabs:~# dpkg -i linux-headers-4.1.15_4.1.15-1_armhf.deb
Gstreamer-imx project uses the waf meta build system, to configure project run:
root@somlabs:~# cd gstreamer-imx root@somlabs:~/gstreamer-imx# ln -s /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ /usr/lib/gstreamer-1.0 root@somlabs:~/gstreamer-imx# ./waf configure --prefix=/usr --kernel-headers=/usr/src/linux-headers-4.1.15/include Setting top to : /root/gstreamer-imx Setting out to : /root/gstreamer-imx/build Checking for 'gcc' (C compiler) : /usr/bin/gcc Checking for compiler switch -O2 : yes ... checking for linux/pxp_device.h : yes PxP elements will be built checking for linux/fb.h and the IPU header linux/ipu.h : yes IPU elements will be built Checking for 'libimxvpuapi >= 0.10.3' : not found could not find installed imxvpuapi library - VPU elements will not be built ... 'configure' finished successfully (25.634s)
Now build and install gstreamer-imx:
root@somlabs:~/gstreamer-imx# ./waf Waf: Entering directory `/root/gstreamer-imx/build' [ 1/35] Compiling src/common/canvas.c [ 2/35] Compiling src/common/fd_object.c [ 3/35] Compiling src/common/phys_mem_allocator.c ... [27/35] Compiling src/compositor/gst-backport/gstimxbpvideoaggregator.c [28/35] Compiling src/compositor/compositor.c [29/35] Linking build/src/compositor/libgstimxcompositor.so [30/35] Linking build/src/blitter/libgstimxblitter.so [31/35] Linking build/src/ipu/libgstimxipu.so [32/35] Linking build/src/pxp/libgstimxpxp.so [33/35] Symlinking build/src/blitter/libgstimxblitter.so [34/35] Symlinking build/src/compositor/libgstimxcompositor.so [35/35] Symlinking build/src/common/libgstimxcommon.so Waf: Leaving directory `/root/gstreamer-imx/build' 'build' finished successfully (1m45.921s) root@somlabs:~/gstreamer-imx# ./waf install Waf: Entering directory `/root/gstreamer-imx/build' + install /usr/lib/libgstimxcommon.so.0.13.0 + symlink /usr/lib/libgstimxcommon.so (to libgstimxcommon.so.0.13.0) + symlink /usr/lib/libgstimxcommon.so.0 (to libgstimxcommon.so.0.13.0) + install /usr/lib/pkgconfig/gstimxcommon.pc ... Waf: Leaving directory `/root/gstreamer-imx/build' 'install' finished successfully (0.584s)
After this step you should be able to see newly installed imx plugins with gst-inspect-1.0 tool (which allows to get documentation on available elements and detailed information on a specific element):
root@somlabs:~/gstreamer-imx# gst-inspect-1.0 | grep imx imxpxp: imxpxpvideotransform: Freescale PxP video transform imxpxp: imxpxpvideosink: Freescale PxP video sink
Inspect imxpxpvideosink element:
root@somlabs:~/gstreamer-imx# gst-inspect-1.0 imxpxpvideosink Factory Details: Rank primary + 1 (257) Long-name Freescale PxP video sink Klass Sink/Video Description Video output using the Freescale PxP API Author Carlos Rafael Giani <dv@pseudoterminal.org> … Element Properties: name : The name of the object flags: readable, writable String. Default: "imxpxpvideosink0" parent : The parent of the object flags: readable, writable Object of type "GstObject"
Hardware-accelerated video-stream playback
Download sample video file:
wget http://craftymind.com/factory/html5video/BigBuckBunny_640x360.mp4
The easiest pipeline for decoding local file:
gst-launch-1.0 playbin uri=file://<file> video-sink=imxpxpvideosink
or
gst-launch-1.0 filesrc location=<file> ! decodebin ! imxpxpvideosink use-vsync=true
An example of more complex pipeline (picture-in-picture with videomixer component – logo.jpg file as a background, mixed with H264 video):
gst-launch-1.0 filesrc location=logo.jpg ! jpegdec ! imagefreeze ! videomixer name=mix sink_1::xpos=80 sink_1::ypos=100 ! imxpxpvideosink sync=false use-vsync=true filesrc location=video_001.mp4 ! decodebin ! mix.
‘Picture-in-picture’ example as a C application:
#include <stdlib.h> #include <glib.h> #include <gst/gst.h> #define GST_VIDEOSINK "imxpxpvideosink" #define SINK1_X_OFFSET 80 #define SINK1_Y_OFFSET 100 #define SYNC_FALSE 1 #define VSYNC_TRUE 1 static GMainLoop *loop = NULL; static guint timeout = 1; /* * bus_call() */ static gboolean bus_call (GstBus *bus, GstMessage *msg, gpointer data) { switch (GST_MESSAGE_TYPE (msg)) { /* due to ImageFreeze component in pipeline - 'EOS' case * should not be reached */ case GST_MESSAGE_EOS: g_print (" [error ] unexpected 'EOS' message\n"); g_main_loop_quit (loop); break; case GST_MESSAGE_ERROR: { gchar *debug; GError *error; gst_message_parse_error (msg, &error, &debug); g_free (debug); g_print (" [error ] %s\n", error->message); g_error_free (error); g_main_loop_quit (loop); break; } default: break; } return TRUE; } /* * timeout_callback() * */ static gboolean timeout_callback (gpointer data) { g_print (" [status] timeout callback\n"); /* FIXME: send EOS to pipeline */ g_main_loop_quit (loop); return FALSE; } static void on_pad_added (GstElement *element, GstPad *pad, gpointer data) { GstElement *vidmixer = (GstElement *) data; GstPad *sink_1; gchar *name; name = gst_pad_get_name (pad); g_print (" [status] new pad %s was created\n", name); /* set timeout */ if (g_strcmp0 (name, "src_0") == 0) { g_print (" [status] set timeout callback\n"); g_timeout_add (timeout, timeout_callback, loop); } g_free (name); /* link decoder with videomixer */ g_print (" [status] on-pad-added callback\n"); gst_element_link (element, vidmixer); /* set X/Y offsets for video sink_1 */ sink_1 = gst_element_get_static_pad (vidmixer, "sink_1"); g_object_set (sink_1, "ypos", SINK1_Y_OFFSET, NULL); g_object_set (sink_1, "xpos", SINK1_X_OFFSET, NULL); gst_object_unref (sink_1); } int main (int argc, char *argv[]) { GstBus *bus; GstElement *pipeline, *im_source, *vd_source, *jpegdec; GstElement *imgfrez, *vidmixer, *decoder, *sink; gchar *logo, *filename; guint bus_watch_id; /* init */ gst_init (&argc, &argv); if (argc != 4) { g_print (" [error ] invalid input args\n"); return -1; } logo = argv[1]; filename = argv[2]; timeout = (guint) atoi(argv[3]); loop = g_main_loop_new (NULL, FALSE); g_print (" [status] configuration: logo=%s\n", filename); g_print (" [status] configuration: filename=%s\n", filename); g_print (" [status] configuration: timeout=%d\n", timeout); /* gstreamer elements */ pipeline = gst_pipeline_new ("somlabs-demo-player"); im_source = gst_element_factory_make ("filesrc", "image-source"); vd_source = gst_element_factory_make ("filesrc", "video-source"); jpegdec = gst_element_factory_make ("jpegdec", "jpeg-decoder"); imgfrez = gst_element_factory_make ("imagefreeze", "image-freeze"); vidmixer = gst_element_factory_make ("videomixer", "video-mixer"); decoder = gst_element_factory_make ("decodebin", "decode-bin"); sink = gst_element_factory_make (GST_VIDEOSINK, "video-sink"); if (!pipeline || !im_source || !vd_source || !jpegdec || !imgfrez || !vidmixer || !decoder || !sink) { g_print (" [error ] cannot create element for pipeline\n"); return -1; } /* set up the pipeline */ g_object_set (G_OBJECT (im_source), "location", logo, NULL); g_object_set (G_OBJECT (vd_source), "location", filename, NULL); #if SYNC_FALSE g_print (" [status] configuration: 'sync'=FALSE\n"); g_object_set (G_OBJECT (sink), "sync", FALSE, NULL); #endif #if VSYNC_TRUE g_print (" [status] configuration: 'use-vsync'=TRUE\n"); g_object_set (G_OBJECT (sink), "use-vsync", TRUE, NULL); #endif /* add message handler */ bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline)); bus_watch_id = gst_bus_add_watch (bus, bus_call, NULL); gst_object_unref (bus); /* add all elements into the pipeline */ gst_bin_add_many (GST_BIN (pipeline), im_source, jpegdec, imgfrez, vidmixer, sink, vd_source, decoder, NULL); /* link elements together */ gst_element_link (im_source, jpegdec); gst_element_link (jpegdec, imgfrez); gst_element_link (vd_source, decoder); gst_element_link (imgfrez, vidmixer); gst_element_link (vidmixer,sink); g_signal_connect (decoder, "pad-added", G_CALLBACK (on_pad_added), vidmixer); /* set the pipeline to "playing" state */ g_print (" [status] pipeline: GST_STATE_PLAYING\n"); gst_element_set_state (pipeline, GST_STATE_PLAYING); /* setup timeout */ g_timeout_add (timeout, timeout_callback, NULL); /* enter to main loop */ g_print (" [status] enter to mainloop...\n"); g_main_loop_run (loop); /* out of the main loop, clean up nicely */ g_print (" [status] pipeline: GST_STATE_NULL\n"); gst_element_set_state (pipeline, GST_STATE_NULL); g_print (" [status] exiting...\n"); gst_object_unref (GST_OBJECT (pipeline)); g_source_remove (bus_watch_id); g_main_loop_unref (loop); return 0; }
To compile and run app:
gcc som-player.c -o som-player `pkg-config --cflags --libs gstreamer-1.0` ./som-player
Ready to use image file
You can also use a prepared image file.
You need to download the image file: SoMLabs-VisionSOM-6ULLl-Hardware-accelerated-video-playback.zip
Then you need to write the image file to microSD card. More information you can find in manuals: How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux or How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows.