View source for Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)

{{PageHeader|Hardware-accelerated video playback on the VisionSOM-6ULL (GStreamer with support for PxP engine)}}

== Prerequisites ==
This tutorial based on Debian 9.2 image for VisionSOM-6ULL module. Please visit [[How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux]] or [[How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows]] for more detailed instruction how to create a bootable microSD card.

To enable support for LCD-TFT display, please download and apply enable-tft-lcd.patch for somlabs-visionsom-6ull.dts device tree file:

<pre>wget http://wiki.somlabs.com/images/c/cd/Enable-tft-lcd.zip
unzip Enable-tft-lcd.zip
patch /home/developer/source/somlabs-dts-1.0/somlabs-visionsom-6ull.dts ./enable-tft-lcd.patch</pre>
 
For more info how to customize, build and upload Device Tree file, please visit [[How to customize Debian 9.2 device tree]].

== Installing GStreamer multimedia framework ==

GStreamer is a pipeline-based multimedia framework which allows a programmer to create a variety of media-handling components, including simple audio playback, audio and video playback, recording and streaming. More info about gstreamer can be found here:

* https://en.wikipedia.org/wiki/GStreamer
* https://gstreamer.freedesktop.org/documentation/
* https://github.com/GStreamer

Install gstreamer1.x packages:

<pre>root@somlabs:~# apt-get update
root@somlabs:~# apt-get install gstreamer1.0-x gstreamer1.0-tools
root@somlabs:~# apt-get install gstreamer1.0-plugins-good 
root@somlabs:~# apt-get install gstreamer1.0-plugins-bad
root@somlabs:~# apt-get install gstreamer1.0-libav</pre>

At this point you could use:

<pre>root@somlabs:~# gst-launch-1.0 videotestsrc ! fbdevsink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
<Ctrl+c>
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.732790131
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...</pre>

to display prebuild image/video pattern on the display, but it will not stretch to the display, and will not be hardware accelerated:

[[File:Wiki-somlabs-gstreamer-1.jpg|center|700px]]

== Adding GStreamer PxP engine support via gstreamer-imx ==

The multimedia performance of i.MX 6ULL processor is enhanced by a Pixel Processing Pipeline (PXP) to support 2D image processing, including color-space conversion, scaling, alpha-blending, and rotation. You can easily add PxP support via GStreamer and Gstreamer-imx plugins. 

Install build dependencies for the gstreamer-imx package:

<pre>root@somlabs:~# apt-get install build-essential autoconf libtool
root@somlabs:~# apt-get install wget python pkg-config
root@somlabs:~# apt-get install libgstreamer1.0-dev
root@somlabs:~# apt-get install libgstreamer-plugins-base1.0-dev</pre>

Clone gstreamer-imx repository (if you don't already have git installed, simply install it with sudo apt install git):

<pre>root@somlabs:~# git clone git://github.com/Freescale/gstreamer-imx.git
Cloning into 'gstreamer-imx'...
remote: Counting objects: 4349, done.
remote: Total 4349 (delta 0), reused 0 (delta 0), pack-reused 4349
Receiving objects: 100% (4349/4349), 1.91 MiB | 1.35 MiB/s, done.
Resolving deltas: 100% (3129/3129), done.</pre>

To compile gstreamer-imx we need a Linux kernel headers with i.MX additions for the PxP subsystems (kernel headers where at least linux/pxp_device.h can be found). Let's install .deb package with kernel headers (prebuilt .deb package is available as a part of somlabs-visionsom-6ull-debian-rootfs-qemu.tar.xz image):

<pre>root@somlabs:~# dpkg -i linux-headers-4.1.15_4.1.15-1_armhf.deb</pre>

Gstreamer-imx project uses the waf meta build system, to configure project run:
 
<pre>root@somlabs:~# cd gstreamer-imx

root@somlabs:~/gstreamer-imx# ln -s /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ /usr/lib/gstreamer-1.0

root@somlabs:~/gstreamer-imx# ./waf configure --prefix=/usr --kernel-headers=/usr/src/linux-headers-4.1.15/include
Setting top to : /root/gstreamer-imx 
Setting out to : /root/gstreamer-imx/build 
Checking for 'gcc' (C compiler) : /usr/bin/gcc 
Checking for compiler switch -O2 : yes 
...
checking for linux/pxp_device.h : yes 
PxP elements will be built 
checking for linux/fb.h and the IPU header linux/ipu.h : yes 
IPU elements will be built 
Checking for 'libimxvpuapi >= 0.10.3' : not found 
could not find installed imxvpuapi library - VPU elements will not be built 
...
'configure' finished successfully (25.634s)</pre>

Now build and install gstreamer-imx:

<pre>root@somlabs:~/gstreamer-imx# ./waf 
Waf: Entering directory `/root/gstreamer-imx/build'
[ 1/35] Compiling src/common/canvas.c
[ 2/35] Compiling src/common/fd_object.c
[ 3/35] Compiling src/common/phys_mem_allocator.c
...
[27/35] Compiling src/compositor/gst-backport/gstimxbpvideoaggregator.c
[28/35] Compiling src/compositor/compositor.c
[29/35] Linking build/src/compositor/libgstimxcompositor.so
[30/35] Linking build/src/blitter/libgstimxblitter.so
[31/35] Linking build/src/ipu/libgstimxipu.so
[32/35] Linking build/src/pxp/libgstimxpxp.so
[33/35] Symlinking build/src/blitter/libgstimxblitter.so
[34/35] Symlinking build/src/compositor/libgstimxcompositor.so
[35/35] Symlinking build/src/common/libgstimxcommon.so
Waf: Leaving directory `/root/gstreamer-imx/build'
'build' finished successfully (1m45.921s)

root@somlabs:~/gstreamer-imx# ./waf install
Waf: Entering directory `/root/gstreamer-imx/build'
+ install /usr/lib/libgstimxcommon.so.0.13.0
+ symlink /usr/lib/libgstimxcommon.so (to libgstimxcommon.so.0.13.0)
+ symlink /usr/lib/libgstimxcommon.so.0 (to libgstimxcommon.so.0.13.0)
+ install /usr/lib/pkgconfig/gstimxcommon.pc
...
Waf: Leaving directory `/root/gstreamer-imx/build'
'install' finished successfully (0.584s)</pre>

After this step you should be able to see newly installed imx plugins with
gst-inspect-1.0 tool (which allows to get documentation on available elements and detailed information on a specific element):

<pre>root@somlabs:~/gstreamer-imx# gst-inspect-1.0 | grep imx
imxpxp:  imxpxpvideotransform: Freescale PxP video transform
imxpxp:  imxpxpvideosink: Freescale PxP video sink</pre>

Inspect imxpxpvideosink element:

<pre>root@somlabs:~/gstreamer-imx# gst-inspect-1.0 imxpxpvideosink
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                Freescale PxP video sink
  Klass                    Sink/Video
  Description              Video output using the Freescale PxP API
  Author                   Carlos Rafael Giani <dv@pseudoterminal.org>
…

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "imxpxpvideosink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"</pre>

== Hardware-accelerated video-stream playback ==

Download sample video file:

<pre>wget http://craftymind.com/factory/html5video/BigBuckBunny_640x360.mp4</pre>

The easiest pipeline for decoding local file:

<pre>gst-launch-1.0 playbin uri=file://<file> video-sink=imxpxpvideosink</pre>

or

<pre>gst-launch-1.0 filesrc location=<file> ! decodebin ! imxpxpvideosink use-vsync=true</pre>

[[File:Wiki-somlabs-gstreamer-2.jpg|700px|frameless|center]]

An example of more complex pipeline (picture-in-picture with videomixer component – logo.jpg file as a background, mixed with H264 video):

<pre>gst-launch-1.0 filesrc location=logo.jpg ! jpegdec ! imagefreeze ! videomixer name=mix sink_1::xpos=80 sink_1::ypos=100 ! imxpxpvideosink sync=false use-vsync=true filesrc location=video_001.mp4 ! decodebin ! mix.</pre>

[[File:Wiki-somlabs-gstreamer-3.jpg|700px|frameless|center]]

‘Picture-in-picture’ example as a C application:

<pre>#include <stdlib.h>
#include <glib.h>
#include <gst/gst.h>

#define GST_VIDEOSINK	"imxpxpvideosink"
#define SINK1_X_OFFSET	80
#define SINK1_Y_OFFSET	100

#define SYNC_FALSE	1
#define VSYNC_TRUE	1

static GMainLoop *loop = NULL;
static guint timeout = 1;

/*
 * bus_call()
 */
static gboolean
bus_call (GstBus      *bus,
          GstMessage  *msg,
          gpointer     data)
{
  switch (GST_MESSAGE_TYPE (msg)) {

    /* due to ImageFreeze component in pipeline - 'EOS' case
     * should not be reached */
    case GST_MESSAGE_EOS:
      g_print ("   [error ] unexpected 'EOS' message\n");
      g_main_loop_quit (loop);
      break;

    case GST_MESSAGE_ERROR: {
      gchar  *debug;
      GError *error;

      gst_message_parse_error (msg, &error, &debug);
      g_free (debug);

      g_print ("   [error ] %s\n", error->message);
      g_error_free (error);

      g_main_loop_quit (loop);
      break;
    }

    default:
      break;
  }

  return TRUE;
}

/*
 * timeout_callback()
 *
 */
static gboolean
timeout_callback (gpointer data)
{
  g_print ("   [status] timeout callback\n");

  /* FIXME: send EOS to pipeline */
  g_main_loop_quit (loop);
  return FALSE;
}
static void
on_pad_added (GstElement  *element,
              GstPad      *pad,
              gpointer     data)
{
  GstElement *vidmixer = (GstElement *) data;
  GstPad     *sink_1;
  gchar      *name;

  name = gst_pad_get_name (pad);
  g_print ("   [status] new pad %s was created\n", name);

  /* set timeout */
  if (g_strcmp0 (name, "src_0") == 0)
    {
      g_print ("   [status] set timeout callback\n");
      g_timeout_add (timeout, timeout_callback, loop);
    }
  g_free (name);

  /* link decoder with videomixer */
  g_print ("   [status] on-pad-added callback\n");
  gst_element_link (element, vidmixer);

  /* set X/Y offsets for video sink_1 */
  sink_1 = gst_element_get_static_pad (vidmixer, "sink_1");
  g_object_set (sink_1, "ypos", SINK1_Y_OFFSET, NULL);
  g_object_set (sink_1, "xpos", SINK1_X_OFFSET, NULL);
  gst_object_unref (sink_1);
}

int
main (int    argc,
      char  *argv[])
{
  GstBus *bus;
  GstElement *pipeline, *im_source, *vd_source, *jpegdec;
  GstElement *imgfrez, *vidmixer, *decoder, *sink;
  gchar *logo, *filename;
  guint bus_watch_id;

  /* init */
  gst_init (&argc, &argv);
  if (argc != 4) {
    g_print ("   [error ] invalid input args\n");
    return -1;
  }

  logo = argv[1];
  filename = argv[2];
  timeout = (guint) atoi(argv[3]);
  loop = g_main_loop_new (NULL, FALSE);

  g_print ("   [status] configuration: logo=%s\n", filename);
  g_print ("   [status] configuration: filename=%s\n", filename);
  g_print ("   [status] configuration: timeout=%d\n", timeout);

  /* gstreamer elements */
  pipeline  = gst_pipeline_new ("somlabs-demo-player");
  im_source = gst_element_factory_make ("filesrc",     "image-source");
  vd_source = gst_element_factory_make ("filesrc",     "video-source");
  jpegdec   = gst_element_factory_make ("jpegdec",     "jpeg-decoder");
  imgfrez   = gst_element_factory_make ("imagefreeze", "image-freeze");
  vidmixer  = gst_element_factory_make ("videomixer",  "video-mixer");
  decoder   = gst_element_factory_make ("decodebin",   "decode-bin");
  sink      = gst_element_factory_make (GST_VIDEOSINK, "video-sink");

  if (!pipeline || !im_source || !vd_source || !jpegdec ||
      !imgfrez || !vidmixer || !decoder || !sink) {
    g_print ("   [error ] cannot create element for pipeline\n");
    return -1;
  }

  /* set up the pipeline */
  g_object_set (G_OBJECT (im_source), "location", logo, NULL);
  g_object_set (G_OBJECT (vd_source), "location", filename, NULL);

#if SYNC_FALSE
  g_print ("   [status] configuration: 'sync'=FALSE\n");
  g_object_set (G_OBJECT (sink), "sync", FALSE, NULL);
#endif

#if VSYNC_TRUE
  g_print ("   [status] configuration: 'use-vsync'=TRUE\n");
  g_object_set (G_OBJECT (sink), "use-vsync", TRUE, NULL);
#endif

  /* add message handler */
  bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
  bus_watch_id = gst_bus_add_watch (bus, bus_call, NULL);
  gst_object_unref (bus);

  /* add all elements into the pipeline */
  gst_bin_add_many (GST_BIN (pipeline), im_source, jpegdec, imgfrez,
                    vidmixer, sink, vd_source, decoder, NULL);

  /* link elements together */
  gst_element_link (im_source, jpegdec);
  gst_element_link (jpegdec, imgfrez);
  gst_element_link (vd_source, decoder);
  gst_element_link (imgfrez, vidmixer);
  gst_element_link (vidmixer,sink);
  g_signal_connect (decoder, "pad-added", G_CALLBACK (on_pad_added), vidmixer);

  /* set the pipeline to "playing" state */
  g_print ("   [status] pipeline: GST_STATE_PLAYING\n");
  gst_element_set_state (pipeline, GST_STATE_PLAYING);

  /* setup timeout */
  g_timeout_add (timeout, timeout_callback, NULL);

  /* enter to main loop */
  g_print ("   [status] enter to mainloop...\n");
  g_main_loop_run (loop);

  /* out of the main loop, clean up nicely */
  g_print ("   [status] pipeline: GST_STATE_NULL\n");
  gst_element_set_state (pipeline, GST_STATE_NULL);

  g_print ("   [status] exiting...\n");
  gst_object_unref (GST_OBJECT (pipeline));
  g_source_remove (bus_watch_id);
  g_main_loop_unref (loop);

  return 0;
}</pre>

To compile and run app:

<pre>gcc som-player.c -o som-player `pkg-config --cflags --libs gstreamer-1.0`
./som-player</pre>


== Ready to use image file ==
You can also use a prepared image file.

You need to download the image file:
[http://ftp.somlabs.com/VisionSOM-6ULL/SoMLabs-VisionSOM-6ULL-Hardware-accelerated-video-playback.zip SoMLabs-VisionSOM-6ULL-Hardware-accelerated-video-playback.zip]

Then you need to write the image file to microSD card. More information you can find in manuals: [[How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Linux]] or [[How to prepare SD Card with Debian 9.2 for VisionSOM-6ULL on Windows]].