Building a Custom GStreamer Plugin for NVIDIA DeepStream

Building a Custom GStreamer Plugin for NVIDIA DeepStream

NVIDIA DeepStream is a powerful framework for building AI-powered video analytics applications, enabling real-time processing of video streams using deep learning models. At its core, DeepStream leverages GStreamer, an open-source multimedia framework, to construct flexible and efficient pipelines. While DeepStream comes with a rich set of pre-built plugins, many advanced use cases—such as custom pre-processing, specialized inference logic, or unique post-processing—require building your own GStreamer plugin. This article provides a practical, step-by-step guide to creating a custom GStreamer plugin for NVIDIA DeepStream, covering requirements, installation, and usage examples.

By the end of this guide, you will understand how to extend DeepStream’s capabilities by writing a simple custom element that modifies video frames before passing them downstream. This approach is widely used in production systems, as noted in resources like the Towards Data Science blog, which often explores custom pipeline components for AI applications.

Requirements

Before diving into plugin development, ensure your system meets the following prerequisites. These steps are based on standard setups recommended in NVIDIA documentation and general AI development practices.

Hardware Requirements

• **NVIDIA GPU with CUDA support**: A Tesla T4, V100, or any modern GeForce/Quadro GPU is recommended for hardware acceleration.
• **Sufficient RAM**: At least 8 GB (16 GB or more for high-resolution streams).

Software Requirements

• **Ubuntu 20.04 or 22.04 LTS**: DeepStream is optimized for these distributions.
• **NVIDIA Driver**: Version 525 or later (check with `nvidia-smi`).
• **CUDA Toolkit**: Version 11.8 or 12.x.
• **GStreamer Development Libraries**: Version 1.18 or later.
• **NVIDIA DeepStream SDK**: Version 6.3 or 7.0 (latest stable).
• **Build Tools**: `gcc`, `g++`, `make`, `cmake`, and `pkg-config`.

Verify Installation

First, confirm that GStreamer and DeepStream are installed:

# Check GStreamer version
gst-launch-1.0 --version

# Check DeepStream installation (look for deepstream-app)
which deepstream-app

# Verify CUDA availability
nvcc --version

If any command fails, refer to NVIDIA’s official DeepStream installation guide (available on their developer site) for troubleshooting.

Step-by-Step Installation

Building a custom GStreamer plugin involves creating a new element that implements the GStreamer plugin interface. DeepStream plugins typically inherit from `GstBaseTransform` or `GstElement`, allowing them to integrate seamlessly with the framework. Below, we outline the process from scratch.

1. Set Up the Development Environment

Create a project directory and initialize a basic plugin structure:

mkdir custom-deepstream-plugin
cd custom-deepstream-plugin
mkdir src include

Install necessary development packages:

sudo apt-get update
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev \
    libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base \
    gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
    gstreamer1.0-plugins-ugly gstreamer1.0-libav \
    libgstrtspserver-1.0-dev libjson-glib-dev

For DeepStream-specific headers, ensure the DeepStream SDK is installed. The common path is `/opt/nvidia/deepstream/deepstream-7.0/`.

2. Write the Plugin Source Code

Create a file named `src/gstmycustomplugin.c`. This example implements a simple filter that converts video frames to grayscale (a common pre-processing step for certain AI models).

#include <gst/gst.h>
#include <gst/video/video.h>
#include <gst/video/gstvideofilter.h>

/* Plugin boilerplate */
GST_DEBUG_CATEGORY_STATIC (gst_my_custom_plugin_debug);
#define GST_CAT_DEFAULT gst_my_custom_plugin_debug

/* Filter structure */
typedef struct _GstMyCustomPlugin {
  GstVideoFilter base_videofilter;
  /* Add custom properties here */
} GstMyCustomPlugin;

typedef struct _GstMyCustomPluginClass {
  GstVideoFilterClass base_videofilter_class;
} GstMyCustomPluginClass;

/* Define the element's pad templates */
static GstStaticPadTemplate sink_factory = GST_STATIC_PAD_TEMPLATE (
    "sink",
    GST_PAD_SINK,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE ("{ GRAY8, NV12 }"))
);

static GstStaticPadTemplate src_factory = GST_STATIC_PAD_TEMPLATE (
    "src",
    GST_PAD_SRC,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE ("{ GRAY8, NV12 }"))
);

/* Transform function: convert to grayscale (simplified) */
static GstFlowReturn
gst_my_custom_plugin_transform_frame_ip (GstVideoFilter *filter,
                                         GstVideoFrame *frame)
{
  GstMapInfo map;
  gst_buffer_map (frame->buffer, &map, GST_MAP_WRITE);

  guint8 *data = map.data;
  guint width = GST_VIDEO_FRAME_WIDTH (frame);
  guint height = GST_VIDEO_FRAME_HEIGHT (frame);
  guint stride = GST_VIDEO_FRAME_PLANE_STRIDE (frame, 0);

  /* Simple grayscale: average R, G, B for each pixel (assumes RGB) */
  for (guint y = 0; y < height; y++) {
    for (guint x = 0; x < width; x++) {
      guint8 *pixel = data + y * stride + x * 3;
      guint8 gray = (pixel[0] + pixel[1] + pixel[2]) / 3;
      pixel[0] = gray;
      pixel[1] = gray;
      pixel[2] = gray;
    }
  }

  gst_buffer_unmap (frame->buffer, &map);
  return GST_FLOW_OK;
}

/* Register the element */
#define PACKAGE "MyCustomPlugin"
#define PACKAGE_NAME "My Custom Plugin"
#define PACKAGE_VERSION "1.0.0"
#define PACKAGE_STRING "MyCustomPlugin 1.0.0"
#define PACKAGE_BUGREPORT "https://example.com"

GST_BOILERPLATE_FULL (GstMyCustomPlugin, gst_my_custom_plugin, GstVideoFilter,
                      GST_TYPE_VIDEO_FILTER, PACKAGE, PACKAGE_VERSION,
                      PACKAGE_STRING, PACKAGE_BUGREPORT);

static void
gst_my_custom_plugin_base_init (gpointer g_class)
{
  GstElementClass *element_class = GST_ELEMENT_CLASS (g_class);

  gst_element_class_set_details_simple (element_class,
      "My Custom Plugin",
      "Filter/Effect/Video",
      "Converts video to grayscale",
      "Author <author@example.com>");

  gst_element_class_add_pad_template (element_class,
      gst_static_pad_template_get (&sink_factory));
  gst_element_class_add_pad_template (element_class,
      gst_static_pad_template_get (&src_factory));
}

static void
gst_my_custom_plugin_class_init (GstMyCustomPluginClass *klass)
{
  GstVideoFilterClass *video_filter_class = GST_VIDEO_FILTER_CLASS (klass);
  video_filter_class->transform_frame_ip = gst_my_custom_plugin_transform_frame_ip;
}

static void
gst_my_custom_plugin_init (GstMyCustomPlugin *filter)
{
  /* Initialize custom properties here */
}

/* Plugin entry point */
static gboolean
plugin_init (GstPlugin *plugin)
{
  GST_DEBUG_CATEGORY_INIT (gst_my_custom_plugin_debug, "mycustomplugin", 0,
                           "My Custom Plugin");
  return gst_element_register (plugin, "mycustomplugin", GST_RANK_NONE,
                               GST_TYPE_MY_CUSTOM_PLUGIN);
}

GST_PLUGIN_DEFINE (GST_VERSION_MAJOR, GST_VERSION_MINOR,
                   mycustomplugin, "My Custom Plugin",
                   plugin_init, PACKAGE_VERSION, GST_LICENSE,
                   PACKAGE, PACKAGE_BUGREPORT)

3. Create the Build System

Create a `CMakeLists.txt` file in the project root:

cmake_minimum_required(VERSION 3.10)
project(mycustomplugin VERSION 1.0.0 LANGUAGES C)

find_package(PkgConfig REQUIRED)
pkg_check_modules(GSTREAMER REQUIRED gstreamer-1.0 gstreamer-video-1.0)
pkg_check_modules(GSTREAMER_BASE REQUIRED gstreamer-base-1.0)

add_library(gstmycustomplugin MODULE src/gstmycustomplugin.c)
target_include_directories(gstmycustomplugin PRIVATE include ${GSTREAMER_INCLUDE_DIRS})
target_link_libraries(gstmycustomplugin PRIVATE ${GSTREAMER_LIBRARIES} ${GSTREAMER_BASE_LIBRARIES})
set_target_properties(gstmycustomplugin PROPERTIES PREFIX "")

4. Build the Plugin

Compile the plugin:

mkdir build
cd build
cmake ..
make

This produces a shared library `libgstmycustomplugin.so`.

5. Install the Plugin

Copy the library to GStreamer’s plugin directory:

sudo cp libgstmycustomplugin.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/
sudo ldconfig

Alternatively, set the `GST_PLUGIN_PATH` environment variable to the build directory for testing:

export GST_PLUGIN_PATH=$PWD

6. Verify the Plugin

Use `gst-inspect-1.0` to confirm the plugin is recognized:

gst-inspect-1.0 mycustomplugin

You should see output detailing the element’s pads and capabilities.

Usage Examples

Now that your custom plugin is built, integrate it into a DeepStream pipeline. Below are two practical examples.

Example 1: Basic Grayscale Conversion

Test the plugin standalone with a video file:

gst-launch-1.0 filesrc location=/path/to/input.mp4 ! \
    qtdemux ! h264parse ! nvv4l2decoder ! \
    nvvidconv ! video/x-raw,format=NV12 ! \
    mycustomplugin ! \
    nvvidconv ! nveglglessink

This pipeline decodes an H.264 video, converts it to NV12 format (common in DeepStream), applies your custom grayscale filter, and displays the output.

Example 2: Integration with DeepStream

For a full DeepStream pipeline, use the plugin as a filter between inference and display:

import sys
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib

# Initialize GStreamer
Gst.init(None)

# Build the pipeline string
pipeline_str = """
    filesrc location=/path/to/video.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! 
    nvstreammux batch-size=1 width=1920 height=1080 ! 
    nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-7.0/samples/configs/deepstream-app/config_infer_primary.txt ! 
    nvtracker ! 
    mycustomplugin ! 
    nvdsosd ! 
    nveglglessink
"""

# Create and run the pipeline
pipeline = Gst.parse_launch(pipeline_str)
pipeline.set_state(Gst.State.PLAYING)

bus = pipeline.get_bus()
msg = bus.timed_pop_filtered(Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS | Gst.MessageType.ERROR)
pipeline.set_state(Gst.State.NULL)

This example inserts `mycustomplugin` after the tracker to apply grayscale transformation before overlay rendering. Adjust the config file path to your DeepStream installation.

Example 3: Command-Line DeepStream App

Alternatively, use `deepstream-app` with a custom configuration file that includes your plugin:

deepstream-app -c deepstream_app_config.txt

In the config file, add your plugin to the pipeline definition:

[primary-gie]
enable=1
config-file-path=config_infer_primary.txt

[tracker]
enable=1

[mycustomplugin]
# No additional configuration needed for this simple plugin

[sink0]
enable=1
type=3  # Fakesink for headless operation

Conclusion

Building a custom GStreamer plugin for NVIDIA DeepStream empowers developers to tailor video analytics pipelines to unique requirements. This guide walked through the entire process—from setting up the development environment and writing a simple grayscale filter to compiling, installing, and integrating the plugin into real pipelines. By mastering this skill, you can extend DeepStream’s capabilities for tasks like custom pre-processing, specialized inference logic, or unique post-processing effects.

The approach outlined here is foundational, as noted in resources like the Towards Data Science blog, which often explores such custom components for AI applications. For more advanced plugins—such as those integrating TensorRT models or handling multiple streams—refer to NVIDIA’s DeepStream plugin API documentation and community examples. With this knowledge, you are ready to build sophisticated, production-grade video analytics systems that leverage the full power of NVIDIA GPUs and GStreamer’s flexibility.