Video analytics

Today , there are billions of  of video cameras in our homes,  phones , ATMs ,  baby monitors , laptops , smart watches , traffic monitoring , IOT devices , bots , you name it. The underlying purpose of most of them  is to capture media streams and optimize the content for further processing.

Stages of video Analytics-

1.Data Acquisition

Data gathered from multiple camera sourced need to be streamed in an aggregated manner to data processing unit in an analytics engine  or for archiving . It may be subjected to overall monitoring such as sunlight hours etc  or detailed low level object identification  such as facial recognition of passengers

2.Transformation to Data Sets

The assimilated data is grouped to operable entities. The identification and classification is done by adding attributes to recognizable shapes ,movements , edges and patterns  .

3.Calculate deviation or compliance

A trained model recognizes normal behavior and differentiation from the same is calculated .


Video content Analytics in surveillance 

Considering the use case for monitoring and surveillance cameras  , there is a growing need for realtime video analytics  , to ” detect and determine the temporal and spatial events “. Consider Surveillance cam recordings as forensic evidence or just monitoring incidents and reporting the specific crucial segments of video , both these usecases involve filtering a vast amount of recorded or steaming media to filter out the exact parts that the authorities are looking for. This involves custom identification and recognition of objects in frames .

There is growing research into extracting real time events of interest , minimizing the search time and maximizing accuracy from computer vision .

Consider following use-cases :

  1. Surveillance cam in solar farms or home based setups to predict sun light hours and forecast energy generation value  . Described in greater details here .
  2. Traffic monitoring cameras :
  • Automatic license / number plate recognition – surveillance cams for traffic need to record vehicle plate number to identify and tag the vehicles as they pass by  .
  • Car  dashboard cams for investigative purposes  post accidents and insurance claims
  • Motion tracking – Mapping the vehicle movement to detect any wrong turns , overtakes , parking etc
  • Scan for QR codes and passes at toll gates.
  • Identifying over-speeding vehicles

3. Security and Law enforcement

  • Trigger alarms or lockdowns  on suspicious activity or intrusion into safe facility
  • Virtual fencing and perimeter breach – Map facial identification from known suspects
  • Detection of left items and acceleration of emergency response

Communication based video analytics 

Unified enterprise communication , conferences , meeting , online webcasts , webinars , social messengers , online project demos are extensively using video analytics for building intuitive use cases and boost innovation around their platform . Few examples of vast number of usecases are

  1. Sentiment Analysis : Capturing emotions by mapping key words to ascertain whether the meeting went , happy , positive , productive or  sad , complaining , negative
  2. Augmented Reality for overlaying information such as interactive manual or a image . Areas for current usage include , e-learning and customer support .
  3. Dynamic masking for privacy

Autonomous Robot purpose Video analytics

Self driving drones , cars and even bots extensive use the feed from wide angle / fish eye lens cameras to create a 3D model of their movement in given space of 3 dimensional coordinates system.

Key technologies includes :

  1. Ego-motion estimation – mapping a 3D space with captured camera feed
  2. Deep Learning ( part of AI) from continuous feed from video cameras to find route around obstacles
  3. Remote monitoring for an unmanned vehicle
  4. Sterile monitoring for a unreachable or hazardous area example war-zone , outer  territorial objects as moon , mars , satellites

Bio-metrics based Video analytics 

Often video feed is now used for advanced search, redaction and facial recognition , which leads to features such as

  • unlocking laptop or phone
  • performing click with blink of eyes
  • creating concentration maps of webpage based on where eyes focused

Read more about role of webrtc in bio- metrics here

Video analytics in Industrial and Retail applications 

Application of video analytics in Industrial landscape are manyfold . On one hand it can be for intelligence and  information gathering such as worker foot count . Machine left unattended etc while on the other hand by using specific image optimization techniques can also audit automation testing of engines , machine parts , rotation counts etc .

  1. Flame and Smoke Detection – images from video streams are analysed for color chrominance, flickering ratio, shape, pattern and moving direction to ascertain a fire hazard.
  2. Collect demographics of the area with people counting
  3. Ensure quality control and procedural compliance
  4. Identify tail-gateing or loitering


List of few companies focusing on Video Analytics :

  1. Avigilon -
  2. 3VR –
  3. Intelli-vision –
  4. IPsotek –
  5. aimetis –


Edge Analytics

Performing data analytics at application level on the edge whole system architecture instead of at the core or data warehouse level. The advantage to the computation at fringes of network instead of a centralized system are faster response times and standalone off grid functionality support .

The humongous data collected over by IOT devices m machinery  , sensors  , servers , firewalls , routers , switches gateways and all other types of components are increasingly getting analysed and acted upon at the edge ,  independently with machine learning instead of data centers and network operation centers . With the help of feedback loops and deep learning one could add data drive intelligence to how operations are performed at critical machines arts such as autonomous bots or industrial setups.

Error Recovery and streaming

To control the incoming data stream , we divide and classify the content into packets , add custom classification header and stream them down to the server. In the event of data congestion of back pressure , some non significant packets are either dropped or added to a new dead queue .  The system  is thus made stable for high availability and BCP / fail-over recovery .




GStreamer ( LGPL )ia a media handling library written in C for applicatioan such as streaming , recording, playback , mixing and editing attributes etc. Even enhnaced applicaiosn such as tsrancoding , media ormat conversion , streaming servers for embeeded devices ( read more about Gstreamer in RPi in my srticle here).
It encompases various codecs, filters and is modular with plugins developement to enhance its capabilities. Media Streaming application developers use it as part of their framework at either the broadcaster’s end or as media player.

gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink

To list all packages of Gstreamer

pkg-config --list-all | grep gstreamer
  • gstreamer-gl-1.0 GStreamer OpenGL Plugins Libraries – Streaming media framework, OpenGL plugins libraries
  • gstreamer-bad-video-1.0GStreamer bad video library – Bad video library for GStreamer elementsgstreamer-tag-1.0 GStreamer Tag Library – Tag base classes and helper functionsgstreamer-bad-base-1.0 GStreamer bad base classes – Bad base classes for GStreamer elements
  • gstreamer-net-1.0GStreamer networking library – Network-enabled GStreamer plug-ins and clockinggstreamer-sdp-1.0 GStreamer SDP Library – SDP helper functions
  • gstreamer-1.0 GStreamer – Streaming media framework
  • gstreamer-bad-audio-1.0 GStreamer bad audio library, uninstalled – Bad audio library for GStreamer elements, Not Installedgstreamer-allocators-1.0 GStreamer Allocators Library – Allocators implementation
  • gstreamer-player-1.0 GStreamer Player – GStreamer Player convenience library
  • gstreamer-insertbin-1.0 GStreamer Insert Bin – Bin to automatically and insertally link elements
  • gstreamer-plugins-base-1.0 GStreamer Base Plugins Libraries – Streaming media framework, base plugins libraries
  • gstreamer-vaapi-glx-1.0 GStreamer VA-API (GLX) Plugins Libraries – Streaming media framework, VA-API (GLX) plugins librariesgstreamer-codecparsers-1.0 GStreamer codec parsers – Bitstream parsers for GStreamer elementsgstreamer-base-1.0 GStreamer base classes – Base classes for GStreamer elements
  • gstreamer-app-1.0 GStreamer Application Library – Helper functions and base classes for application integration
  • gstreamer-vaapi-drm-1.0 GStreamer VA-API (DRM) Plugins Libraries – Streaming media framework, VA-API (DRM) plugins librariesgstreamer-check-1.0 GStreamer check unit testing – Unit testing helper library for GStreamer modules
  • gstreamer-vaapi-1.0 GStreamer VA-API Plugins Libraries – Streaming media framework, VA-API plugins libraries
  • gstreamer-controller-1.0 GStreamer controller – Dynamic parameter control for GStreamer elements
  • gstreamer-video-1.0 GStreamer Video Library – Video base classes and helper functions
  • gstreamer-vaapi-wayland-1.0 GStreamer VA-API (Wayland) Plugins Libraries – Streaming media framework, VA-API (Wayland) plugins libraries
  • gstreamer-fft-1.0 GStreamer FFT Library – FFT implementation
  • gstreamer-mpegts-1.0 GStreamer MPEG-TS – GStreamer MPEG-TS support
  • gstreamer-pbutils-1.0 GStreamer Base Utils Library – General utility functions
  • gstreamer-vaapi-x11-1.0 GStreamer VA-API (X11) Plugins Libraries – Streaming media framework, VA-API (X11) plugins libraries
  • gstreamer-rtp-1.0 GStreamer RTP Library – RTP base classes and helper functions
  • gstreamer-rtsp-1.0 GStreamer RTSP Library – RTSP base classes and helper functions
  • gstreamer-riff-1.0 GStreamer RIFF Library – RIFF helper functions
  • gstreamer-audio-1.0 GStreamer Audio library – Audio helper functions and base classes
  • gstreamer-plugins-bad-1.0 GStreamer Bad Plugin libraries – Streaming media framework, bad plugins libraries
  • gstreamer-rtsp-server-1.0 gst-rtsp-server – GStreamer based RTSP server

At the time of writing this article Gstreamer an much early version in 1.X , which was newer than its then stable version 0.x. Since then the library has updated many fold. summarising release highlights for major versions as the blog was updated over time .

Gstreamer 1.8.0 – 24 March 2016

  • Features Hardware-accelerated zero-copy video decoding on Android
  • New video capture source for Android using the android.hardware.Camera API
  • Windows Media reverse playback support (ASF/WMV/WMA)
  • tracing system provides support for more sophisticated debugging tools
  • high-level GstPlayer playback convenience API
  • Initial support for the new Vulkan API
  • Improved Opus audio codec support: Support for more than two channels; MPEG-TS demuxer/muxer can handle Opus; sample-accurate encoding/decoding/transmuxing with Ogg, Matroska, ISOBMFF (Quicktime/MP4), and MPEG-TS as container; new codec utility functions for Opus header and caps handling in pbutils library. The Opus encoder/decoder elements were also moved to gst-plugins-base (from -bad), and the opus RTP depayloader/payloader to -good.
  • Asset proxy support in the GStreamer Editing Services

GStreamer 1.16.0 – 19 April 2019.

  • GStreamer WebRTC stack gained support for data channels for peer-to-peer communication based on SCTP, BUNDLE support, as well as support for multiple TURN servers.
  • AV1 video codec support for Matroska and QuickTime/MP4 containers and more configuration options and supported input formats for the AOMedia AV1 encoder
  • Closed Captions and other Ancillary Data in video
  • planar (non-interleaved) raw audio
  • GstVideoAggregator, compositor and OpenGL mixer elements are now in -base
  • New alternate fields interlace mode where each buffer carries a single field
  • WebM and Matroska ContentEncryption support in the Matroska demuxer
  • new WebKit WPE-based web browser source element
  • Video4Linux: HEVC encoding and decoding, JPEG encoding, and improved dmabuf import/export
  • Hardware-accelerated Nvidia video decoder gained support for VP8/VP9 decoding, whilst the encoder gained support for H.265/HEVC encoding.
  • Many improvements to the Intel Media SDK based hardware-accelerated video decoder and encoder plugin (msdk): dmabuf import/export for zero-copy integration with other components; VP9 decoding; 10-bit HEVC encoding; video post-processing (vpp) support including deinterlacing; and the video decoder now handles dynamic resolution changes.
  • ASS/SSA subtitle overlay renderer can now handle multiple subtitles that overlap in time and will show them on screen simultaneously
  • Meson build feature-complete (with the exception of plugin docs) and it is now the recommended build system on all platforms. The Autotools build is scheduled to be removed in the next cycle.
  • GStreamer Rust bindings and Rust plugins module
  • GStreamer Editing Services allows directly playing back serialized edit list with playbin or (uri)decodebin

Ref :