Today , there are billions of of video cameras in our homes, phones , ATMs , baby monitors , laptops , smart watches , traffic monitoring , IOT devices , bots , you name it. The underlying purpose of most of them is to capture media streams and optimize the content for further processing.
Stages of video Analytics-
Data gathered from multiple camera sourced need to be streamed in an aggregated manner to data processing unit in an analytics engine or for archiving . It may be subjected to overall monitoring such as sunlight hours etc or detailed low level object identification such as facial recognition of passengers
2.Transformation to Data Sets
The assimilated data is grouped to operable entities. The identification and classification is done by adding attributes to recognizable shapes ,movements , edges and patterns .
3.Calculate deviation or compliance
A trained model recognizes normal behavior and differentiation from the same is calculated .
Video content Analytics in surveillance
Considering the use case for monitoring and surveillance cameras , there is a growing need for realtime video analytics , to ” detect and determine the temporal and spatial events “. Consider Surveillance cam recordings as forensic evidence or just monitoring incidents and reporting the specific crucial segments of video , both these usecases involve filtering a vast amount of recorded or steaming media to filter out the exact parts that the authorities are looking for. This involves custom identification and recognition of objects in frames .
There is growing research into extracting real time events of interest , minimizing the search time and maximizing accuracy from computer vision .
Consider following use-cases :
Surveillance cam in solar farms or home based setups to predict sun light hours and forecast energy generation value . Described in greater details here .
Traffic monitoring cameras :
Automatic license / number plate recognition – surveillance cams for traffic need to record vehicle plate number to identify and tag the vehicles as they pass by .
Car dashboard cams for investigative purposes post accidents and insurance claims
Motion tracking – Mapping the vehicle movement to detect any wrong turns , overtakes , parking etc
Scan for QR codes and passes at toll gates.
Identifying over-speeding vehicles
3. Security and Law enforcement
Trigger alarms or lockdowns on suspicious activity or intrusion into safe facility
Virtual fencing and perimeter breach – Map facial identification from known suspects
Detection of left items and acceleration of emergency response
Communication based video analytics
Unified enterprise communication , conferences , meeting , online webcasts , webinars , social messengers , online project demos are extensively using video analytics for building intuitive use cases and boost innovation around their platform . Few examples of vast number of usecases are
Sentiment Analysis : Capturing emotions by mapping key words to ascertain whether the meeting went , happy , positive , productive or sad , complaining , negative
Augmented Reality for overlaying information such as interactive manual or a image . Areas for current usage include , e-learning and customer support .
Dynamic masking for privacy
Autonomous Robot purpose Video analytics
Self driving drones , cars and even bots extensive use the feed from wide angle / fish eye lens cameras to create a 3D model of their movement in given space of 3 dimensional coordinates system.
Key technologies includes :
Ego-motion estimation – mapping a 3D space with captured camera feed
Deep Learning ( part of AI) from continuous feed from video cameras to find route around obstacles
Remote monitoring for an unmanned vehicle
Sterile monitoring for a unreachable or hazardous area example war-zone , outer territorial objects as moon , mars , satellites
Bio-metrics based Video analytics
Often video feed is now used for advanced search, redaction and facial recognition , which leads to features such as
unlocking laptop or phone
performing click with blink of eyes
creating concentration maps of webpage based on where eyes focused
Read more about role of webrtc in bio- metrics here
Video analytics in Industrial and Retail applications
Application of video analytics in Industrial landscape are manyfold . On one hand it can be for intelligence and information gathering such as worker foot count . Machine left unattended etc while on the other hand by using specific image optimization techniques can also audit automation testing of engines , machine parts , rotation counts etc .
Flame and Smoke Detection – images from video streams are analysed for color chrominance, flickering ratio, shape, pattern and moving direction to ascertain a fire hazard.
Collect demographics of the area with people counting
Ensure quality control and procedural compliance
Identify tail-gateing or loitering
List of few companies focusing on Video Analytics :
Performing data analytics at application level on the edge whole system architecture instead of at the core or data warehouse level. The advantage to the computation at fringes of network instead of a centralized system are faster response times and standalone off grid functionality support .
The humongous data collected over by IOT devices m machinery , sensors , servers , firewalls , routers , switches gateways and all other types of components are increasingly getting analysed and acted upon at the edge , independently with machine learning instead of data centers and network operation centers . With the help of feedback loops and deep learning one could add data drive intelligence to how operations are performed at critical machines arts such as autonomous bots or industrial setups.
Error Recovery and streaming
To control the incoming data stream , we divide and classify the content into packets , add custom classification header and stream them down to the server. In the event of data congestion of back pressure , some non significant packets are either dropped or added to a new dead queue . The system is thus made stable for high availability and BCP / fail-over recovery .
GStreamer ( LGPL )ia a media handling library written in C for applicatioan such as streaming , recording, playback , mixing and editing attributes etc. Even enhnaced applicaiosn such as tsrancoding , media ormat conversion , streaming servers for embeeded devices ( read more about Gstreamer in RPi in my srticle here).
It encompases various codecs, filters and is modular with plugins developement to enhance its capabilities. Media Streaming application developers use it as part of their framework at either the broadcaster’s end or as media player.
continue : Streaming / broadcasting Live Video call to non webrtc supported browsers and media players
httontinuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC )
gstreamer-audio-1.0 GStreamer Audio library – Audio helper functions and base classes
gstreamer-plugins-bad-1.0 GStreamer Bad Plugin libraries – Streaming media framework, bad plugins libraries
gstreamer-rtsp-server-1.0 gst-rtsp-server – GStreamer based RTSP server
At the time of writing this article Gstreamer an much early version in 1.X , which was newer than its then stable version 0.x. Since then the library has updated many fold. summarising release highlights for major versions as the blog was updated over time .
Project : Making and IP survillance system using gstreamer and Janus
To build a turn-key easily deployable surveillance solution
Paring of Android Mobile with box
Live streaming from Box to Android
Video Recording inside the box
Auto parsing of recorded video around motion detection
2 way audio
Inbuild Media Control Unit
Efficient use of bandwidth
Secure session while live-streaming
Authentication ( OTP / username- password)
Livestreaming on Opus / vp8
Session Security and keepalives for live-streaming sessions
Sync local videos to cloud storage
Record and playback with timeline and events
Parsing and restructuring video ( transcoding may also be required )
Coturn server for NAT and ICE
Web platform on box ( user interface )+ NoSQL
Web platform on Cloud server ( Admin interface )+ NoSQL
REST APIs for third party add-ons ( Node based )
Android demo app for receiving the live stream and feeds
Varrying experiments and working gstreamer commands
Local Network Stream
To create /dev/video0
To stream on rtspserver using rpicamsrc using h264 parse
Contains code for Android and ios Publishers , players on various platforms including HLS and Flash , streamings servers , Wowza playing mosules , webrtc broadcast
Gstreamer 1.8.0 – 24 March 2016
Features Hardware-accelerated zero-copy video decoding on Android
New video capture source for Android using the android.hardware.Camera API
Windows Media reverse playback support (ASF/WMV/WMA)
tracing system provides support for more sophisticated debugging tools
high-level GstPlayer playback convenience API
Initial support for the new Vulkan API
Improved Opus audio codec support: Support for more than two channels; MPEG-TS demuxer/muxer can handle Opus; sample-accurate encoding/decoding/transmuxing with Ogg, Matroska, ISOBMFF (Quicktime/MP4), and MPEG-TS as container; new codec utility functions for Opus header and caps handling in pbutils library. The Opus encoder/decoder elements were also moved to gst-plugins-base (from -bad), and the opus RTP depayloader/payloader to -good.
Asset proxy support in the GStreamer Editing Services
GStreamer 1.16.0 – 19 April 2019.
GStreamer WebRTC stack gained support for data channels for peer-to-peer communication based on SCTP, BUNDLE support, as well as support for multiple TURN servers.
AV1 video codec support for Matroska and QuickTime/MP4 containers and more configuration options and supported input formats for the AOMedia AV1 encoder
Closed Captions and other Ancillary Data in video
planar (non-interleaved) raw audio
GstVideoAggregator, compositor and OpenGL mixer elements are now in -base
New alternate fields interlace mode where each buffer carries a single field
WebM and Matroska ContentEncryption support in the Matroska demuxer
new WebKit WPE-based web browser source element
Video4Linux: HEVC encoding and decoding, JPEG encoding, and improved dmabuf import/export
Hardware-accelerated Nvidia video decoder gained support for VP8/VP9 decoding, whilst the encoder gained support for H.265/HEVC encoding.
Improvements to the Intel Media SDK based hardware-accelerated video decoder and encoder plugin (msdk): dmabuf import/export for zero-copy integration with other components; VP9 decoding; 10-bit HEVC encoding; video post-processing (vpp) support including deinterlacing; and the video decoder now handles dynamic resolution changes.
ASS/SSA subtitle overlay renderer can now handle multiple subtitles that overlap in time and will show them on screen simultaneously
Meson build feature-complete (with the exception of plugin docs) and it is now the recommended build system on all platforms. The Autotools build is scheduled to be removed in the next cycle.
GStreamer Rust bindings and Rust plugins module
GStreamer Editing Services allows directly playing back serialized edit list with playbin or (uri)decodebin