Today , there are billions of of video cameras in our homes, phones , ATMs , baby monitors , laptops , smart watches , traffic monitoring , IOT devices , bots , you name it. The underlying purpose of most of them is to capture media streams and optimize the content for further processing.
Stages of video Analytics-
1.Data Acquisition
Data gathered from multiple camera sourced need to be streamed in an aggregated manner to data processing unit in an analytics engine or for archiving . It may be subjected to overall monitoring such as sunlight hours etc or detailed low level object identification such as facial recognition of passengers
2.Transformation to Data Sets
The assimilated data is grouped to operable entities. The identification and classification is done by adding attributes to recognizable shapes ,movements , edges and patterns .
3.Calculate deviation or compliance
A trained model recognizes normal behavior and differentiation from the same is calculated .
Video content Analytics in surveillance
Considering the use case for monitoring and surveillance cameras , there is a growing need for realtime video analytics , to ” detect and determine the temporal and spatial events “. Consider Surveillance cam recordings as forensic evidence or just monitoring incidents and reporting the specific crucial segments of video , both these usecases involve filtering a vast amount of recorded or steaming media to filter out the exact parts that the authorities are looking for. This involves custom identification and recognition of objects in frames .
There is growing research into extracting real time events of interest , minimizing the search time and maximizing accuracy from computer vision .
Consider following use-cases :
Surveillance cam in solar farms or home based setups to predict sun light hours and forecast energy generation value . Described in greater details here .
Traffic monitoring cameras :
Automatic license / number plate recognition – surveillance cams for traffic need to record vehicle plate number to identify and tag the vehicles as they pass by .
Car dashboard cams for investigative purposes post accidents and insurance claims
Motion tracking – Mapping the vehicle movement to detect any wrong turns , overtakes , parking etc
Scan for QR codes and passes at toll gates.
Identifying over-speeding vehicles
3. Security and Law enforcement
Trigger alarms or lockdowns on suspicious activity or intrusion into safe facility
Virtual fencing and perimeter breach – Map facial identification from known suspects
Detection of left items and acceleration of emergency response
Communication based video analytics
Unified enterprise communication , conferences , meeting , online webcasts , webinars , social messengers , online project demos are extensively using video analytics for building intuitive use cases and boost innovation around their platform . Few examples of vast number of usecases are
Sentiment Analysis : Capturing emotions by mapping key words to ascertain whether the meeting went , happy , positive , productive or sad , complaining , negative
Augmented Reality for overlaying information such as interactive manual or a image . Areas for current usage include , e-learning and customer support .
Dynamic masking for privacy
Autonomous Robot purpose Video analytics
Self driving drones , cars and even bots extensive use the feed from wide angle / fish eye lens cameras to create a 3D model of their movement in given space of 3 dimensional coordinates system.
Key technologies includes :
Ego-motion estimation – mapping a 3D space with captured camera feed
Deep Learning ( part of AI) from continuous feed from video cameras to find route around obstacles
Remote monitoring for an unmanned vehicle
Sterile monitoring for a unreachable or hazardous area example war-zone , outer territorial objects as moon , mars , satellites
Bio-metrics based Video analytics
Often video feed is now used for advanced search, redaction and facial recognition , which leads to features such as
unlocking laptop or phone
performing click with blink of eyes
creating concentration maps of webpage based on where eyes focused
Read more about role of webrtc in bio- metrics here
Video analytics in Industrial and Retail applications
Application of video analytics in Industrial landscape are manyfold . On one hand it can be for intelligence and information gathering such as worker foot count . Machine left unattended etc while on the other hand by using specific image optimization techniques can also audit automation testing of engines , machine parts , rotation counts etc .
Flame and Smoke Detection – images from video streams are analysed for color chrominance, flickering ratio, shape, pattern and moving direction to ascertain a fire hazard.
Collect demographics of the area with people counting
Ensure quality control and procedural compliance
Identify tail-gateing or loitering
List of few companies focusing on Video Analytics :
Performing data analytics at application level on the edge whole system architecture instead of at the core or data warehouse level. The advantage to the computation at fringes of network instead of a centralized system are faster response times and standalone off grid functionality support .
The humongous data collected over by IOT devices m machinery , sensors , servers , firewalls , routers , switches gateways and all other types of components are increasingly getting analysed and acted upon at the edge , independently with machine learning instead of data centers and network operation centers . With the help of feedback loops and deep learning one could add data drive intelligence to how operations are performed at critical machines arts such as autonomous bots or industrial setups.
Error Recovery and streaming
To control the incoming data stream , we divide and classify the content into packets , add custom classification header and stream them down to the server. In the event of data congestion of back pressure , some non significant packets are either dropped or added to a new dead queue . The system is thus made stable for high availability and BCP / fail-over recovery .
GStreamer ( LGPL )is a media handling library written in C for applications such as streaming, recording, playback, mixing and editing attributes so on. Even enhanced applications such as transcoding , media format conversion , streaming servers for embedded devices ( read more about Gstreamer in RPi in my article here).
It is a pipeline based multimedia framework.It encompas various codecs, filters and is modular with plugins development to enhance its capabilities. Media Streaming application developers use it as part of their framework at either the broadcaster’s end or as a media player.
allowing developers to extend its functionality by adding new plugins.
Plugins provide support for various codecs, formats, filters, and input/output devices.
Pipeline-Based Processing:
GStreamer uses a pipeline model where multimedia data flows through a series of interconnected elements (e.g., sources, filters, sinks).
Each element performs a specific task, such as decoding, encoding, or rendering.
Cross-Platform Support:
GStreamer runs on Linux, Windows, macOS, Android, and other platforms, making it versatile for different environments.
Wide Range of Supported Formats:
GStreamer supports a vast array of audio and video formats, including MP3, AAC, H.264, VP9, and more.
It can handle both compressed and uncompressed media streams.
Real-Time Streaming:
GStreamer is capable of real-time streaming, making it suitable for applications like video conferencing, IP cameras, and live broadcasting.
Customizable and Extensible GStreamer’s API allows for fine-grained control over media processing.
C: The native API for GStreamer.
Python: Bindings for Python .
C++: Bindings for C++.
Rust: Bindings for Rust.
Integration with Other Technologies, frameworks and libraries, such as OpenCV for computer vision, PulseAudio for audio, and WebRTC for real-time communication.
Core Concepts:
Elements: The basic building blocks of a GStreamer pipeline. Examples include:
Source elements: Capture or generate data (e.g., reading from a file or a webcam).
Filter elements: Process data (e.g., decoding, encoding, or applying effects).
Sink elements: Output data (e.g., rendering video to a screen or writing to a file).
Pads: Connection points on elements where data flows in (sink pads) or out (source pads).
Pipeline: A container for elements that defines the flow of media data.
Bus: A communication channel for messages (e.g., errors, state changes) between the pipeline and the application.
Some Use Cases:
Server-Side Processing: GStreamer can be used for transcoding, mixing, or analyzing media on servers.
Media Players: GStreamer is used in media players like Totem and Rhythmbox.
Streaming / broadcasting Live Video call to non webrtc supported browsers and media players
Attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc .
As the title of this article suggests I am going to pen my attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc. Some of the high level archietctures for streaming Webrtc Video to multiple endpoints…
This blog is in continuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC )
gstreamer-audio-1.0 GStreamer Audio library – Audio helper functions and base classes
gstreamer-plugins-bad-1.0 GStreamer Bad Plugin libraries – Streaming media framework, bad plugins libraries
gstreamer-rtsp-server-1.0 gst-rtsp-server – GStreamer based RTSP server
At the time of writing this article Gstreamer an much early version in 1.X , which was newer than its then stable version 0.x. Since then the library has updated many fold. summarising release highlights for major versions as the blog was updated over time .
Ex1 : Capture video from your webcam and save it as an MP4 file
Project : Making and IP survillance system using gstreamer and Janus
To build a turn-key easily deployable surveillance solution
Features :
Paring of Android Mobile with box
Live streaming from Box to Android
Video Recording inside the box
Auto parsing of recorded video around motion detection
Event listeners
2 way audio
Inbuild Media Control Unit
Efficient use of bandwidth
Secure session while live-streaming
Modules
Authentication ( OTP / username- password)
Livestreaming on Opus / vp8
Session Security and keepalives for live-streaming sessions
Sync local videos to cloud storage
Record and playback with timeline and events
Parsing and restructuring video ( transcoding may also be required )
Coturn server for NAT and ICE
Web platform on box ( user interface )+ NoSQL
Web platform on Cloud server ( Admin interface )+ NoSQL
REST APIs for third party add-ons ( Node based )
Android demo app for receiving the live stream and feeds
Varrying experiments and working gstreamer commands
Local Network Stream
To create /dev/video0
modprobe bcm2835-v4l2
To stream on rtspserver using rpicamsrc using h264 parse. Adjust the pipeline based on your specific requirements (e.g., resolution, bitrate, or codec).
Telecom R & D 