Gstreamer

GStreamer ( LGPL )is a media handling library written in C for applications such as streaming, recording, playback, mixing and editing attributes so on. Even enhanced applications such as transcoding , media format conversion , streaming servers for embedded devices ( read more about Gstreamer in RPi in my article here).

It is a pipeline based multimedia framework.It encompas various codecs, filters and is modular with plugins development to enhance its capabilities. Media Streaming application developers use it as part of their framework at either the broadcaster’s end or as a media player.

Pipeline simple example1

gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink

ex2

gst-launch-1.0 filesrc location=file_example_MP4_480_1_5MG.mp4 ! qtdemux ! h264parse ! avdec_h264 ! autovideosink

Key Features of GStreamer include

1. Modular and Plugin-Based Architecture:
   • allowing developers to extend its functionality by adding new plugins.
   • Plugins provide support for various codecs, formats, filters, and input/output devices.
2. Pipeline-Based Processing:
   • GStreamer uses a pipeline model where multimedia data flows through a series of interconnected elements (e.g., sources, filters, sinks).
   • Each element performs a specific task, such as decoding, encoding, or rendering.
3. Cross-Platform Support:
   • GStreamer runs on Linux, Windows, macOS, Android, and other platforms, making it versatile for different environments.
4. Wide Range of Supported Formats:
   • GStreamer supports a vast array of audio and video formats, including MP3, AAC, H.264, VP9, and more.
   • It can handle both compressed and uncompressed media streams.
5. Real-Time Streaming:
   • GStreamer is capable of real-time streaming, making it suitable for applications like video conferencing, IP cameras, and live broadcasting.
6. Customizable and Extensible GStreamer’s API allows for fine-grained control over media processing.
   • C: The native API for GStreamer.
   • Python: Bindings for Python .
   • C++: Bindings for C++.
   • Rust: Bindings for Rust.
7. Integration with Other Technologies, frameworks and libraries, such as OpenCV for computer vision, PulseAudio for audio, and WebRTC for real-time communication.

Core Concepts:

• Elements: The basic building blocks of a GStreamer pipeline. Examples include:
  • Source elements: Capture or generate data (e.g., reading from a file or a webcam).
  • Filter elements: Process data (e.g., decoding, encoding, or applying effects).
  • Sink elements: Output data (e.g., rendering video to a screen or writing to a file).
• Pads: Connection points on elements where data flows in (sink pads) or out (source pads).
• Pipeline: A container for elements that defines the flow of media data.
• Bus: A communication channel for messages (e.g., errors, state changes) between the pipeline and the application.

Some Use Cases:

Server-Side Processing: GStreamer can be used for transcoding, mixing, or analyzing media on servers.

Media Players: GStreamer is used in media players like Totem and Rhythmbox.

gst-launch-1.0 playbin uri=file://($pwd)/file_example_MP4_480_1_5MG.mp4

Also you can play with Debug options

GST_DEBUG=2 gst-launch-1.0 filesrc location=yourfile.mp4 ! qtdemux name=demux\ 
demux.video_0 ! queue ! decodebin ! autovideosink \
demux.audio_0 ! queue ! decodebin ! autoaudiosink

Video Editing: It can be used for non-linear video editing and processing.

Streaming: GStreamer is used in applications like IPTV, video conferencing, and live streaming.
One can use any sample source fot testruns

• videotestsrc: Generates test video patterns.
• audiotestsrc: Generates test audio tones.
• v4l2src: Captures video from a webcam (Linux).
• ximagesrc: Captures the screen (Linux).

Embedded Systems: It is popular in embedded devices for multimedia playback and capture.

GStreamer-1.8.1 rtsp server and client on ubuntu – Install and configuration for a RTSP Streaming server and Client

GStreamer-1.8.1 rtsp server and client on ubuntu

crtmpserver + ffmpeg

Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

 Attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc .

Code Snippets

To list all packages of Gstreamer

pkg-config --list-all | grep gstreamer

• gstreamer-gl-1.0 GStreamer OpenGL Plugins Libraries – Streaming media framework, OpenGL plugins libraries
• gstreamer-bad-video-1.0GStreamer bad video library – Bad video library for GStreamer elements
• gstreamer-tag-1.0 GStreamer Tag Library – Tag base classes and helper functions
• gstreamer-bad-base-1.0 GStreamer bad base classes – Bad base classes for GStreamer elements
• gstreamer-net-1.0GStreamer networking library – Network-enabled GStreamer plug-ins and clocking
• gstreamer-sdp-1.0 GStreamer SDP Library – SDP helper functions
• gstreamer-1.0 GStreamer – Streaming media framework
• gstreamer-bad-audio-1.0 GStreamer bad audio library, uninstalled – Bad audio library for GStreamer elements, Not Installedgstreamer-allocators-1.0 GStreamer Allocators Library – Allocators implementation
• gstreamer-player-1.0 GStreamer Player – GStreamer Player convenience library
• gstreamer-insertbin-1.0 GStreamer Insert Bin – Bin to automatically and insertally link elements
• gstreamer-plugins-base-1.0 GStreamer Base Plugins Libraries – Streaming media framework, base plugins libraries
• gstreamer-vaapi-glx-1.0 GStreamer VA-API (GLX) Plugins Libraries – Streaming media framework, VA-API (GLX) plugins librariesgstreamer-codecparsers-1.0 GStreamer codec parsers – Bitstream parsers for GStreamer elementsgstreamer-base-1.0 GStreamer base classes – Base classes for GStreamer elements
• gstreamer-app-1.0 GStreamer Application Library – Helper functions and base classes for application integration
• gstreamer-vaapi-drm-1.0 GStreamer VA-API (DRM) Plugins Libraries – Streaming media framework, VA-API (DRM) plugins librariesgstreamer-check-1.0 GStreamer check unit testing – Unit testing helper library for GStreamer modules
• gstreamer-vaapi-1.0 GStreamer VA-API Plugins Libraries – Streaming media framework, VA-API plugins libraries
• gstreamer-controller-1.0 GStreamer controller – Dynamic parameter control for GStreamer elements
• gstreamer-video-1.0 GStreamer Video Library – Video base classes and helper functions
• gstreamer-vaapi-wayland-1.0 GStreamer VA-API (Wayland) Plugins Libraries – Streaming media framework, VA-API (Wayland) plugins libraries
• gstreamer-fft-1.0 GStreamer FFT Library – FFT implementation
• gstreamer-mpegts-1.0 GStreamer MPEG-TS – GStreamer MPEG-TS support
• gstreamer-pbutils-1.0 GStreamer Base Utils Library – General utility functions
• gstreamer-vaapi-x11-1.0 GStreamer VA-API (X11) Plugins Libraries – Streaming media framework, VA-API (X11) plugins libraries
• gstreamer-rtp-1.0 GStreamer RTP Library – RTP base classes and helper functions
• gstreamer-rtsp-1.0 GStreamer RTSP Library – RTSP base classes and helper functions
• gstreamer-riff-1.0 GStreamer RIFF Library – RIFF helper functions
• gstreamer-audio-1.0 GStreamer Audio library – Audio helper functions and base classes
• gstreamer-plugins-bad-1.0 GStreamer Bad Plugin libraries – Streaming media framework, bad plugins libraries
• gstreamer-rtsp-server-1.0 gst-rtsp-server – GStreamer based RTSP server

At the time of writing this article Gstreamer an much early version in 1.X , which was newer than its then stable version 0.x. Since then the library has updated many fold. summarising release highlights for major versions as the blog was updated over time .

Ex1 : Capture video from your webcam and save it as an MP4 file

> v4l2-ctl --list-devices
    Integrated_Webcam_HD: Integrate (usb-0000:00:14.0-5):
    /dev/video0

gst-launch-1.0 v4l2src ! videoconvert ! x264enc ! mp4mux ! filesink location=webcam_output.mp4

Ex 2: Record Screen and Save as MP4

gst-launch-1.0 ximagesrc ! videoconvert ! x264enc ! mp4mux ! filesink location=screen_record.mp4

After interrupt

Project : Making and IP survillance system using gstreamer and Janus

To build a turn-key easily deployable surveillance solution 

Features :

1. Paring of Android Mobile with box
2. Live streaming from Box to Android
3. Video Recording inside the  box
4. Auto parsing of recorded video around motion detection 
5. Event listeners 
6. 2 way audio
7. Inbuild Media Control Unit
8. Efficient use of bandwidth 
9. Secure session while live-streaming

Modules

1. Authentication ( OTP / username- password)
2. Livestreaming on Opus / vp8 
3. Session Security and keepalives for live-streaming sessions
4. Sync local videos to cloud storage 
5. Record and playback with timeline and events 
6. Parsing and restructuring video ( transcoding may also be required ) 
7. Coturn server for NAT and ICE
8. Web platform on box ( user interface )+ NoSQL
9. Web platform on Cloud server ( Admin interface )+ NoSQL
10.  REST APIs for third party add-ons ( Node based )
11. Android demo app for receiving the live stream and feeds

Varrying experiments and working gstreamer commands

Local Network Stream 

To create /dev/video0

modprobe bcm2835-v4l2

To stream on rtspserver using rpicamsrc using h264 parse. Adjust the pipeline based on your specific requirements (e.g., resolution, bitrate, or codec).

./gst-rtsp-server-1.4.4/examples/test-launch --gst-debug=2 '(rpicamsrc num-buffers=5000 ! 'video/x-h264,width=1080,height=720,framerate=30/1' ! h264parse ! rtph264pay name=pay0 pt=96 )'

./test-launch “( tcpclientsrc host=127.0.0.1 port=5000 ! gdpdepay ! rtph264pay name=pay0 pt=96 )”

pipe raspivid to tcpserversink

raspivid -t 0 -w 800 -h 600 -fps 25 -g 5 -b 4000000 -vf -n -o - | gst-launch-1.0 -v fdsrc ! h264parse ! gdppay ! tcpserversink host=127.0.0.1 port=5000;

Stream Video over local Network with 15 fps

raspivid -n -ih -t 0 -rot 0 -w 1280 -h 720 -fps 15 -b 1000000 -o - | nc -l -p 5001

streaming video over local network with 30FPS and higher bitrate

raspivid -n -t 0 -rot 0 -w 1920 -h 1080 -fps 30 -b 5000000 -o - | nc -l -p 5001

Recording

Audio record to file
Using arecord :

arecord -D plughw:1 -c1 -r 48000 -f S16_LE -t wav -v file.wav;

Using pulse :
pulseAudio -D

gst-launch-1.0 -v pulsesrc device=hw:1 volume=8.0 ! audio/x-raw,format=S16LE ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! flvmux ! filesink location = "testaudio.flv";

Video record to file ( mpg)

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! 'video/x-h264,width=640,height=480’ ! mux. avimux name=mux ! filesink location=testvideo2.mpg;

Video record to file ( flv )

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! flvmux ! filesink location="testvieo.flv";

Video record to file ( h264)
gst-launch-1.0 -e rpicamsrc bitrate=500000 ! filesink location=”raw3.h264″;

Video record to file ( mp4)

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! mp4mux ! filesink location=video.mp4;

Audio + Video record to file ( flv)

gst-launch-1.0 -e /
rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! muxout. /
pulsesrc volume=8.0 ! /
queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. /
flvmux name=muxout streamable=true ! filesink location ='test44.flv';

Audio + Video record to file ( flv) using pulsesrc

gst-launch-1.0 -v --gst-debug-level=3 pulsesrc device="alsa_input.platform-asoc-simple-card.0.analog-stereo" volume=5.0 mute=FALSE ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! audioresample ! audioconvert ! voaacenc ! aacparse ! flvmux ! filesink location="voicetest.flv";

Audio + Video record to file (mp4)

gst-launch-1.0 -e /
rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=10/1 !s h264parse ! muxout. /
pulsesrc volume=4.0 ! /
queue ! audioconvert ! voaacenc ! muxout. /
flvmux name=muxout streamable=true ! filesink location = 'test224.mp4';

Streaming

stream raw Audio over RTMP to srtmpsink

gst-launch-1.0 pulsesrc device=hw:1 volume=8.0 ! /
audio/x-raw,format=S24LE ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! flvmux ! rtmpsink location = “rtmp://192.168.0.3:1935/live/test”;

stream AACpparse Audio over RTMP to srtmpsink

gst-launch-1.0 -v --gst-debug-level=3 pulsesrc device="alsa_input.platform-asoc-simple-card.0.analog-stereo" volume=5.0 mute=FALSE ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! audioresample ! audioconvert ! voaacenc ! aacparse ! flvmux ! rtmpsink location="rtmp://www.altani.com:1935/voice/1/test";

stream Video over RTMP

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=6/1 ! h264parse ! /
flvmux ! rtmpsink location = ‘rtmp://52.66.125.31:1935/live/test live=1’;

stream Audio + video over RTMP from rpicamsrc , framerate 10

gst-launch-1.0 rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! muxout. pulsesrc volume=8.0 ! queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. flvmux name=muxout streamable=true ! rtmpsink location ='rtmp://www.altanai.com/live/test44';

stream Audio + video over RTMP from rpicamsrc , framerate 30

gst-launch-1.0 rpicamsrc bitrate=500000 ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! muxout. pulsesrc ! queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. flvmux name=muxout ! queue ! rtmpsink location ='rtmp://www.altanai.com/live/test44';

VOD ( video On Demand )

Stream h264 file over RTMP

gst-launch-1.0 -e filesrc location="raw3.h264" ! video/x-h264 ! h264p
arse ! flvmux ! rtmpsink location = 'rtmp://www.altanai.com/live/test';

Stream flv file over RTMP

gst-launch-1.0 -e filesrc location=”testvieo.flv” ! /
video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! /
flvmux ! rtmpsink location = 'rtmp://192.168.0.3:1935/live/test';

Github Repo for Livestreaming

https://github.com/altanai/Livestreaming

Debugging Error while runnign Gstreamer pipeline

Keep Ubuntu updated

Keep Gstreamer and its plugin updated

sudo apt update
sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly

References :

https://gstreamer.freedesktop.org