Gstreamer

GStreamer ( LGPL )ia a media handling library written in C for applicatioan such as streaming , recording, playback , mixing and editing attributes etc. Even enhnaced applicaiosn such as tsrancoding , media ormat conversion , streaming servers for embeeded devices ( read more about Gstreamer in RPi in my srticle here).
It encompases various codecs, filters and is modular with plugins developement to enhance its capabilities. Media Streaming application developers use it as part of their framework at either the broadcaster’s end or as media player.

gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink

More detailed reading :

GStreamer-1.8.1 rtsp server and client on ubuntu – Install and configuration for a RTSP Streaming server and Client https://telecom.altanai.com/2016/05/20/gstreamer-1-8-1-rtsp-server-and-client-on-ubuntu/

crtmpserver + ffmpeg –

https://telecom.altanai.com/2016/06/19/crtmpserver-ffmpeg

Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

 attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc .

https://telecom.altanai.com/2015/02/17/streaming-broadcasting-live-video-call-to-non-webrtc-supported-browsers-and-media-players/

continue : Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

httontinuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC )

https://telecom.altanai.com/2015/02/26/continue-streaming-broadcasting-live-video-call-to-non-webrtc-supported-browsers-and-media-players/

TO continue with basics of gstreamer keep reading

To list all packages of Gstreamer

pkg-config --list-all | grep gstreamer
  • gstreamer-gl-1.0 GStreamer OpenGL Plugins Libraries – Streaming media framework, OpenGL plugins libraries
  • gstreamer-bad-video-1.0GStreamer bad video library – Bad video library for GStreamer elements
  • gstreamer-tag-1.0 GStreamer Tag Library – Tag base classes and helper functions
  • gstreamer-bad-base-1.0 GStreamer bad base classes – Bad base classes for GStreamer elements
  • gstreamer-net-1.0GStreamer networking library – Network-enabled GStreamer plug-ins and clocking
  • gstreamer-sdp-1.0 GStreamer SDP Library – SDP helper functions
  • gstreamer-1.0 GStreamer – Streaming media framework
  • gstreamer-bad-audio-1.0 GStreamer bad audio library, uninstalled – Bad audio library for GStreamer elements, Not Installedgstreamer-allocators-1.0 GStreamer Allocators Library – Allocators implementation
  • gstreamer-player-1.0 GStreamer Player – GStreamer Player convenience library
  • gstreamer-insertbin-1.0 GStreamer Insert Bin – Bin to automatically and insertally link elements
  • gstreamer-plugins-base-1.0 GStreamer Base Plugins Libraries – Streaming media framework, base plugins libraries
  • gstreamer-vaapi-glx-1.0 GStreamer VA-API (GLX) Plugins Libraries – Streaming media framework, VA-API (GLX) plugins librariesgstreamer-codecparsers-1.0 GStreamer codec parsers – Bitstream parsers for GStreamer elementsgstreamer-base-1.0 GStreamer base classes – Base classes for GStreamer elements
  • gstreamer-app-1.0 GStreamer Application Library – Helper functions and base classes for application integration
  • gstreamer-vaapi-drm-1.0 GStreamer VA-API (DRM) Plugins Libraries – Streaming media framework, VA-API (DRM) plugins librariesgstreamer-check-1.0 GStreamer check unit testing – Unit testing helper library for GStreamer modules
  • gstreamer-vaapi-1.0 GStreamer VA-API Plugins Libraries – Streaming media framework, VA-API plugins libraries
  • gstreamer-controller-1.0 GStreamer controller – Dynamic parameter control for GStreamer elements
  • gstreamer-video-1.0 GStreamer Video Library – Video base classes and helper functions
  • gstreamer-vaapi-wayland-1.0 GStreamer VA-API (Wayland) Plugins Libraries – Streaming media framework, VA-API (Wayland) plugins libraries
  • gstreamer-fft-1.0 GStreamer FFT Library – FFT implementation
  • gstreamer-mpegts-1.0 GStreamer MPEG-TS – GStreamer MPEG-TS support
  • gstreamer-pbutils-1.0 GStreamer Base Utils Library – General utility functions
  • gstreamer-vaapi-x11-1.0 GStreamer VA-API (X11) Plugins Libraries – Streaming media framework, VA-API (X11) plugins libraries
  • gstreamer-rtp-1.0 GStreamer RTP Library – RTP base classes and helper functions
  • gstreamer-rtsp-1.0 GStreamer RTSP Library – RTSP base classes and helper functions
  • gstreamer-riff-1.0 GStreamer RIFF Library – RIFF helper functions
  • gstreamer-audio-1.0 GStreamer Audio library – Audio helper functions and base classes
  • gstreamer-plugins-bad-1.0 GStreamer Bad Plugin libraries – Streaming media framework, bad plugins libraries
  • gstreamer-rtsp-server-1.0 gst-rtsp-server – GStreamer based RTSP server

At the time of writing this article Gstreamer an much early version in 1.X , which was newer than its then stable version 0.x. Since then the library has updated many fold. summarising release highlights for major versions as the blog was updated over time .

Project : Making and IP survillance system using gstreamer and Janus

To build a turn-key easily deployable surveillance solution 

Features :

  1. Paring of Android Mobile with box
  2. Live streaming from Box to Android
  3. Video Recording inside the  box
  4. Auto parsing of recorded video around motion detection 
  5. Event listeners 
  6. 2 way audio
  7. Inbuild Media Control Unit
  8. Efficient use of bandwidth 
  9. Secure session while live-streaming

Modules

  1. Authentication ( OTP / username- password)
  2. Livestreaming on Opus / vp8 
  3. Session Security and keepalives for live-streaming sessions
  4. Sync local videos to cloud storage 
  5. Record and playback with timeline and events 
  6. Parsing and restructuring video ( transcoding may also be required ) 
  7. Coturn server for NAT and ICE
  8. Web platform on box ( user interface )+ NoSQL
  9. Web platform on Cloud server ( Admin interface )+ NoSQL
  10.  REST APIs for third party add-ons ( Node based )
  11. Android demo app for receiving the live stream and feeds

Varrying experiments and working gstreamer commands

Local Network Stream 

To create /dev/video0

modprobe bcm2835-v4l2

To stream on rtspserver using rpicamsrc using h264 parse

./gst-rtsp-server-1.4.4/examples/test-launch --gst-debug=2 '(rpicamsrc num-buffers=5000 ! 'video/x-h264,width=1080,height=720,framerate=30/1' ! h264parse ! rtph264pay name=pay0 pt=96 )'

./test-launch “( tcpclientsrc host=127.0.0.1 port=5000 ! gdpdepay ! rtph264pay name=pay0 pt=96 )”

pipe raspivid to tcpserversink

raspivid -t 0 -w 800 -h 600 -fps 25 -g 5 -b 4000000 -vf -n -o - | gst-launch-1.0 -v fdsrc ! h264parse ! gdppay ! tcpserversink host=127.0.0.1 port=5000;

Stream Video over local Network with 15 fps

raspivid -n -ih -t 0 -rot 0 -w 1280 -h 720 -fps 15 -b 1000000 -o - | nc -l -p 5001

streaming video over local network with 30FPS and higher bitrate

raspivid -n -t 0 -rot 0 -w 1920 -h 1080 -fps 30 -b 5000000 -o - | nc -l -p 5001

Recording

Audio record to file
Using arecord :

arecord -D plughw:1 -c1 -r 48000 -f S16_LE -t wav -v file.wav;

Using pulse :
pulseAudio -D

gst-launch-1.0 -v pulsesrc device=hw:1 volume=8.0 ! audio/x-raw,format=S16LE ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! flvmux ! filesink location = "testaudio.flv";

Video record to file ( mpg)

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! 'video/x-h264,width=640,height=480’ ! mux. avimux name=mux ! filesink location=testvideo2.mpg;

Video record to file ( flv )

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! flvmux ! filesink location="testvieo.flv";

Video record to file ( h264)
gst-launch-1.0 -e rpicamsrc bitrate=500000 ! filesink location=”raw3.h264″;

Video record to file ( mp4)

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! mp4mux ! filesink location=video.mp4;

Audio + Video record to file ( flv)

gst-launch-1.0 -e /
rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! muxout. /
pulsesrc volume=8.0 ! /
queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. /
flvmux name=muxout streamable=true ! filesink location ='test44.flv';

Audio + Video record to file ( flv) using pulsesrc

gst-launch-1.0 -v --gst-debug-level=3 pulsesrc device="alsa_input.platform-asoc-simple-card.0.analog-stereo" volume=5.0 mute=FALSE ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! audioresample ! audioconvert ! voaacenc ! aacparse ! flvmux ! filesink location="voicetest.flv";

Audio + Video record to file (mp4)

gst-launch-1.0 -e /
rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=10/1 !s h264parse ! muxout. /
pulsesrc volume=4.0 ! /
queue ! audioconvert ! voaacenc ! muxout. /
flvmux name=muxout streamable=true ! filesink location = 'test224.mp4';

Streaming

stream raw Audio over RTMP to srtmpsink

gst-launch-1.0 pulsesrc device=hw:1 volume=8.0 ! /
audio/x-raw,format=S24LE ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! flvmux ! rtmpsink location = “rtmp://192.168.0.3:1935/live/test”;

stream AACpparse Audio over RTMP to srtmpsink

gst-launch-1.0 -v --gst-debug-level=3 pulsesrc device="alsa_input.platform-asoc-simple-card.0.analog-stereo" volume=5.0 mute=FALSE ! audio/x-raw,format=S16LE,rate=48000,channels=1 ! audioresample ! audioconvert ! voaacenc ! aacparse ! flvmux ! rtmpsink location="rtmp://www.altani.com:1935/voice/1/test";

stream Video over RTMP

gst-launch-1.0 -e rpicamsrc bitrate=500000 ! /
video/x-h264,width=320,height=240,framerate=6/1 ! h264parse ! /
flvmux ! rtmpsink location = ‘rtmp://52.66.125.31:1935/live/test live=1’;

stream Audio + video over RTMP from rpicamsrc , framerate 10

gst-launch-1.0 rpicamsrc bitrate=500000 ! video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! muxout. pulsesrc volume=8.0 ! queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. flvmux name=muxout streamable=true ! rtmpsink location ='rtmp://www.altanai.com/live/test44';

stream Audio + video over RTMP from rpicamsrc , framerate 30

gst-launch-1.0 rpicamsrc bitrate=500000 ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! muxout. pulsesrc ! queue ! audioconvert ! voaacenc bitrate=65536 ! aacparse ! muxout. flvmux name=muxout ! queue ! rtmpsink location ='rtmp://www.altanai.com/live/test44';

VOD ( video On Demand )

Stream h264 file over RTMP

gst-launch-1.0 -e filesrc location="raw3.h264" ! video/x-h264 ! h264p
arse ! flvmux ! rtmpsink location = 'rtmp://www.altanai.com/live/test';

Stream flv file over RTMP

gst-launch-1.0 -e filesrc location=”testvieo.flv” ! /
video/x-h264,width=320,height=240,framerate=10/1 ! h264parse ! /
flvmux ! rtmpsink location = 'rtmp://192.168.0.3:1935/live/test';

Github Repo for Livestreaming

https://github.com/altanai/Livestreaming

Contains code for Android and ios Publishers , players on various platforms including HLS and Flash , streamings servers , Wowza playing mosules , webrtc broadcast

Gstreamer 1.8.0 – 24 March 2016

Features Hardware-accelerated zero-copy video decoding on Android

New video capture source for Android using the android.hardware.Camera API

Windows Media reverse playback support (ASF/WMV/WMA)

tracing system provides support for more sophisticated debugging tools

high-level GstPlayer playback convenience API

Initial support for the new Vulkan API

Improved Opus audio codec support: Support for more than two channels; MPEG-TS demuxer/muxer can handle Opus; sample-accurate encoding/decoding/transmuxing with Ogg, Matroska, ISOBMFF (Quicktime/MP4), and MPEG-TS as container; new codec utility functions for Opus header and caps handling in pbutils library. The Opus encoder/decoder elements were also moved to gst-plugins-base (from -bad), and the opus RTP depayloader/payloader to -good.

Asset proxy support in the GStreamer Editing Services

GStreamer 1.16.0 – 19 April 2019.

GStreamer WebRTC stack gained support for data channels for peer-to-peer communication based on SCTP, BUNDLE support, as well as support for multiple TURN servers.

AV1 video codec support for Matroska and QuickTime/MP4 containers and more configuration options and supported input formats for the AOMedia AV1 encoder

Closed Captions and other Ancillary Data in video

planar (non-interleaved) raw audio

GstVideoAggregator, compositor and OpenGL mixer elements are now in -base

New alternate fields interlace mode where each buffer carries a single field

WebM and Matroska ContentEncryption support in the Matroska demuxer

new WebKit WPE-based web browser source element

Video4Linux: HEVC encoding and decoding, JPEG encoding, and improved dmabuf import/export

Hardware-accelerated Nvidia video decoder gained support for VP8/VP9 decoding, whilst the encoder gained support for H.265/HEVC encoding.

Improvements to the Intel Media SDK based hardware-accelerated video decoder and encoder plugin (msdk): dmabuf import/export for zero-copy integration with other components; VP9 decoding; 10-bit HEVC encoding; video post-processing (vpp) support including deinterlacing; and the video decoder now handles dynamic resolution changes.

ASS/SSA subtitle overlay renderer can now handle multiple subtitles that overlap in time and will show them on screen simultaneously

Meson build feature-complete (with the exception of plugin docs) and it is now the recommended build system on all platforms. The Autotools build is scheduled to be removed in the next cycle.

GStreamer Rust bindings and Rust plugins module

GStreamer Editing Services allows directly playing back serialized edit list with playbin or (uri)decodebin

References :

https://gstreamer.freedesktop.org

GStreamer-1.8.1 rtsp server and client on ubuntu

GStreamer is a streaming media framework, based on graphs of filters which operate on media data.

Gstreamer is constructed using a pipes and filter architecture.
The basic structure of a stream pipeline is that you start with a stream source (camera, screengrab, file etc) and end with a stream sink (screen window, file, network etc). The ! are called pads and they connect the filters.

Data that flows through pads is described by caps (short for capabilities). Caps can be though of as mime-type (e.g. audio/x-raw, video/x-raw) along with mime-type (e.g. width, height, depth).

Source Code

Download the latest archives from https://gstreamer.freedesktop.org/src/

Source code on git : https://github.com/GStreamer

Primarily 3 files are required

  1. gstreamer-1.8.1.tar.xz
  2. gst-plugins-base-1.8.1.tar.xz
  3. gst-rtsp-server-1.8.1.tar.xz

If the destination machine is a ec2 instance one can also scp the tar.xz file there

To extract the tar.xz files use tar -xf <filename> it will create a folder for each package.

Prerequisites

build-essentials

sudo apt-get install build-essentials

bison

flex

GLib >= 2.40.0

GLib package contains low-level libraries useful for providing data structure handling for C, portability wrappers and interfaces for such runtime functionality as an event loop, threads, dynamic loading and an object system.

sudo apt-get install libglib2.0-dev

gstreamer

Installing gstreamer 1.8.1 . Gstreamer create a media stream with elements and properties as will be shown on  later sections of this tutorial .

cd gstreamer-1.8.1
./configure
make
sudo make install

Screenshot from 2016-05-19 16-51-29.png

Screenshot from 2016-05-19 16-55-27.png

Screenshot from 2016-05-19 16-56-05.png

after installation  export the path

export LD_LIBRARY_PATH=/usr/local/lib

then verify the installation of the gstreamer by

gst-inspect-1.0

provides information on installed gstreamer modules ie print out a long list ( about 123 in my case ) plugin that are installed such as coreelements:

capsfilter: CapsFilter ximagesink: ximagesink: Video sink videorate: videorate: Video rate adjuster typefindfunctions: image/x-quicktime: qif, qtif, qti typefindfunctions: video/quicktime: mov, mp4 typefindfunctions: application/x-3gp: 3gp typefindfunctions: audio/x-m4a: m4a typefindfunctions: video/x-nuv: nuv typefindfunctions: video/x-h265: h265, x265, 265 typefindfunctions: video/x-h264: h264, x264, 264 typefindfunctions: video/x-h263: h263, 263 typefindfunctions: video/mpeg4: m4v typefindfunctions: video/mpeg-elementary: mpv, mpeg, mpg typefindfunctions: application/ogg: ogg, oga, ogv, ogm, ogx, spx, anx, axa, axv typefindfunctions: video/mpegts: ts, mts typefindfunctions: video/mpeg-sys: mpe, mpeg, mpg typefindfunctions: audio/x-gsm: gsm

gst plugins

Now build the plugins

cd gst-plugins-base-1.8.1
./configure
make
sudo make install

 

gst plugins good

cd gst-plugins-good-1.8.1.tar
./configure
 make
sudo make install

RTSP Server

Now make and install the rtsp server

cd gst-rtsp-server-1.8.1
./configure

last few lines from console traces

Configuration
Version : 1.8.1
Source code location : .
Prefix : /usr/local
Compiler : gcc -std=gnu99
CGroups example : no

make

It will compile the examples .

sudo make install

 

stream video test src

~/mediaServer/gst-rtsp-server-1.8.1/examples]$./test-launch --gst-debug=0 &quot;( videotestsrc ! video/x-raw,format=(yuv),width=352,height=288,framerate=15/1 ! x264enc ! rtph264pay name=pay0 pt=96 )&quot;
stream ready at rtsp://127.0.0.1:8554/test

Ref:

Manual for developers : https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-rtsp-server/html/index.html


Simplest pipeline

gst-launch-1.0 fakesrc ! fakesink

➜ ~ gst-launch-1.0 fakesrc ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
To stop press ctrl +c ^
Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:48.004547887 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... [/sourcecode ] or to display to a audiovideosink gst-launch-1.0 videotestsrc ! autovideosink
Screenshot from 2016-05-20 12-31-18.png To capture webcam
gst-launch v4l2src ! xvimagesink

Screenshot from 2016-05-20 13-06-56.png