GStreamer-1.8.1 rtsp server and client on ubuntu

GStreamer is a streaming media framework, based on graphs of filters which operate on media data.

Gstreamer is constructed using a pipes and filter architecture.
The basic structure of a stream pipeline is that you start with a stream source (camera, screengrab, file etc) and end with a stream sink (screen window, file, network etc). The ! are called pads and they connect the filters.

Data that flows through pads is described by caps (short for capabilities). Caps can be though of as mime-type (e.g. audio/x-raw, video/x-raw) along with mime-type (e.g. width, height, depth).

Source Code

Download the latest archives from https://gstreamer.freedesktop.org/src/

Source code on git : https://github.com/GStreamer

Primarily 3 files are required

  1. gstreamer-1.8.1.tar.xz
  2. gst-plugins-base-1.8.1.tar.xz
  3. gst-rtsp-server-1.8.1.tar.xz

If the destination machine is a ec2 instance one can also scp the tar.xz file there

To extract the tar.xz files use tar -xf <filename> it will create a folder for each package.

Prerequisites

build-essentials

sudo apt-get install build-essentials

bison

flex

GLib >= 2.40.0

GLib package contains low-level libraries useful for providing data structure handling for C, portability wrappers and interfaces for such runtime functionality as an event loop, threads, dynamic loading and an object system.

sudo apt-get install libglib2.0-dev

gstreamer

Installing gstreamer 1.8.1 . Gstreamer create a media stream with elements and properties as will be shown on  later sections of this tutorial .

cd gstreamer-1.8.1
./configure
make
sudo make install

Screenshot from 2016-05-19 16-51-29.png

Screenshot from 2016-05-19 16-55-27.png

Screenshot from 2016-05-19 16-56-05.png

after installation  export the path

export LD_LIBRARY_PATH=/usr/local/lib

then verify the installation of the gstreamer by

gst-inspect-1.0

provides information on installed gstreamer modules ie print out a long list ( about 123 in my case ) plugin that are installed such as coreelements:

capsfilter: CapsFilter ximagesink: ximagesink: Video sink videorate: videorate: Video rate adjuster typefindfunctions: image/x-quicktime: qif, qtif, qti typefindfunctions: video/quicktime: mov, mp4 typefindfunctions: application/x-3gp: 3gp typefindfunctions: audio/x-m4a: m4a typefindfunctions: video/x-nuv: nuv typefindfunctions: video/x-h265: h265, x265, 265 typefindfunctions: video/x-h264: h264, x264, 264 typefindfunctions: video/x-h263: h263, 263 typefindfunctions: video/mpeg4: m4v typefindfunctions: video/mpeg-elementary: mpv, mpeg, mpg typefindfunctions: application/ogg: ogg, oga, ogv, ogm, ogx, spx, anx, axa, axv typefindfunctions: video/mpegts: ts, mts typefindfunctions: video/mpeg-sys: mpe, mpeg, mpg typefindfunctions: audio/x-gsm: gsm

gst plugins

Now build the plugins

cd gst-plugins-base-1.8.1
./configure
make
sudo make install

 

gst plugins good

cd gst-plugins-good-1.8.1.tar
./configure
 make
sudo make install

RTSP Server

Now make and install the rtsp server

cd gst-rtsp-server-1.8.1
./configure

last few lines from console traces

Configuration
Version : 1.8.1
Source code location : .
Prefix : /usr/local
Compiler : gcc -std=gnu99
CGroups example : no

make

It will compile the examples .

sudo make install

 

stream video test src

~/mediaServer/gst-rtsp-server-1.8.1/examples]$./test-launch --gst-debug=0 &quot;( videotestsrc ! video/x-raw,format=(yuv),width=352,height=288,framerate=15/1 ! x264enc ! rtph264pay name=pay0 pt=96 )&quot;
stream ready at rtsp://127.0.0.1:8554/test

Ref:

Manual for developers : https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-rtsp-server/html/index.html


Simplest pipeline

gst-launch-1.0 fakesrc ! fakesink

➜ ~ gst-launch-1.0 fakesrc ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
To stop press ctrl +c ^
Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:48.004547887 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... [/sourcecode ] or to display to a audiovideosink gst-launch-1.0 videotestsrc ! autovideosink
Screenshot from 2016-05-20 12-31-18.png To capture webcam
gst-launch v4l2src ! xvimagesink

Screenshot from 2016-05-20 13-06-56.png

Setting up ubuntu ec2 t2 micro for webrtc and socketio

Setting up a ec2 instance on AWS for web real time communication platform over nodejs and socket.io using WebRTC.

Primarily a Web Call  , Chat and conference platform uses WebRTC for the media stream and socketio for the signalling . Additionally used technologies are nosql for session information storage , REST Apis for getting sessions details to third parties.

Below is a comprehensive setup if ec2 t2.micro free tier instance, installation with a webrtc project module and samples of customisation and usage .

Technologies used are listed below :

Server

  1. ec2 instance t2.micro covered under free tier
  2. domain name
  3. SSL certificate

Core module for Web Calling feature

  1. WebRTC
  2. Node.js
  3. socket.io

UI components

  1. javascript
  2. css
  3. html5
  4. bootstrap
  5. jquerry

Supporting setup for session management

  1. Code version-ing  and maintenance
  2. git
  3. npm

Sample Project https://github.com/altanai/webrtcdevelopment

Amazon’s free tier ec2

Amazon EC2 : These are elastic compute general purpose storage servers that mean that they can resize the compute capacity in the cloud based on load . 750 hours per month of Linux, RHEL, or SLES t2.micro instance usage. Expires 12 months after sign-up.

Some other products are also covered under free tier which may come in handy for setting up the complete complatorm. Here is a quick summary

Amazon S3 : it is a storage server. Can be used to store media file like image s, music , videos , recorded video etc .

Amazon RDS : It a relational database server . If one is using mysql or postgress for storing session information or user profile data . It is good option .

Amazon SES : email service. Can be used to send invites and notifications to users over mail for scheduled sessions or missed calls .

Amazon CloudFront : It is a CDN ( content delivery network ) . If one wants their libraries to be widly available without any overheads . CDN is a good choice .

Alternatively any server from Google cloud , azure free tier or digital ocean or even heroku can be used for WebRTC code deployment . Note that webrtc capture now requires htps in domain name.

Server Setup

Set up environment by installing nvm  , npm  and git ( source version control)

1. NVM ( node version manager )

cURL:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash

or Wget:

wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

To check installation

command -v nvm
nvm

2. NPM( node package manager)

sudo apt-get install npm
Screenshot from 2016-05-16 12-41-42

2. Git

sudo apt-get install git
Screenshot from 2016-05-17 11-25-01

 SSL certificates

Since 2015 it has become mandatory to have only https origin request WebRTC’s getUserMedia API ie Voice, video, geolocation , screen sharing require https origins.
Note that this does not apply to case where its required to only serve peer’s media Stream or using Datachannels . Voice, video, geolocation , screen sharing now require https origins

For A POC purpose here is th way of generating a self signed certificate
Transport Layer Security and/or Secure Socket Layer( TLS/SSL) is a public/private key infrastructure.Following are the steps

1.create a private key

 openssl genrsa -out webrtc-key.pem 2048

2.Create a “Certificate Signing Request” (CSR) file

 openssl req -new -sha256 -key webrtc-key.pem -out webrtc-csr.pem

3.Now create a self-signed certificate with the CSR,

 openssl x509 -req -in webrtc-csr.pem -signkey webrtc-key.pem -out webrtc-cert.pem

However in production or actual implementation it is highly recommended to use a signed certificate by CA as For examples include

Web Server

create https certificate using self generate or purchased SSL certificates using fs , node-static and https modules . To know how to create self generated SSL certificates follow section above on SSL certificates.

var fs = require(‘fs’);
var _static = require(‘node-static’);
var https = require(‘https’);

var file = new _static.Server(&amp;amp;amp;amp;amp;amp;amp;quot;./&amp;amp;amp;amp;amp;amp;amp;quot;, {
cache: 3600,
gzip: true,
indexFile: &amp;amp;amp;amp;amp;amp;amp;quot;index.html&amp;amp;amp;amp;amp;amp;amp;quot;
});

var options = {
key: fs.readFileSync(‘ssl_certs/webrtc-key.pem’),
cert: fs.readFileSync(‘ssl_certs/webrtc-cert.pem’),
ca: fs.readFileSync(‘ssl_certs/webrtc-csr.pem’),
requestCert: true,
rejectUnauthorized: false
};

var app = https.createServer(options, function(request, response){
request.addListener(‘end’, function () {
file.serve(request, response);
}).resume();
});

app.listen(&amp;amp;amp;amp;amp;amp;amp;quot;8080&amp;amp;amp;amp;amp;amp;amp;quot;);

Web servers work with the HTTP (and HTTPS) protocol which is TCP based. As a genral rule TCP establishes connection whereas UDP send data packets

Scoketio signalling server as npm

Socket.io determines which of the following real-time communication method is suited to the particular client and its network bandwidth .

  • WebSocket
  • Adobe Flash Socket
  • AJAX long polling
  • AJAX multipart streaming
  • Forever Iframe
  • JSONP Polling

The socket.io server needs a HTTP Server for initial handshake. The general steps for socketio signalling server

1.require socket.io and keep the reference

var io = require(‘socket.io’)

2.Create your http / https server outline in section on webserver

3.bind your http and https servers (.listen)

io.listen(app, {
    log: false,
    origins: ‘*:*’
})

4. Optionally set transport

io.set(‘transports’, [
‘websocket’
])

5.setup io events

io.sockets.on(‘connection’, function (socket) {
//Do domething
})

Note that Socket.io or websockets require an http server for the initial handshake.

Install ssocketio npm module

npm install socket.io

Complete code for signalling server

const io = require("socket.io")
    .listen(app, {
        log: false,
        origins: "*:*"
    });
io.set("transports", ["websocket"])

var channels = {};

io.sockets.on("connection", function (socket) {

    console.log("connection");
    var initiatorChannel = "";

    if (!io.isConnected) {
        io.isConnected = true;
    }

    socket.on("namespace", function (data) {
        onNewNamespace(data.channel, data.sender);
    })

    socket.on("new-channel", function (data) {
        if (!channels[data.channel]) {
            initiatorChannel = data.channel;
        }
        console.log("new channel", data.channel, "by", data.sender)
        channels[data.channel] = {
            channel: data.channel,
            users: [data.sender]
        };

    })

    socket.on("join-channel", function (data) {
        console.log("Join channel", data.channel, "by", data.sender)
        channels[data.channel].users.push(data.sender);
    })

    socket.on("presence", function (channel) {
        var isChannelPresent = !!channels[channel.channel];
        console.log("presence for channel ", isChannelPresent)
        socket.emit("presence", isChannelPresent)
    })

    socket.on("disconnect", function (channel) {
        // handle disconnected event
    })

    socket.on("admin_enquire", function (data) {
        switch (data.ask) {
            case "channels":
                socket.emit("response_to_admin_enquire", channels)
                break;
            case "channel_clients":
                socket.emit("response_to_admin_enquire", io.of("/" + data.channel).clients());
                break;
            default :
                socket.emit("response_to_admin_enquire", channels)
        }
    })
})



function onNewNamespace(channel, sender) {
    console.log("onNewNamespace", channel);
    io.of("/" + channel).on("connection", function(socket) {
        var username;
        if (io.isConnected) {
            io.isConnected = false;
            socket.emit("connect", true)
        }
    
        socket.on("message", function (data) {
            if (data.sender == sender) {
                if(!username) username = data.data.sender;
                socket.broadcast.emit("message", data.data)
            }
        })
    
        socket.on("disconnect", function() {
            if(username) {
                socket.broadcast.emit("user-left", username)
                username = null;
            }
        })
    })
}

WebRTC main HTML5  project

This is the front  end section of the whole exercise . It contains JavaScript , css and html5 to make a webrtc call

<body id="pagebody">
<div id="elementToShare" className="container-fluid">
    <!-- ................................ top panel ....................... -->
    <div className="row topPanelClass">
        <div id="topIconHolder">
            <ul id="topIconHolder_ul">
                <li hidden><span id="username" className="userName" hidden>a</span></li>
                <li hidden><span id="numbersofusers" className="numbers-of-users" hidden></span></li>
                <li><span id="HelpButton" className="btn btn-info glyphicon glyphicon-question-sign topPanelButton"
                          data-toggle="modal" data-target="#helpModal"> Help </span></li>
            </ul>
        </div>
    </div>
    <!-- .............alerts................. -->
    <div className="row" id="alertBox" hidden="true"></div>
    <!-- .......................... Row ................................ -->
    <div className="row thirdPanelClass">
        <div className="col-xs-12 videoBox merge" id="videoHold">
            <div className="row users-container merge" id="usersContainer">
                <div className="CardClass" id="card">

                    <!-- when no remote -->
                    <div id="local" className="row" hidden="">
                        <video name="localVideo" autoPlay="autoplay" muted="true"/>
                    </div>
                    <!-- when remote is connected -->
                    <div id="remote" className="row" style="display:inline" hidden>
                        <div className="col-sm-6 merge" className="leftVideoClass" id="leftVideo">
                            <video name="video1" hidden autoPlay="autoplay" muted="true"></video>
                        </div>
                        <div className="col-sm-6 merge" className="rightVideoClass" id="rightVideo">
                            <video name="video2" hidden autoPlay="autoplay"></video>
                        </div>
                    </div>
                </div>
            </div>
        </div>
    </div>
    <!--modal help -->
    <div className="modal fade" id="helpModal" role="dialog">
        <div className="modal-dialog modal-lg">
            <div className="modal-content">
                <div className="modal-header">
                    <button type="button" className="close" data-dismiss="modal">&times;</button>
                    <h4 className="modal-title">Help</h4>
                </div>
                <div className="modal-body">
                    WebRTC Runs in only https due to getusermedia security contraints
                </div>
                <div className="modal-footer">
                    <button type="button" className="btn btn-default" data-dismiss="modal">Close</button>
                </div>
            </div>
        </div>
    </div>
</div>
</body>

the document start script that invokes the JS script

$('document').ready(function () {

    sessionid = init(true);

    var local = {
        localVideo: "localVideo",
        videoClass: "",
        userDisplay: false,
        userMetaDisplay: false
    };

    var remote = {
        remotearr: ["video1", "video2"],
        videoClass: "",
        userDisplay: false,
        userMetaDisplay: false
    };

    webrtcdomobj = new WebRTCdom(
        local, remote
    );

    var session = {
        sessionid: sessionid,
        socketAddr: "https://localhost:8084/"
    };

    var webrtcdevobj = new WebRTCdev(session, null, null, null);

    startcall();
});

Screenshot from 2016-05-17 12-12-37.png

Common known issues:

1.Opening page https://<web server ip>:< web server port>/index.html says insecure

This is beacuse the self signed certificates produced by open source openSSL is not recognized by a trusted third party Certificate Agency.
A CA ( Certificate Authority ) issues digital certificate to certify the ownership of a public key for a domain.

To solve the access issue goto https://<web server ip>:< web server port> and given access permission such as outlined in snapshot below

image

2.Already have given permission to Web Server , page loads but yet no activity .

if you open developer console ( ctrl+shift+I on google chrome ) you will notice that there migh be access related errros in red .
If you are using different server for web server and signalling server or even if same server but different ports you need to explicity go to the signalling server url and port and give access permission for the same reason as mentione above.

3.no webcam capture on opening the page

This could happen due to many reasons

  •  page is not loaded on https
  • browser is not webrtc compatible
  • Media permission to webcam are blocked
  • the machine does have any media capture devices attached
  •  Driver issues in the client machine while accessing webcams and mics .

4.socketio + code: 0, message: “Transport unknown”

Due to the version  v1.0.x of socket.io while performing handshake . To auto correct this , downgrade to v0.9.x