Monthly Archives: May 2016

GStreamer-1.8.1 rtsp server and client on ubuntu

GStreamer is a streaming media framework, based on graphs of filters which operate on media data.

Gstreamer is constructed using a pipes and filter architecture.
The basic structure of a stream pipeline is that you start with a stream source (camera, screengrab, file etc) and end with a stream sink (screen window, file, network etc). The ! are called pads and they connect the filters.

Data that flows through pads is described by caps (short for capabilities). Caps can be though of as mime-type (e.g. audio/x-raw, video/x-raw) along with mime-type (e.g. width, height, depth).

Source Code

Download the latest archives from https://gstreamer.freedesktop.org/src/

Source code on git : https://github.com/GStreamer

Primarily 3 files are required

  1. gstreamer-1.8.1.tar.xz
  2. gst-plugins-base-1.8.1.tar.xz
  3. gst-rtsp-server-1.8.1.tar.xz

If the destination machine is a ec2 instance one can also scp the tar.xz file there

To extract the tar.xz files use tar -xf <filename> it will create a folder for each package.

Prerequisites

build-essentials

sudo apt-get install build-essentials

bison

flex

GLib >= 2.40.0

GLib package contains low-level libraries useful for providing data structure handling for C, portability wrappers and interfaces for such runtime functionality as an event loop, threads, dynamic loading and an object system.

sudo apt-get install libglib2.0-dev

gstreamer

Installing gstreamer 1.8.1 . Gstreamer create a media stream with elements and properties as will be shown on  later sections of this tutorial .

cd gstreamer-1.8.1
./configure
make
sudo make install

Screenshot from 2016-05-19 16-51-29.png

Screenshot from 2016-05-19 16-55-27.png

Screenshot from 2016-05-19 16-56-05.png

after installation  export the path

export LD_LIBRARY_PATH=/usr/local/lib

then verify the installation of the gstreamer by

gst-inspect-1.0

provides information on installed gstreamer modules ie print out a long list ( about 123 in my case ) plugin that are installed such as coreelements:

capsfilter: CapsFilter ximagesink: ximagesink: Video sink videorate: videorate: Video rate adjuster typefindfunctions: image/x-quicktime: qif, qtif, qti typefindfunctions: video/quicktime: mov, mp4 typefindfunctions: application/x-3gp: 3gp typefindfunctions: audio/x-m4a: m4a typefindfunctions: video/x-nuv: nuv typefindfunctions: video/x-h265: h265, x265, 265 typefindfunctions: video/x-h264: h264, x264, 264 typefindfunctions: video/x-h263: h263, 263 typefindfunctions: video/mpeg4: m4v typefindfunctions: video/mpeg-elementary: mpv, mpeg, mpg typefindfunctions: application/ogg: ogg, oga, ogv, ogm, ogx, spx, anx, axa, axv typefindfunctions: video/mpegts: ts, mts typefindfunctions: video/mpeg-sys: mpe, mpeg, mpg typefindfunctions: audio/x-gsm: gsm

gst plugins

Now build the plugins

cd gst-plugins-base-1.8.1
./configure
make
sudo make install

 

gst plugins good

cd gst-plugins-good-1.8.1.tar
./configure
 make
sudo make install

RTSP Server

Now make and install the rtsp server

cd gst-rtsp-server-1.8.1
./configure

last few lines from console traces

Configuration
Version : 1.8.1
Source code location : .
Prefix : /usr/local
Compiler : gcc -std=gnu99
CGroups example : no

make

It will compile the examples .

sudo make install

 

stream video test src

~/mediaServer/gst-rtsp-server-1.8.1/examples]$./test-launch --gst-debug=0 &quot;( videotestsrc ! video/x-raw,format=(yuv),width=352,height=288,framerate=15/1 ! x264enc ! rtph264pay name=pay0 pt=96 )&quot;
stream ready at rtsp://127.0.0.1:8554/test

Ref:

Manual for developers : https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-rtsp-server/html/index.html


Simplest pipeline

gst-launch-1.0 fakesrc ! fakesink

➜ ~ gst-launch-1.0 fakesrc ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
To stop press ctrl +c ^
Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:48.004547887 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ... [/sourcecode ] or to display to a audiovideosink gst-launch-1.0 videotestsrc ! autovideosink
Screenshot from 2016-05-20 12-31-18.png To capture webcam
gst-launch v4l2src ! xvimagesink

Screenshot from 2016-05-20 13-06-56.png

Setting up ubuntu ec2 t2 micro for webrtc and socketio

Setting up a ec2 instance on AWS for web real time communication platform over nodejs and socket.io using WebRTC .

Primarily a Web Call  , Chat and conference platform uses WebRTC for the media stream and socketio for the signalling . Additionally used technologies are nosql for session information storage , REST Apis foe getting sessions details to third parties.

Below is a comprehensive setup if ec2 t2.micro free tier instance  ,  installation with a webrtc project module and samples of customization and usuage .

Technologies used are listed below :

Server

  1. ec2 instance t2.micro covered under free tier
  2. domain name
  3. SSL certificate

Core module for Web Calling feature

  1. WebRTC
  2. Node.js
  3. socket.io

UI components

  1. javascript
  2. css
  3. html5
  4. bootstrap
  5. jquerry

Supporting setup for session management

  1. Code version-ing  and maintenance
  2. git
  3. npm

Amazon’s free tier ec2

Amazon EC2
ec2 instances are elastic compute general purpose storage servers that mean that they can resize the compute capacity in the cloud based on load .
750 hours per month of Linux, RHEL, or SLES t2.micro instance usage
Expires 12 months after sign-up.

Some other products are also covered under free tier which may come in handy for setting up the complete complatorm .Here is a quick summary

1.Amazon S3
it is a storage server. Can be used to store media file like image s, music , videos , recorded video etc .

2.Amazon RDS
It a relational database server . If one is using mysql or postgress for storing session information or user profile data . It is good option .

3.Amazon SES
email service. Can be used to send invites and notifications to users over mail for scheduled sessions or missed calls .

4.Amazon CloudFront
It is a CDN ( content delivery network ) . If one wants their libraries to be widly available without any overheads . CDN is a good choice .

Server Setup

Set up environment by installing nvm  , npm  and git ( source version control)

1. NVM ( node version manager )

cURL:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash

or Wget:

wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.31.1/install.sh | bash&amp;amp;amp;amp;amp;amp;amp;amp;amp;lt;/code&amp;amp;amp;amp;amp;amp;amp;amp;amp;gt;

To check installation

command -v nvm
nvm

2. NPM( node package manager)

sudo apt-get install npm

Screenshot from 2016-05-16 12-41-42

2. Git

sudo apt-get install git

Screenshot from 2016-05-17 11-25-01

 SSL certificates

Since 2015 it has become mandatory to have only https origin request WebRTC’s getUserMedia API ie Voice, video, geolocation , screen sharing require https origins.
Note that this does not apply to case where its required to only serve peer’s media Stream or using Datachannels . Voice, video, geolocation , screen sharing now require https origins

For A POC purpose here is th way of generating a self signed certificate
Transport Layer Security and/or Secure Socket Layer( TLS/SSL) is a public/private key infrastructure.Following are the steps

1.create a private key
openssl genrsa -out webrtc-key.pem 2048

2.Create a “Certificate Signing Request” (CSR) file
openssl req -new -sha256 -key webrtc-key.pem -out webrtc-csr.pem

3.Now create a self-signed certificate with the CSR,
openssl x509 -req -in webrtc-csr.pem -signkey webrtc-key.pem -out webrtc-cert.pem

However in production or actual implementation it is highly recommended to use a signed certificate by CA as For examples include
Godaddy (https://ca.godaddy.com/web-security/ssl-certificate) , Comoddo (https://ssl.comodo.com/) , Global Sign (https://www.globalsign.com/en/ssl/managed-ssl/) , Symantec (https://www.symantec.com/ssl-certificates) etc .

Web Server

create https certificate using self generate or purchased SSL certificates using fs , node-static and https modules . To know how to create self generated SSL certificates follow section above on SSL certificates.

var fs = require(‘fs’);
var _static = require(‘node-static’);
var https = require(‘https’);

var file = new _static.Server(&amp;amp;amp;amp;amp;amp;amp;quot;./&amp;amp;amp;amp;amp;amp;amp;quot;, {
cache: 3600,
gzip: true,
indexFile: &amp;amp;amp;amp;amp;amp;amp;quot;index.html&amp;amp;amp;amp;amp;amp;amp;quot;
});

var options = {
key: fs.readFileSync(‘ssl_certs/webrtc-key.pem’),
cert: fs.readFileSync(‘ssl_certs/webrtc-cert.pem’),
ca: fs.readFileSync(‘ssl_certs/webrtc-csr.pem’),
requestCert: true,
rejectUnauthorized: false
};

var app = https.createServer(options, function(request, response){
request.addListener(‘end’, function () {
file.serve(request, response);
}).resume();
});

app.listen(&amp;amp;amp;amp;amp;amp;amp;quot;8080&amp;amp;amp;amp;amp;amp;amp;quot;);

Web servers work with the HTTP (and HTTPS) protocol which is TCP based. As a genral rule TCP establishes connection whereas UDP send data packets

 

Scoketio signalling server as npm

Socket.io determines which of the following real-time communication method is suited to the particular client and its network bandwidth .

  • WebSocket
  • Adobe Flash Socket
  • AJAX long polling
  • AJAX multipart streaming
  • Forever Iframe
  • JSONP Polling

The socket.io server needs a HTTP Server for initial handshake.

The general steps for socketio signalling server are:

1.require socket.io and keep the reference. like
var io = require(‘socket.io’)

2.Create your http / https server
outline in section on webserver

3.bind your http and https servers (.listen)
io.listen(app, {
log: false,
origins: ‘*:*’
});

4. Optionally set transport
io.set(‘transports’, [
‘websocket’
]);

4.setup io events as
io.sockets.on(‘connection’, function (socket) {

//Do domething
});

Note that Socket.io or websockets require an http server for the initial handshake.
&amp;amp;lt;pre&amp;amp;gt;Install ssocketio npm module&amp;amp;lt;/pre&amp;amp;gt;&amp;amp;lt;pre&amp;amp;gt;
npm install socket.io
[/sourcecode ]

Complete code for signalling server

var io = require(‘socket.io’).listen(app, {
log: false,
origins: ‘*:*’
});

io.set(‘transports’, [
‘websocket’
]);

var channels = {};

io.sockets.on(‘connection’, function (socket) {

console.log(&amp;amp;amp;amp;amp;amp;amp;quot;connection &amp;amp;amp;amp;amp;amp;amp;quot;);
var initiatorChannel = ”;

if (!io.isConnected) {
io.isConnected = true;
}

socket.on(‘namespace’,function(data){
onNewNamespace(data.channel, data.sender);
});

socket.on(‘new-channel’, function (data) {
if (!channels[data.channel]) {
initiatorChannel = data.channel;
}
console.log(&amp;amp;amp;amp;amp;amp;amp;quot;————new channel &amp;amp;amp;amp;amp;amp;amp;quot;, data.channel , &amp;amp;amp;amp;amp;amp;amp;quot; by &amp;amp;amp;amp;amp;amp;amp;quot; , data.sender);
channels[data.channel] = {
channel: data.channel,
users:[data.sender]
};

});

socket.on(‘join-channel’, function (data) {
console.log(&amp;amp;amp;amp;amp;amp;amp;quot;————join&amp;amp;amp;amp;amp;amp;amp;amp;nbsp;channel &amp;amp;amp;amp;amp;amp;amp;quot;, data.channel , &amp;amp;amp;amp;amp;amp;amp;quot; by &amp;amp;amp;amp;amp;amp;amp;quot; , data.sender);
channels[data.channel].users.push(data.sender);
});

socket.on(‘presence’, function (channel) {
var isChannelPresent = !! channels[channel.channel];
console.log(&amp;amp;amp;amp;amp;amp;amp;quot;presence for channel &amp;amp;amp;amp;amp;amp;amp;quot; ,isChannelPresent);
socket.emit(‘presence’, isChannelPresent);
});

socket.on(‘disconnect’, function (channel) {
});

socket.on(&amp;amp;amp;amp;amp;amp;amp;quot;admin_enquire&amp;amp;amp;amp;amp;amp;amp;quot;,function(data){
switch (data.ask){
case &amp;amp;amp;amp;amp;amp;amp;quot;channels&amp;amp;amp;amp;amp;amp;amp;quot;:
socket.emit(‘response_to_admin_enquire’, channels);
break;
case &amp;amp;amp;amp;amp;amp;amp;quot;channel_clients&amp;amp;amp;amp;amp;amp;amp;quot;:
socket.emit(‘response_to_admin_enquire’, io.of(‘/’ + data.channel).clients());
break;
default :
socket.emit(‘response_to_admin_enquire’, channels);
}

});

});

function onNewNamespace(channel, sender) {
console.log(&amp;amp;amp;amp;amp;amp;amp;quot; —–&amp;amp;amp;amp;amp;amp;amp;amp;gt; onNewNamespace &amp;amp;amp;amp;amp;amp;amp;quot;, channel);

io.of(‘/’ + channel).on(‘connection’, function (socket) {

var username;
if (io.isConnected) {
io.isConnected = false;
socket.emit(‘connect’, true);
}

socket.on(‘message’, function (data) {
if (data.sender == sender) {
if(!username) username = data.data.sender;
socket.broadcast.emit(‘message’, data.data);
}
});

socket.on(‘disconnect’, function() {
if(username) {
socket.broadcast.emit(‘user-left’, username);
username = null;
}
});
});
}

 

WebRTC main HTML5  project

This is the front  end section of the whole exercise . It contains JavaScript , css and html5 to make a webrtc call

<html lang=en>
<head>
<title>WebRTC Call</title>

<meta http-equiv=Content-Type content="text/html; charset=UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">

	<link rel=stylesheet href="https://ec2-54-193-124-35.us-west-1.compute.amazonaws.com:8084/minScripts/webrtcdevelopment_header.css">
<script src="https://ec2-54-193-124-35.us-west-1.compute.amazonaws.com:8084/minScripts/webrtcdevelopment_header.js"> </script>

<style type="text/css">
video{
width:100% !important;
}
body{
background: #2B2B2B;
}
</style>
</head>

<body id="pagebody">
<div id="elementToShare" class="container-fluid">
<!-- ................................ top panel ....................... -->
<div class="row topPanelClass" >
<div id="topIconHolder" >
<ul id="topIconHolder_ul">
	<li hidden> <span id="username" class="userName" hidden>a</span></li>
	<li hidden> <span id="numbersofusers" class="numbers-of-users" hidden></span></li>
	<li> <span id="HelpButton" class="btn btn-info glyphicon glyphicon-question-sign topPanelButton" data-toggle="modal" data-target="#helpModal" > Help </span></li>
</ul>
</div>
</div>
<!-- .............alerts................. -->
<div class="row" id="alertBox" hidden="true"></div>
<!-- .......................... Row ................................ -->
<div class="row thirdPanelClass">
<div class="col-xs-12 videoBox merge" id="videoHold">
<div class="row users-container merge" id="usersContainer" >
<div class="CardClass" id="card">

<!-- when no remote -->
<div id="local" class="row" hidden="">
<video name="localVideo" autoplay="autoplay" muted="true" />
</div>
<!-- when remote is connected -->
<div id ="remote" class="row" style="display:inline" hidden>
<div class="col-sm-6 merge" class="leftVideoClass" id="leftVideo">
<video name="video1" hidden autoplay="autoplay" muted="true" ></video>
</div>
<div class="col-sm-6 merge" class="rightVideoClass" id="rightVideo" >
<video name="video2" hidden autoplay="autoplay" ></video>
</div>
</div>
</div>
</div>
</div>
</div>
<!--modal help -->
<div class="modal fade" id="helpModal" role="dialog">
<div class="modal-dialog modal-lg">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal">&times;</button>
<h4 class="modal-title">Help</h4>
</div>
<div class="modal-body">
WebRTC Runs in only https due to getusermedia security contraints
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div>
</div>
</body>

	<link rel=stylesheet href="https://ec2-54-193-124-35.us-west-1.compute.amazonaws.com:8084/minScripts/webrtcdevelopment.css">
<script src="https://ec2-54-193-124-35.us-west-1.compute.amazonaws.com:8084/minScripts/webrtcdevelopment.js"> </script>

<script>
$('document').ready(function(){

 sessionid= init(true);

 var local={
 localVideo: "localVideo",
 videoClass:"",
 userDisplay:false,
 userMetaDisplay:false 
 };

 var remote={
 remotearr: ["video1" , "video2"],
 videoClass:"",
 userDisplay:false,
 userMetaDisplay:false 
 };

 webrtcdomobj= new WebRTCdom(
 local,remote
 );

 var session ={
 sessionid : sessionid,
 socketAddr: "https://localhost:8084/"
 };

 var webrtcdevobj = new WebRTCdev ( session, null, null , null );

 startcall();
});
</script>
</html>
Screenshot from 2016-05-17 12-12-37.png

Common known issues:

1.Opening page https://<web server ip>:< web server port>/index.html says insecure

This is beacuse the self signed certificates produced by open source openSSL is not recognized by a trusted third party Certificate Agency.
A CA ( Certificate Authority ) issues digital certificate to certify the ownership of a public key for a domain.

To solve the access issue goto https://<web server ip>:< web server port> and given access permission such as outlined in snapshot below

image

2.Already have given permission to Web Server , page loads but yet no activity .

if you open developer console ( ctrl+shift+I on google chrome ) you will notice that there migh be access related errros in red .
If you are using different server for web server and signalling server or even if same server but different ports you need to explicity go to the signalling server url and port and give access permission for the same reason as mentione above.
3.no webcam capture on opening the page

This could happen due to many reasons

  •  page is not loaded on https
  • browser is not webrtc compatible
  • Media permission to webcam are blocked
  • the machine does have any media capture devices attached
  •  Driver issues in the client machine while accessing webcams and mics .

4.socketio + code: 0, message: “Transport unknown”

Due to the version  v1.0.x of socket.io while performing handshake . To auto correct this , downgrade to v0.9.x