Anywhere anytime Telemedicine communication tool accessible on any device. The solution provides a low eight signalling server which drops out as soon as call is connected thus ensuring absolutely private calls without relaying or involving any central server in any call related data or media . This ensure doctor patient details are not processed , stored or recorded by our servers.
The solution enables doctors / nurses / medical practitioners and patients to do
High definition Audio/video calls
End to end encrypted p2p chats
Integration with HMS ( hospital management system ) to fetch history of the patients
Screens sharing to show reports without transferring them as files
Include more concerned people of doctors using Mesh based peer to peer conferencing feature.
Confidentialty and Privacy
For privacy and security of certain health information only HIPAA (Health Insurance Portability and Accountability Act of 1996) compliant video-conferencing tools can only be used for Telemedicine in US.
Telemedicine scenario Callflow
Calllfow for Attended Call Transfer and 2 way conference in a Telemedicine scenario between Patient , hospital attendant , doctor and a nurse
Codecs signifies the media stream’s compession and decompression. For peers to have suceesfull excchange of media, they need a common set of codecs to agree upon for the session . The list codecs are sent between each other as part of offeer and answer or SDP in SIP.
As WebRTC provides containerless bare mediastreamgtrackobjects. Codecs for these tracks is not mandated by webRTC . Yet the codecs are specified by two seprate RFCs
RFC 7878 WebRTC Audio Codec and Processing Requirements specifies least the Opus codec as well as G.711’s PCMA and PCMU formats.
RFC 7742 WebRTC Video Processing and Codec Requirnments specifies support for VP8 and H.264’s Constrained Baseline profile for video .
In WebRTC video is protected using Datagram Transport Layer Security (DTLS) / Secure Real-time Transport Protocol (SRTP). In this article we are going to dicuss Audio/Video Codecs processing requirnments only.
The MediaStreamTrack interface typically represents a stream of data of audio or video and a MediaStream may contain zero or more MediaStreamTrack objects.
The objects RTCRtpSender and RTCRtpReceiver can be used by the application to get more fine grained control over the transmission and reception of MediaStreamTracks.
Media Flow in VoIP systemMedia Flow in WebRTC Call
Video
Video Capture insync with hardware’s capabilities
WebRTC compatible browsers are required to support Whie-balance , light level , autofocus from video source
Video Capture Resolution
Minimum WebRTC video attributes unless specified in SDP ( Session Description protocl ) is minimum 20 FPS and resolution 320 x 240 pixels.
Also supports mid stream resilution changes such as in screen source fromdesktop sharinig .
SDP attributes for resolution, frame rate, and bitrate
SDP allows for codec-independent indication of preferred video resolutions using a=imageattr to indicate the maximum resolution that is acceptable.
Sender must send limiting the encoded resolution to the indicated maximum size, as the receiver may not be capable of handling higher resolutions.
Dynamic FPS control based on actual hardware encoding :
video source capture to adjust frame rate accroding to low bandwidth , poor light conditions and harware supported rate rather than force a higher FPS .
Stream Orientation
support generating the R0 and R1 bits of the Coordination of Video Orientation (CVO) mechanism and sharing with peer
Codecs
WebRTC is free and opensource and its woring bodies promote royality free codecs too. The working groups RTCWEB and IETF make the sure of the fact that non-royality beraning codec are mandatory while other codecs can be optional in WebRTC non browsers .
WebRTC Browsers MUST implement the VP8 video codec as described in RFC6386] and H.264 Constrained Baseline as described in [H264].
RFC 7442 WebRTC Video Codec and Processing Requirements
most of the codesc below follow Lossy DCT(discrete cosine transform (DCT) based algorithm for encoding.
Sample SDP from offer in Chrome browser v80 for Linux incliudes these profile :
Developed by on2 and then acquired and opensource by google . Now free of royality fees.
Supported conatiner – 3GP, Ogg, WebM
No limit on frame rate or data rate and provides maximum resolution of 16384×16384 pixels.
libvpx encoder library.
VP8 encoders must limit the streams they send to conform to the values indicated by receivers in the corresponding max-fr and max-fs SDP attributes. encode and decode pixels with an implied 1:1 (square) aspect ratio.
supported simulcast
VP9
Video Processor 9 (VP9) is the successor to the older VP8 and comparable to HEVC as they both have simillar bit rates .
Open and free of royalties and any other licensing requirements Its supported Containers are – MP4, Ogg, WebM
H264/AVC constrained
AVC’s Constrained Baseline (CBP ) profile compliant with WebRTC
Constrained Baseline Profile Level 1.2 and H.264 Constrained High Profile Level 1.3 . Contrained baseline is a submet of the main profile , suited to low dealy , low complexity. suited to lower processing device like mobile videos
Multiview Video Coding – can have multiple views of the same scene ,such as stereoscopic video.
Other profiles , which are not supporedt are Baseline (BP) , Extended (XP), Main (MP) , High (HiP) , Progressive High (ProHiP) , High 10 (Hi10P), High 4:2:2 (Hi422P) and High 4:4:4 Predictive
Its supported containers are 3GP, MP4, WebM
Parameter settings:
packetization-mode
max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br
sprop-parameter-sets: H.264 allows sequence and picture information to be sent both in-band and out-of-band. WebRTC implementations must signal this information in-band.
Supplemental Enhancement Information (SEI) “filler payload” and “full frame freeze” messages( used while video switching in MCU streams )
It is a propertiary , patented codec , mianted by MPEG / ITU
AV1 (AOMedia Video 1)
open format designed by the Alliance for Open Media royality free especially designed for internet video HTML element and WebRTC higher data compression rates than VP9 and H.265/HEVC
offers 3 profiles in increasing support for color depths and chroma subsampling. main, high, and professional
supports HDR supports Varible Frame Rate
Supported container are ISOBMFF, MPEG-TS, MP4, WebM
Graph for Video Track in chrome://webrtc-internals
Other RTP parameters
RTX(regtranmission ) – packet loss recovery technique for real-time applications with relaxed delay bounds.
Non WebRTC supported Video codecs
Need active realtime media transcoding
H.263
Already used for video conferencing on PSTN (Public Switched Telephone Networks), RTSP, and SIP (IP-based videoconferencing) systems. suited for low bandwidth networks Although it is not comaptible with WebRTC but many media gateways incldue realtime transcoding existed between H263 based SIP systems and vp8 based webrtc ones to enable video communication between them
H.265 / HEVC
proprietary format and is covered by a number of patents. Licensing is managed by MPEG LA .
Container – Mp4
Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints
With the rise of Internet of Things many Endpoints especially IP cameras connected to Raspberry Pi like SOC( system on chiops )n wanted to stream directly to the browser within theor own provate network or even on public network using TURN / STUN.
The figure below shows how such a call flow is possible between an IP cemera ( such as Baby Cam ) and its parent monitoring it over a WebRTC suppported mobile phone browser . The process includes streaming teh content from IOT device on RTSP stream and using realtime trans-coding between H264 and VP8
Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints
Audio
Audio Level
audio level for speech transmission to avoid users having to manually adjust the playback and to facilitate mixing in conferencing applications.
normalization considering frequencies above 300 Hz, regardless of the sampling rate used.
adapted to avoid clipping, either by lowering the gain to a level below -19 dBm0 or through the use of a compressor.
GAIN calculation
If the endpoint has control over the entire audio-capture path like a regular phone the gain should be adjusted in such a way that an average speaker would have a level of 2600 (-19 dBm0) for active speech.
If the endpoint does not have control over the entire audio capture like software endpoint then the endpoint SHOULD use automatic gain control (AGC) to dynamically adjust the level to 2600 (-19 dBm0) +/- 6 dB.
For music- or desktop-sharing applications, the level SHOULD NOT be automatically adjusted, and the endpoint SHOULD allow the user to set the gain manually.
Acoustic Echo Cancellation (AEC)
Endpoints shoudl allow echo control mechsnisms
Codecs
WebRTC endpoints are should implement audio codecs: OPUS and PCMA / PCMU, along with Comforrt Noise and DTMF events.
Trace for audio codecs supported in chrome (Version 80.0.3987.149 (Official Build) (64-bit) on ubuntu)
For all cases where the endpoint is able to process audio at a sampling rate higher than 8 kHz, it is w3C recommenda that Opus be offered before PCMA/PCMU.
AAC (Advanvced Audio Encoding)
part of the MPEG-4 (H.264) standard supported congainers – MP4, ADTS, 3GP
Lossy compression but has number pf profiles suiting each usecase like high quality surround sound to low-fidelity audio for speech-only use.
G.711 (PCMA and PCMU)
ITU published Pulse Code Modulation (PCM) with either µ-law or A-law encoding. vital to interface with the standard teelcom network and carriers
Fixed 64Kbpd bit rate
supports 3GP container formats
G.711 PCM (A-law) is known as PCMA and G.711 PCM (µ-law) is known as PCMU
G.722
ncoded using Adaptive Differential Pulse Code Modulation (ADPCM) which is suited for voice compression conatiners used 3GP, AMR-WB
Comfort noise (CN)
artificial background noise which is used to fill gaps in a transmission instead of using pure silence
avoids jarring or RTP Timeout
for streams encoded with G.711 or any other supported codec that does not provide its own CN. Use of Discontinuous Transmission (DTX) / CN by senders is optional
Internet Low Bitrate Codec (iLBC)
opensource narrow band codec designed specifically for streaming voice audio
Internet Speech Audio Codec (iSAC)
designed for voice transmissions which are encapsulated within an RTP stream.
DTMF and ‘audio/telephone-event’ media type
endpoints may send DTMF events at any time and should suppress in-band dual-tone multi-frequency (DTMF) tones, if any.
This article is aimed at explaning the intricacies and detailed offer answer flow in webrtc handshake and JSEP . You can read the following artciles on WebRTC as prereq before reading through this one
WebRTC API – Peerconnection , getUserMedia , Datachannel , DataStaats
JSEP (JavaScript Session Establishment Protocol)
JSEP (JavaScript Session Establishment Protocol) is used during signalling via w3c RTCPeerConnectionAPI interface to set up a multimedia session. The multimedia session description specfies the crtical components of setting up a session between local and remote such as transport ports , protcol , profiles . It also handles the intercation with ICE state machine
Offer/Answer Excahange Flow
prereq : Setup Client side for the caller PeerConnectionFactory to generate PeerConnections PeerConnection for every connection to remote peer MediaStream audio and video from client device
Side initiating the session creates a offer by CreateOffer() API
5. Remote party stores the answer in its local config using setLocalDescription() API
6. Answer is transferred to Initiator side using choice of signalling ( SIP , WS , HTTP, XMPP .. ) again
7. Initiating side stores it use setRemoteDescription() API
Interfaces of webrtc and tracks to stream addition
Process to perform webrtc handshake
Webrtc call setup and incoming call callflow between remote peer , peerconnection actory , peerconnection and application
setup a callreceive a call
Signalling state Transitions on PeerConnection
As the caller initiates a new RTCPeerConnection() , the RTCSignalingState state is “stable” as remote and local descriptions are empty
As the caller initiates call and calls createOffer() , he now has offer SDP and procced to store offer locally with setLocalDescription(offer) the RTCSignalingState state is “have-local-offer” . After than caller send the offer to callee over signalling channel
Simillarily as the calle recives the offer , it starts with RTCSignalingState stable and then proceeds to store the Remote’s offer using setRemoteDescription(offer) , its state is now “have-remote-offer”
The callee generates a provsional answer and for caller and stores it locally , state transitiosn to “have-local-pranswer” . the pranswer SDP is send to caller over signalling channel again .
Caller stores the callee’s pr answer SDP and state updates to “have-remote-pranswer”
Once there is no offer/answer exchange in progress the state again changes to ” stable “.
State schanges to “closed” if RTCpeerConnection is closed
Media Section : An m= section is generated for each RtpTransceiver that has been added to the PeerConnection. For the initial offer since no ports are available yet , dummy port 9 can be sadded. However if it is bundle only then port value is set to 0. Later the port value will be set to the port value of default ICE candidate.
DTLS filed “UDP/TLS/RTP/SAVPF” is followed by the list of codecs in order of priority.
“c=” line in msection too must be filled with dummy values if IP 0.0.0.0 as no candidates are available yet .
For each media format on the m= line, “a=rtpmap” for “rtx” with the clock rate of codec and “a=fmtp” to reference the payload type of the primary codec. “a=rtcp-fb” specified RTCP feedback
When createOffer is called a second (or later) time, or is called after a local description has already been installed, the processig is different due to gathered ICE candidates . However the <session-version> is not changed .
Additionally m section is updated if RtpTransceiver is added or removed
Each “m=” and c=” line MUST be filled in with the port, relevant RTP profile, and address of the default candidate for the m= section
If the m= section is not bundled into another m= section, update the “a=rtcp” with port and address of RTCP camdidate and add “a=camdidate” with “a=end-of-candidates”
Local Answer created by side receiving the session/ Callee
When createAnswer is called for the first time after a remote description has been provided, the result is known as the initial answer.
Each offered m= section will have an associated RtpTransceiver
Remote Destination / Callee can reject the m section by setting port in m line to 0 . It can reject msection if neither of the offered media format are supported , RtpTransceiver is stoopped etc.
For the initial offer the dummy port value of 9 is set as no ICE candudate is avaible yet . Simillarly “c=” line must contain the “dummy” value “IN IP4 0.0.0.0” too.
The <proto> field MUST be set to exactly match the <proto> field for the corresponding m= line in the offer.
If the answer contains any “a=ice-options” attributes where “trickle” is listed as an attribute, update the PeerConnection canTrickle property to be true.
Modifying Offer/answer SDP
SDP returned from createOffer or createAnswer MUST NOT be changed before passing it to setLocalDescription. After calling setLocalDescription with an offer or answer, the application MAY modify the SDP to reduce its capabilities before sending it to the far side
Assume we have a MCU at location and want the video stream to relay via a Media Server.
SDP Parsing
SDP is used for session parsing and contians sequence of line with key value pairs. SDP is read, line-by-line, and converted to a data structure that contains the deserialized information.
Line “v=” , “o=”,”b=” and “a=” are processed . The “i=”, “u=”, “e=”, “p=”, “t=”, “r=”, “z=”, and “k=” lines are not used by this specification; they MUST be checked for syntax but their values are not used. Line “c=” is checked for syntax and ICE mismatch detection
“a= ” attribute could be : “a=group” , “s=”ice-lite” , “a=ice-pwd”, “a=ice-options” , “a=fingerprint”, “a=setup” , a=tls-id”, “a=identity” , “a=extmap”
Media Section Parsing
Line “m=” for media , proto , port , fmt in RTP
Attributes “a=” can be
“a=rtpmap” or “a=fmtp”
map from an RTP payload type number to a media encoding name that identifies the payload format.
dierction as “a=sendrecv” , a=recvonly , a=sendonly , a=inactive
Muxing as “a=rtcp-mux” ,
“a=rtcp-mux-only”
RTCP attributes “a=rtcp” , “a=rtcp-rsize”
Line “c=” is checked .
Line “b=” for bandiwtdh , bwtype
Attribites for “a=” could be “a=ice-ufrag”, “a=”ice-pwd”, “a=ice-options” , “a=candidate”, “a=remote-candidate” , a=end-of-candidates” and “a=fingerprint”
Semantics Verification
Interactive Connectivity Establishment (ICE) for NAT traversal
Protocols using offer/answer are difficult to operate through Network Address Translators (NATs) since flow of media packets require IP addresses and ports of media sources and sinks within their messages. Also realtime media emphasises on reduced latency and decreased packet loss .
An extension to the offer/answer model, and works by including a multiplicity of IP addresses and ports in SDP offers and answers, which are then tested for connectivity by peer-to-peer connectivity checks. Checks done by STUN and TURN also allows for address selection for multi-homed and dual-stack hosts
ICE allows the agents to discover enough information about their topologies to potentially find one or more paths by which they can communicate. Then it systematically tries all possible pairs (in a carefully sorted order) until it finds one or more that work.
ICE Gathering
Caller and callee performs checks to finalize the protocol and routing needed to establish a peer connection . Number of candudates are proposed till they mutually agree upon one . Peerconnection then uses that candiadte detaisl to initiate the connection .
While Applying a Local Description at the media engine level if m= section is new, WebRTC media stacks begins gathering candidates for it.
RTCPeerconnection specified canTrickleIceCandidates . ICE trickling is the process of continuing to send candidates after the initial offer or answer has already been sent to the other peer.
ICE TransportRole is responsible for Choosing a candidate pair
ICE layer sets one peer as controlling and other as controlled agent . The controling agent makes the final decision as to which candidate pair to choose.
An agent identifies all CANDIDATE whic is a transport address. Types:
HOST CANDIDATE – directly from a local interface which could be Wifi, Virtual Private Network (VPN) or Mobile IP (MIP) if an agent is multihomed ( private and public networks) , it obtains a candidate from each IP address and includes all candidates in its offer.
STUN or TURN to obtain additional candidates. Types
translated addresses on the public side of a NAT (SERVER REFLEXIVE CANDIDATES)
addresses on TURN servers (RELAYED CANDIDATES)
Mapping Server Reflexive address
Agent sends the TURN Allocate request from IP address and port X:x, NAT will create a binding X1′:x1′, mapping this server reflexive candidate to the host candidate X:x ( BASE). Outgoing packets sent from the host candidate will be translated by the NAT to the server reflexive candidate. Incoming packets sent to the server reflexive candidate will be translated by the NAT to the host candidate and forwarded to the agent.
Allocate Request and response fom TURN – Informing the agent of this relayed candidate
Only STUN based Binding
agent sends a STUN Binding request to its STUN server which will get server reflexive candidate and send back Binding response.
STUN Binding request for connectivity checks on CANDIDATE PAIRS
The candidates are carried in attributes in the SDP offer . The remote peer also follows this process and gather and send lits own sorted list of candidates. Hence CANDIDATE PAIRS from both sides are formed.
PEER REFLEXIVE CANDIDATES – connectivity checks can produce aditional candidates espceialy around symmetric NAT
Since the same address is used for STUN. and media ( RTP/RTCP) Demultiplexing based on packet contents helps to identify which one is which.
Checks : ICE checks are performed in a specific sequence, so that high-priority candidate pairs are checked first.
TRIGGERED CHECKS – accelerates the process of finding a valid candidate ORDINARY CHECKS – agent works through ordered prioritised check list by sending a STUN request for the next candidate pair on the list periodically.
Checks ensure maintaining frozen candidates and pairs with some foundation for media stream
Each candidate pair in the check list has a foundation and a state. States for candidates pairs
1.Waiting: A check has not been performed for this pair, and can be performed as soon as it is the highest-priority Waiting pair onthe check list.
2. In-Progress: A check has been sent for this pair, but the transaction is in progress.
3. Succeeded: A check for this pair was already done and produced a successful result.
4. Failed: A check for this pair was already done and failed, either never producing any response or producing an unrecoverable failure response.
5. Frozen: A check for this pair hasn’t been performed, and it can’t yet be performed until some other check succeeds, allowing this pair to unfreeze and move into the Waiting state.
selecting low-latency media paths can use various techniques such as actual round-trip time (RTT) measurement controlling agent gets to nominate which candidate pairs will get used for media amongst the ones that are valid. Ways regular nomination and aggressive nomination
TBD
To read More on WebRTC Communication as a platform
A CPasS ( communication platform as a service ) is cloud based communication platform alo B2B cloud communications platform that provides real time communication capabilities. This should be easily integrable with any given external environment or application of the customer, without him worrying about building backend infrastructure or interfaces .
Traditionally , with IP protected protocols , licensed codecs maintaining a signalling protocol stack , network interfaces building communication platform was a costly affair. Cisco , Facetime , Skype were the only OTT ( over the top) players taking away from telco’s call revenue .
However with the advent of standardize , open source protocol and codecs plenty of CPaaS providers have crowded the market making more supply than there is demand. A customer wanting to quickly integrate real time communications on his platform has many options to choose from. This article provides an insight to how CPaaS solution are architectured and programmed
Sample CPass Architecture build on open source technologies
Call server + Media Server that can be interacted with via UA
Comm clients like sipphones , webrtc client , SDK ( software development kits ) or libraries for desktop , embedded and/or mobile platforms .
APIs that can trigger automated calls and perform preprogrammed routing.
Rich documentation and samples to build various apps such as call centre solutions , interactive auto-attendant using IVR , DTMF , conference solutions etc .
Some CPaaS providers also add features like transcribing ,transcoding , recording , playback etc to provide edge over other CPaaS providers
Datacentre vs Cloud server
-tbd
Advantages of using a CPaaS vs building your own RTC platform
Tech insights and experiences
companies who have been catering to telco and communication domain make robust solutions based on industry best practices which beats novice solution build in a fortnight anyday
keeping up with emerging trends
Market trends like new codecs , rich communication services , multi tenancy, contextual communication , NLP , other ML based enhancements are provided by CPaaS company
Auto Scaling , High Availability
A firm specializing in CPaaS solution has already thought of clustering and autoscaling to meet peak traffic requirements and backup/replication on standby servers to activate incase of failure
CAPEX and OPEX
using a Cpaas saves on human resources, infrastructure, and time to market. It saves tremendously on underlying IT infrastructure and many a times provides flexible pricing models
In a nutshell I have come across so many small size startups trying to build CPaaS solution from scratch but only realising it after weeks of trying to build a MVP that they are stuck with firewall, NAT, media quality or interoperability issues . Since there are so many solution already out in the market it is best to instead use them as underlying layer and build applications services using it such as callcentre or CRM services etc .
The purpose of this article if to demo the process of using Kamailio + RTP Engine to enable SIP based WebRTC call to a traditional SIP UA like Xlite. Kamailio Will thus provide not only call routing but also NATing , TLS and websocket support for webrtc endpoints.
For this bridging of SRTP from WebRTC endpoint like JSSIP to RTP for SIP UA like Xlite , we will use RTP engine .
Kamailio
Kamailio with modules like websocket , TLS , NATHelper which help it to support websocket based SIP which default kamailio configuration doesn’t.
Snippets from kamailio config to support webrtc endpoints is below . the detailed explanation is at kamailio config for webrtc
output should be like
…
Certificate is to be certified until Jun 25 11:02:41 2020 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
and files genrated shoudl look like
/home/ubuntu# ls
certs kamailio1_cert.pem kamailio1_cert_req.pem privkey.pem
copy the newly created certs to their respective paths
Websocket is considered a transport option just as TCP or UDP in kamailio config , hence just as one defines IP addr and ports for TCP, UDP protocol , we need to define the same for WS or WSS
check if port in R-URI meant for ws or wss, did not receive websocket or secure websocket
if (($Rp == MY_WS_PORT || $Rp == MY_WSS_PORT) && !(proto == WS || proto == WSS)) {
xlog("L_WARN", "SIP request received on $Rp\n");
sl_send_reply("403", "Forbidden");
exit;
}
request_route for websocket , included checking is client is behind NAT using nat_uac_test methods from NAThelper. If it is then for REGISTER methods do fix_nated_register and for other add_contact_alias
Like most other WebRTC libraries , JSSIP is event driven and provides provide core WEBRTC API like getUserMedia and RTP PeerConnection providing STUN,ICE,DTLS, SRTP features. It also integrated with rtcninja to provide cross browser accessibility. The differentiators with JSSIP lies in the fact that it supports SIP stack over websockets
instantiate JSSIP websocket interface with kamailio IP
var socket = new JsSIP.WebSocketInterface('wss://<kamailio_ip>:443');
Add configuration for registeration . Note if not using kamailio as proxy to SBC, it is recommended to add regiseteration features to provide user reachability for incoming calls and NAT pings
var configuration = { sockets : socket, uri : 'sip:username@example.com', password : 'password' };
create UA and start
var ua = new JsSIP.UA(configuration); ua.start();
SIP over WEBSOCKET messages and kamailio processing
REGISTER sip JSSIP UA with user altanai , domina voiptelcom.com
INVITE from user1 altanai to john , notice that TO header doesnt have tag . this will handy for recognizing whether it is first message of dialog of=r and indialog message such as ACK , RE-INVITE , BYE etc
100 trying from callee . Note the to and from headers remain same for request or responses . This is send automatically by kamailio for INVITE.
SIP/2.0 100 trying -- your call is important to us
Via: SIP/2.0/WSS ipoct61ao12v.invalid;branch=z9hG4bK4220209;rport=17502;received=x.x.x.x
To: sip:john@voiptelco.com
From: sip:altanai@voiptelco.com;tag=2q0lecmbsn
Call-ID: s8bnv5869fp68d1ju8c1
CSeq: 1799 INVITE
Server: kamailio (5.2.3 (x86_64/linux))
Content-Length: 0
180 ringing from Callee , note the addition of contact header
since ACK is a Within dialog message and sequential request withing a dialog should take the path determined by record-routing , we first check if it has to tag. Having a to tag validates that it is a in-dialog request .
After this validate if is loose_route() and has no destination URI $du , then try to add rui alias using handle_ruri_alias() , if that fails , reject the request .
If it is not loose_route() and method is ACK then check if the ACK matches a transaction t_check_trans() ie is stateful . If it is then relay otherwise reject.
route(WITHINDLG);
...
route[WITHINDLG] {
if (has_totag()) {
if (loose_route()) {
if ($du == "") {
if (!handle_ruri_alias()) {
xlog("L_ERR", "Bad alias <$ru>\n");
sl_send_reply("400", "Bad Request");
exit;
}
}
route(RELAY);
} else {
if ( is_method("ACK") ) {
if ( t_check_trans() ) {
t_relay();
exit;
} else {
exit;
}
}
sl_send_reply("404", "Not Found");
}
exit;
}
}
Also required to convert ICE packet fromWebRTC to non ICE for Xlite.
Unified communication services build around WebRTC should be vendor agnostic and multi-tenant and be supported by other Communication Service Providers (CSPs), SIP trunks, PBXs, Telecom Equipment Manufacturers (TEMs), and Communication Platform as a Service (CPaaS). This can happen if all endpoints adhere to SIP standards in most updated RFC. However since not all are on the boat , Session border controllers are a great way to mitigate the differences and provide seamless connectivity to signalling and media , which could be between WebRTC, SIP or PSTN, from TDM to IP .
Session Border Controllers ( SBC ) assist in controlling the signalling and usually also the media streams involved in calls and sessions.
They are often part of a VOIP network on the border where there are 2 peer networks of service providers such as backbone network and access network of corporate communication system which is behind firewall.
A more complex example is that of a large corporation where different departments have security needs for each location and perhaps for each kind of data. In this case, filtering routers or other network elements are used to control the flow of data streams. It is the job of a session border controller to assist policy administrators in managing the flow of session data across these borders. – wikipedia
SBC act like a SIP-aware firewall with proxy/B2BUA.
What is B2BUA?
A Back to back user agent ( B2BUA ) is a proxy-like server that splits a SIP transaction in two pieces:
on the side facing User Agent Client (UAC), it acts as server;
on the side facing User Agent Server (UAS) it acts as a client.
B2BUAs keep state information about active dialog. Read more here .
Remote Access
SBC mostly have public url address for teleworkers and a internal IP for enterprise/ inner LAN . This enables users connected to enterprise LAN ( who do not have public address ) to make a call to user outside of their network. During this process SBC takes care of following while relaying packets .
Security
Connectivity
Qos
Regulatory
Media Services
Statistics and billing information
Topology hiding
SBC hides and anonymize secure information like IP ports before forwarding message to outside world . This helps protect the internal node of Operators such as PSTN gateways or SIP proxies from revealing outside.
Explaining the functions of SBC in detail
1. Security
SBCs are often used by corporations along with firewalls and intrusion prevention systems (IPS) to enable VoIP calls to and from a protected enterprise network. VoIP service providers use SBCs to allow the use of VoIP protocols from private networks with Internet connections using NAT, and also to implement strong security measures that are necessary to maintain a high quality of service. The security features includes :
Prevent malicious attacks on network such as DOS, DDos.
Intrusion detection
cryptographic authentication
Identity/URL based access control
Blacklisting bad endpoints
Malformed packet protection
Encryption of signaling (via TLS and IPSec) and media (SRTP)
Stateful signalling and Validation
Toll Fraud – detect who is intending to use the telecom services without paying up
2. Connectivity
As SBC offers IP-to-IP network boundary, it recives SIP request from users like REGISTER , INVITE and routes them towards destination, making their IP. During this process it performs various operations like
NAT traversal
IPv4 to IPv6 inter-working
VPN connectivity
SIP normalization via SIP message and header manipulation
Multi vendor protocol normalization
Further Routing features includes : Least Cost Routing based on MoS ( Mean Opinion Score ) : Choosing a path based on MoS is better than chooisng any random path .
Protocol translations between SIP, SIP-I, H.323.
In essence SBC achieve interoperability, overcoming some of the problems that firewalls and network address translators (NATs) present for VoIP calls.
Automatic Rerouting
connectivity loss from UA for whole branch is detected by timeouts . But they can also be detected by audio trough SIP OPTIONS by SBC . In such connectivity loss , SBC decides rerouting or sending back 504 to caller .
4. QoS
To introduce performance optimization and business rules in call management QoS is very important . This includes the following :
Traffic policing
Resource allocation
Rate limiting
Call Admission Control (CAC)
ToS/DSCP bit setting
Recording and Audit of messages , voice calls , files
System and event logging
5. Regulatory
Govt policies ( such as ambulance , police ) and/ or enterprise policies may require some calls to be holding priority over others . This can also be configured under SBC as emergency calls and prioritization.
Some instances may require communication provider to comply with lawful bodies and provide session information or content , this is also called as Lawful interception (LI) . This enables security officials to collect specific information rather than examining all the traffic that passes through a particular router. This is also part of SBC. 6. Media services
Many of the new generation of SBCs also provide built-in digital signal processors (DSPs) to enable them to offer border-based media control and services such as- DTMF relay , Media transcoding , Tones and announcements etc.
WebRTC enabled SBC’s also provide conversion between DTLS-SRTP, to and from RTCP/RTP. Also transcoding for Opus into G7xx codecs
and ability to relay VP8/VP9 and H.264 codecs.
7. Statistics and billing information
SBC have an interface with and OSS/BSS systems for billing process , as almost all traffic that pass through the edge of the network passes via SBC. For this reason it is also used to gather Statistics and usage-based information like bandwidth, memory and CPU. PCAP traces of both signaling and media information of specific sessions .
New feature rich SBCs also have built-in digital signal processors (DSPs). Thus able to provide more control over session’s media/voice . They also add services like Relay and Interworking, Media Transcoding, Tones and Announcements, DTMF etc.
Session Border Controller for WebRTC , SIP , PSTN , IP PBX and Skype for business .
Diagram Component Description
Gateways provide compression or decompression, control signaling, call routing, and packetizing.
PSTN Gateway : Converts analog to VOIP and vice versa . Only audio no support for rich multimedia .
VOIP Gateway : A VoIP Gateway acts like a translator converting digital telecom lines to VoIP . VOIP gateway often also include voice and fax. They also have interfaces to Soft switches and network management systems.
WebRTC Gateway : They help in providing NAT with ICE-lite and STUN connectivity for peers behind policies and Firewall .
SIP trunking : Enterprises save on significant operation cost by switching to IP /SIP trunking in place of TDM (Time Division Multiplexing). Read more on SIP trunk and VPN here.
SIP Server : A Telecom application server ( SIP Server ) is useful for building VAS ( Value Added Services ) and other fine grained policies on real time services . Read more on SIP Servers here .
VOIP/SIP service Provider : There are many Worldwide SIP Service providers such as Verizon in USA , BT in europe, Swisscom in Switzerland etc .
Building a SBC
The latest trends in Telecommunications industry demand an open standardized SBC to cater to growing and large array of SIP Trunking, Unified Multimedia Communications UC&C, VoLTE, VoWi-Fi, RCS and OTT services worldwide . Building an SBC requires that it meet the following prime requirements :
software centric
Cloud Deploybale
Rich multimedia (audio , video , files etc) processing
open interfaces
The end product should be flexible to be deployed as COTS ( Commercial Off the shelf) product or as a virtual network function in the NFV cloud.
Multi Configuration , should be supported such as Hosted or Cloud deployed .
Overcome inconsistencies in SIP from different Vendors
Security and Lawful Interception
Carrier Grade Scaling
Flow Diagram
Thus we see how SBC became important part of comm systems developed over SIP and MGCP. SBC offer B2BUA ( Back to Back user agent) behavior to control both signalling and media traffic.
Setting up a ec2 instance on AWS for web real time communication platform over nodejs and socket.io using WebRTC.
Primarily a Web Call , Chat and conference platform uses WebRTC for the media stream and socketio for the signalling . Additionally used technologies are nosql for session information storage , REST Apis for getting sessions details to third parties.
Below is a comprehensive setup if ec2 t2.micro free tier instance, installation with a webrtc project module and samples of customisation and usage .
Amazon EC2 : These are elastic compute general purpose storage servers that mean that they can resize the compute capacity in the cloud based on load . 750 hours per month of Linux, RHEL, or SLES t2.micro instance usage. Expires 12 months after sign-up.
Some other products are also covered under free tier which may come in handy for setting up the complete complatorm. Here is a quick summary
Amazon S3 : it is a storage server. Can be used to store media file like image s, music , videos , recorded video etc .
Amazon RDS : It a relational database server . If one is using mysql or postgress for storing session information or user profile data . It is good option .
Amazon SES : email service. Can be used to send invites and notifications to users over mail for scheduled sessions or missed calls .
Amazon CloudFront : It is a CDN ( content delivery network ) . If one wants their libraries to be widly available without any overheads . CDN is a good choice .
Alternatively any server from Google cloud , azure free tier or digital ocean or even heroku can be used for WebRTC code deployment . Note that webrtc capture now requires htps in domain name.
Server Setup
Set up environment by installing nvm , npm and git ( source version control)
Since 2015 it has become mandatory to have only https origin request WebRTC’s getUserMedia API ie Voice, video, geolocation , screen sharing require https origins.
Note that this does not apply to case where its required to only serve peer’s media Stream or using Datachannels . Voice, video, geolocation , screen sharing now require https origins
For A POC purpose here is th way of generating a self signed certificate
Transport Layer Security and/or Secure Socket Layer( TLS/SSL) is a public/private key infrastructure.Following are the steps
1.create a private key
openssl genrsa -out webrtc-key.pem 2048
2.Create a “Certificate Signing Request” (CSR) file
create https certificate using self generate or purchased SSL certificates using fs , node-static and https modules . To know how to create self generated SSL certificates follow section above on SSL certificates.
var fs = require(‘fs’);
var _static = require(‘node-static’);
var https = require(‘https’);
var file = new _static.Server(&amp;amp;amp;amp;amp;amp;quot;./&amp;amp;amp;amp;amp;amp;quot;, {
cache: 3600,
gzip: true,
indexFile: &amp;amp;amp;amp;amp;amp;quot;index.html&amp;amp;amp;amp;amp;amp;quot;
});
the document start script that invokes the JS script
$('document').ready(function () {
sessionid = init(true);
var local = {
localVideo: "localVideo",
videoClass: "",
userDisplay: false,
userMetaDisplay: false
};
var remote = {
remotearr: ["video1", "video2"],
videoClass: "",
userDisplay: false,
userMetaDisplay: false
};
webrtcdomobj = new WebRTCdom(
local, remote
);
var session = {
sessionid: sessionid,
socketAddr: "https://localhost:8084/"
};
var webrtcdevobj = new WebRTCdev(session, null, null, null);
startcall();
});
Common known issues:
1.Opening page https://<web server ip>:< web server port>/index.html says insecure
This is beacuse the self signed certificates produced by open source openSSL is not recognized by a trusted third party Certificate Agency.
A CA ( Certificate Authority ) issues digital certificate to certify the ownership of a public key for a domain.
To solve the access issue goto https://<web server ip>:< web server port> and given access permission such as outlined in snapshot below
2.Already have given permission to Web Server , page loads but yet no activity .
if you open developer console ( ctrl+shift+I on google chrome ) you will notice that there migh be access related errros in red . If you are using different server for web server and signalling server or even if same server but different ports you need to explicity go to the signalling server url and port and give access permission for the same reason as mentione above.
3.no webcam capture on opening the page
This could happen due to many reasons
page is not loaded on https
browser is not webrtc compatible
Media permission to webcam are blocked
the machine does have any media capture devices attached
Driver issues in the client machine while accessing webcams and mics .
Bot to clean roads and outdoors for a better and cleaner India. It lifts up small objects like plastic cups,wrappers,leaves etc.
The droid also provides real-time camera stream and detects obstruction to re-route itself. It can communicates over GSM ,wifi and BLE . It can also be remote navigated via browsers or android.
Working :
Litter comes between rotating brushes
Litter is picked by brushes and pushed upwards
Brushes push it towards the tray
Is is inspired by Swach Bharat Abhiyaan in India , its an effort to contribute to society and welfare and well-being through technology . Following are some diagrams for the current and the previous versions , along with major delta points .
RamuDroid v1.0
Remote Streaming and movement via motors switched manually. Communication over Ethernet.
Dashboard /console Screen
v3.0
Cleaning garbage on public roads and outdoors through robot . Remote navigation and control of control through web page and camera live streaming .
v3 web console
v6
Clean roads , pick up litter ( wrappers, leaves , cups , plastics bits etc ) . communicated over BLE ,Wifi and 3G n/w . Auto buzzer when meet with an obstruction in way . Flash Lights . Enhanced Design .
For the last couple of weeks , I have been working on the concept of rendering 3D graphics on WebRTC media stream using different JavaScript libraries as part of a Virtual Reality project .
What is Augmented Reality ?
Augmented reality (AR) is viewing a real-world environment with elements that are supplemented by computer-generated sensory inputs such as sound, video, graphics , location etc.
How is it diff. from Virtual Reality ?
Virtual Reality – replaces the real world with simulated one , user is isolated from real life , Examples – Oculus Rift & Kinect
Augmented Reality – blending of virtual reality and real life , user interacts with real world through digital overlays , Examples – Google glass & Holo Lens
Methods for rendering augmented Reality
Computer Vision
Object Recognition
Eye Tracking
Face Detection and substitution
Emotion and gesture picker
Edge Detection
web based Augmented Reality solution
Components for a Web base end to end AR solution
Web :
WebRTC getusermedia
Web Speech API
WebGL
css
svg
HTML5 canvas
sensor API
H/w :
Graphics driver
microphone and camera
sensors
3D :
Geometry and Math Utilities
3D Model Loaders and models
Lights, Materials,Shaders, Particles,
Animation
WebRTC
Web based Real Time communications
Definition for browser’s media stream and data
Awaiting more standardization , on a API level at the W3C and at the protocol level at the IETF.
Enable browser to browser applications for voice calling, video chat and P2P file sharing without plugins.
Enables web browsers with Real-Time Communications (RTC) capabilities
MIT : Free, open project
Code snippet for WebRTC API
1.To begin with WebRTC we first need to validate that the browser has permission to access the webcam.
Display the video as a plane which can be viewed from various angles in a given background landscape. Credits for code : https://stemkoski.github.io/Three.js/
1.Use code from slide 10 to get user’s webcam input through getUserMedia
Make a Screen , camera and renderer as previously described
Give orbital CONTROLS for viewing the media plane from all angles
controls = new THREE.OrbitControls( camera, renderer.domElement );
Add point LIGHT to scene
Make the FLOOR with an image texture
var floorTexture = new THREE.ImageUtils.loadTexture( 'imageURL.jpg' );
floorTexture.wrapS = floorTexture.wrapT = THREE.RepeatWrapping;
floorTexture.repeat.set( 10, 10 );
var floorMaterial = new THREE.MeshBasicMaterial({map: floorTexture, side: THREE.DoubleSide});
var floorGeometry = new THREE.PlaneGeometry(1000, 1000, 10, 10);
var floor = new THREE.Mesh(floorGeometry, floorMaterial);
floor.position.y = -0.5;
floor.rotation.x = Math.PI / 2;
scene.add(floor);</pre>
<pre>
6. Add Fog
scene.fog = new THREE.FogExp2( 0x9999ff, 0.00025 );
7.Add video Image Context and Texture.
video = document.getElementById( 'monitor' );
videoImage = document.getElementById( 'videoImage' );
videoImageContext = videoImage.getContext( '2d' );
videoImageContext.fillStyle = '#000000';
videoImageContext.fillRect( 0, 0, videoImage.width, videoImage.height );
videoTexture = new THREE.Texture( videoImage );
videoTexture.minFilter = THREE.LinearFilter;
videoTexture.magFilter = THREE.LinearFilter;
var movieMaterial=new THREE.MeshBasicMaterial({map:videoTexture,overdraw:true,side:THREE.DoubleSide});
var movieGeometry = new THREE.PlaneGeometry( 100, 100, 1, 1 );
var movieScreen = new THREE.Mesh( movieGeometry, movieMaterial );
movieScreen.position.set(0,50,0);
scene.add(movieScreen);
WebRTC has the potential to drive the Live Streaming broadcasting area with its powerful no plugin , no installation , open standard policy . However the only roadblock is the VP8 codec which differs from the traditional H264 codec that is used by almost all the media servers , media control units , etc .
This post is first in the series of building a WebRTC based broadcasting solution. Note that a p2p session differs from a broadcasting session as Peer-to-peer session applies to bidirectional media streaming where as broadcasting only applies unidirectional media flow.
Scalable Broadcasting and Live streaming alternatives
1. WebRTC multi peers
Since WebRTC is p2p technology , it is convenient to build a network of webrtc client viewers which can pass on the stream to 3 other peers in different session. In this fashion a fission chain like structure is created where a single stream originated to first peer is replicated to 3 others which is in turn replicated to 9 peers etc .
WebRTC Scalable Streaming Server -WebRTC multi peers
Advantages :
Scalable without the investment of media servers
No additional space required at service providers network .
Disadvantage :
The entire set of end clients to a node get disconnected if a single node is broken .
Since sessions are dynamically created , it is difficult to maintain a map with fallback option in case of service disruption from any single node .
Client incur bandwidth load of 2 Mbps( stream incoming peer ) incoming and 6 Mbps ( for 3 connected peers ) outgoing data .
2. Torrent based WebRTC chain
To over come the shortcoming of previous approach of tree based broadcasting , it is suggested to use a chained broadcasting mechanism .
WebRTC Scalable Streaming Server- Single chain connection
To improvise on this mechanism for incresing efficieny for slow bandwidth connections we can stop their outgoing stream converting them to only consumers . This way the connection is mapped and arranged in such a fashion that every alternate peer is connected to 2 peers for stream replication. The slow bandwidth clients can be attached as independent endpoints .
3. WebRTC Relay nodes for multiple peers
The aim here is to build a career grade WebRTC stream broadcasting platform , which is capable of using the WebRTC’s mediastream and peerconnection API , along with repeaters to make a scalable broadcasting / live streaming solution using socketio for behavior control and signalling .
Algorithm :
At the Publisher’s end
1. GetUserMedia
2. Start Room “liveConf”
3. Add outgoing stream to session “liveConf “ with peer “BR” in 1 way transport .
1 outgoing audio stream -> 1 MB in 1 RTP port
1 outgoing video -> 1 MB 1 more RTP port
Total Required 2 MB and 2 RTP ports
At the Repeater layer (high upload and download bandwidth )
4. Peer “BR” opens parallel room “liveConf_1” , “liveConf_2” with 4 other peers “Repeater1 “, “Repeater2” , so on
5. Repetare1 getRemoteStream from “liveConf_1” and add as localStream to “liveConf_1_1”
Here the upload bandwidth is high and each repeater is capable of handling 6 outgoing streams . Therefore total 4 repeaters can handle upto 24 streams very easily
At the Viewer’s end
6. Viewer Joins room ”liveConf_1_1”
7. Play the incoming stream on WebRTC browser video element”
WebRTC Relay nodes for multiple peers
Advantages:
As 6 viewers can connect to 1 repeater for feed , total of 24 viewers will require only 4 repeaters.
Only 2 MB consumption at publishers end and 2MB at each viewer’s end.
4. WebRTC recorder to Broadcasting Media Server VOD
This process is essentially NOT a live streaming solution but a Video On Demand type of implementation for a recorded webRTC stream .
Figure shows a WebRTC node which can record the webrtc files as webm . Audio and video can be together recorded on fireox . With chrome one needs to merge a separately recorded webm ( video) and wav ( audio ) file to make a single webm file containing both audio and video and them store in VOD server’s repo.
WebRTC Chunk recorder to Broadcasting Media Server VOD
Although inherently Media Server do not support webm format but few new age lightweight media servers such as Kurento are capable of this .
Advantages :
Can solve the end goal of broadcasting from a webrtc browser to multiple webrtc browsers without incurring extra load on any client machine ( Obviously assuming that Media Server handles the distribution of video and load sharing automatically )
Disadvantages:
It is not livestreaming
For significantly longer recorded stream the delta in delay of streaming increases considerably . Ideally this delta should be no more than 5 minutes .
TFX is a modular widget based WebRTC communication and collaboration solution. It is a customizable solution where developers can create and add their own widget over the underlying WebRTC communication mechanism . It can support extensive set of user activity such as video chat , message , play games , collaborate on code , draw something together etc . It can go as wide as your imagination . This post describes the process of creating widgets to host over existing TFX platform .
Prerequisites
It is required to have TFX Chrome extension installed and running from Chrome App Store under above . To do this follow the steps described in TangoFX v0.1 User’s manual.
Test TFX Sessions ?
TFX Sessions uses the browser’s media API’s , like getUserMedia and Peerconnection to establish p2p media connection . Before media can traverse between 2 end points the signalling server is required to establish the path using Offer- Answer Model . This can be tested by making unit test cases on these function calls .
TFX Sessions uses socketio based handshake between peers to ascertain that they are valid endpoints to enter in a communication session . This is determined by SDP ( Session Description Parameters ) . The same can be observed in chrome://webrtc-internals/ traces and graphs .
How to make widgets using TFX API ?
Step 1: To make widgets for TFX , just write your simple web program which should consist of one main html webpage and associated css and js files for it .
Step 2 : Find an interesting idea which is requires minimal js and css . Remember it is a widget and not a full fleshed web project , however js frameworks like requirejs , angularjs , emberjs etc , work as well.
Step 3: Make a compact folder with the name of widget and put the respective files in it. For example the html files or view files would go to src folder , javascript files would goto js folder , css files would goto css folder , pictures to picture folder , audio files to sound folder and so on .
Step 4 : Once the widget is performing well in standalone environment , we can add a sync file to communicate the peer behaviors across TFX network . For this we primarily use 2 methods .
SendMessage : To send the data that will be traversed over DataChannel API of TFX . The content is in json format and will be shared with the peers in the session .
OnMessage : To receive the message communicated by the TFX API over network
Step 5: Submit the application to us or test it yourself by adding the plugin description in in widgetmanifest.json file . Few added widgets are
Step 6 : For proper orientation of the application make sure that overflow is hidden and padding to left is atleast 60 px so that it doesnt overlap with panel padding-left: 60px; overflow: hidden;
Step 7 : Voila the widget is ready to go .
Simple Messaging Widget
For demonstration purpose I have summarised the exact steps followed to create the simple messaging widget which uses WebRTC ‘s Datachannel API in the back and TFX SendMessage & OnMessage API to achieve
Step 1 : Think of a general chat scenario as present in various messaging si
Step 2: Made a folder structure with separation for js , css and src. Add the respective files in folder. It would look like following figure:
//send message when mouse is on mesage dicv ans enter is hit
$("#messages").keyup(function(event){
if(event.keyCode == 13){
var msg=$('#MessageBox').val();
//send to peer
var data ={
"msgcontent":msg
}
sendMessage(data);
addMessageLog(msg);
$("#MessageBox").val('');
}
});
function addMessageLog(msg){
//add text to text area for message log for self
$('#MessageHistoryBox').text( $('#MessageHistoryBox').text() + '\n'+ 'you : '+ msg);
}
// handles send message
function sendMessage(message) {
var widgetdata={
"type":"plugin",
"plugintype":"relaymsg",
"action":"update",
"content":message
};
// postmessage
window.parent.postMessage(widgetdata,'*');
}
//to handle incoming message
function onmessage(evt) {
//add text to text area for message log from peer
if(evt.data.msgcontent!=null ){
$('#MessageHistoryBox').text( $('#MessageHistoryBox').text() +'\n'+ 'other : '+ evt.data.msgcontent );
}
}
window.addEventListener("message",onmessage,false);
Step 6: The end result is :
Developing a cross origin Widget ( XHR)
Let us demonstrate the process and important points to create a cross- origin widget :
step 1 : Develop a separate web project and run it on a https
step 2 : Add the widget frame in TFX . Following is the code I added to make an XHR request over GET
var xmlhttp;
xmlhttp=new XMLHttpRequest();
xmlhttp.onreadystatechange=function()
{
if (xmlhttp.readyState==4 &amp;&amp; xmlhttp.status==200)
{
document.getElementById("myDiv").innerHTML=xmlhttp.responseText;
}
}
xmlhttp.open("GET","https://192.168.0.119:8000/TFXCrossSiteProj/files/document1.txt",true);
xmlhttp.send();
step 3 : Using self made https we have have to open the url separately in browser and give it explicit permission to open in advanced setting. Make sure the original file is visible to you at the widgets url .
step 4: Adding permission to manifest for access the cross origin requests
Step 5 : Rest of the process are similar to develop a regular widget ie css and js .
Step 6: Resulting widget on TFX
Note 1 : In absence of changes to manifest file the cross origin request is meet with a Access-Control-Allow-Origin error .
Note 2: While using POST the TFX responds with Failed to load resource: the server responded with a status of 404 (Not Found)
Note 3: Also if instead of https http is used the TFX still responds with Failed to load resource: the server responded with a status of 404 (Not Found)
STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) are protocols that can be used to provide NAT traversal for VoIP and WebRTC. These projects provide a VoIP media traffic NAT traversal server and gateway.
TURN Server is a VoIP media traffic NAT traversal server and gateway.
This article describes working with some of the TURN servers
rfc5766-turn-server ( legacy )
COTURN
Xirsys TURN service
Twillio’s TURN service
Asterisk ICE resolution
Kamailio NAT and STUN modules in Server
We know that a STUN only server such as the one below , may not work under firewalls like in enterperises or even in universities resulting in black video , one way video , inconcsistent streaming or even no video experience .
// google STUN
var iceservers_array=[{urls: 'stun:stun.l.google.com:19302'}] ;
To overcome this we rely on a publicaly avaiable TURN server which can step in and do ICE exchange to seup relay routes for the media stream. Some of the options for both self hosted and TURN as a service are described below .
rfc5766-turn-server
It is a legacy project mainly archives for reference( links at the end of section) . This is a VoIP gateway for inter network communication which is opensource MIT licensed .
platforms supported : Any client platform is supported, including Android, iOS, Linux, OS X, Windows, and Windows Phone. This project can be successfully used on other *NIX platforms ( Aamazon EC2) too. It supports flat file or Database based user management system ( MySQL , postgress , redis ). The source code project contains , TURN server , TURN client messaging library and some sample scripts to test various modules like protocol , relay , security etc .
Protocols : Protocols between the TURN client and the TURN server – UDP, TCP, TLS, and DTLS. Relay protocol – UDP , TCP .
Authentication : The authentication mechanism is using key which is calculated over the user name, the realm, and the user password. Key for the HMAC depends on whether long-term or short-term credentials are in use. For long-term credentials, the key is 16 bytes: key = MD5(username “:” realm “:” SASLprep(password))
Installation : Since I used my Ubuntu Software center for installing the RFC turn server 5677 .
The content got stored inside /usr/share/rfc5766-turn-server. Also install mysql for record keeping
sudo apt-get install mysql-server
Intall MySQL workbench to monitor the values feed into the turn database server in MysqL. connect to MySQL instance using the following screenshot
The database formed with mysql after successful operation is as follows . We shall notice that the initial db is absolutely null
Terminal Commands
These terminal command ( binary images ) get stored inside etc/init.d after installing
turnadmin – Its turn relay administration tool used for generating , updating keys and passwords . For generating a key to get long term crdentaial use -k command and for aading or updateing a long -term user use the -a command. Therefore a simple command to generate a key is
or we can also check using the terminal based mySQL client
mysql> use turn;
Database changed
mysql> select * from turnusers_lt;
+------------+----------------------------------+
| name | hmackey |
+------------+----------------------------------+
| altanai | 57bdc681481c4f7626bffcde292c85e7 |
| turnwebrtc | 6066cbe0b5ee14439b2ddfc177268309 |
+------------+----------------------------------+
2 rows in set (0.00 sec)
turnserver – Its command to handle the turnserver itself . We can use the simple turnserver command to start it without any db support using just turnserver. Screenshot for this is
We can use a database like mysql to start it with db connection string
turnutils_uclient: emulates multiple UDP,TCP,TLS or DTLS clients.
turnutils_peer: simple stateless UDP-only “echo” server. For every incoming UDP packet, it simply echoes it back.
turnutils_stunclient: simple STUN client example that implements RFC 5389 ( using STUN as endpoint to determine the IP address and port allocated to it , keep-alive , check connectivity etc) and RFC 5780 (experimental NAT Behavior Discovery STUN usage) .
turnutils_rfc5769check: checks the correctness of the STUN/TURN protocol implementation. This program will perform several checks and print the result on the screen. It will exit with 0 status if everything is OK, and with (-1) if there was an error in the protocol implementation.
Test
1. Test vectors from RFC 5769 to double-check that our STUN/TURN message encoding algorithms work properly. Run the utility to check all protocols :
$ cd examples
$ ./scripts/rfc5769.sh
2. TURN functionality test (bare minimum TURN example).
If everything compiled properly, then the following programs must run together successfully, simulating TURN network routing in local loopback networking environment:
console 1 :
$ ./scripts/basic/relay.sh
console2 :
$ ./scripts/peer.sh
If the client application produces output and in approximately 22 seconds prints the jitter, loss and round-trip-delay statistics, then everything is fine.
Insert the above piece of code on peer connection config . Now call from one network environment to another . For example call from a enterprise network behind a Wifi router to a public internet datacard webrtc agent . The call should connect with video flowing smoothly between the two .
TURN working accross Enterprise firrewall on a WebRTC video platform written with SimpeWebRTC lbrary . Project tango FX
Project Coturn evolved from rfc5766-turn-server project with many new advanced TURN specs beyond the original RFC 5766 document. The databses supported are : SQLite , MySQL , PostgreSQL , Redis , MongoDB
Protocols :
The implementation fully supports the following client-to-TURN-server protocols: UDP , TCP , TLS SSL3/TLS1.0/TLS1.1/TLS1.2; ECDHE , DTLS versions 1.0 and 1.2. Supported relay protocols UDP (per RFC 5766) and TCP (per RFC 6062)
Authetication :
Supported message integrity digest algorithms:
HMAC-SHA1, with MD5-hashed keys (as required by STUN and TURN standards)
HMAC-SHA256, with SHA256-hashed keys (an extension to the STUN and TURN specs)
Install libopenssl and libevent plus its dev or extra libraries . OpenSSL has to be installed before libevent2 for TLS beacuse When libevent builds it checks whether OpenSSL has been already installed, and its version.
RFC 5766 -Traversal Using Relays around NAT (TURN):Relay Extensions to Session Traversal Utilities for NAT (STUN)
RFC 6062 -Traversal Using Relays around NAT (TURN) Extensions for TCP Allocations
RFC 6156 – IPv6 extension for TURN
RFC 7443 – ALPN support for STUN & TURN
Datagram Transport Layer Security (DTLS) as Transport for Traversal Using Relays around NAT (TURN) . It facilitate the resolution of TURN URIs into the IP address and port of TURN servers supporting DTLS as a transport protocol.
Mobile ICE (MICE) – Mobility with TURN . It minimizes traffic disruption caused by changing IP address during a mobility event by shorter network path .
Chorme WebRTC Internals Page for COTURN connection
Xirsys
Xirsys is a provider for WebRTC infrastructure which included stun and turn server hosting as well .
The process of using their services includes singing up for a account and choosing whether you want a paid service capable of handling more calls simultaneously or free one handling only upto 10 concurrent turn connections .
The dashboard appears like this :
To receive the api one need to make a one time call to their service , the result of which contains the keys to invoke the turn services from webrtc script .
window.addEventListener("load", function (e) {
let xhr = new XMLHttpRequest();
xhr.onreadystatechange = function ($evt) {
if (xhr.readyState == 4 && xhr.status == 200) {
let res = JSON.parse(xhr.responseText);
console.log("response: ", res);
iceservers_array.push(res.v.iceServers);
alert( iceservers_array);
}
};
xhr.open("PUT", "https://global.xirsys.net/_turn/webrtc", true);
xhr.setRequestHeader("Authorization","Basic " + btoa("altanai:<sec rettoken>"));
xhr.setRequestHeader("Content-Type","application/json");
xhr.send(JSON.stringify({"format": "urls"}));
});
The resulting output should look like ( my keys are hidden with a red rectangle ofcourse )
The process of adding a TURN / STUN to your webrtc script in JS is as follows :
iceServers:[
{"url":"stun:turn2.xirsys.com"},
{"username":"< put your API username>","url":"turn:turn2.xirsys.com:443?transport=udp","credential":"< put your API credentail>"},
{"username":"< put your API username>","url":"turn:turn2.xirsys.com:443?transport=tcp","credential":"< put your API credentail>"}]
// before unload update status on main site
window.addEventListener("load", function (e) {
let xhr = new XMLHttpRequest();
xhr.onreadystatechange = function ($evt) {
if (xhr.readyState == 4 && xhr.status == 200) {
let res = JSON.parse(xhr.responseText);
console.log("response: ", res);
iceservers_array = res.iceServers;
console.log("iceservers_array: ", iceservers_array);
CallSessionBegins();
}
};
xhr.open("POST", "https://<ourdomain>:3000/token", true);
xhr.setRequestHeader("Content-Type","application/json");
xhr.send(JSON.stringify({"format": "urls"}));
});
Asterisk TURN module and service
Asterisk is an open source carrer grade SIP server which also provides Firewall traversal . A github repo containing some asterisk dialplan examples is https://github.com/altanai/asteriskexamples. An article discussing Asterisk and its implementation
Asterisk is a framework or toolkit designed for VOIP systems . It can support Enterprise communication systems like PBXs, call distributors, VoIP gateways , conference bridges etc . It is open source and free to use . It is developed in C and runs in linux . Technically , Asterisk has protocol support for many … Continue reading Asterisk – installation and dial plans for WebRTC supported PJSIP clients
In this article we discuss Nating in a SIP Server like kamailio . Types of NAT pings , its behaviour and types .Also some implementation of some of the Kamailios modules like NAT Traversal module NAT helper module STUN module A repository for extensive application of kamailio modules for various usecases is https://github.com/altanai/kamailioexamples DNS sub-system … Continue reading Kamailio DNS and NAT
We know that WebRTC is web based real-time communications on browser-based platform using the browser’s media application programming interface (API) and adding our JavaScript & HTML5 t control the media flow .
WebRTC has enabled developers to build apps/ sites / widgets / plugins capable of delivering simultaneous voice/video/data/screen-sharing capability in a peer to peer fashion.
But something which escapes our attention is the way in which media ia traversing across the network. Ofcourse the webrtc call runs very smoothly when both the peers are on open public internet without any restrictions or firewall blocks . But the real problem begins when one of the peer is behind a Corporate/Enterprise network or using a different Internet service provider with some security restrictions . In such a case the normal ICE capability of WebRTC is not enough , what is required is a NAT traversal mechanism .
STUN and TURN server protocols handle session initiations with handshakes between peers in different network environments . In case of a firewall blocking a STUN peer-to-peer connection, the system fallback to a TURN server which provides the necessary traversing mechanism through the NAT.
Lets study from the start ie ICE . What is it and why is it used ?
ICE (Interactive Connectivity Establishment ) framework ( mandatory by WebRTC standards ) find network interfaces and ports in Offer / Answer Model to exchange network based information with participating communication clients. ICE makes use of the Session Traversal Utilities for NAT (STUN) protocol and its extension, Traversal Using Relay NAT (TURN)
ICE is defined by RFC 5245 – Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols.
WebRTC needs SDP Offer to be send to the clientB Javascript code from clientA Javascript code . Client B uses this SDP offer to generate an SDP Answer for client A. The SDP ( as seen on chrome://webrtc-internals/ ) includes ICE candidates which punchs open ports in the firewalls.
However incase both sides are symmetric NATs the media flow gets blocked. For such a case TURN is used which tries to give a public ip and port mapped to internal ip and port so as to provide an alternative routing mechanism like a packet-mirror. It can open a DTLS connection and use it to key the SRTP-DTLS media streams, and to send DataChannels over DTLS.
In order to Understand this better consider various scenarios
1 . No Firewall present on either peer . Both connected to open public internet .
Diagrammatic representation of this shown as follows :
WebRTC signalling and media flow on Open public network
In this case there is no restriction to signal or media flow and the call takes places smoothly in p2p fashion.
2. Either one or both the peer ( could be many in case of multi conf call ) are present behind a firewall or restrictive connection or router configured for intranet .
In such a case the signal may pass with the use of default ICE candidates or simple ppensource google Stun server such as
However the media is restricted resulting in a black / empty / no video situation for both peers . To combat such situation a relay mechanism such as TURN is required which essentially maps public ip to private ips thus creating a alternative route for media and data to flow through .
WebRTC media flow when peers are behind NAT . Uses TURN relay mechanism
3. When the TURN server is also behind a firewall . The config file of the turn server need to be altered to map the public and private IP
The diagrammatic description of this is as follows :
WebRTC media flow when peers are behind NAT and TURN server is behind NAT as well . TURN config files bind a public interface to private interface address .
This blog is in continuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC )
Attempt 4: Stream the content to a WebRTC endpoint which is hidden in a video call . Pick the stream from vp8 object URL send to a streaming server
This process involved the following components :
WebRTC API : simplewebrtc on Chrome
Transfer mechanism from client to Streaming server: webrtc media channel
Problems : No streaming server is qualified to handle a direct webrtc input and stream it on network .
Attempt 4.1 : Stream the content to a WebRTC endpoint . Do WebRTC Endpoint to RTP Endpoint bridge using Kurento APIs.
Use the RTP port and ip address to input into a ffmpeg or gstreamer or VLC terminal command and out put a live H264 stream on another ip and port address .
This process involved the following components :
API : Kurento
Transfer mechanism : HTML5 webrtc client -> application server hosting java -> media server -> application for webrtc media to RTP media conversation -> RTP player
Screenshots of attempts with Wowza to stream from a ip and port
problems :
The stream was black ie no video content .
Attempt 4.2 : Build a WebRTC Endpoint to Http endpoint in kurento and force the video audio encoding to be that of H264 and PCMU.
code for adding constraints to output media and forcing choice of codecs
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
HttpGetEndpoint httpEndpoint=new HttpGetEndpoint.Builder(pipeline).build();
org.kurento.client.Fraction fr= new org.kurento.client.Fraction(1, 30);
VideoCaps vc= new VideoCaps(VideoCodec.H264,fr);
httpEndpoint.setVideoFormat(vc);
AudioCaps ac= new AudioCaps(AudioCodec.PCMU, 65536);
httpEndpoint.setAudioFormat(ac);
webRtcEndpoint.connect(httpEndpoint);
code for using gstreamer filter to force the output in raw format . It is a alternate solution to above
//basic media operation of 1 pipeline and 2 endpoinst
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
//adding Gstream filters
GStreamerFilter filter1 = new GStreamerFilter.Builder(pipeline, "videorate max-rate=30").withFilterType(FilterType.VIDEO).build();
GStreamerFilter filter2 = new GStreamerFilter.Builder(pipeline, "capsfilter caps=video/x-h264,width=1280,height=720,framerate=30/1").withFilterType(FilterType.VIDEO).build();
GStreamerFilter filter3 = new GStreamerFilter.Builder(pipeline, "capsfilter caps=audio/x-mpeg,layer=3,rate=48000").withFilterType(FilterType.AUDIO).build();
//connecting all poin ts to one another
webRtcEndpoint.connect (filter1);
filter1.connect (filter2);
filter2.connect (filter3);
filter3.connect (rtpEndpoint);
// RTP SDP offer and answer
String requestRTPsdp = rtpEndpoint.generateOffer();
rtpEndpoint.processAnswer(requestRTPsdp);
problem : The output is still webm
Attempt 5 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Wowza streaming server
This process involved the following components
WebRTC Stream and object URL of the blob containing VP8 media
Kurento WebRTC Endpoint bridge to generate SDP
Wowza Streaming server
code for kurento to generate a SDP file from WebRTC to RTP bridge
@RequestMapping(value = "/rtpsdp", method = RequestMethod.POST)
private String processRequestrtpsdp(@RequestBody String sdpOffer)
throws IOException, URISyntaxException, InterruptedException {
//basic media operation of 1 pipeline and 2 endpoinst
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
//connecting all poin ts to one another
webRtcEndpoint.connect (rtpEndpoint);
// RTP SDP offer and answer
String requestRTPsdp = rtpEndpoint.generateOffer();
rtpEndpoint.processAnswer(requestRTPsdp);
// write the SDP conector to an external file
PrintWriter out = new PrintWriter("/tmp/test.sdp");
out.println(requestRTPsdp);
out.close();
HttpGetEndpoint httpEndpoint = new HttpGetEndpoint.Builder(pipeline).build();
PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline, requestRTPsdp).build();
httpEndpoint.connect(rtpEndpoint);
player.connect(httpEndpoint);
// Playing media and opening the default desktop browser
player.play();
String videoUrl = httpEndpoint.getUrl();
System.out.println(" ------- video URL -------------"+ videoUrl);
// send the response to front client
String responseSdp = webRtcEndpoint.processOffer(sdpOffer);
return responseSdp;
}
problems : wowza doesnt not recognize the WebRTC SDP and play the video
screenshot of wowza with SDP input
Attempt 5.1 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Default Ubuntu media player
Attempt 5.2 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over VLC using socket input
problem : nothing plays
screenshot of VLC connected to play from socket and failure to play anything
Attempt 5.3: Create a WebRTC endpoint and connected it to RTP endpoint via media pipelines . Also make the RTP SDP offer and answering the same . Play with ffnpeg / ffplay / gst playbin
Could not find codec parameter for stream1 ( video:h263, none ) .Other errors types are , Could not write header for output file output file is empty nothing was encoded
Error screenshots of trying to play the RTP SDP file with ffmpeg
Attempt 6 : Use a WebRTC capable media and streaming server ( eg Kurento ) to pick a live stream of VP8 .
Convert the VP8 to H264 ( ffmpeg / RTP endpoint ) .
Convert H264 to Mp4 using MP4 parser and pass to a streaming server ( wowza)
As the title of this article suggests I am going to pen my attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc .
I am currently attempting to do this by making my own MP4 engine from WebRTC feed . However I am sharing my past experiments in hope of helping someone whose objective is not the same as mine and might get some help from these threads .
It uses API fromwebrtc-experiment.com. The broadcast is in one direction only where the viewrs are never asked for their mic / webcam permission .
problem : The broadcast is for WebRTC browsers only and doesnt support non webrtc players / browsers
Attempt 1.1: Stream the media directly to nodejs through websocket
window.addEventListener('DOMContentLoaded', function() {
var v = document.getElementById('v');
navigator.getUserMedia = (navigator.getUserMedia ||
navigator.webkitGetUserMedia ||
navigator.mozGetUserMedia ||
navigator.msGetUserMedia);
if (navigator.getUserMedia) {
// Request access to video only
navigator.getUserMedia(
{
video:true,
audio:false
},
function(stream) {
var url = window.URL || window.webkitURL;
v.src = url ? url.createObjectURL(stream) : stream;
v.play();
var ws = new WebSocket('ws://localhost:3000', 'echo-protocol');
waitForSocketConnection(ws, function(){
console.log(" url.createObjectURL(stream)-----", url.createObjectURL(stream))
ws.send(stream);
console.log("message sent!!!");
});
},
function(error) {
alert('Something went wrong. (error code ' + error.code + ')');
return;
}
);
}
else {
alert('Sorry, the browser you are using doesn\'t support getUserMedia');
return;
}
});
//Make the function wait until the connection is made...
function waitForSocketConnection(socket, callback){
setTimeout(
function () {
if (socket.readyState === 1) {
console.log("Connection is made")
if(callback != null){
callback();
}
return;
} else {
console.log("wait for connection...")
waitForSocketConnection(socket, callback);
}
}, 5); // wait 5 milisecond for the connection...
}
problem : The video is in form of buffer and doesnot play
Attempt 2: Record the WebRTC media ( 5 secs each ) into chunks of webm format-> transfer them to other end -> append the chunks together like a regular file
This process involved the following components :
Recorder Javascript library : RecordJs
Transfer mechanism : Record using RecordRTC.js -> send to other end for media server -> stitching together the small webm files into big one at runtime and play
Programs :
Code for video recorder
navigator.getUserMedia(videoConstraints, function(stream) {
video.onloadedmetadata = function() {
video.width = 320;
video.height = 240;
var options = {
type: isRecordVideo ? 'video' : 'gif',
video: video,
canvas: {
width: canvasWidth_input.value,
height: canvasHeight_input.value
}
};
recorder = window.RecordRTC(stream, options);
recorder.startRecording();
};
video.src = URL.createObjectURL(stream);
}, function() {
if (document.getElementById('record-screen').checked) {
if (location.protocol === 'http:')
alert('&amp;amp;amp;amp;lt;https&amp;amp;amp;amp;gt; is mandatory to capture screen.');
else
alert('Multi-capturing of screen is not allowed. Capturing process is denied. Are you enabled flag: "Enable screen capture support in getUserMedia"?');
} else
alert('Webcam access is denied.');
});
Code for video append-er
var FILE1 = '1.webm';
var FILE2 = '2.webm';
var FILE3 = '3.webm';
var FILE4 = '4.webm';
var FILE5 = '5.webm';
var NUM_CHUNKS = 5;
var video = document.querySelector('video');
window.MediaSource = window.MediaSource || window.WebKitMediaSource;
if (!!!window.MediaSource) {
alert('MediaSource API is not available');
}
var mediaSource = new MediaSource();
video.src = window.URL.createObjectURL(mediaSource);
function callback(e) {
var sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="vorbis,vp8"');
GET(FILE1, function(uInt8Array) {
var file = new Blob([uInt8Array], {type: 'video/webm'});
var i = 1;
(function readChunk_(i) {
var reader = new FileReader();
reader.onload = function(e) {
sourceBuffer.appendBuffer(new Uint8Array(e.target.result));
if (i == NUM_CHUNKS) mediaSource.endOfStream();
else {
if (video.paused) {
video.play(); // Start playing after 1st chunk is appended.
}
readChunk_(++i);
}
};
reader.readAsArrayBuffer(file);
})(i); // Start the recursive call by self calling.
});
}
mediaSource.addEventListener('sourceopen', callback, false);
mediaSource.addEventListener('webkitsourceopen', callback, false);
mediaSource.addEventListener('webkitsourceended', function(e) {
logger.log('mediaSource readyState: ' + this.readyState);
}, false);
// function get the video via XHR
function GET(url, callback) {
var xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType = 'arraybuffer';
xhr.send();
xhr.onload = function(e) {
if (xhr.status != 200) {
alert("Unexpected status code " + xhr.status + " for " + url);
return false;
}
callback(new Uint8Array(xhr.response));
};
}
Shortcoming of this approach
The webm files failed to play on most of the media players
The recorder can only either record video or audio file at a time .
Attempt 2.1: Record the WebRTC media ( 5 secs each ) into chunks of webm format ( RecordRTC.js) > Use Kurento JS script ( kws-media-api,js) to make a HTTP Endpoint to recorded Webm files -> append the chunks together like a regular file at runtime
function getByID(id) {
return document.getElementById(id);
}
var recordAudio = getByID('record-audio'),
recordVideo = getByID('record-video'),
stopRecordingAudio = getByID('stop-recording-audio'),
stopRecordingVideo = getByID('stop-recording-video'),
broadcasting=getByID('broadcasting');
var canvasWidth_input = getByID('canvas-width-input'),
canvasHeight_input = getByID('canvas-height-input');
var video = getByID('video');
var audio = getByID('audio');
var videoConstraints = {
audio: false,
video: {
mandatory: {},
optional: []
}
};
var audioConstraints = {
audio: true,
video: false
};
const ws_uri = 'ws://localhost:8888/kurento';
var URL_SMALL="http://localhost:8080/streamtomp4/approach1/5561840332.webm";
var audioStream;
var recorder;
recordAudio.onclick = function() {
if (!audioStream)
navigator.getUserMedia(audioConstraints, function(stream) {
if (window.IsChrome) stream = new window.MediaStream(stream.getAudioTracks());
audioStream = stream;
audio.src = URL.createObjectURL(audioStream);
audio.muted = true;
audio.play();
// "audio" is a default type
recorder = window.RecordRTC(stream, {
type: 'audio'
});
recorder.startRecording();
}, function() {});
else {
audio.src = URL.createObjectURL(audioStream);
audio.muted = true;
audio.play();
if (recorder) recorder.startRecording();
}
window.isAudio = true;
this.disabled = true;
stopRecordingAudio.disabled = false;
};
stopRecordingAudio.onclick = function() {
this.disabled = true;
recordAudio.disabled = false;
audio.src = '';
if (recorder)
recorder.stopRecording(function(url) {
audio.src = url;
audio.muted = false;
audio.play();
document.getElementById('audio-url-preview').innerHTML = '&amp;amp;amp;amp;lt;a href="' + url + '" target="_blank"&amp;amp;amp;amp;gt;Recorded Audio URL&amp;amp;amp;amp;lt;/a&amp;amp;amp;amp;gt;';
});
};
recordVideo.onclick = function() {
recordVideoOrGIF(true);
};
function recordVideoOrGIF(isRecordVideo) {
navigator.getUserMedia(videoConstraints, function(stream) {
video.onloadedmetadata = function() {
video.width = 320;
video.height = 240;
var options = {
type: isRecordVideo ? 'video' : 'gif',
video: video,
canvas: {
width: canvasWidth_input.value,
height: canvasHeight_input.value
}
};
recorder = window.RecordRTC(stream, options);
recorder.startRecording();
};
video.src = URL.createObjectURL(stream);
}, function() {
if (document.getElementById('record-screen').checked) {
if (location.protocol === 'http:')
alert('&amp;amp;amp;amp;lt;https&amp;amp;amp;amp;gt; is mandatory to capture screen.');
else
alert('Multi-capturing of screen is not allowed. Capturing process is denied. Are you enabled flag: "Enable screen capture support in getUserMedia"?');
} else
alert('Webcam access is denied.');
});
window.isAudio = false;
if (isRecordVideo) {
recordVideo.disabled = true;
stopRecordingVideo.disabled = false;
} else {
recordGIF.disabled = true;
stopRecordingGIF.disabled = false;
}
}
stopRecordingVideo.onclick = function() {
this.disabled = true;
recordVideo.disabled = false;
if (recorder)
recorder.stopRecording(function(url) {
video.src = url;
video.play();
document.getElementById('video-url-preview').innerHTML = '&amp;amp;amp;amp;lt;a href="' + url + '" target="_blank"&amp;amp;amp;amp;gt;Recorded Video URL&amp;amp;amp;amp;lt;/a&amp;amp;amp;amp;gt;';
});
};
/*----broadcasting -------------*/
function onerror(error)
{
console.log( " error occured");
console.error(error);
};
broadcast.onclick = function() {
var videoOutput = document.getElementById("videoOutput");
KwsMedia(ws_uri, function(error, kwsMedia)
{
if(error) return onerror(error);
// Create pipeline
kwsMedia.create('MediaPipeline', function(error, pipeline)
{
if(error) return onerror(error);
// Create pipeline media elements (endpoints &amp;amp;amp;amp;amp; filters)
pipeline.create('PlayerEndpoint', {uri: URL_SMALL},
function(error, player)
{
if(error) return console.error(error);
pipeline.create('HttpGetEndpoint', function(error, httpGet)
{
if(error) return onerror(error);
// Connect media element between them
player.connect(httpGet, function(error, pipeline)
{
if(error) return onerror(error);
// Set the video on the video tag
httpGet.getUrl(function(error, url)
{
if(error) return onerror(error);
videoOutput.src = url;
console.log(url);
// Start player
player.play(function(error)
{
if(error) return onerror(error);
console.log('player.play');
});
});
});
// Subscribe to HttpGetEndpoint EOS event
httpGet.on('EndOfStream', function(event)
{
console.log("EndOfStream event:", event);
});
});
});
});
},
onerror);
}
problem : dissecting the live video into small the files and appending to each other on reception is an expensive , time and resource consuming process . Also involves heavy buffering and other problems pertaining to real-time streaming .
Attempt 2.2 : Send the recorded chunks of webm to a port on linux server . Use socket programming to pick up these individual files and play using VLC player from UDP port of the Linux Server
Attempt 2.3: Send the recorded chunks of webm to a port on linux server socket . Use socket programming to pick up these individual webm files and convert to H264 format so that they can be send to a media server.
This process involved the following components :
Recorder Javascript library : RecordJs
Transfer mechanism :WebRTC endpoint -> Call handler ( Record in chunks ) -> ffmpeg / gstreamer to put it on RTP -> streaming server like wowza – > viewers
Programs : Use HTML webpage Webscoket connection -> nodejs program to write content from websocket to linux socket -> nodejs program to read that socket and print the content on console
Program to transfer the webm recorder files over websocket to nodejs program
//Make the function wait until the connection is made...
function waitForSocketConnection(socket, callback){
setTimeout(
function () {
if (socket.readyState === 1) {
console.log("Connection is made")
if(callback != null){
callback();
}
return;
} else {
console.log("wait for connection...")
waitForSocketConnection(socket, callback);
}
}, 5); // wait 5 milisecond for the connection...
}
function previewFile() {
var preview = document.querySelector('img');
var file = document.querySelector('input[type=file]').files[0];
var reader = new FileReader();
reader.onloadend = function () {
preview.src = reader.result;
console.log(" reader result ", reader.result);
var video=document.getElementById("v");
video.src=reader.result;
console.log(" video played ");
var ws = new WebSocket('ws://localhost:3000', 'echo-protocol');
waitForSocketConnection(ws, function(){
ws.send(reader.result);
console.log("message sent!!!");
});
}
if (file) {
// converts to base64 encoded string of the file data
//reader.readAsDataURL(file);
reader.readAsBinaryString(file);
} else {
preview.src = "";
}
}
Program for Linux Sockets sender which creates the socket for the webm files
var net = require('net');
var fs = require('fs');
var socketPath = '/tmp/tfxsocket';
var http = require('http');
var stream = require('stream');
var util = require('util');
var WebSocketServer = require('ws').Server;
var port = 3000;
var serverUrl = "localhost";
var socket;
/*----------http server -----------*/
var server= http.createServer(function (request, response) {
});
server.listen(port, serverUrl);
console.log('HTTP Server running at ',serverUrl,port);
/*------websocket server ----------*/
var wss = new WebSocketServer({server: server});
wss.on("connection", function(ws) {
console.log("websocket connection open");
ws.on('message', function (message) {
console.log(" stream recived from broadcast client on port 3000 ");
var s = require('net').Socket();
s.connect(socketPath);
s.write(message);
console.log(" send the stream to socketPath",socketPath);
});
ws.on("close", function() {
console.log("websocket connection close")
});
});
Program for Linux Socket Listener using nodejs and socket . Here the socket is in node /tmp/mysocket
var net = require('net');
var client = net.createConnection("/tmp/mysocket");
client.on("connect", function() {
console.log("connected to mysocket");
});
client.on("data", function(data) {
console.log(data);
});
client.on('end', function() {
console.log('server disconnected');
});
Output 1: Video Buffer displayed
Output 2 : Random data from Video displayed
ffmpeg format of transfering the content from socket to UDP IP and port
ffmpeg -i unix://tmp/mysocket -f format udp://192.168.0.119:8083
problems of this approach : The video was on a passing stage from the socket and contained no information as such when tried to play / show console
Attempt 3 : Send the live WebRTC stream from Kurento WebRTC endpoint to Kurento HTTP endpoint . play using Mozilla VLC web plugin
