Author: altanai

VoIP and Media Streaming Engineer. https://telecom.altanai.com/

Energy Efficient VoIP systems

Data Centres are the concentrated processing units for the amazing Internet that is driving the technological innovation of our generation and has become the backbone of our global economy. DataCentres not only process , store and carry textual data rather a vast amount of computing is for multimedia content which could range from social media to, video streaming or VoIP calls. In this article let us analyze the energy effiiciency , carbon footprint and scope of improvements for a VoIP related data centre which hosts SIP and related RTC technology signalling and media servers and process CDRs and/or media files for playback or recordings.

Just like a regular IT datacentre , storage, computing power and network capacity define the usage of the server.Also unobstructed electricty of of paramount importance as any blackout could drop ongoing calls and lead to loss of revenue for the service provider not to forget the loss caused to parties engaged in call.

Increasing Power consumption by telecom Sector over the years

Global PPA(power purchase agreements) volumes by sector, 2009-2019, IEA, Global PPA volumes by sector, 2009-2019, IEA, Paris https://www.iea.org/data-and-statistics/charts/global-ppa-volumes-by-sector-2009-2019
source : CDP The 3% solution 2013 [19]

Typical VoIP Setup : Whether a cloud Infrstrcture provider of a hosted data centre , an aproximate number of 7 servers is required even for SME ( small to medium enterprises ) communication system and VoIP systems

  • 2 signalling servers primary and standby for HA ,
  • 2 media server for MCU of media bridges or IVR playback etc ,
  • 1 for CDR , logs or call analytics , stats and other supplementary operation
  • 1 for dev or engineering team .
  • 1 edge server could be API server or a gateway or laodbalancer.
Sample voIP system

VoIP solutions are more energy expensive, unless aggressive power saving schemes are in place

Comparison of energy efficiency in PSTN and VoIP systems [14]

While PSTN and other hybrid scenarios relied on audio only communication the embedded systems involved took great pain to make then energy efficient which is not really the case with all digital and software based VoIP.

Power Consumption

Mobile phone :  Typical smartphone with 4,000mAh ( 4 Ah) battery that gets 1 full cycle of usage a day. Daily consumption =4Ah*3.7V=14.8 Wh

Laptop : With  14–15″ screen, a laptop can draw 60 watts power in active use depending on model. Runing 8 hours a day can be 60 * 8 = 480 Wh ( 0.480 kWh) energy consumed in a day.

Desktop PC : Runing at 50-60 Hz frequency , can upto draw 200 W power in active use. For 8 hours energy usage 200 * 8 = 1600 Wh ( 1.6 kWh ) energy a day.

Server : Even though servers are virtual to the request maker , they caters to the request on the other end of the internet.

ServerPurposeServer CPU consumptionClientsClient CPU consumption
ApplicationHosts an application, which can be run through a web browser or customized client software.mediumAny network device with access.low
ComputingMakes available CPU and memory to the client. This type of server might be a supercomputer or mainframe.highAny networked computer that requires more CPU power and RAM to complete an activity.medium
DatabaseMaintains and provides access to any database.lowAny form of software that requires access to structured data.low
FileMakes available shared files and folders across a network.mediumAny client that needs access to shared resources.low
GameProvisions a multiplayer game environment.highPersonal computers, tablets, smartphones, or game consoles.high
MailHosts your email and makes it available across the network.mediumUser of email applications.low
MediaEnables media streaming of digital video or audio over a network.highWeb and mobile applications.high
PrintShares printers over a network.lowAny device that needs to print.low
WebHosts webpages either on the internet or on private internal networks.mediumAny device with a browser.medium
CPU consumption of various server types and their clients

Typically runing on 850 Wh ( 0.850 kwh ) of energy in an hour and since server are usually up 24*7 that totals to

0.850 * 24 = 20.4 kWh a day [2].

VoIP System ( 7 VM’s) : For a setup of 7 VM’s ( could on a the same PM), total energy consumed in a day

20.4 * 7 = 142.8 kWh.

Data centre: The data centre building consists of the infrastructure to support the servers, disks and networking equipment it contains. However, for simplicity, I will only use the consumption of servers and ignore the cooling units, networking, backup batteries charging, generators, lightning, fire suppression, maintenance etc.

High tier DC can have 100 Megawatts of capacity having each rack was using 25 kW of power in a 52U Rack. 100,000 kW / 25 kW = 4,000 racks * 52(U) = 208000 1U servers. This number scales down depending on how much energy each server uses and idle servers.

Total energy 100,000 kW * 24 hours = 2400000 kWh

Carbon Footprint

Carbon footprint in the context of this article refers to the amount of greenhouse gas ( consisting majorly of Co2) caused by electricity consumption. The unit is carbon emission equivalent of the total amount of electricity consumed kg CO2 per kWh.

In doing this calculation I have assumed 0.233 kg CO2 per kWh which could be less or more depending on the generation profile of the electricity provider as well as the heat produced by the machine.

Laptop: Aside from the production which could be 61.4 kg (135.5 lbs) of Co2, a 60W laptop will produce 0.112 kg co2 eq per day.

Desktop PC: Aside from production cost and heating, the GWG and co2 eq emission from running a desktop for a day ( 8 hours) produces 1.6* 0.233 = 0.3728 kg CO2 per kWh

Server : 20.4 * 0.233 = 4.7532 kg CO2 per kWh per day .

VoIP System ( 7 VM’s): Again ignoring the GWG emission of associated components, 142.8 * 0.233 = 33.2724 kg CO2 per kWh per day. It is to be noted that DC’s ( datacentres) use the term PUE ( Power Usage Effectiveness) to showcase their energy efficiency and energy efficiency certification uses the same in ratings.

Data centre: electrical carbon footprint( approximate calculation not counting the cooling, infra maintenance, lightning and possibly idle servers in datacentre) is 2400000 * 0.233 = 559200 kg CO2 per kWh per day

It is to be noted that a common figure should not be extrapolated like this to derive carbon emission. The emission depends on the fuel mix of the electricity generation as well as the life cycle assessment (LCA) of carbon equivalent emission. Countries with heavy reliance on renewables have lower co2 footprint per kWh ~ 0.013 kg co2 per kWh Sweden while others may have higher such as 0.819 kg CO2 per kWh Estonia [1].

Flatten the Curve from Tech and Internet usage

Rack servers tend to be the main perpetrators of wasting energy and represent the largest portion of the IT
energy load in a typical data center.

A decade ago, small enterprise IT facilities were quick to create data centres for hosting applications from hospitals, banks, insurance companies. While some of these is likely to have been upgrade to shared server instances runing on IaaS providers, most of them are still serving traffic or stays there for the lack of effort to upgrade.

With the advancement in p2p technlogies such as dApps , bitcoion network , p2p webrtc streaming , more edge computed ML continue to create disruptions in existing trend , most likely to result in in many fold increase in consumption.

According to the Cambridge Center for Alternative Finance (CCAF), Bitcoin currently consumes around 110 Terawatt Hours per year — 0.55% of global electricity production

Harward Business Review [12]

“the emissions generated by watching 30 minutes of Netflix (1.6 kg of CO2) is the same as driving almost four miles.” 

EnergyInnovation [13]

Cloud Computing and Energy efficiency

Cloud computing ( SaaS, PaaS , IaaS and also CPaaS) minimize power consumption and consequently IT costs via virtualization, clustering and dynamic configuration.

With cloud infrastrcture vendors such as Amazon , Google , microsoft .. and their adoption of energy efficiency computing and credible transparency has alleviated some of the stress that could have been made if onsite self – hosted data centres were used as often in mainstream as a decade ago.

Even as cloud providers gives on -demand access to shared resources in large scale distributed computing , the ease of getting on board has inturn created a surge in cloud hosted online applications consequently high power consumption, more operation costs and higher CO2 emissions.

Components of energy Consumption in Data Centre

As shown CPU, Memory, and Storage incur 45% of the costs and consume 26% of the total energy , however power distribution and colling cost 25% but consumer >50% of total energy.

Energy forcast for Data Centres

As reported by nature [3] the widely cited forcasts suggested thte total electrcity demand of ICT ( Informatioin and Communication technology ) will accelerate and while consumer devices such as smart TV , laptops and mobile are becoming energy effcient , the data centres and network devices will demand bigger portions. Reported in 2018 , 200 Twh( terawatt hours) of energy was being consumed by data centers . Although there are no figures for the telecom or specifically IP cloud telephony , the assumption that enormous multimedia data flows in every session is enouogh to assume the figure must be huge.

Energy eficiency in data centres have also been the subject of many papers and studies. Many of the tech advancements and measures have so far been able to keep the growth in energy requirnments by tech sector to a linear/ flat one.

past and projected growth rate of total US data center energy use from 2000 until 2020. It also illustrates how much faster data center energy use would grow if the industry, hypothetically, did not make any further efficiency improvements after 2010. (Source: US Department of Energy, Lawrence Berkeley National Laboratory)

Some noteworthy innovations made in Data centre for energy efficiency include –

  1. Star efficiency requirnments
  • Average server utilization
  • Server power scaling at low utilization
  • Average power draw of hard disk drives
  • Average power draw of network ports
  • Average infrastructure efficiency (i.e., PUE)

PUE = Total Facility power / IT equipment power

Standard 2.0, Good 1.4 , Better 1.1

Low PUE indicates greater efficicny since more power would then be used by It gear . Idealistically 1 should be the perfect score where all power was used only by the IT gears.

2. Optimizing the cooling system which takes a lot of focus is also not touched upon here but can be understood in great detail from very many sources including one here on how google uses AI for cooling its Datacentres [6]

3. Throttle-down drive ,a device that reduces energy consumption on idle processors, so that when a server is running at its typical 20% utilization it is not drawing full power

Energy efficiency is vital to not only productivity and performance but also to carbon neutral tech and economy. There is ample scope to designing energy efficient applications and platfroms. Some approaches are described below:

Energy Efficiency in VoIP Architecture and design

Low Energy consumption not only lowers operating cost but also helps the enviornment by reducing carbon emission.

1.Server Virtualization

By consolidating multiple independant servers to a single underlying physical server helps retain the logical sepration while also maintaining the energy costs and maximizinng utilization . VM’s( Virtual machines) are instances of virtaulized portions on the same server and can be independetly accesed using its own IP and network settings.

To reduce electricity usage in our labs and data centers, we use smart power distribution units to monitor
our lab equipment. We increase server utilization by using virtual machines. Our Cisco Customer
Experience labs use a check-in, check-out system of automation pods to allow lab employees to set up
configurations virtually and then release equipment when they are finished with it.

Cisco 2020 Environment Technical Review [20]

Models to place VMs on PM ( physical machine ) have been proposed by Dong et al[8] , Huang[9]  ,Tian et al [10]

2.Decommissioning old / outdated servers

While this is the most obvious way to increase efficiency , it is also the toughest since legacy applications or a small portion of it may be running on a server that service providers are not keen on updating or updates do not exist and it is past end of life yet somehow still in use. It is important to identify such components. Check if maybe an old glassfish or bea weblogic SIP servlet server needs updating and/or migration !

3.Plan HA ( high availability ) efficiently

Redundant servers take only if at all any , partial loads so they can be activated in full swing when failover happens in other server. With quick load up times and forward looking monitoring , the analyzers can monitor logs for upcoming failure or predictable downtime and infra script can bring up pre designed containers in seconds if not minutes. It isn’t wise to create more than 1 standby server which does no essential work but consumes as much power.

4.Consolidate individual applications on a Server

Map the maximum precitable load and deduce the percentage comsuption with teh same . In view of these figures it is best to consolidate applications servers to be run on a single server . A distributed microservice based architecture can also support consolidation by runing each major application in its own dockerized container. Consolidation ensures that

  • All data can be stored and accessed centrally, which reduces the likelihood of data duplication.
  • while a server is drawing full power , it is also showing relataible utilization.
  • Single point to prevent intrusion , provide security and fix vulnerabilities against malware like ( ransomware , viruses , spyware , trojans)

5.Reduce redundancy

While it is a common practise to store multiple copies of data such as CDR ( call detail records ) and archiev historical logs for later auditing , it is not the most energy efficient way since it ends up wasting stoarge space. It is infact a better approach to skim only the crtical parts and diacard the rest and definetely implement background tasks to compress the older and less referenced logs.

6.Power management

Powering down idle server or putting unused server to sleep is an effective way to reduce operating power but is often ignored by the IT department in view of risking slower performance and failure in call continuity in case a server does go down. However power management leads to potential energy savings and should be weighted accordingly.

7.Common Storage such as Network Attached Storage

Power consumption is roughly linear to the number of storage modules used. Storage redundancy needs to be
right-sized to avoid rapid consumption of avaible storage space , CPU cycles to refer and index them, its associated power consumption [7].

The process of maximizing storage capacity utilization by drawing from a common pool of shared storage on need baisis also allows for flexixbility.

It is sensible to take the data offline thereby reducing clutter on production system and make the existing data quickly retrievable.

8.Sharing other IT resources

Central Processing Units (CPU), disk drives, and memory optimizes electrical usage. Short term load shifting combined with throttling resources up and down as demand dictates improves long term hardware energy efficiency. [7]

Hardware based approaches such as energy star rating, air conditoning , placement of server racks , air flow , cabling etc have not been touched upon in this article they can be read from energystar report here [5] .

9. DMZ / Perimeter network

The perimeter network (also known as DMZ, demilitarized zone, and screened subnet) is a zone where resources and services accessible from outside the organization are available. Often used as barrier between internal secure green zone within company and outside partners / suppliers such as external organization gateways.

  • Load balancers
  • API gateways
  • SBC ( Session Border controllers)
  • Media Gateways

Ways to cut down on CPU consumption in DMZ machines

  1. Scrutinize incoming traffic only , trust outgoing traffic .

2. Use hardware / network firewalls to monitor and block instead of software defined ones . Hardware firewall can be a standalone physical device or form part of another device on your network. Physical devices like routers, for example, already have a built-in firewall. 

Other types of firewalls

  • Application-layer firewalls can be a physical appliance, or software-based, like a plug-in or a filter. These types of firewalls target your applications. For example, they could affect how requests for HTTP connections are inspected across each of your applications.
  • Packet filtering firewalls scrutinize each data packet as it travels through your network. Based on rules you configure, they decide whether to block the specific packet or not. For example firewalls can block SSH/RDP for remote management.
  • Circuit-level firewalls check whether TCP and UDP connections across your network are valid before data is exchanged. For example, this type of firewall might first check whether the source and destination addresses, the user, the time, and date meet certain defined rules.
  • Proxy server firewalls secure the traffic into and out of a network by monitor, filter, and cache data requests to and from the network.

Energy Efficiency in VoIP Applications and algorithms

In theory, energy efficient algorithms would take less processing power , run fewer CPU cycles and consume less memory. For the experiments with WebRTC and SIP VoIP systems CPU performance can be reliable factor to consider for carbon emissions . Here is list of approaches to include energy as of the parameters in programing for RTC applications.

  1. Take advanatge of Multi Core applications

Multi-core processor chips allow simultaneous processing of multiple tasks, which leads to higher efficiency. Same power source and shared cooling leads to better efficiency . It is the same logic which applied to consolidating one power supply for a rach isntead of individual power supply to each servers on rack.

2. Reduce Buffering

Input/Output buffer pile up comuted packets or blocks which will come inot use in near future but may be discarded all together in event of skip or shutdown. For example in case of video on Demand ( VoD) , a buffered video of 1 hour is of not much use if viewer decides to cancel the video session after 10 minutes .

3. Optimize memeory access algorithms

4. Network energy Management to vary as per demand

The newer generations of network equipment pack more throughput per unit of power. There are active energy management measures that can also be applied to reduce energy usage as network demand varies. In a telecoomunication system , almost always a tradeof between power consumption and network performance is made.

  1. Quick switching of speed of the network to match the amount of data that is currently transmitted. A demand following streaming session will maingtain the QoS , avoid imbalance while also reducing power consumption.

2. Avoid sudden burst and peaks and/or align them with energy availaibility .

Metrics

  • computational performance (i.e., computations/second per server),
  • electrical efficiency of computations (i.e., computations per kWh),
  • storage capacity (i.e., TB per drive), and
  • port speeds (i.e., Gb per port)

5. Task Scheduling algorithms

Some recent researched frameworks and models take Co2 emission into prespective , while allocating resources according to queuing model. The most efficient ones not only bring down the carbon footprint but also the high operating cost [11].

Scheduling and monitoring techniques have been applied to achieve a cost effective and power-aware cloud environment by reducing the resource exploitation

6. Centralised operation – RTP topology ( Mesh , MCU and SFU)

Instead of operating many servers at low CPU utilization, at edge of client’s end, combines the processing
power onto fewer servers that operate at higher utilization.

Modern machine learning programs are computationally intensive, and their integration in VoIP systems for tagging , sentiment analysis , voice quality analysis is increasingly adding additional strain already heavy processing of media server in transcoding and multiplexing .

Media Server using SFU ( Selective Forwarding unit) to transmit mediastrem

As an example a SFU client sends one upstream but receives 4 downstreams which reduces the load on server but increases on clients .

7. Distributing workload based on server performance

Aggregating tasks and runing them as Serverless , asynchronous jobs instead of standalone processes is very efficient way to cut down idle runing wastage. Additioally catagorizing server workloads based on server performance can also reduce power consumption by using idle servers efficiently. Thermal aware workload distribution also helps reducing power consumption and consequently electricity consumption in cooling .

8 . Reduce reauthetication and challemge response mechanism when it can be avoided.

There exists multiple modes to authenticate and authorize users and application access to server content

Over the network

  • password based auth ,
  • third party based auth ( Oauth)
  • 2 factors authetication( phone/sms based) ,
  • multi factor auth ( sms / email / other media) ,
  • token auth ( custom USB device/ smart card ) ,
  • biometric auth (physical human charecteristics / scanners ) ,
  • transactional auth ( location , hour of day , browser/ machine type)

Computer recognition authentication

  • CAPTCHA
  • Single sign-on

Authentication protocols

  • Kerbos – Key Distribution Center (KDC) using a Ticker gransting Server ( TGS)
  • TLS/SSL

A callflow involves AAA while creating the session and may require occsional re authetication to reafform the user is intended one. Doing re-authtication too often increases the power consumption and can be countered by caching and timeout mechanism.

Point of presence and handover using Carbon footprint in different demographics

  1. Include Carbon emission from Datacentre in condieration before engaging the server in call path from load balancer gateway

2. Use point of presence ( PoP) for server according to their carbon emission factor in the demography .

Us states carbon emission rate from electricity generation (2018 report ) Source : [16]
UK greenhouse gas reporting source : [17]

Energy Efficiency in WebRTC browser applications and native applications

In a Video conferencing the over browser, WebRTC has emerged as te the default standard . The efficiency of sch webrtc browser based video conferencing web applications can be enhanced in the following ways :

1.Use VoIP Push Notifications to Avoid Persistent Connections

2. Voice Activity detection ( Mute the spectators ) and join with video true , audio false for attendeees

Energy efficiency in VoIP phones

If all eligible VoIP phones sold in the United States were ENERGY STAR certified, the energy cost savings would grow to more than $65 million each year and 1.2 billion pounds of annual greenhouse gas emissions would be prevented, equivalent to the emissions from more than 119,000 vehicles.

Energystart [15]

Low-energy-consuming embedded hardware on most phones keep the average consumption low . A analog phone can consume power between 0.07 W to 9.27 W while a VoIP phone can consume 0.1W to 3.5 W of standby power.

Off mode power is often less than standby power since phone is on low power model during idle hours such as night . According to energy star Sund transmission mechnism also plays a key role and hybrid phones consume more power.

Power allowance (W) for each of the below features of the device:

  • 1.0 watt for Gigabit Ethernet
  • 0.2 watt for Energy Efficiency Ethernet 802.3az compliant Gigabit Ethernet

Additional proxy incentive(W) for the ability to maintain network presence while in a low power mode and intelligently wake when needed

  • 0.3 watt for base capability
  • 0.5 watt for remote wake

Government bodies and groups to track Energy efficiency of Telecom and IP telephony

  • Alliance for Telecommunications Industry Solutions (ATIS)
  • Telecommunications Energy Efficiency Ratio (TEER)
  • measurement method covers all power conversion and power distribution from the front end of the
    system to the data wire plug, including application-specific integrated circuits (ASICs).
  • European Telecommunications Standards Institute (ETSI)
  • International Telecommunication Union (ITU)
  • U.S. Department of Energy (DOE), Environmental Protection Agency (EPA)

External links

Amazon : https://sustainability.aboutamazon.com/environment/sustainable-operations/carbon-footprint

Cisco : https://www.cisco.com/c/dam/m/en_us/about/csr/esg-hub/_pdf/2020_Environment_Technical_Review.pdf

3CX : https://askozia.com/voip/how-can-i-save-energy-with-green-voip-and-my-ip-pbx/

The purpose of the article is to raise awareness about carbon footprint from application programs to archietcture designs techniques to data centres and commuulative performance. It gives a direction to stakeholders (customers , programmers , architects , mangers , … ) to choose less carbon emitting approach whenever possible since every bit counts to help the environment.

References

[1] rensmart.com https://www.rensmart.com/Calculators/KWH-to-CO2

[2] https://www.zdnet.com/article/toolkit-calculate-datacenter-server-power-usage/

[3] nature : https://www.nature.com/articles/d41586-018-06610-y

[4] Center of Expertise for Energy Efficiency in Data Centers at the US Department of Energy’s Lawrence Berkeley National Laboratory in Berkeley, California. https://datacenters.lbl.gov/

[5] energy Star – https://www.energystar.gov/sites/default/files/asset/document/DataCenter-Top12-Brochure-Final.pdf

[6] https://www.blog.google/inside-google/infrastructure/safety-first-ai-autonomous-data-center-cooling-and-industrial-control/

[7] https://www.energy.gov/sites/default/files/2013/10/f3/eedatacenterbestpractices.pdf

[8] Yin K, Wang S, Wang G, Cai Z, Chen Y. Optimizing deployment of VMs in cloud computing environment. In: Proceedings of the 3rd international conference on computer science and network technology. IEEE; 2013. p. 703–06.

[9] Huang W, Li X, Qian Z. An energy efficient virtual machine placement algorithm with balanced resource utilization. In: Proceedings of the seventh IEEE international conference on innovative mobile and internet services in ubiquitous computing; 2013. p. 313–19.

[10] W. Tian, C.S. Yeo, R. Xue, Y. Zhong Power-aware schedulingof real-time virtual machines in cloud data centers considering fixed processing intervalsProc IEEE, 1 (2012), pp. 269-273

[11] H. Chen, X. Zhu, H. Guo, J. Zhu, X. Qin, J. Wu Towards energy-efficient scheduling for real-time tasks under uncertain Cloud computing environmentJ Syst Softw, 99 (2015), pp. 20-35

[12] https://hbr.org/2021/05/how-much-energy-does-bitcoin-actually-consume

[13] https://energyinnovation.org/2020/03/17/how-much-energy-do-data-centers-really-use/

[14] F. Bota, F. Khuhawar, M. Mellia and M. Meo, “Comparison of energy efficiency in PSTN and VoIP systems,” 2012 Third International Conference on Future Systems: Where Energy, Computing and Communication Meet (e-Energy), 2012, pp. 1-4, doi: 10.1145/2208828.2208834. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.720.446&rep=rep1&type=pdf

[15] https://www.energystar.gov/products/office_equipment/voice_over_internet_protocol_voip_phone

[16] egrid summary table 2018 for carbon emission rate in Us states : https://www.epa.gov/sites/default/files/2020-01/documents/egrid2018_summary_tables.pdf

[17] UK greenhourse gas reporting – https://www.gov.uk/government/publications/greenhouse-gas-reporting-conversion-factors-2018

[19] http://assets.worldwildlife.org/publications/575/files/original/The_3_Percent_Solution_-_June_10.pdf?1371151781

[20] https://www.cisco.com/c/dam/m/en_us/about/csr/esg-hub/_pdf/2020_Environment_Technical_Review.pdf

TeleMedicine and WebRTC

Anywhere anytime Telemedicine communication tool accessible on any device.  The solution provides a low eight signalling server which drops out as soon as call is connected thus ensuring absolutely private calls without relaying or involving any central server in any call related data or media . This ensure doctor patient details are not processed , stored or recorded by our servers.

The solution enables doctors / nurses / medical practitioners and patients  to do

  • High definition Audio/video calls 
  • End to end encrypted p2p chats 
  • Integration with HMS ( hospital management system ) to fetch history of the patients 
  • Screens sharing to show reports without transferring them as files 
  • Include more concerned people of doctors using Mesh based peer to peer conferencing feature.      

Confidentialty and Privacy

For privacy and security of certain health information only HIPAA (Health Insurance Portability and Accountability Act of 1996) compliant video-conferencing tools can only be used for Telemedicine in US.

Telemedicine scenario Callflow

Calllfow for Attended Call Transfer and 2 way conference in a Telemedicine scenario between Patient , hospital attendant , doctor and a nurse

References :

Performance of WebRTC sites and electron apps

This post is about making performance enhancements to a WebRTC app so that they can be used in the area which requires sensitive data to be communicated, cannot afford downtime, fast response and low RTT, need to be secure enough to withstand and hacks and attacks.

Best practices for WebRTC single page applications

As a check the communication clients become single page driven , a lot of authetication , heartbeat sync , web workers , signalling event driven flow management resides on the same page along with the actuall CPU consumption for the audio video resources and media streams . This in turns can make the webpage heavy and many a times could result in crash due to being ” unresponsive”.

Here are some my best to-dos for making sure the webrtc communication client page runs efficiently

Visual stability and CLS ( Cummulative Layout Shift)

CLS metrics measures the sum total of all individual layout shift scores for every unexpected layout shift that occurs during the entire lifespan of the page.

To have a good user interactionn experiences, the DOM elements should display as less movement as possible so that page appears stable . In the opposite case for a flickering page ( maybe due to notification DOM dynamically pushing the other layout elements ) it is difficult to precisely interact with the page elements such as buttons .

Minimize main thread work

The main thread is where a browser processes runs all the JavaScript in your page, as well as to perform layout, reflows, and garbage collection. therefore long js processes can block the thread and make the page unresponsive .

Depriciation of XMLHTTP request on main thread

Reduce Javacsipt execution Time

Unoptimized JS code takes longer to execute and impacts network , parse-compileand memory cost.

If your JavaScript holds on to a lot of references, it can potentially consume a lot of memory. Pages appear janky or slow when they consume a lot of memory. Memory leaks can cause your page to freeze up completely.

Some effective tips to spedding up JS execution include

  • minifying and compressing code
  • Removing the unused code and console.logs
  • Apply caching to save lookup time

Caching

-tbd

Security Concerns

In one of my previous posts I have mentioned about Security threats to WebRTC Solution . It includes mainly 4 ways in which WebRTC Solution Providers and Users are vulnerable , which includes

  1. Identity Management ,
  2. Browser Security ,
  3. Authentication and
  4. Media encryption.

WebRTC Security

WebRTC Security
Identity Management ,
Browser Security ,
Authentication and
Media encryption.
Browser Threat Model
Best practices for the Webrtc comm agents
ICE TURN challenges
DTLS
SRTP

Cookies – Security vs persistsnt state

Cross-site request forgery (CSRF) attacks rely on the fact that cookies are attached to any request to a given origin, no matter who initiates the request.

While adding cookies we must ensure that if SameSite =None , the cookies must be secure 

Set-Cookie: widget_session=abc123; SameSite=None; Secure

SameSite to Strict, your cookie will only be sent in a first-party context. In user terms, the cookie will only be sent if the site for the cookie matches the site currently shown in the browser’s URL bar. 

Set-Cookie: promo_shown=1; SameSite=Strict

You can test this behavior as of Chrome 76 by enabling chrome://flags/#cookies-without-same-site-must-be-secure and from Firefox 69 in about:config by setting network.cookie.sameSite.noneRequiresSecure.

Performance monitoring

Key Performance Indicators (KPIs) are used to evaluate the performance of a website . It is crticial that a webrtc web page must be light weight to acocmodate the signalling control stack javscript libs to be used for offer answer handling and communicating with the signaller on open sockets or long polling mechnism .

Lighthouse results

Lighthouse tab in chrome developer tools shows relavnat areas of imporevemnt on the webpage from performmace , Accesibility , Best Practices , Search Engine optimization and progressive Web App

Also shsows individual categories and comments

Time to render and Page load

Page attributes under Chrome developers control depicts the page load and redering time for every element includeing scripts and markup. Specifically it has

  • Time to Title
  • Time to render
  • Time to inetract

Networking attributes to be cofigured based on DNS mapping and host provider. These Can be evalutaed based on chrome developer tool reports

Task interaction time

Other page interaction crtiteria includes the frames their inetraction and timings for the same.

In the screenhosta ttcjed see the loading tasks which basically depcits the delay by dom elements under transitions owing to user interaction . This ideally should be minimum to keep the page responsive.

Page’s total memeory

measureMemory()

performance.measureUserAgentSpecificMemory

The above functions ( old and new ) estimates the memory usage of the entire web page

these calls can be used to correlate new JS code with the impact on memery and subsewuntly find if there are any memeory leaks . Can also use these memery metrics to do A/B testing .

DNS lookup Time

Services such as Pingdom (https://tools.pingdom.com/) or WebPageTest can quickly calculate your website’s DNS lookup times

Load / sgtress tresting can be perfomed over tools such as LoadStorm , JMeter

Page weight and PRPL

Loading assests over CDN , minfying sripts and reducing over all weight of the page are good ways to keep the pagae light and active and prevent any chrome tab crashes .

PRPL expands to Push/preload , Render , PreCache , Lazy load

  • Render the initial route as soon as possible.
  • Pre-cache remaining assets.
  • Lazy load other routes and non-critical assets.

Preload is a declarative fetch request that tells the browser to request a resource as soon as possible. Hence should be used for crticial assests . 

<link rel="preload" as="script" href="critical.js">

The non critical compoenents could then be loaded on async .

Lazy load must be used for large files like js paylaods which are costly to load. To send a smaller JavaScript payload that contains only the code needed when a user initially loads your application, split the entire bundle and lazy load chunks on demand.

Web Workers

Web Workers are a simple means for web content to run scripts in background threads.The Worker interface spawns real OS-level threads

By acting as a proxy, service workers can fetch assets directly from the cache rather than the server on repeat visits. 

Conversion Rates over analytics Tools

Google Analytics is a good way of deducing Converdsation and bounce rates

Codecs weight

Vide codecs Vp8 vs VP9

SFU vs MCU or p2p media flow

References :

5G

In the course of evolution of RAN ( Radio Access layer) technologies 5G outsmarts 4G-2010 which comes in succession after 3G-2000, 2.5G, 2G -1990 and 1G/PSTN -1980 respectively. Among the most striking features of 5G –

  • entirely IP based
  • ability to connect 100x more devices ( IOT favourable )
  • speed upto 10 Gbit/s
  • high peak bit rate
  • high data volume per unit area
  • virtually 0 latency hence high response time

Thus it can accomodate the rapid growth of rich mulimedia application like OTT streaming of HD content, gaming , Augmented reality so on while enabling devices connected to Internet of Things sto onboard the telecommunication backbone with high system spectral efficiency and ubiquitious connectivity .

Infact 5G has seen maximum investment in year 2020 in revamping infrastrcuture as compared to other technologies such as IoT or even Cloud. This could be partly due to high rise in high speed communication for streaming and remote communication owining to steep rise in remote learning adn working from home scenarious.

img source statista – global-telecom-industry-priority-investment-areas

Spectrum

5G is specified to operate over range 1 GHz to 100 GHz.

  • Low-band spectrum (below 2.5 GHz) – excellent coverage,
  • mid- band spectrum (2.5–10 GHz) – a combination of good coverage and very high bitrates,
  • high band-spectrum (10–100 GHz) – the bandwidths needed for the highest bitrates (up to 20 Gb/s) and lowest latencies

Workplan for 5G standardisation and release

The Workplan started in 2014 and is ongoing as of now (2018)

image source : 3GPP “Getting ready for 5G”

3GPP is the standard defining body for telecom and has specified almost all RAN technologies like GSM , GPRS , W-CDMA , UMTS , EDGE , HSPAand LTE before .

Applications of 5G

5G targets three main use case

  • enhanced mobile broadband (eMBB),
  • massive machine type communications (mMTC)
  • ultra-reliable low latency communications (URLLC) (also called critical machine type communications (cMTC))
sources : whitepaper ericsson

General Data Protection Regulation (GDPR) in VoIP

GDPR , Europe’s digital privacy legislation passed in 2018, replaces the 1995 EU Data Protection Directive. It is rules designed to give EU citizens more control over their personal data & strengthen privacy rights. It aims to simplify the regulatory environment for business and citizens.

To read about other Certificates , compliances and Security in VoIP which summaries

  • HIPAA (Health Insurance Portability and Accountability Act) ,
  • SOX( Sarbanes Oxley Act of 2002),
  • Privacy Related Compliance certificates like COPPA (Children’s Online Privacy Protection Act ) of 1998,
  • CPNI (Customer Proprietary Network Information) 2007,
  • GDPR (General Data Protection Regulation)  in European Union 2018,
  • California Consumer Privacy Act (CCPA) 2019,
  • Personal Data Protection Bill (PDP) – India 2018 and
  • also specifications against Robocalls and SPIT ( SPAM over Internet Telephony) among others

Multinational companies will predominantly be regulated by the supervisory authority where they have their “main establishment” or headquarter. However, the issue concerning GDPR is that it not only applies to any organisation operating within the EU, but also to any organisations outside of the EU which offer goods or services to customers or businesses in the EU.

Key Principles of GDPR are

  • Lawfulness, fairness and transparency
  • Purpose limitation
  • Data minimisation
  • Accuracy
  • Storage limitation
  • Integrity and confidentiality (security)
  • Accountability

GDPR consists of 7 projects (DPO, Impact assessment, Portability, Notification of violations, Consent, Profiling, Certification and Lead authority) that will strengthen the control of personal data throughout the European Union.

Stakeholders

stakeholders of data protection regulation are
Data Subject – an individual, a resident of the European Union, whose personal data are to be protected

Data Controller – an institution, business or a person processing the personal data e.g. e-commerce website.

Data Protection Officer – a person appointed by the Data Controller responsible for overseeing data protection practices.

Data Processor – a subject (company, institution) processing a data on behalf of the controller. It can be an online CRM app or company storing data in the cloud.

Data Authority – a public institution monitoring implementation of the regulations in the specific EU member country.

Extra-Territorial Scope

Any VoIP service provider may feel that since they are not based out of EU such as officially headquartered in the Asia Pacific or US region they may not be legally binding to GDPR. However, GDPR expands the territorial and material scope of EU data protection law.  It applies to both controllers and processors established in the EU, and those outside the EU, who offer goods or services to or monitor EU data subject.

VoIP service providers as Data Processors

A processor is a “person, public authority, agency or other body which processes personal data on behalf of the controller”.
Most VoIP service providers are multinational in nature with services offered directly or indirectly to all regions. The GDPR imposes direct statutory obligations on data processors, which means they will be subject to direct enforcement by supervisory authorities, fines, and compensation claims by data subjects. However, a processor’s liability will be limited to the extent that it has not complied with it’s statutory and contractual obligations.

Data minimization – It is now a good practise to store and process as less user’s personal data as necessary to render our services effectively. Also to maintain data for only a stipulated time ( approx 90 days of CDR for call details and logs )

Record Keeping, Accountability and governance

To show compliance with GDPR, a service provider maintain detailed records of processing activities. Also, they must implement technological and organisational measures to ensure, and be able to demonstrate, that processing is performed in accordance with the GDPR. Some ways to apply these are :

  • Contracts: putting written contracts in place with organisations that process personal data on your behalf
  • maintaining documentation of your processing activities
  • Organisational policies focus on Data protection by design and default – two-factor auth, strong passwords to guard against brute-force, encryption, focus on security in architecture
  • Rish analysis and impact assessments: for uses of personal data that are likely to result in a high risk to individuals’ interests
  • Audit by Data protection officer
  • Clear Codes of conduct
  • Certifications

As for a VOIP landscape thankfully every call or message session is followed by a CDR ( Calld Detail Record ) or MDR ( Message Detail Record).

Additionally, assign a unique signature to every data-access client the VoIP system and log every read/write operation carried out on data stores whether persistent datastores or system caches.

Privacy Notices to Subjects

User profile data such as :

  • Basic identity information, name, address and ID numbers
  • Web data such as location, IP address, cookie data and RFID tags
  • Health and genetic data
  • Bio-metric data
  • Racial or ethnic data
  • Political opinions
  • Sexual orientation

is protected strictly under GDPR rules

A service provider should provide indepth information to data subjects when collecting their personal data, to ensure fairness and transparency. They must provide the information in an easily accessible form, using clear and plain language.

Consent

The GDPR introduces a higher bar for relying on consent , requiring clear affirmative action. Silence, pre ticked boxes or inactivity will not be sufficient to constitute consent. Data subjects can withdraw their consent at any time, and it must be easy for them to do so.

Lawful basis for processing Data now include

In Article 6 of the GDPR , there are six available lawful bases for processing.

(a) Consent: the individual has given clear consent for you to process their personal data for a specific purpose.

(b) Contract: the processing is necessary for a contract you have with the individual, or because they have asked you to take specific steps before entering into a contract.

(c) Legal obligation: the processing is necessary for you to comply with the law (not including contractual obligations).

(d) Vital interests: the processing is necessary to protect someone’s life.

(e) Public task: the processing is necessary for you to perform a task in the public interest or for your official functions, and the task or function has a clear basis in law.

(f) Legitimate interests: the processing is necessary for your legitimate interests or the legitimate interests of a third party, unless there is a good reason to protect the individual’s personal data which overrides those legitimate interests.

File such as PCAPS , Recordings and transcripts of calls hold sensitive information from end users , these should be encryoted and inaccssible to even the dev teams within the org without explicit consent of end user .

Individuals’ Rights

The GDPR provides individuals with new and enhanced rights to Data subjects who will have more control over the processing of their personal data. A data subject access request can only be refused if it is manifestly unfounded or excessive, in particular because of its repetitive character.

Rights of Data Subjets include

  • Right of Access
  • Right to Rectification
  • Right to Be Forgotten
  • Right to Restriction of Processing
  • Right to Data Portability
  • Right to Object
  • Right to Object to Automated Decisionmaking

For a VoIP service provider if a user opts for redaction then none of his calls or messages should be traced in logs . Also replace distinguishable end user identifier such as phone number and sip uri with *** charecters

Provide option for “Account Deletion” and purge account – If a user wished to close his/her account , his/her detaisl should be deleted form the sustem except for the bare bones detaisl which are otherwise required for legal , taxation and accounting requirnments

Breach Notification

A controller is a “person, public authority, agency or other body which, alone or jointly with others, determines the purposes and means of processing of personal data”,

A controller will have a mandatory obligation to notify his supervisory authority of a data breach within 72 hours unless the breach is unlikely to result in a risk to the rights of data subjects. Will also have to notify affected data subjects where the breach is likely to result in a “high risk” to their rights. A processor, however, will only be obliged to report data breaches to controllers

International Data Transfers

Data transfers to countries outside the EEA(European Economic Area) continue to be prohibited unless that country ensures an adequate level of protection. The GDPR retains existing transfer mechanisms and provides for additional mechanisms, including approved codes of conduct and certification schemes.

The GDPR prohibits any non-EU court, tribunal or regulator from ordering the disclosure of personal data from EU companies unless it requests such disclosure under an international agreement, such as a mutual legal assistance treaty.

One of the biggest challenges for a service provider is the identification & categorization of GDPR impacted data sets in disparate locations across the enterprise. A dev team must flag tables, attributes and other data objects that are categorically covered under GDPR regulations and then ensure that they are not transferred to a server outside of EU.

In the present age of Virtual shared server instance, cloud computing and VoIP protocol it is operational a very tough task for a communication service provider to ensure that data is not transferred outside of EU such as a VoIP call from origination in US and destination in EU will require information exchanges via SDP, vcard , RTP stream via media proxies etc.

Sanctions

The GDPR provides supervisory authorities with wide-ranging powers to enforce compliance, including the power to impose significant fines. You will face fines of up to €20m or 4% of your total worldwide annual turnover of the preceding financial year. In addition, data subjects can sue you for pecuniary or non-pecuniary damages (i.e. distress). Supervisory authorities will have a discretion as to whether to impose a fine and the level of that fine.

Data Protection officer (DPO)

Under the terms of GDPR, an organisation must appoint a Data Protection Officer (DPO) if it carries out large-scale processing of special categories of data, carries out large scale monitoring of individuals such as behaviour tracking or is a public authority.

Reference :

Media Architecture , RTP topologies

With the sudden onset of Covid-19 and building trend of working-from-home , the demand for building scalable conferncing solution and virtual meeting room has skyrocketed . Here is my advice if you are building a auto- scalable conferencing solution

This article is about media server setup to provide mid to high scale conferencing solution over SIP to various endpoints including SIP softphones , PBXs , Carrier/PSTN and WebRTC.

Point to Point

Endpoints communicating over unicast
RTP and RTCP tarffic is private between sender and reciver even if the endpoints contains multiple SSRC’s in RTP session.

Advantages of P2p Disadvantages of p2p
  • Facilitates private communication between the parties
  • Only limitaion to number of stream between the partcipants are the physical limiations such as bandwidth, num of available ports

    • Point to Point via Middlebox

    Same as above but with a middle-box involved

    Translator

    mostly used interoperability for non-interoperable endpoints such as transcoding the codecs or transport convertion
    does not use an SSRC of its own and keeps the SSRC for an RTP stream across the translation.

    Subtypes of Multibox :

    Transport/Relay Anchoring

    Roles like NAT traversal by pinning the media path to a public address domain relay or TURN server

    Middleboxes for auditing or privacy control of participant’s IP

    Other SBC ( Session Border Gateways) like characteristics are also part of this topology setup

    Transport translator

    interconnecting networks like multicast to unicast

    media packetization to allow other media to connect to the session like non-RTP protocols

    Media translator

    modified the media inside of RTP streams commonly known as transcoding

    can do up to full encoding/decoding of RTP streams

    in many cases it can also act on behalf of non-RTP supported endpoints , receiving and responding to feedback reports ad performing FEC ( forward error corrected )

    Back-To-Back RTP Session

    Mostly like middlebox like translator but establishes separate legs RTP session with the endpoints, bridging the two sessions.

    Takes complete responsibility of forwarding the correct RTP payload and maintain the relation between the SSRC and CNAMEs

    Advantages of Back-To-Back RTP SessionDisadvantages of Back-To-Back RTP Session
    B2BUA / media bridge take responsibility tpo relay and manages congestion
  • It can be subjected to MIM attack or have a backdoor to eavesdrop on conversations

    • Point to Point using Multicast

    Any-Source Multicast (ASM)

    traffic from any particpant sent to the multicat group address reaches all other partcipants

    Source-Specific Multicast (SSM)

    Selective Sender stream to the multicast group which streams it to the recibers

    Point to Multipoint using Mesh

    many unicast RTP streams making a mesh

    Point to Multipoint + Translator

    Some more variants of this topology are Point to Multipoint with Mixer

    Media Mixing Mixer

    receives RTP streams from several endpoints and selects the stream(s) to be included in a media-domain mix. The selection can be through

    static configuration or by dynamic, content-dependent means such as voice activation. The mixer then creates a single outgoing RTP stream from this mix.

    Media Switching Mixer

    RTP mixer based on media switching avoids the media decoding and encoding operations in the mixer, as it conceptually forwards the encoded media stream.

    The Mixer can reduce bitrate or switch between sources like active speakers.

    SFU ( Selective Forwarding Unit)

    Middlebox can select which of the potential sources ( SSRC) transmitting media will be sent to each of the endpoints. This transmission is set up as an independent RTP Session.

    Extensively used in videoconferencing topologies with scalable video coding as well as simulcasting.

    Advantges of SFUDisadvatages of SFU
    Low lanetncy and low jitter buffer requirnment by avoiding re encondingunable to manage network and control bitrate

    On a high level, one can safely assume that given the current average internet bandwidth, for count of peers between 3-6 mesh architectures make sense however any number above it requires centralized media architecture.

    Among the centralized media architectures, SFU makes sense for atmost 6-15 people in a conference however is the number of participants exceed that it may need to switch to MCU mode.

    Other Hybrid Topologies

    There are various topologies for multi-endpoint conferences. Hybrid topologies include forward video while mixing audio or auto-switching between the configuration as load increases or decreases or by a paid premium of free plan

    Hybrid model

    Some endpoints receive forwarded streams while others receive mixed/composited streams.

    Serverless models

    Centralized topology in which one endpoint serves as an MCU or SFU.

    Used by Jitsi and Skype

    Point to Multipoint Using Video-Switching MCUs

    Much like MCU but unlike MCU can switch the bitrate and resolution stream based on the active speaker, host or presenter, floor control like characteristics.

    This setup can embed the characteristics of translator, selector and can even do congestion control based on RTCP

    To handle a multipoint conference scenario it acts as a translator forwarding the selected RTP stream under its own SSRC, with the appropriate CSRC values and modifies the RTCP RRs it forwards between the domains

    Cascaded SFUs

    SFU chained reduces latency while also enabling scalability however takes a toll on server network as well as endpoint resources

    Transport Protocols

    Before getting into an in-depth discussion of all possible types of Media Architectures in VoIP systems, let us learn about TCP vs UDP

    TCP is a reliable connection-oriented protocol that sends REQ and receives ACK to establish a connection between communicating parties. It sequentially ends packets which can be resent individually when the receiver recognizes out of order packets. It is thus used for session creation due to its errors correction and congestion control features.

    Once a session is established it automatically shifts to RTP over UDP. UDP even though not as reliable, not guarantying non-duplication and delivery error correction is used due to its tunnelling methods where packets of other protocols are encapsulated inside of UDP packet. However to provide E2E security other methods for Auth and encryption are used.

    Audio PCAP storage and Privacy constraints for Media Servers

    A Call session produces various traces for offtime monitoring and analysis which can include

    CDR ( Call Detail Records ) – to , from numbers , ring time , answer time , duration etc

    Signalling PCAPS – collected usually from SIP application server containing the SIP requests, SDP and responses. It shows the call flow sequences for example, who sent the INVITE and who send the BYE or CANCEL. How many times the call was updated or paused/resumed etc .

    Media Stats – jitter , buffer , RTT , MOS for all legs and avg values

    Audio PCAPS – this is the recording of the RTP stream and RTCP packets between the parties and requires explicit consent from the customer or user . The VoIP companies complying with GDPR cannot record Audio stream for calls and preserve for any purpose like audit , call quality debugging or an inspection by themselves.

    Throwing more light on Audio PCAPS storage, assuming the user provides explicit permission to do so , here is the approach for carrying out the recording and storage operations.

    Firther more , strict accesscontrol , encryption and annonymisation of the media packets is necessary to obfuscate details of the call session.

    References :

    To learn about the difference between Media Server tologies

    • centralized vs decentralised,
    • SFU vs MCU ,
    • multicast vs unicast ,

    Read – SIP conferecning and Media Bridge

    SIP conferencing and Media Bridges

    SIP is the most popular signalling protocol in VOIP ecosystem. It is most suited to a caller-callee scenario , yet however supporting scalable conferences on VOIP is a market demand. It is desired that SIP must for multimedia stream but also provide conference control for building communication and collaboration apps for new and customisable solutions.

    To read more about buildinga scalable VoIP Server Side architecture and

    • Clustering the Servers with common cache for High availiability and prompt failure recovery
    • Multitier archietcture ie seprartion between Data/session and Application Server /Engine layer
    • Micro service based architecture ie diff between proxies like Load balancer, SBC, Backend services , OSS/BSS etc
    • Containerization and Autoscalling

    Read – VoIP/ OTT / Telecom Solution startup’s strategy for Building a scalable flexible SIP platform

    VoIP/ OTT / Telecom Solution startup’s strategy for Building a scalable flexible SIP platform

    I have been contemplating points that make for a successful developer  to develop solutions and services for a  Telecom Application Server.  The trend has shown many variations from pure IN programs like VPN , Prepaid billing logic to SIP servlets for call parking , call completion. From SIP servlets to JAISNLEE open standard based communication. … Continue reading VoIP/ OTT / Telecom Solution startup’s strategy for Building a scalable flexible SIP platform

    WebRTC Audio/Video Codecs

    Codecs signifies the media stream’s compession and decompression. For peers to have suceesfull excchange of media, they need a common set of codecs to agree upon for the session . The list codecs are sent  between each other as part of offeer and answer or SDP in SIP.

    As WebRTC provides containerless bare mediastreamgtrackobjects. Codecs for these tracks is not mandated by webRTC . Yet the codecs are specified by two seprate RFCs

    RFC 7878 WebRTC Audio Codec and Processing Requirements specifies least the Opus codec as well as G.711’s PCMA and PCMU formats.

    RFC 7742 WebRTC Video Processing and Codec Requirnments specifies support for  VP8 and H.264’s Constrained Baseline profile for video .

    In WebRTC video is protected using Datagram Transport Layer Security (DTLS) / Secure Real-time Transport Protocol (SRTP). In this article we are going to dicuss Audio/Video Codecs processing requirnments only.

    Quick links : If you are new to WebRTC read : Introduction to WebRTC is at https://telecom.altanai.com/2013/08/02/what-is-webrtc/ and Layers of WebRTC at https://telecom.altanai.com/2013/07/31/webrtc/

    WebRTC Media Stack

    Media Stream Trcaks in WebRTC

    The MediaStreamTrack interface typically represents a stream of data of audio or video and a MediaStream may contain zero or more MediaStreamTrack objects.

    The objects RTCRtpSender and RTCRtpReceiver can be used by the application to get more fine grained control over the transmission and reception of MediaStreamTracks.

    Media Flow in VoIP system
    Media Flow in WebRTC Call

    Video

    Video Capture insync with hardware’s capabilities

    WebRTC compatible browsers are required to support Whie-balance , light level , autofocus from video source

    Video Capture Resolution

    Minimum WebRTC video attributes unless specified in SDP ( Session Description protocl ) is minimum 20 FPS and resolution 320 x 240 pixels. 

    Also supports mid stream resilution changes such as in screen source fromdesktop sharinig .

    SDP attributes for resolution, frame rate, and bitrate

    SDP allows for codec-independent indication of preferred video resolutions using a=imageattr to indicate the maximum resolution that is acceptable. 

    Sender must send limiting the encoded resolution to the indicated maximum size, as the receiver may not be capable of handling higher resolutions.

    Dynamic FPS control based on actual hardware encoding :

    video source capture to adjust frame rate accroding to low bandwidth , poor light conditions and harware supported rate rather than force a higher FPS .

    Stream Orientation

    support generating the R0 and R1 bits of the Coordination of Video Orientation (CVO) mechanism and sharing with peer

    Codecs

    WebRTC is free and opensource and its woring bodies promote royality free codecs too. The working groups RTCWEB and IETF make the sure of the fact that non-royality beraning codec are mandatory while other codecs can be optional in WebRTC non browsers .

    WebRTC Browsers MUST implement the VP8 video codec as described in
    RFC6386] and H.264 Constrained Baseline as described in [H264].

    RFC 7442 WebRTC Video Codec and Processing Requirements

    most of the codesc below follow Lossy DCT(discrete cosine transform (DCT) based algorithm for encoding.

    Sample SDP from offer in Chrome browser v80 for Linux incliudes these profile :

    m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123
a=rtcp-mux
a=rtcp-rsize

a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98

a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100

a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=102

a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=127

a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125

a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 red/90000
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=124

    VP8

    Developed by on2 and then acquired and opensource by google . Now free of royality fees.

    Supported conatiner – 3GP, Ogg, WebM

    No limit on frame rate or data rate and provides maximum resolution of 16384×16384 pixels.

    libvpx encoder library.

    VP8 encoders must limit the streams they send to conform to the values indicated by receivers in the corresponding max-fr and max-fs SDP attributes.
    encode and decode pixels with an implied 1:1 (square) aspect ratio.

    supported simulcast

    VP9

    Video Processor 9 (VP9) is the successor to the older VP8 and comparable to HEVC as they both have simillar bit rates .

    Open and free of royalties and any other licensing requirements
    Its supported Containers are – MP4, Ogg, WebM

    H264/AVC constrained

    AVC’s Constrained Baseline (CBP ) profile compliant with WebRTC

    Constrained Baseline Profile Level 1.2 and H.264 Constrained High Profile Level 1.3 . Contrained baseline is a submet of the main profile , suited to low dealy , low complexity. suited to lower processing device like mobile videos

    Multiview Video Coding – can have multiple views of the same scene ,such as stereoscopic video.

    Other profiles , which are not supporedt are Baseline (BP) , Extended (XP), Main (MP) , High (HiP) , Progressive High (ProHiP) , High 10 (Hi10P), High 4:2:2 (Hi422P) and High 4:4:4 Predictive

    Its supported containers are 3GP, MP4, WebM

    Parameter settings:

    • packetization-mode
    • max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br
    • sprop-parameter-sets: H.264 allows sequence and picture information to be sent both in-band and out-of-band. WebRTC implementations must signal this information in-band.
    • Supplemental Enhancement Information (SEI) “filler payload” and “full frame freeze” messages( used while video switching in MCU streams )

    It is a propertiary , patented codec , mianted by MPEG / ITU

    AV1 (AOMedia Video 1)

    open format designed by the Alliance for Open Media
    royality free
    especially designed for internet video HTML element and WebRTC
    higher data compression rates than VP9 and H.265/HEVC

    offers 3 profiles in increasing support for color depths and chroma subsampling.
    main,
    high, and
    professional

    supports HDR
    supports Varible Frame Rate

    Supported container are ISOBMFF, MPEG-TS, MP4, WebM

    Stats for Video based media stream track

    timestamp 04/05/2020, 14:25:59
ssrc 3929649593
isRemote false
mediaType video
kind video
trackId RTCMediaStreamTrack_sender_2
transportId RTCTransport_0_1
codecId RTCCodec_1_Outbound_96
[codec] VP8 (payloadType: 96)
firCount 0
pliCount 9
nackCount 476
qpSum 912936
[qpSum/framesEncoded] 32.86666666666667
mediaSourceId RTCVideoSource_2
packetsSent 333664
[packetsSent/s] 29.021823604499957
retransmittedPacketsSent 0
bytesSent 342640589
[bytesSent/s] 3685.7715977714947
headerBytesSent 8157584
retransmittedBytesSent 0
framesEncoded 52837
[framesEncoded/s] 30.022576142586164
keyFramesEncoded 31
totalEncodeTime 438.752
[totalEncodeTime/framesEncoded_in_ms] 3.5333333333331516
totalEncodedBytesTarget 335009905
[totalEncodedBytesTarget/s] 3602.7091371103397
totalPacketSendDelay 20872.8
[totalPacketSendDelay/packetsSent_in_ms] 6.89655172416302
qualityLimitationReason bandwidth
qualityLimitationResolutionChanges 20
encoderImplementation libvpx
    Graph for Video Track in chrome://webrtc-internals

    Other RTP parameters

    RTX(regtranmission ) – packet loss recovery technique for real-time applications with relaxed delay bounds.

    Non WebRTC supported Video codecs

    Need active realtime media transcoding

    H.263

    Already used for video conferencing on PSTN (Public Switched Telephone Networks), RTSP, and SIP (IP-based videoconferencing) systems.
    suited for low bandwidth networks
    Although it is not comaptible with WebRTC but many media gateways incldue realtime transcoding existed between H263 based SIP systems and vp8 based webrtc ones to enable video communication between them

    H.265 / HEVC

    proprietary format and is covered by a number of patents. Licensing is managed by MPEG LA .

    Container – Mp4

    Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints

    With the rise of Internet of Things many Endpoints especially IP cameras connected to Raspberry Pi like SOC( system on chiops )n wanted to stream directly to the browser within theor own provate network or even on public network using TURN / STUN.

    The figure below shows how such a call flow is possible between an IP cemera ( such as Baby Cam ) and its parent monitoring it over a WebRTC suppported mobile phone browser . The process includes streaming teh content from IOT device on RTSP stream and using realtime trans-coding between H264 and VP8

    Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints

    Audio

    Audio Level

    audio level for speech transmission to avoid users having to manually adjust the playback and to facilitate mixing in conferencing applications.

    normalization considering frequencies above 300 Hz, regardless of the sampling rate used.

    adapted to avoid clipping, either by lowering the gain to a level below -19 dBm0 or through the use of a compressor.

    GAIN calculation

    • If the endpoint has control over the entire audio-capture path like a regular phone
      the gain should be adjusted in such a way that an average speaker would have a level of 2600 (-19 dBm0) for active speech.
    • If the endpoint does not have control over the entire audio capture like software endpoint
      then the endpoint SHOULD use automatic gain control (AGC) to dynamically adjust the level to 2600 (-19 dBm0) +/- 6 dB.
    • For music- or desktop-sharing applications, the level SHOULD NOT be automatically adjusted, and the endpoint SHOULD allow the user to set the gain manually.

    Acoustic Echo Cancellation (AEC)

    Endpoints shoudl allow echo control mechsnisms

    Codecs

    WebRTC endpoints are should implement audio codecs: OPUS and PCMA / PCMU, along with Comforrt Noise and DTMF events.

    Trace for audio codecs supported in chrome (Version 80.0.3987.149 (Official Build) (64-bit) on ubuntu)

    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    …
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtcp-fb:111 transport-cc
    a=fmtp:111 minptime=10;useinbandfec=1
    a=rtpmap:103 ISAC/16000
    a=rtpmap:104 ISAC/32000
    a=rtpmap:9 G722/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:106 CN/32000
    a=rtpmap:105 CN/16000
    a=rtpmap:13 CN/8000
    a=rtpmap:110 telephone-event/48000
    a=rtpmap:112 telephone-event/32000
    a=rtpmap:113 telephone-event/16000
    a=rtpmap:126 telephone-event/8000

    Opus

    stabdardised by IETF

    container- Ogg, WebM, MPEG-TS, MP4

    supportes multiple comptression algorithms

    For all cases where the endpoint is able to process audio at a sampling rate higher than 8 kHz, it is w3C recommenda that Opus be offered before PCMA/PCMU.

    AAC (Advanvced Audio Encoding)

    part of the MPEG-4 (H.264) standard
    supported congainers – MP4, ADTS, 3GP

    Lossy compression but has number pf profiles suiting each usecase like high quality surround sound to low-fidelity audio for speech-only use.

    G.711 (PCMA and PCMU)

    ITU published Pulse Code Modulation (PCM) with either µ-law or A-law encoding.
    vital to interface with the standard teelcom network and carriers

    Fixed 64Kbpd bit rate

    supports 3GP container formats

    G.711 PCM (A-law) is known as PCMA and G.711 PCM (µ-law) is known as PCMU

    G.722

    ncoded using Adaptive Differential Pulse Code Modulation (ADPCM) which is suited for voice compression
    conatiners used 3GP, AMR-WB

    Comfort noise (CN)

    artificial background noise which is used to fill gaps in a transmission instead of using pure silence

    avoids jarring or RTP Timeout

    for streams encoded with G.711 or any other supported codec that does not provide its own CN.
    Use of Discontinuous Transmission (DTX) / CN by senders is optional

    Internet Low Bitrate Codec (iLBC)

    opensource narrow band codec
    designed specifically for streaming voice audio

    Internet Speech Audio Codec (iSAC)

    designed for voice transmissions which are encapsulated within an RTP stream.

    DTMF and ‘audio/telephone-event’ media type

    endpoints may send DTMF events at any time and should suppress in-band dual-tone multi-frequency (DTMF) tones, if any.

    DTMF events list
    | 0 | DTMF digit “0”
    | 1 | DTMF digit “1”
    | 2 | DTMF digit “2”
    | 3 | DTMF digit “3”
    | 4 | DTMF digit “4”
    | 5 | DTMF digit “5”
    | 6 | DTMF digit “6”
    | 7 | DTMF digit “7”
    | 8 | DTMF digit “8”
    | 9 | DTMF digit “9”
    | 10 | DTMF digit “*”
    | 11 | DTMF digit “#”
    | 12 | DTMF digit “A”
    | 13 | DTMF digit “B”
    | 14 | DTMF digit “C”
    | 15 | DTMF digit “D”

    Stats for Audio Media track 

    timestamp 04/05/2020, 14:25:59
ssrc 3005719707
isRemote fals
mediaType audio
kind audio
trackId RTCMediaStreamTrack_sender_1
transportId RTCTransport_0_1
codecId RTCCodec_0_Outbound_111
[codec] opus (payloadType: 111)
mediaSourceId RTCAudioSource_1
packetsSent 88277
[packetsSent/s] 50.03762690431027
retransmittedPacketsSent 0
bytesSent 1977974
[bytesSent/s] 150.11288071293083
headerBytesSent 2118648
retransmittedBytesSent 0
    Graphs in chrome://webrtc-internals for Audio

    DataChannel

    m=application 9 UDP/DTLS/SCTP webrtc-datachannel
    c=IN IP4 0.0.0.0
    b=AS:30
    a=ice-ufrag:blj+
    a=ice-pwd:Ytdofc24WZYWRAnyNSNhuF4F
    a=ice-options:trickle
    a=fingerprint:sha-256 18:2F:B9:13:A1:BA:33:0C:D0:59:DB:83:9A:EA:38:0B:D7:DC:EC:50:20:6E:89:54:CC:E8:70:10:80:2B:8C:EE
    a=setup:active
    a=mid:2
    a=sctp-port:5000
    a=max-message-size:262144

    Stats for Datachannel

    Statistics RTCDataChannel_1
timestamp 04/05/2020, 14:25:59
label sctp
protocol
datachannelid 1
state open
messagesSent 1
[messagesSent/s] 0
bytesSent 228
[bytesSent/s] 0
messagesReceived 1
[messagesReceived/s] 0
bytesReceived 228
[bytesReceived/s] 0

    Refrenecs :

    Attacks on SIP Networks

    Major standards bodies including 3GPP, ITU-T, and ETSI have all adopted SIP as the core signalling protocol for services such as LTE , VoIP, conferencing, Video on Demand (VoD), IPTV (Internet Television), presence, and Instant Messaging (IM) etc. With the continous evolution of SIP as the defacto VoIP protocol , we need to underatdn the risk mitigartion practices around it .

    I have written about VoIP and security in these blogs before

    For Security around web browser based calling via webrtc i have written

    • Webrtc Security , which describes browser threat modal , access to local resource , Same Orogin Policy (SOP) and Cross Resource Sharing ( CORS) as well as Location sharing , ICE , TUEN and threats to privacy with screen sharing , microgone camera long term access and probable mid call attacks .
    • Genric secrutity of web Application build around hosting platform of webrtc. Includs concepts like Identity management , browser security – cross site security amd clickjacking , Authetication of devices and applications , Media Encryption and regex checking.

    Also Written about VoIP security at protocol level with SRTP /DTLS using TLS and specifically using available pre added security modules on kamailio SIP server. It describes Sanity checks , ACL lists with permissions , hiding topology details , countering Flood using pike and Fail2Ban as well as Traffic monitoring and detection .

    In this article we will cover types of attacks on SIP systems

    Types of attacks on SIP based systems

    Registration Hijacking

    malicious registrations on registrar by a third party who modifies From header field of a SIP request.

    exmaple implementation :
    attacker de-registers all existing contacts for a URI
    attacker can also register their own device as the appropriate contact address, thereby directing all requests for the affected user to him

    solution – Autheticaion of user

    Impersonating a Server

    attacker impersonates the remote server
    user’s request can now be intercepted by some other party
    user’s request may be forwarded to insecure locations

    Solution –

    confidentiality, integrity, and authentication of proxy servers

    Proxy/redirect sever, and registrars SHOULD possess a site certificate issued by CA which could be validated by UA

    Temparing Message bodies

    If users are relying on SIP message bodies to communicate either of

    • session encryption keys for a media session
    • MIME bodies
    • SDP
    • encapsulated telephony signals
      Then the atackers on proxy server can modify the session key or can act as a man-in-the-middle and do eaves droppng

    exmaple implementation :
    attacker can point RTP media streams to a wiretapping device
    can changes Subject header field to appear to users as spam

    solution – end to end ecryption over TLS + Digest Authorization

    mid-session threats like tearing down session

    Request forging
    attacker learns the params of the session like To , From tags etc then he can alter ongoing session parameters and even bring it down

    example implementation :
    attacker inserts a BYE in a ongoing session thereby tearing it down
    can insert re INVITE and redierct the stream to wiretaping device

    solution – authetication on every request
    signing and encrypting of MIME bodies, and transference of credentials with S/MIME

    Denial of Service and Amplification

    DOS attacks – rendering a particular network element unavailable, usually by directing an excessive amount of network traffic at its interfaces.
    dDOS – multiple network hosts to flood a target host with a large amount of network traffic.

    Can be created by sending falsified sip requests to other parties such that numerous transactions originating in the backwards direction comes to the target server created congestion.

    exmaple implementation :
    attackers creates a falsified source IP address and a corresponding Via header field that identify a targeted host as the originator of the request. Then send this to large number of SIP network element . This geneerates DOS aimed at target.

    attackers uses falsified Route header field values in a request that identify the target host and then send such messages to forking proxies that will amplify messaging sent to the target.

    Flooding with register attacks can deplete available memory and disk resources of a registrar by registering huge numbers of bindings.
    Flooding a stateful proxy server causes it to consume computational expense associated with processing a SIP transaction

    Solution –
    detect flooding and pike in traffic and use ipban to block
    challenge questionable requests with only a single 401 (Unauthorized) or 407 (Proxy Authentication Required), forgoing the normal response retransmission algorithm, and thus behaving statelessly towards unauthenticated requests.

    Security mchanisms

    Full encryption vs hop by hop encrption

    SIP mssages cannot be encrypted end-to-end in their entirety since
    message fields such as the Request-URI, Route, and Via need to be visible to proxies in most network architectures
    so that SIP requests are routed correctly.
    proxy servers need to also update the message with via headers

    Thus SIP uses low level security along with hop by hop encrption and auth headers to verify the identity of proxy servers

    Transport and Network Layer Security

    IPsec – used where set of hosts or administrative domains have an existing trust relationship with one another.

    TLS – used where hop-by-hop security is required between hosts with no pre-existing trust association.

    SIPS URI Scheme

    Used as an address-of-record for a particular user, signifies that each hop over which the request is forwarded, must be secured with TLS

    HTTP Authentication

    Reuse of the HTTP Digest authentication via 401 and 407 response codes that implement challenge for autehtication
    provides replay protection and one-way authentication.

    S/MIME

    allows SIP UAs to encrypt MIME bodies within SIP, securing these bodies end-to-end without affecting message headers.
    provides end-to-end confidentiality and integrity for message bodies

    nonce-count

    provides replay protection

    SIP over TLS

    SIP messages can be secured using TLS. There is also TLS for Datagrams called DTLS.

    Security of SIP signalling is different from security of protocols used in concert with SIP like RTP , RTCP. and that will be covered in later topics of this article.

    TLS operation consists of two phases: handshake phase and bulk data encryption phase

    Handshake phase

    Prepare algorithm to be used during TLS session

    Server Authentication

    server sends its certificate to the client, which then verifies the certificate using a certificate authority’s (CA’s) public key.

    Client Authentication

    Server sends an additional CertificateRequest message to request the client’s certificate. The client responds with

    1. Certificate message containing the client certificate with the client public key and
    2. CertificateVerify message containing a digest signature of the handshake messages signed by clients private key

    Server authenticates client by client’s public key , since only client holding correct private key can sign the message.

    prepare the shared secret for bulk data encryption

    client generate a pre_master_secret, and encrypt it using the server’s public key obtained from the server’s certificate. The server decrypts the pre_master_secret using its own private key.
    Both the server and client then compute a master_secret they share based on the same pre_master_secret. The master_secret is further used to generate the shared symmetric keys for bulk data encryption and message authentication

    Public key cryptographic operations such as RSA are much more expensive than shared key cryptography. This is why TLS uses public key cryptography to establish the shared secret key in the handshake phase, and then uses symmetric key cryptography with the negotiated shared secret as the data encryption key.

    Stateless proxy servers do not maintain state information about the SIP session and therefore tend to be more scalable. However, many standard application functionalities, such as authentication, authorization, accounting, and call forking require the proxy server to operate in a stateful
    mode by keeping different levels of session state information.

    Steps :

    1. The SIP proxy server enforces proxy authentication with
      407 Proxy Authentication Required challenge.
    2. UAC provides credentials that verify its claimed identity (e.g., based on MD5 [34] digest algorithm) and retransmits in authorization header

    Security of RTP

    confidentiality protection of the RTP session and integrity protection of the RTP/RTCP packets requires source authentication of all the packets to ensure no man-in-the-middle (MITM) attack is taking place.

    end to end media encryption – SRTP ( Secure RTP )

    encodes the voice into encrypted IP packages and transport those via the internet from the transmitter  to receive

    References

    • The Impact of TLS on SIP Server Performance – Charles Shen† Erich Nahum‡ Henning Schulzrinne† Charles Wright , Department of Computer Science, Columbia University,IBM T.J. Watson Research Center

    HTTP/2 – offer/answer signaling for WebRTC call

    HTTP ( Hyper Text Transfer Protocol ) is the top application layer protocol atop the Tarnsport layer ( TCP ) and the Network layer ( IP )

    HTTP/1.1

    release in 1997. Since HTTP/1 allowed only 1 req at a time , HTTP/1.1

    Allows one one outstanding connection on a TCP session but allowed request pieplinig to achieve concurency.

    HTTP/2

    In 2015, HTTP/2 was released which aimed at reducing latency while delivering heavy graphics, videos and other media cpmponents on web page especially on mobile sites .
    optimizes server push and service workers

    FRAMES

    A key differenet between Http/1.1 and HTTP/2 is the fact that former transmites requests and reponses in plaintext whereas the later encapsulates them into binary format , proving more features and scope for optimzation.

    Thus at protocol level , it is all about frames of bytes which are part of stream.

    “enables a more efficient use of network resources and a reduced perception of latency by introducing header field compression and allowing multiple concurrent exchanges on the same connection. It also introduces unsolicited push of representations from servers to clients.”

    Hypertext Transfer Protocol Version 2 (HTTP/2) draft-ietf-httpbis-http2-latest
    https://http2.github.io/http2-spec/

    It is important to know that Browsers only implement HTTP/2 under HTTPS, thus TLS connection is must for whichw e need certs ad keys signed by CA ( either self signed using openssl , signed by public CA like godaddy , verisign or letsencrypt)

    Compatibility Layer between HTTP1.1 and HTTP2.0 in node

    Nodejs >9 provides http2 as native module. Exmaple of using http2 with compatibility layer

    const http2 = require('http2');
const options = {
 key: 'ss/key', // path to key
 cert: 'ssl/cert' // path to cert
};

const server = http2.createSecureServer(options, (req, res) => {
    req.addListener('end', function () {
        file.serve(req, res);
    }).resume();
});
server.listen(8084);

    in replacement for existing server http/https server

    const https = require('https');
app = https.createServer(options, function (request, response) {
    request.addListener('end', function () {
        file.serve(request, response);
    }).resume();
});

app.listen(8084);

    Socket.io/ Websocket over HTTP2

    The WebSocket Protocol uses the HTTP/1.1 Upgrade mechanism to transition a TCP connection from HTTP into a WebSocket connection

    Due to its multiplexing nature, HTTP/2 does not allow connection-wide header fields or status codes, such as the Upgrade and Connection request-header fields or the 101 (Switching Protocols) response code. These are all required for opening handshake.

    Ideally the code shouldve looekd like this with backward compatiability layer , but continue reading update ..

    var app = http2
    .createSecureServer(options, (req, res) => {
        req.addListener('end', function () {
            file.serve(req, res);
        }).resume();
    })
    .listen(properties.http2Port);

var io = require('socket.io').listen(app);
io.origins('*:*');
io.on('connection', onConnection); // evenet handler onconnection

    Error during WebSocket handshake: Unexpected response code: 403

    update May 2020 : I tried using the http2 server with websocket like mentioned above ,h owever many many hours of working around WSS over HTTP2 secure server , I consistencly kept faccing the ECONNRESET issues after couple of seconds , which would crash the server

    client 403
    server ECONNRESET

    Therefore leaving the web server to server htmll conetnt I reverted the siganlling back to HTTPs/1.1 given the reasons for sticking with WSS is low latency and existing work that was already put in.

    Example Repo : https://github.com/altanai/webrtcdevelopment/tree/htt2.0

    Reading Further of exploring HTTP CONNECT methods for setting WS handshake . Will update this section in future if it works .

    Streams

    A “stream” is an independent, bidirectional sequence of frames exchanged between the client and server within an HTTP/2 connection.
    A single HTTP/2 connection can contain multiple concurrently open streams, with either endpoint interleaving frames from multiple streams.

    Core http2 module provides new core API (Http2Stream), accessed via a “stream” listener:

    const http2 = require('http2');
const options = {
 key: 'ss/key', // path to key
 cert: 'ssl/cert' // path to cert
};

const server = http2.createSecureServer(options, (stream, headers) => {
    stream.respond({ ':status': 200 });
    stream.end('some text!');
});
server.listen(3000);

    Other features

    • stream multiplexing
    • stream Prioritization
    • header compression
    • Flow Control
    • support for trailer

    Persistent , one connection per origin.

    With the new binary framing mechanism in place, HTTP/2 no longer needs multiple TCP connections to multiplex streams in parallel; each stream is split into many frames, which can be interleaved and prioritized. As a result, all HTTP/2 connections are persistent, and only one connection per origin is required,

    Server Push

    bundling multiple assets and resources into a single HTTP/2  and lets the srever proactively push resources to client’s cache .

    Server issues PUSH_PROMISE , client validates whether it needs the resource of not. If the client matches it then they will load like regular GET call

    The PUSH_PROMISE frame includes a header block that contains a complete set of request header fields that the server attributes to the request.

    After sending the PUSH_PROMISE frame, the server can begin delivering the pushed response as a response on a server-initiated stream that uses the promised stream identifier.

    Client receives a PUSH_PROMISE frame and can either chooses to accept the pushed response or if it does not wish to receive the pushed response from the server it can can send a RST_STREAM frame, using either the CANCEL or REFUSED_STREAM code and referencing the pushed stream’s identifier.

    Push Stream Support

    -tbd

    respondWithFile() and respondWithFD() APIs can send raw file data that bypasses the Streams API.

    Related technologies

    MIME

    Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of email messages to support text in character sets other than ASCII, as well as attachments of audio, video, images, and application programs. Message bodies may consist of multiple parts, and header information may be specified in non-ASCII character sets.

    Email messages + MIME : transmitted with standard protocols, such as the Simple Mail Transfer Protocol (SMTP), the Post Office Protocol (POP), and the Internet Message Access Protocol (IMAP).

    MIME in HTTP in WWW : servers insert a MIME header field at the beginning of any Web transmission. Clients use the content type or media type header to select an appropriate viewer application for the type of data indicated. Browsers typically contain GIF and JPEG image viewers.

    MIME header fields

    MIME version

    MIME-Version: 1.0

    Content Type

    Content-Type: text/plain

    multipart/mixed , text/html, image/jpeg, audio/mp3, video/mp4, and application/msword

    content disposition

    Content-Disposition: attachment; filename=genome.jpeg;
  modification-date="Wed, 12 Feb 1997 16:29:51 -0500";

    Content-Transfer-Encoding

    References :

    Certificates, compliances and Security in VoIP

    This article describes various Certificates and compliances, Bill and Acts on data privacy, Security and prevention of Robocalls as adopted by countries around the world pertaining to Interconnected VoIP providers, telecommunications services, wireless telephone companies etc

    Compliance certificates by Industry types

    HIPAA (Health Insurance Portability and Accountability Act)

    Deals with privacy and security of personal medical records and electronic health care transaction

    Applicability  : If voip company handles medical information

    Includes : 

    • Not allowed Voice mail transcription
    • Should have End-to-End Encryption
    • Restrict  using unsecured WiFi networks to prevent Snooping
    • User security , strong password rules  and mandatory monthly change
    • Secure Firmware on VoIP phones
    • Maintaining Call and Access Logs

    SOX( Sarbanes Oxley Act of 2002)

    Also known as SOX, SarbOX or Public Company Accounting Reform and Investor Protection Act

    Applicability : if managing the communications operations of a regulated, publicly traded company 

    Includes : 

    • Retain records which include financial and other sensitive data
    • ways employees are provided or denied access to records or data based on their roles and responsibilities
    • do information audit by a trusted third party. 
    • Retention and deletion of files such as audio files like voicemails, text messages, video clips, declared paper records, storage, and logs of communications activities
    • Physical and digital security controls around cloud-based VoIP applications and the networks

    Privacy Related Compliance certificates

    COPPA (Children’s Online Privacy Protection Act ) of 1998 

    prohibits deceptive marketing to children under the age of 13, or collecting personal information without disclosure to their parents. 

    any information is to be passed on to a third party, must be easy for the child’s guardian to review and/or protect

    2011 amendment  requires that the data collected was erased after a period of time,

    2014 FTC issued guidelines that apps and app stores require “verifiable parental consent.”

    CPNI (Customer Proprietary Network Information) 2007

    CPNI (Customer Proprietary Network Information) in united states is the information that communication providers  acquire about their subscribers. This Individually identifiable information that is created by a customer’s relationship with a provider, such as data about the frequency, duration, and timing of calls, the information on a customer’s bill, and call identifying information. This processing information is governed strictly by FCC and certification should be renewed on an annual basis

    Provider can pass along that information to marketers to sell other services, as long as the customer is notified

    In 2007, the FCC explicitly extended the application of the Commission’s CPNI rules of the Telecommunications Act of 1996 to providers of interconnected VoIP service.

    CALEA

    Communications Assistance for Law Enforcement Act (CALEA) conduct electronic surveillance by imposing specific obligations on “telecommunications carriers” for assisting law enforcement, including delivering call interception and call identification functionality to the government with a minimum of interference to customer service and privacy.

    Read more about CALEA and its roles in VoIP here Regulatory and Legal Considerations with WebRTC development

    GDPR (General Data Protection Regulation)  in European Union 2018

    Supersedes the 1995 Data Protection Directive

    Establishes requirements of organizations that process data, defines the rights of individuals to manage their data, and outlines penalties for those who violate these rights.

    No personal data may be processed unless this processing is done under one of six lawful bases specified by the regulation (consent, contract, public task, vital interest, legitimate interest or legal requirement). When the processing is based on consent the data subject has the right to revoke it at any time.

    Controllers must notify Supervising Authorities (SA)s of a personal data breach within 72 hours of learning of the breach.

    California Consumer Privacy Act (CCPA) 2019

    consumer rights relating to the access to, deletion of, and sharing of personal information that is collected by businesses. 

    Allows consumers to know whether their personal data is sold or disclosed , to whom .

    Allows opt-out right for sales of personal information

    Right to deletion – to request a business to delete any personal information about a consumer collected from that consumer

    Personal Data Protection Bill (PDP) – India 2018

    This bill introduces various private and sensitive protection frameworks  like restriction on retention of personal data, Right to correction and erasure (such as right to be forgotten) , Prohibition and transparency of processing of personal data. It also classifies data fiduciaries  including certain social media intermediaries. 

    The Bill amends the Information Technology Act, 2000 to delete the provisions related to compensation payable by companies for failure to protect personal data.

    Other data privacy acts similar to GDPR 

    • South Korea’s Personal Information Protection Act  2011
    • Brazil’s Lei Geral de Proteçao de Dados (LGPD)  2020
    • Privacy Amendment (Notifiable Data Breaches) to Australia’s Privacy Act 2018
    • Japan’s Act on Protection of Personal Information 2017
    • Thailand Personal Data Protection Act (PDPA) 2020

    Features offered by VOIP companies for Data privacy 

    • Access Control & Logging
    • Auto Data Redaction / Account Deletion policy 
    • SIEM (Security information and event management) alerts 
    • Information security , Encrypted Storage For Recordings & Transcripts
    • Disclosing all third party services that are involved in data processing too
    • Role Based Access Control and 2 Factor Authentication
    • Data Security Audits and appointing  data protection officer to oversee GDPR compliance

    Against Robocalls and SPIT ( SPAM over Internet Telephony)

     2009 Truth in Caller ID Act 

    Telephone Consumer Protection Act of 1991

    Implementation of Do not call registry against use of robocalls, automatic dialers, and other methods of communication

    Do-Not-Call Implementation Act of 2003

    if a business has an established relationship with a customer, it can continue to call them for up to 18 months. If a consumer calls the company, say, to ask for information about the product or service, the company has three months to get back to him.

    if the customer asks to not receive calls, the company must stop calling, or be subject to fines.

    Exemptions – Calls from a not-for-profit B organisation , informational messages as flight cancellations , Calls from sales and debt collectors etc

    Personal Data Privacy and Security Act 2009

    Implemented to curb  identity theft and computer hacking. Sensitive personal identifiable information includes : victim’s name, social security number, home address, fingerprint/biometrics data, date of birth, and bank account numbers.

    Any company that is breached must notify the affected individuals by mail, telephone, or email, and the message must include information on the company and how to get in touch with credit reporting agencies

    If the breach involves government or national security , company must also contact the Secret Service within fourteen days 

    TRACED Act (Telephone Robocall Abuse Criminal Enforcement and Deterrence) 2019

    Canadian Radio-television and Telecommunications Commission (CRTC) 2018 -32

    A solution mechanism has already been standardised and active in adoption called STIR / SHAKEN ( Secure Telephony Identity Revisited / Signature-based Handling of Asserted information using toKENs) described in another article here.

    Emergency services 

    FCC E911 E911 / VoIP E911 rules

    Unlike traditional telephone connections, which are tied to a physical location, VOIP’s packet switched technology allows a particular number to be anywhere making it more difficult for it to reach localised services like emergency numbers of Public Safety Answering Points (PSAPs) . Thus FCC regulations as well as the New and Emerging Technologies 911 Improvement Act of 2008 (NET 911 Act), interconnected VoIP providers are required to provide 911 and E911 service. 

    …

    Ref : 

    WebRTC APIs

    Contents

    • Media Capture and streams
    • Peer to peer Connection-
      • RTCPeerConnection, RTCConfiguration, ICE, Offer/Answer, states
    • RTP Media API
      • RTCRtpSender
      • RTCRtpReceiver
      • RTCRtpTransreciver
    • RTCDtlsTransport
    • RTCIceCandidate
      • ICE gathering
    • RTCIceTransport Interface
    • Peer-to-peer Data API
    • Peer-to-peer DTMF
    • Statistics

    Peer-to-peer connections

    creates p2p communication channel

    RTCConfiguration Dictionary

    dictionary RTCConfiguration {
  sequence<RTCIceServer> iceServers;
  RTCIceTransportPolicy iceTransportPolicy;
  RTCBundlePolicy bundlePolicy;
  RTCRtcpMuxPolicy rtcpMuxPolicy;
  DOMString peerIdentity;
  sequence<RTCCertificate> certificates;
  [EnforceRange] octet iceCandidatePoolSize = 0;
};

    RTCIceCredentialType Enum

    enum RTCIceCredentialType {
  "password",
  "oauth"
};

    supports OAuth 2.0 based authentication. The application, acting as the OAuth Client, is responsible for refreshing the credential information and updating the ICE Agent with fresh new credentials before the accessToken expires. The OAuth Client can use the RTCPeerConnection setConfiguration method to periodically refresh the TURN credentials.

    RTCOAuthCredential Dictionary
    describe the OAuth auth credential information which is used by the STUN/TURN client (inside the ICE Agent) to authenticate against a STUN/TURN server

    dictionary RTCOAuthCredential {
  required DOMString macKey;
  required DOMString accessToken;
};

    RTCIceServer Dictionary

    Describe the STUN and TURN servers that can be used by the ICE Agent to establish a connection with a peer.

    dictionary RTCIceServer { required (DOMString or sequence<DOMString>) urls; DOMString username; (DOMString or RTCOAuthCredential) credential; RTCIceCredentialType credentialType = "password"; };

    Example :

     [{urls: 'stun:stun1.example.net'},
  {urls: ['turns:turn.example.org', 'turn:turn.example.net'],
    username: 'user',
    credential: 'myPassword',
    credentialType: 'password'},
  {urls: 'turns:turn2.example.net',
    username: '22BIjxU93h/IgwEb',
    credential: {
      macKey: 'WmtzanB3ZW9peFhtdm42NzUzNG0=',
      accessToken: 'AAwg3kPHWPfvk9bDFL936wYvkoctMADzQ5VhNDgeMR3+ZlZ35byg972fW8QjpEl7bx91YLBPFsIhsxloWcXPhA=='
    },
    credentialType: 'oauth'}
];

    RTCIceTransportPolicy Enum

    ICE candidate policy [JSEP] to select candidates for the ICE connectivity checks

    • relay – use only media relay candidates such as candidates passing through a TURN server. It prevents the remote endpoint/unknown caller from learning the user’s IP addresses
    • all – ICE Agent can use any type of candidate when this value is specified.

    RTCBundlePolicy Enum

    • balanced – Gather ICE candidates for each media type (audio, video, and data). If the remote endpoint is not bundle-aware, negotiate only one audio and video track on separate transports.
    • max-compat – Gather ICE candidates for each track. If the remote endpoint is not bundle-aware, negotiate all media tracks on separate transports.
    • max-bundle – Gather ICE candidates for only one track. If the remote endpoint is not bundle-aware, negotiate only one media track. If the remote endpoint is bundle-aware, all media tracks and data channels are bundled onto the same transport.

    If the value of configuration.bundlePolicy is set and its value differs from the connection’s bundle policy, throw an InvalidModificationError.

    RTCRtcpMuxPolicy Enum

    what ICE candidates are gathered to support non-multiplexed RTCP.

    • negotiate – Gather ICE candidates for both RTP and RTCP candidates. If the remote-endpoint is capable of multiplexing RTCP, multiplex RTCP on the RTP candidates. If it is not, use both the RTP and RTCP candidates separately.
    • require – Gather ICE candidates only for RTP and multiplex RTCP on the RTP candidates. If the remote endpoint is not capable of rtcp-mux, session negotiation will fail.

    If the value of configuration.rtcpMuxPolicy is set and its value differs from the connection’s rtcpMux policy, throw an InvalidModificationError. If the value is “negotiate” and the user agent does not implement non-muxed RTCP, throw a NotSupportedError.

    Offer/Answer Options – VoiceActivityDetection

    dictionary RTCOfferAnswerOptions {
  boolean voiceActivityDetection = true;
};

    capable of detecting “silence” 

    dictionary RTCOfferOptions : RTCOfferAnswerOptions {
  boolean iceRestart = false;
};

    dictionary RTCAnswerOptions : RTCOfferAnswerOptions {};

    An RTCPeerConnection object has a signaling state, a connection state, an ICE gathering state, and an ICE connection state.

    RTCSignalingState Enum

    “stable”,
    “have-local-offer”,
    “have-remote-offer”,
    “have-local-pranswer”,
    “have-remote-pranswer”,
    “closed”

    RTCIceGatheringState Enum

    “closed”,
    “failed”,
    “disconnected”,
    “new”,
    “connecting”,
    “connected”

    RTCIceConnectionState Enum

    “closed”,
    “failed”,
    “disconnected”,
    “new”,
    “checking”,
    “completed”,
    “connected”

    An RTCPeerConnection object has an operations chain which ensures that only one asynchronous operation in the chain executes concurrently.

    Also an RTCPeerConnection object MUST not be garbage collected as long as any event can cause an event handler to be triggered on the object. When the object’s internal slot is true ie closed, no such event handler can be triggered and it is therefore safe to garbage collect the object.

    RTCPeerConnection Interface

    interface RTCPeerConnection : EventTarget {

Promise<RTCSessionDescriptionInit> createOffer(optional RTCOfferOptions options);

Promise<RTCSessionDescriptionInit> createAnswer(optional RTCAnswerOptions options);

Promise<void> setLocalDescription(optional RTCSessionDescriptionInit description);
  
readonly attribute RTCSessionDescription? localDescription;
readonly attribute RTCSessionDescription? currentLocalDescription;
readonly attribute RTCSessionDescription? pendingLocalDescription;
  
Promise<void> setRemoteDescription(optional RTCSessionDescriptionInit description);
  
readonly attribute RTCSessionDescription? remoteDescription;
readonly attribute RTCSessionDescription? currentRemoteDescription;
readonly attribute RTCSessionDescription? pendingRemoteDescription;

Promise<void> addIceCandidate(optional RTCIceCandidateInit candidate);
readonly attribute RTCSignalingState signalingState;
readonly attribute RTCIceGatheringState iceGatheringState;
readonly attribute RTCIceConnectionState iceConnectionState;
readonly attribute RTCPeerConnectionState connectionState;
readonly attribute boolean? canTrickleIceCandidates;
void restartIce();
static sequence<RTCIceServer> getDefaultIceServers();

RTCConfiguration getConfiguration();
void setConfiguration(RTCConfiguration configuration);
void close();

attribute EventHandler onnegotiationneeded;
attribute EventHandler onicecandidate;
attribute EventHandler onicecandidateerror;
attribute EventHandler onsignalingstatechange;
attribute EventHandler oniceconnectionstatechange;
attribute EventHandler onicegatheringstatechange;
attribute EventHandler onconnectionstatechange;
};

    CreateOffer() – generates a blob of SDP that contains an RFC 3264 offer with the supported configurations for the session, including

    • descriptions of the local MediaStreamTracks attached to this RTCPeerConnection,
    • codec/RTP/RTCP capabilities
    • ICE agent (usernameFragment, password , local candiadtes etc )
    • DTLS connection
    var pc = new RTCPeerConnection();
pc.createOffer({
     mandatory: {
        OfferToReceiveAudio: true,
        OfferToReceiveVideo: true
    },
    optional: [{
        VoiceActivityDetection: false
    }]
}).then(function(offer) {
	return pc.setLocalDescription(offer);
})
.then(function() {
// Send the offer to the remote through signaling server
})
.catch(handleError);

    CreateAnswer() – generates an SDPanswer with the supported configuration for the session that is compatible with the parameters in the remote configuration

    var pc = new RTCPeerConnection();
pc.createAnswer({
  OfferToReceiveAudio: true
  OfferToReceiveVideo: true
})
.then(function(answer) {
  return pc.setLocalDescription(answer);
})
.then(function() {
  // Send the answer to the remote through signaling server
})
.catch(handleError);

    Codec preferences of an m= section’s associated transceiver is said to be the value of the RTCRtpTranceiver with the following filtering applied

    • If direction is “sendrecv”, exclude any codecs not included in the intersection of RTCRtpSender.getCapabilities(kind).codecs and RTCRtpReceiver.getCapabilities(kind).codecs.
    • If direction is “sendonly”, exclude any codecs not included in RTCRtpSender.getCapabilities(kind).codecs.
    • If direction is “recvonly”, exclude any codecs not included in RTCRtpReceiver.getCapabilities(kind).codecs.

    Legacy Interface Extensions

    partial interface RTCPeerConnection {
 
Promise<void> createOffer(
 RTCSessionDescriptionCallback successCallback,
 RTCPeerConnectionErrorCallback failureCallback,
 optional RTCOfferOptions options);
 
Promise<void> setLocalDescription(
optional RTCSessionDescriptionInit description,
VoidFunction successCallback,                                  RTCPeerConnectionErrorCallback failureCallback);

Promise<void> createAnswer(
RTCSessionDescriptionCallback successCallback,
RTCPeerConnectionErrorCallback failureCallback);
  
Promise<void> setRemoteDescription(
optional RTCSessionDescriptionInit description,
VoidFunction successCallback,
                                     RTCPeerConnectionErrorCallback failureCallback);
  
Promise<void> addIceCandidate(
RTCIceCandidateInit candidate,
VoidFunction successCallback,
RTCPeerConnectionErrorCallback failureCallback);
};

    Session Description Model

    enum RTCSdpType {
  "offer",
  "pranswer",
  "answer",
  "rollback"
};

    interface RTCSessionDescription {
  readonly attribute RTCSdpType type;
  readonly attribute DOMString sdp;
  [Default] object toJSON();
};

    dictionary RTCSessionDescriptionInit {
  RTCSdpType type;
  DOMString sdp = "";
};

    RTCPriorityType
    Priority and QoS Model which can be

    “very-low”,
    “low”,
    “medium”,
    “high”

    RTP Media API

    Send and receive MediaStreamTracks over a peer-to-peer connection.
    Tracks, when added to an RTCPeerConnection, result in signaling; when this signaling is forwarded to a remote peer, it causes corresponding tracks to be created on the remote side.

    The actual encoding and transmission of MediaStreamTracks is managed through objects called RTCRtpSenders. Similarly, the reception and decoding of MediaStreamTracks is managed through objects called RTCRtpReceivers. These are associated with one track.

    RTCRtpTransceivers are created implicitly when the application attaches a MediaStreamTrack to an RTCPeerConnection via the addTrack(), or explicitly when the application uses the addTransceiver(). They are also created when a remote description is applied that includes a new media description. 

    rtpTransceiver = RTCPeerConnection.addTransceiver(trackOrKind, init);

    trackOrKind should be either audio or video othereise a TypeError is thrown

    init is optiona . It can contain direction , sendEncodings , streams

    RTCPeerConnection Interface

    partial interface RTCPeerConnection {
  
sequence<RTCRtpSender> getSenders();
sequence<RTCRtpReceiver> getReceivers();
sequence<RTCRtpTransceiver> getTransceivers();

RTCRtpSender addTrack(MediaStreamTrack track, MediaStream... streams);
void removeTrack(RTCRtpSender sender);
RTCRtpTransceiver addTransceiver((MediaStreamTrack or DOMString) trackOrKind, optional RTCRtpTransceiverInit init);
attribute EventHandler ontrack;
};

    RTCRtpTransceiverInit

    dictionary RTCRtpTransceiverInit {
  RTCRtpTransceiverDirection direction = "sendrecv";
  sequence<MediaStream> streams = [];
  sequence<RTCRtpEncodingParameters> sendEncodings = [];
};

    RTCRtpTransceiverDirection can be either of

    “sendrecv”,
    “sendonly”,
    “recvonly”,
    “inactive”,
    “stopped”

    RTCRtpSender Interface

    Allows an application to control how a given MediaStreamTrack is encoded and transmitted to a remote peer. 

    interface RTCRtpSender {
readonly attribute MediaStreamTrack? track;
readonly attribute RTCDtlsTransport? transport;
readonly attribute RTCDtlsTransport? rtcpTransport;
static RTCRtpCapabilities? getCapabilities(DOMString kind);
Promise<void> setParameters(RTCRtpSendParameters parameters);
RTCRtpSendParameters getParameters();
Promise<void> replaceTrack(MediaStreamTrack? withTrack);
void setStreams(MediaStream... streams);
Promise<RTCStatsReport> getStats();
};

    RTCRtpParameters Dictionary

    dictionary RTCRtpParameters {
  required sequence<RTCRtpHeaderExtensionParameters> headerExtensions;
  required RTCRtcpParameters rtcp;
  required sequence<RTCRtpCodecParameters> codecs;
};

    RTCRtpSendParameters Dictionary

    dictionary RTCRtpSendParameters : RTCRtpParameters {
  required DOMString transactionId;
  required sequence<RTCRtpEncodingParameters> encodings;
  RTCDegradationPreference degradationPreference = "balanced";
  RTCPriorityType priority = "low";
};

    RTCRtpReceiveParameters Dictionary

    dictionary RTCRtpReceiveParameters : RTCRtpParameters {
  required sequence<RTCRtpDecodingParameters> encodings;
};

    RTCRtpCodingParameters Dictionary

    dictionary RTCRtpCodingParameters {
  DOMString rid;
};

    RTCRtpDecodingParameters Dictionary

    dictionary RTCRtpDecodingParameters : RTCRtpCodingParameters {};

    RTCRtpEncodingParameters Dictionary

    dictionary RTCRtpEncodingParameters : RTCRtpCodingParameters {
  octet codecPayloadType;
  RTCDtxStatus dtx;
  boolean active = true;
  unsigned long ptime;
  unsigned long maxBitrate;
  double maxFramerate;
  double scaleResolutionDownBy;
};

    RTCDtxStatus Enum

    disabled- Discontinuous transmission is disabled.
    enabled- Discontinuous transmission is enabled if negotiated.

    RTCDegradationPreference Enum

    enum RTCDegradationPreference {
  "maintain-framerate",
  "maintain-resolution",
  "balanced"
};

    RTCRtcpParameters Dictionary

    dictionary RTCRtcpParameters {
  DOMString cname;
  boolean reducedSize;
};

    RTCRtpHeaderExtensionParameters Dictionary

    dictionary RTCRtpHeaderExtensionParameters {
  required DOMString uri;
  required unsigned short id;
  boolean encrypted = false;
};

    RTCRtpCodecParameters Dictionary

    dictionary RTCRtpCodecParameters {
  required octet payloadType;
  required DOMString mimeType;
  required unsigned long clockRate;
  unsigned short channels;
  DOMString sdpFmtpLine;
};

    payloadType – identify this codec.
    mimeType – codec MIME media type/subtype. Valid media types and subtypes are listed in [IANA-RTP-2]
    clockRate – expressed in Hertz
    channels – number of channels (mono=1, stereo=2).
    sdpFmtpLine – “format specific parameters” field from the “a=fmtp” line in the SDP corresponding to the codec

    RTCRtpCapabilities Dictionary

    dictionary RTCRtpCapabilities {
  required sequence<RTCRtpCodecCapability> codecs;
  required sequence<RTCRtpHeaderExtensionCapability> headerExtensions;
};

    RTCRtpCodecCapability Dictionary

    dictionary RTCRtpCodecCapability {
  required DOMString mimeType;
  required unsigned long clockRate;
  unsigned short channels;
  DOMString sdpFmtpLine;
};

    RTCRtpHeaderExtensionCapability Dictionary

    dictionary RTCRtpHeaderExtensionCapability {
  DOMString uri;
};

    Example JS code to RTCRtpCapabilities

    const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
const capabilities = RTCRtpSender.getCapabilities('audio');

    Output :

    codecs: Array(13)
0: {channels: 2, clockRate: 48000, mimeType: "audio/opus", sdpFmtpLine: "minptime=10;useinbandfec=1"}
1: {channels: 1, clockRate: 16000, mimeType: "audio/ISAC"}
2: {channels: 1, clockRate: 32000, mimeType: "audio/ISAC"}
3: {channels: 1, clockRate: 8000, mimeType: "audio/G722"}
4: {channels: 1, clockRate: 8000, mimeType: "audio/PCMU"}
5: {channels: 1, clockRate: 8000, mimeType: "audio/PCMA"}
6: {channels: 1, clockRate: 32000, mimeType: "audio/CN"}
7: {channels: 1, clockRate: 16000, mimeType: "audio/CN"}
8: {channels: 1, clockRate: 8000, mimeType: "audio/CN"}
9: {channels: 1, clockRate: 48000, mimeType: "audio/telephone-event"}
10: {channels: 1, clockRate: 32000, mimeType: "audio/telephone-event"}
11: {channels: 1, clockRate: 16000, mimeType: "audio/telephone-event"}
12: {channels: 1, clockRate: 8000, mimeType: "audio/telephone-event"}
length: 13

headerExtensions: Array(6)
0: {uri: "urn:ietf:params:rtp-hdrext:ssrc-audio-level"}
1: {uri: "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"}
2: {uri: "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"}
3: {uri: "urn:ietf:params:rtp-hdrext:sdes:mid"}
4: {uri: "urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id"}
5: {uri: "urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id"}
length: 6

    RTCRtpReceiver Interface

    allows an application to inspect the receipt of a MediaStreamTrack.

    interface RTCRtpReceiver {
  readonly attribute MediaStreamTrack track;
  readonly attribute RTCDtlsTransport? transport;
  readonly attribute RTCDtlsTransport? rtcpTransport;
  static RTCRtpCapabilities? getCapabilities(DOMString kind);
  RTCRtpReceiveParameters getParameters();
  sequence<RTCRtpContributingSource> getContributingSources();
  sequence<RTCRtpSynchronizationSource> getSynchronizationSources();
  Promise<RTCStatsReport> getStats();
};

    dictionary RTCRtpContributingSource 

    dictionary RTCRtpContributingSource {
  required DOMHighResTimeStamp timestamp;
  required unsigned long source;
  double audioLevel;
  required unsigned long rtpTimestamp;
};

    dictionary RTCRtpSynchronizationSource 

    dictionary RTCRtpSynchronizationSource : RTCRtpContributingSource {
  boolean voiceActivityFlag;
};

    voiceActivityFlag of type boolean – Only present for audio receivers. Whether the last RTP packet, delivered from this source, contains voice activity (true) or not (false).

    RTCRtpTransceiver Interface

    Each SDP media section describes one bidirectional SRTP (“Secure Real Time Protocol”) stream. RTCRtpTransceiver describes this permanent pairing of an RTCRtpSender and an RTCRtpReceiver, along with some shared state. It is uniquely identified using its mid property.

    Thus it is combination of an RTCRtpSender and an RTCRtpReceiver that share a common mid. An associated transceiver( with mid) is one that’s represented in the last applied session description.

    interface RTCRtpTransceiver {

readonly attribute DOMString? mid;
[SameObject] readonly attribute RTCRtpSender sender;
[SameObject] readonly attribute RTCRtpReceiver receiver;

attribute RTCRtpTransceiverDirection direction;
readonly attribute RTCRtpTransceiverDirection? currentDirection;

void stop();

void setCodecPreferences(sequence<RTCRtpCodecCapability> codecs);
};

    Method stop() – Irreversibly marks the transceiver as stopping, unless it is already stopped. This will immediately cause the transceiver’s sender to no longer send, and its receiver to no longer receive.
    stopping transceiver will cause future calls to createOffer to generate a zero port in the media description for the corresponding transceiver and stopped transceiver will cause future calls to createOffer or createAnswer to generate a zero port in the media description for the corresponding transceiver

    Methods setCodecPreferences() – overrides the default codec preferences used by the user agent.

    Example setting codec Preferebec for OPUS in audio

    peer = new RTCPeerConnection();    
const transceiver = peer.addTransceiver('audio');
const audiocapabilities = RTCRtpSender.getCapabilities('audio');
let codec = [];
codec.push(audiocapabilities.codecs[0]);
transceiver.setCodecPreferences(codec);

    Before setting codec preference for OPUS

    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    …
    a=recvonly
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtcp-fb:111 transport-cc
    a=fmtp:111 minptime=10;useinbandfec=1
    a=rtpmap:103 ISAC/16000
    a=rtpmap:104 ISAC/32000
    a=rtpmap:9 G722/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:106 CN/32000
    a=rtpmap:105 CN/16000
    a=rtpmap:13 CN/8000
    a=rtpmap:110 telephone-event/48000
    a=rtpmap:112 telephone-event/32000
    a=rtpmap:113 telephone-event/16000
    a=rtpmap:126 telephone-event/8000

    After setting codec preference for OPUS audio

    m=audio 9 UDP/TLS/RTP/SAVPF 111
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    …
    a=sendrecv
    a=msid:hcgvWcGG7WhdzboWk79q39NiO8xkh4ArWhbM f15d77bb-7a6f-4f41-80cd-51a3c40de7b7
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtcp-fb:111 transport-cc
    a=fmtp:111 minptime=10;useinbandfec=1

    RTCDtlsTransport

    Access to information about the Datagram Transport Layer Security (DTLS) transport over which RTP and RTCP packets are sent and received by RTCRtpSender and RTCRtpReceiver objects, as well other data such as SCTP packets sent and received by data channels.
    Each RTCDtlsTransport object represents the DTLS transport layer for the RTP or RTCP component of a specific RTCRtpTransceiver, or a group of RTCRtpTransceivers if such a group has been negotiated via [BUNDLE].

    interface RTCDtlsTransport : EventTarget {
  [SameObject] readonly attribute RTCIceTransport iceTransport;
  readonly attribute RTCDtlsTransportState state;
  sequence<ArrayBuffer> getRemoteCertificates();
  attribute EventHandler onstatechange;
  attribute EventHandler onerror;
};

    RTCDtlsTransportState Enum

    “new”- DTLS has not started negotiating yet.
    “connecting” – DTLS is in the process of negotiating a secure connection and verifying the remote fingerprint.
    “connected”- DTLS has completed negotiation of a secure connection and verified the remote fingerprint.
    “closed” – transport has been closed intentionally like close_notify alert, or calling close().
    “failed” – transport has failed as the result of an error like failure to validate the remote fingerprint

    RTCDtlsFingerprint dictionary 

    dictionary RTCDtlsFingerprint {
  DOMString algorithm;
  DOMString value;
};

    Protocols multiplexed with RTP (e.g. data channel) share its component ID. This represents the component-id value 1 when encoded in candidate-attribute while ICE candadte for RTCP has component-id value 2 when encoded in candidate-attribute.

    RTCTrackEvent

    The track event uses the RTCTrackEvent interface.

    interface RTCTrackEvent : Event {
  readonly attribute RTCRtpReceiver receiver;
  readonly attribute MediaStreamTrack track;
  [SameObject] readonly attribute FrozenArray<MediaStream> streams;
  readonly attribute RTCRtpTransceiver transceiver;
};

    dictionary RTCTrackEventInit

    dictionary RTCTrackEventInit : EventInit {
  required RTCRtpReceiver receiver;
  required MediaStreamTrack track;
  sequence<MediaStream> streams = [];
  required RTCRtpTransceiver transceiver;
};

    RTCIceCandidate

    This interface candidate Internet Connectivity Establishment (ICE) configuration used to setup RTCPeerconnection. To facilitate routing of media on given peer connection, both endpoints exchange several candidates and then one candidate out of the lot is chosen which will be then used to initiate the connection.

    • candidate – transport address for the candidate that can be used for connectivity checks.
    • component – candidate is an RTP or an RTCP candidate
    • foundation – unique identifier that is the same for any candidates of the same type , helps optimize ICE performance while prioritizing and correlating candidates that appear on multiple RTCIceTransport objects.
    • ip , port
    • priority
    • protocol – tcp/udp
    • relatedAddress , relatedPort
    • sdpMid – candidate’s media stream identification tag
    • sdpMLineIndex

    usernameFragment – randomly-generated username fragment (“ice-ufrag”) which ICE uses for message integrity along with a randomly-generated password (“ice-pwd”).

    Interfaces for Connectivity Establishment

    describes ICE candidates

    interface RTCIceCandidate {
  readonly attribute DOMString candidate;
  readonly attribute DOMString? sdpMid;
  readonly attribute unsigned short? sdpMLineIndex;
  readonly attribute DOMString? foundation;
  readonly attribute RTCIceComponent? component;
  readonly attribute unsigned long? priority;
  readonly attribute DOMString? address;
  readonly attribute RTCIceProtocol? protocol;
  readonly attribute unsigned short? port;
  readonly attribute RTCIceCandidateType? type;
  readonly attribute RTCIceTcpCandidateType? tcpType;
  readonly attribute DOMString? relatedAddress;
  readonly attribute unsigned short? relatedPort;
  readonly attribute DOMString? usernameFragment;
  RTCIceCandidateInit toJSON();
};

    RTCIceProtocol can be either tcp or udp

    TCP candidate type which can be either of

    • active – An active TCP candidate is one for which the transport will attempt to open an outbound connection but will not receive incoming connection requests.
    • passive – A passive TCP candidate is one for which the transport will receive incoming connection attempts but not attempt a connection.
    • so – An so candidate is one for which the transport will attempt to open a connection simultaneously with its peer.

    UDP candidate type

    • host – actual direct IP address of the remote peer
    • srflx – server reflexive ,  generated by a STUN/TURN server
    • prflx – peer reflexive ,IP address comes from a symmetric NAT between the two peers, usually as an additional candidate during trickle ICE
    • relay – generated using TURN

    ICE Candidate UDP Host

    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 51577 typ host generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 51382 typ host generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 53600 typ host generation 0 ufrag muSq network-id 1 network-cost 10

    ICE Candidate TCP Host

    Notice TCP host camdidates for mid 0 , 1 and 3 for video , audio and data media types

    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10

    ICE Candidate UDP Srflx

    Notice 3 candidates for 3 streams sdpMid 0,1 and 2

    sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27177 typ srflx raddr 192.168.1.114 rport 53600 generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27176 typ srflx raddr 192.168.1.114 rport 51382 generation 0 ufrag muSq network-id 1 network-cost 10
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27175 typ srflx raddr 192.168.1.114 rport 51577 generation 0 ufrag muSq network-id 1 network-cost 10

    ICE Candidate (host)

    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:2880323124 1 udp 2122260223 192.168.1.116 61622 typ host generation 0 ufrag jsPO network-id 1 network-cost 10

    usernameFragment – randomly-generated username fragment (“ice-ufrag”) which ICE uses for message integrity along with a randomly-generated password (“ice-pwd”).

    a=ice-ufrag:D+yg
    a=ice-pwd:2ep6CXO0FP/JfS4ue/dfjeQM
    a=ice-options:trickle

    RTCPeerConnectionIceEvent

    interface RTCPeerConnectionIceEvent : Event {
  readonly attribute RTCIceCandidate? candidate;
  readonly attribute DOMString? url;
};

    RTCPeerConnectionIceErrorEvent

    interface RTCPeerConnectionIceErrorEvent : Event {
  readonly attribute DOMString hostCandidate;
  readonly attribute DOMString url;
  readonly attribute unsigned short errorCode;
  readonly attribute USVString errorText;
};

    RTCIceTransport Interface

    Access to information about the ICE transport over which packets are sent and received. Each RTCIceTransport object represents the ICE transport layer for the RTP or RTCP component of a specific RTCRtpTransceiver, or a group of RTCRtpTransceivers if such a group has been negotiated via [BUNDLE].

    interface RTCIceTransport : EventTarget {
  readonly attribute RTCIceRole role;
  readonly attribute RTCIceComponent component;
  readonly attribute RTCIceTransportState state;
  readonly attribute RTCIceGathererState gatheringState;
  sequence<RTCIceCandidate> getLocalCandidates();
  sequence<RTCIceCandidate> getRemoteCandidates();
  RTCIceCandidatePair? getSelectedCandidatePair();
  RTCIceParameters? getLocalParameters();
  RTCIceParameters? getRemoteParameters();
  attribute EventHandler onstatechange;
  attribute EventHandler ongatheringstatechange;
  attribute EventHandler onselectedcandidatepairchange;
};

    RTCIceParameters Dictionary

    dictionary RTCIceParameters {
  DOMString usernameFragment;
  DOMString password;
};

    RTCIceCandidatePair Dictionary

    dictionary RTCIceCandidatePair {
  RTCIceCandidate local;
  RTCIceCandidate remote;
};

    RTCIceGathererState Enum

    “new”,
    “gathering”,
    “complete”

    RTCIceTransportState Enum

    • “new” – ICE agent is gathering addresses or is waiting to be given remote candidates 
    • “checking” –
    • “connected” – Found a working candidate pair, but still performing connectivity checks to find a better one.
    • “completed” – Found a working candidate pair and done performing connectivity checks.
    • “disconnected”,
    • “failed”,
    • “closed”

    RTCIceRole Enum

    “unknown”, // agent who role is not yet defined
    “controlling”, // controlling agent
    “controlled” // controlled agent

    RTCIceComponent Enum

    “rtp”, // ICE Transport is used for RTP (or RTCP multiplexing)
    “rtcp” // ICE Transport is used for RTCP

    Peer-to-peer Data API

    -tbd

    Peer-to-peer DTMF

    -tbd

    Statistics Model

    The browser maintains a set of statistics for monitored objects, in the form of stats objects.
    A group of related objects may be referenced by a selector( like MediaStreamTrack that is sent or received by the RTCPeerConnection).

    Statistics API extends the RTCPeerConnection interface

    partial interface RTCPeerConnection {
  Promise<RTCStatsReport> getStats(optional MediaStreamTrack? selector = null);
  attribute EventHandler onstatsended;
};

    Method getStats()- Gathers stats for the given selector and reports the result asynchronously.

    RTCStatsReport Object

    map between strings that identify the inspected objects (id attribute in RTCStats instances), and their corresponding RTCStats-derived dictionaries.

    interface RTCStatsReport {
  readonly maplike<DOMString, object>;
};

    RTCStats Dictionary

    stats object constructed by inspecting a specific monitored object. 

    dictionary RTCStats {
  required DOMHighResTimeStamp timestamp;
  required RTCStatsType type;
  required DOMString id;
};

    RTCStatsEvent

    Constructor (DOMString type, RTCStatsEventInit eventInitDict)]

interface RTCStatsEvent : Event {
  readonly attribute RTCStatsReport report;
};

    dictionary RTCStatsEventInit

    dictionary RTCStatsEventInit : EventInit {
  required RTCStatsReport report;
};

    Stats

    • RTCRTPStreamStats, attributes ssrc, kind, transportId, codecId
    • RTCReceivedRTPStreamStats, all required attributes from its inherited dictionaries, and also attributes packetsReceived, packetsLost, jitter, packetsDiscarded
    • RTCInboundRTPStreamStats, all required attributes from its inherited dictionaries, and also attributes trackId, receiverId, remoteId, framesDecoded, nackCount
    • RTCRemoteInboundRTPStreamStats, all required attributes from its inherited dictionaries, and also attributes localId, bytesReceived, roundTripTime
    • RTCSentRTPStreamStats, with all required attributes from its inherited dictionaries, and also attributes packetsSent, bytesSent
    • RTCOutboundRTPStreamStats, with all required attributes from its inherited dictionaries, and also attributes trackId, senderId, remoteId, framesEncoded, nackCount
    • RTCRemoteOutboundRTPStreamStats, with all required attributes from its inherited dictionaries, and also attributes localId, remoteTimestamp
    • RTCPeerConnectionStats, with attributes dataChannelsOpened, dataChannelsClosed
    • RTCDataChannelStats, with attributes label, protocol, dataChannelIdentifier, state, messagesSent, bytesSent, messagesReceived, bytesReceived
    • RTCMediaStreamStats, with attributes streamIdentifer, trackIds
    • RTCMediaHandlerStats with attributes trackIdentifier, ended
    • RTCSenderVideoTrackAttachmentStats, with all required attributes from its inherited dictionaries
    • RTCSenderAudioTrackAttachmentStats, with all required attributes from its inherited dictionaries
    • RTCAudioHandlerStats with attribute audioLevel
    • RTCVideoHandlerStats with attributes frameWidth, frameHeight, framesPerSecond
    • RTCVideoSenderStats with attribute framesSent
    • RTCVideoReceiverStats with attributes framesReceived, framesDecoded, framesDropped, partialFramesLost
    • RTCCodecStats, with attributes payloadType, codecType, mimeType, clockRate, channels, sdpFmtpLine
    • RTCTransportStats, with attributes bytesSent, bytesReceived, rtcpTransportStatsId, selectedCandidatePairId, localCertificateId, remoteCertificateId
    • RTCIceCandidatePairStats, with attributes transportId, localCandidateId, remoteCandidateId, state, priority, nominated, bytesSent, bytesReceived, totalRoundTripTime, currentRoundTripTime
    • RTCIceCandidateStats, with attributes address, port, protocol, candidateType, url
    • RTCCertificateStats, with attributes fingerprint, fingerprintAlgorithm, base64Certificate, issuerCertificateId

    Ref :

    CLI/NCLI, Robocalls and STIR/SHAKEN

    To understand the need for implementing an identification verification technique in Internet protocol based network to network communication system , we need to evaluate the existing problem plaguing the VoIP setup .

    What is Call ID spoofing ? 

    Vulnerability of existing interconnection phone system which is used by robo-callers to mask their identity or to make it appear the call is from a legitimate source, usually originates from voice-over-IP (VOIP) systems.

    In this context understand the Caller Line identification CLI/ NCLI techniques used by VoIP and OTT( over the top) providers today.

    CLI (Caller Line Identification)

    If call goes out on a CLI route ( White Route ) the received party will likely see your callerID information

    • Lawful – Termination is legal on the remote end ie abiding country’s telco infrastructure and stable
    • Expensive – usually with direct or via leased line (TDM) interconnections with the tier-1 carriers.

    Non-CLI (Non-Caller Line Identification)

    The Caller ID is not visible at the call
    If call goes out on a Non-CLI route (Grey Route) goes out on a non-CLI routes they will see either a blocked call or some generic number.

    • Unlawful – questionable legality or maybe violating some providers AUP(Acceptable Use Policy ) on the remote end.
    • Cheaper – low quality , usually via VoIP-GSM gateways

    Example include robocalls , tele-marketting / spam etc which are unwilling to share their Caller Id for call receiver, to not be blocked or cancelled.

    To overcome the problem of non-verifiable spam , robocalls a suite of protocols and procedures are proposed that can combat caller ID spoofing on VOIP and connected public telephone networks.

    STIR/SHAKEN

    Secure Telephony Identity Revisited / Signature-based Handling of Asserted information using toKENs

    Used by robocallers to mask their identity or to make it appear the call is from a legitimate source
    usually orignates from voice-over-IP (VOIP) systems

    STIR

    Standards developed by the Internet Engineering Task Force (IETF) 

    For telecommunication service providers implement  certificate management system to create and manage the public and private keys, digital certificates used to sign and verify Caller ID details. 

    Adds information to the SIP headers that allow the endpoints along the system to positively identify the origin of the data , such as JSON web tokens encrypted with the provider’s private key, encoded using Base64,

    There are three levels of verification, or “attestation”

    • A : Full Attestation
      indicates that the provider recognizes the entire phone number as being registered with the originating subscriber.
    • B : Partial Attestation
      call originated with a known customer but the entire number cannot be verified,
    • C : Gateway Attestation
      call can only be verified as coming from a known gateway

    How can the Public Key Infrastructure be used ? 

    In an interconnection network , each telephone service provider will obtain its digital certificate from a certificate authority (CA)  that is trusted by other telephone service providers. Calling party signs the SIP Header  caller ID as legitimate . The called party verifies that the calling number is authentic

    STIR

    Originating service provider’s encrypted SIP Identity Header includes the following data:

    1. Attestation level
    2. Date and Time
    3. Calling and Called Numbers
    4. Orig ID for analytics and/or traceback purposes among others
    5. Location of certificate repository
    6. Signature
    7. Encryption algorithm

    FCC has also assigned the role of a Secure Telephone Identity Policy Administrator (STI-PA) which oversees that CAs do not provide certificate to spoofing robocallers and enforce the framework for STIR /SHAKEN .

    Sample Identity header in SIP requst

    INVITE sip:bob@biloxi.example.org SIP/2.0
Via: SIP/2.0/TLS pc33.atlanta.example.com;branch=z9hG4bKnashds8
To: Bob
From: Alice ;tag=1928301774
Call-ID: a84b4c76e66710
CSeq: 314159 INVITE
Max-Forwards: 70
Date: Thu, 21 Feb 2002 13:02:03 GMT
Contact:
Identity:
"ZYNBbHC00VMZr2kZt6VmCvPonWJMGvQTBDqghoWeLxJfzB2a1pxAr3VgrB0SsSAaifsRdiOPoQZYOy2wrVghuhcsMbHWUSFxI6p6q5TOQXHMmz6uEo3svJsSH49thyGnFVcnyaZ++yRlBYYQTLqWzJ+KVhPKbfU/pryhVn9Yc6U="
Identity-Info: https://atlanta.example.com/atlanta.cer;alg=rsa-sha1
Content-Type: application/sdp
Content-Length: 147

v=0
o=UserA 2890844526 2890844526 IN IP4 pc33.atlanta.example.com
s=Session SDP
c=IN IP4 pc33.atlanta.example.com
t=0 0
m=audio 49172 RTP/AVP 0
a=rtpmap:0 PCMU/8000

    SHAKEN

    STIR is based on the SIP protocol and is designed to work with calls being routed through a VOIP network. Since traditional endpoints like POTS and SS7 networks also should be covered under this call authenticity framework , SHAKEN was developed to manage call via IP-to-telephone gateways .

    Developed by the Alliance of Telecommunications Industry Solutions (ATIS)

    Working Steps  :

    1. When a call is initiated, a SIP INVITE is received by the originating service provider.
    2. Originating service provider verifies the call source and number to determine how to confirm validity.
      1. Full Attestation (A) — The service provider authenticates the calling party AND confirms they are authorized to use this number. An example would be a registered subscriber.
      2. Partial Attestation (B) — The service provider verifies the call origination but cannot confirm that the call source is authorized to use the calling number. An example would be a calling number from behind an enterprise PBX.
      3. Gateway Attestation (C) — The service provider authenticates the call’s origin but cannot verify the source. An example would be a call received from an international gateway.
    3. Create a SIP Identity header that contains information on the calling number, called number, attestation level, and call origination, along with the certificate thus caller ID “signed” as legitimate
    4. SIP INVITE with the SIP Identity header with the certificate is sent to the destination service provider.
    5. Destination service provider verifies the identity of the header and certificate.

    Diagrammatic depiction of flow of how Telecom carriers to digitally validates authenticity before receiving or handoff through their network

    SHAKEN

    References

    Video Codecs – H264 , H265 , AV1

    Article discusses the popularly adopted current standards for video codecs( compression / decompression) namely MPEG2, H264, H265 and AV1

    MPEG 2

    MPEG-2 (a.k.a. H.222/H.262 as defined by the ITU)
    generic coding of moving pictures and associated audio information
    combination of lossy video compression and lossy audio data compression methods, which permit storage and transmission of movies using currently available storage media and transmission bandwidth.

    better than MPEG 1

    evolved out of the shortcomings of MPEG-1 such as audio compression system limited to two channels (stereo) , No standardized support for interlaced video with poor compression , Only one standardized “profile” (Constrained Parameters Bitstream), which was unsuited for higher resolution video.

    Application

    • over-the-air digital television broadcasting and in the DVD-Video standard.
    • TV stations, TV receivers, DVD players, and other equipment
    • MOD and TOD – recording formats for use in consumer digital file-based camcorders.
    • XDCAM – professional file-based video recording format.
    • DVB – Application-specific restrictions on MPEG-2 video in the DVB standard:

    H264

    Advanced Video Coding (AVC), or H.264 or aka MPEG-4 AVC or ITU-T H.264 / MPEG-4 Part 10 ‘Advanced Video Coding’ (AVC)
    introduced in 2004

    Better than MPEG2

    40-50% bit rate reduction compared to MPEG-2

    Support Up to 4K (4,096×2,304) and 59.94 fps
    21 profiles ; 17 levels

    Compression Model

    Video compression relies on predicting motion between frames. It works by comparing different parts of a video frame to find the ones that are redundant within the subsequent frames ie not changed such as background sections in video. These areas are replaced with a short information, referencing the original pixels(intraframe motion prediction) using mathematical function and direction of motion

    Hybrid spatial-temporal prediction model
    Flexible partition of Macro Block(MB), sub MB for motion estimation
    Intra Prediction (extrapolate already decoded neighbouring pixels for prediction)
    Introduced multi-view extension
    9 directional modes for intra prediction
    Macro Blocks structure with maximum size of 16×16
    Entropy coding is CABAC(Context-adaptive binary arithmetic coding) and CAVLC(Context-adaptive variable-length coding )

    Applications

    • most deployed video compression standard
    • Delivers high definition video images over direct-broadcast satellite-based television services,
    • Digital storage media and Blu-Ray disc formats,
    • Terrestrial, Cable, Satellite and Internet Protocol television (IPTV)
    • Security and surveillance systems and DVB
    • Mobile video, media players, video chat

    H265

    High Efficiency Video Coding (HEVC), or H.265 or MPEG-H HEVC
    video compression standard designed to substantially improve coding efficiency
    stream high-quality videos in congested network environments or bandwidth constrained mobile networks
    Jan 2013
    product of collaboration between the ITU Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG).

    • HVEC wikipedia

    better than H264

    overcome shortage of bandwidth, spectrum, storage
    bandwidth savings of approx. 45% over H.264 encoded content

    resolutions up to 8192×4320, including 8K UHD
    Supports up to 300 fps
    3 approved profiles, draft for additional 5 ; 13 levels
    Whereas macroblocks can span 4×4 to 16×16 block sizes, CTUs can process as many as 64×64 blocks, giving it the ability to compress information more efficiently.

    multiview encoding – stereoscopic video coding standard for video compression that allows for the efficient encoding of video sequences captured simultaneously from multiple camera angles in a single video stream. It also packs a large amount of inter-view statistical dependencies.

    Compression Model

    Enhanced Hybrid spatial-temporal prediction model
    CTU ( coding tree units) supporting larger block structure (64×64) with more variable sub partition structures

    Motion Estimation – Intra prediction with more nodes, asymmetric partitions in Inter Prediction)
    Individual rectangular regions that divide the image are independent

    Paralleling processing computing – decoding process can be split up across multiple parallel process threads, taking advantage multi-core processors.

    Wavefront Parallel Processing (WPP)- sort of decision tree that grants a more productive and effectual compression.
    33 directional nodes – DC intra prediction , planar prediction. , Adaptive Motion Vector Prediction
    Entropy coding is only CABAC

    Applications

    • cater to growing HD content for multi platform delivery
    • differentiated and premium 4K content

    reduced bitrate enables broadcasters and OTT vendors to bundle more channels / content on existing delivery mediums
    also provide greater video quality experience at same bitrate

    Using ffmpeg for H265 encoding

    I took a h264 file (640×480) , duration 30 seconds of size 39,08,744 bytes (3.9 MB on disk) and converted using ffnpeg

    After conversion it was a HEVC (Parameter Sets in Bitstream) , MPEG-4 movie – 621 KB only !!! without any loss of clarity.

    > ffmpeg -i pivideo3.mp4 -c:v libx265 -crf 28 -c:a aac -b:a 128k output.mp4                                              ffmpeg version 4.1.4 Copyright (c) 2000-2019 the FFmpeg developers   built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)   configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.4_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-12.0.1.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-12.0.1.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr   libavutil      56. 22.100 / 56. 22.100   libavcodec     58. 35.100 / 58. 35.100   libavformat    58. 20.100 / 58. 20.100   libavdevice    58.  5.100 / 58.  5.100   libavfilter     7. 40.101 /  7. 40.101   libavresample   4.  0.  0 /  4.  0.  0   libswscale      5.  3.100 /  5.  3.100   libswresample   3.  3.100 /  3.  3.100   libpostproc    55.  3.100 / 55.  3.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pivideo3.mp4':   Metadata:     major_brand     : isom     minor_version   : 1     compatible_brands: isomavc1     creation_time   : 2019-06-23T04:58:13.000000Z   Duration: 00:00:29.84, start: 0.000000, bitrate: 1047 kb/s     Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480, 1046 kb/s, 25 fps, 25 tbr, 25k tbn, 50k tbc (default)     Metadata:       creation_time   : 2019-06-23T04:58:13.000000Z       handler_name    : h264@GPAC0.5.2-DEV-revVersion: 0.5.2-426-gc5ad4e4+dfsg5-3+deb9u1 Codec AVOption b (set bitrate (in bits/s)) specified for output file #0 (output.mp4) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream. Stream mapping:   Stream #0:0 -> #0:0 (h264 (native) -> hevc (libx265)) Press [q] to stop, [?] for help x265 [info]: HEVC encoder version 3.1.2+1-76650bab70f9 x265 [info]: build info [Mac OS X][clang 10.0.1][64 bit] 8bit+10bit+12bit x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 x265 [info]: Main profile, Level-3 (Main tier) x265 [info]: Thread pool created using 4 threads x265 [info]: Slices                              : 1 x265 [info]: frame threads / pool features       : 2 / wpp(8 rows) x265 [warning]: Source height < 720p; disabling lookahead-slices x265 [info]: Coding QT: max CU size, min CU size : 64 / 8 x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3 x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00 x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2 x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0 x265 [info]: References / ref-limit  cu / depth  : 3 / off / on x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1 x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60 x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip signhide tmvp b-intra x265 [info]: tools: strong-intra-smoothing deblock sao Output #0, mp4, to 'output.mp4':   Metadata:     major_brand     : isom     minor_version   : 1     compatible_brands: isomavc1     encoder         : Lavf58.20.100     Stream #0:0(und): Video: hevc (libx265) (hev1 / 0x31766568), yuv420p, 640x480, q=2-31, 25 fps, 12800 tbn, 25 tbc (default)     Metadata:       creation_time   : 2019-06-23T04:58:13.000000Z       handler_name    : h264@GPAC0.5.2-DEV-revVersion: 0.5.2-426-gc5ad4e4+dfsg5-3+deb9u1       encoder         : Lavc58.35.100 libx265 frame=  746 fps= 64 q=-0.0 Lsize=     606kB time=00:00:29.72 bitrate= 167.2kbits/s speed=2.56x     video:594kB audio:0kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 2.018159% x265 [info]: frame I:      3, Avg QP:27.18  kb/s: 1884.53  x265 [info]: frame P:    179, Avg QP:27.32  kb/s: 523.32   x265 [info]: frame B:    564, Avg QP:35.17  kb/s: 38.69    x265 [info]: Weighted P-Frames: Y:5.6% UV:5.0% x265 [info]: consecutive B-frames: 1.6% 3.8% 9.3% 53.3% 31.9%  encoded 746 frames in 11.60s (64.31 fps), 162.40 kb/s, Avg QP:33.25

    if you get error like 

    Unknown encoder 'libx265'

    then reinstall ffmpeg with h265 support

    AV1

    Realtime High quality video encoder
    product of product of the Alliance for Open Media (AOM)
    Contained by Matroska , WebM , ISOBMFF , RTP (WebRTC)

    • Wikipedia AV1

    better than H265

    AV1 is royalty free and overcomes the patent complexities around H265/HVEC

    Applications

    • Video transmission over internet , voip , multi conference
    • Virtual / Augmented reality
    • self driving cars streaming
    • intended for use in HTML5 web video and WebRTC together with the Opus audio format

    Audio and Acoustic Signal Processing

    Audio signals are electronic representations of sound waves—longitudinal waves which travel through air, consisting of compressions and rarefactions and Audio Signal Processing focuses on the computational methods for intentionally altering auditory signals or sounds, in order to achieve a particular goal.

    Application of audio Signal processing in general

    • storage
    • data compression
    • music information retrieval
    • speech processing ( emotion recognition/sentiment analysis , NLP)
    • localization
    • acoustic detection
    • Transmission / Broadcasting – enhance their fidelity or optimize for bandwidth or latency.
    • noise cancellation
    • acoustic fingerprinting
    • sound recognition ( speaker Identification , biometric speech verification , voice commands )
    • synthesis – electronic generation of audio signals. Speech synthesisers can generate human like speech.
    • enhancement (e.g. equalization, filtering, level compression, echo and reverb removal or addition, etc.)

    Effects for audio streams processing

    • delay or echo
      To simulate reverberation effect, one or several delayed signals are added to the original signal. To be perceived as echo, the delay has to be of order 35 milliseconds or above.
      Implemented using tape delays or bucket-brigade devices.
    • flanger
      delayed signal is added to the original signal with a continuously variable delay (usually smaller than 10 ms).
      signal would fall out-of-phase with its partner, producing a phasing comb filter effect and then speed up until it was back in phase with the master
    • phaser
      signal is split, a portion is filtered with a variable all-pass filter to produce a phase-shift, and then the unfiltered and filtered signals are mixed to produce a comb filter.
    • chorus
      delayed version of the signal is added to the original signal. above 5 ms to be audible. Often, the delayed signals will be slightly pitch shifted to more realistically convey the effect of multiple voices.
    • equalization
      frequency response is adjusted using audio filter(s) to produce desired spectral characteristics. Frequency ranges can be emphasized or attenuated using low-pass, high-pass, band-pass or band-stop filters.
      overdrive effects such as the use of a fuzz box can be used to produce distorted sounds, such as for imitating robotic voices or to simulate distorted radiotelephone traffic
    • pitch shift
      shifts a signal up or down in pitch. For example, a signal may be shifted an octave up or down. This is usually applied to the entire signal, and not to each note separately. Blending the original signal with shifted duplicate(s) can create harmonies from one voice.
    • time stretching
      changing the speed of an audio signal without affecting its pitch.
    • resonators
      emphasize harmonic frequency content on specified frequencies. These may be created from parametric EQs or from delay-based comb-filters.
    • modulation
      change the frequency or amplitude of a carrier signal in relation to a predefined signal.
    • compression
      reduction of the dynamic range of a sound to avoid unintentional fluctuation in the dynamics. Level compression is not to be confused with audio data compression, where the amount of data is reduced without affecting the amplitude of the sound it represents.
    • 3D audio effects
      place sounds outside the stereo basis
    • reverse echo
      swelling effect created by reversing an audio signal and recording echo and/or delay while the signal runs in reverse.
    • wave field synthesis
      spatial audio rendering technique for the creation of virtual acoustic environments

    ASP application in Telephony and mobile phones, by ITU (International Telegraph Union)

    • Acoustic echo control
      aims to eliminate the acoustic feedback, which is particularly problematic in the speakerphone use-case during bidirectional voice
    • Noise control
      microphone doesn’t only pick up the desired speech signal, but often also unwanted background noise. Noise control tries to minimize those unwanted signals . Multi-microphone AASP, has enabled the suppression of directional interferers.
    • Gain control
      how loud a speech signal should be when leaving a telephony transmitter as well as when it is being played back at the receiver. Implemented either statically during the handset design stage or automatically/adaptively during operation in real-time.
    • Linear filtering
      ITU defines an acceptable timbre range for optimum speech intelligibility. AASP in the form of linear filtering can help the handset manufacturer to meet these requirements.
    • Speech coding: from analog POTS based call to G.711 narrowband (approximately 300 Hz to 3.4 kHz) speech coder is a big leap in terms of call capacity. other speech coders with varying tradeoffs between compression ratio, speech quality, and computational complexity have been also made available. AASP provides higher quality wideband speech (approximately 150 Hz to 7 kHz).

    ASP applications in music playback

    AASP is used to provide audio post-processing and audio decoding capabilities for mobile media consumption needs, such as listening to music, watching videos, and gaming

    • Post-processing
      techniques as equalization and filtering allow the user to adjust the timbre of the audio such as bass boost and parametric equalization. Other techniques like adding reverberation, pitch shift, time stretching etc
    • Audio (de)coding: audio contianers like mp3 and AAC define how music is distributed, stored, and consumed also in Online music streaming services

    ASP for virtual assitants

    Virtual Assistance include a variety of servies from Apple’s Siri, Microsoft’s Cortana , Google’s Now , Alexa etc. ASP is used in

    • Speech enhancement
      multi-microphone speech pickup using beamforming and noise suppression to isolate the desired speech prior to forwarding it to the speech recognition engine.
    • Speech recognition (speech-to-text): this draws ideas from multiple disciplinary fields including linguistics, computer science, and AASP. Ongoing work in acoustic modeling is a major contribution to recognition accuracy improvement in speech recognition by AASP.
    • Speech synthesis (text-to-speech): this technology has come a very long way from its very robotic sounding introduction in the 1930s to making synthesized speech sound more and more natural.

    Other areas of ASP

    • Virtual reality (VR) like VR headset / gaming simulators use three-dimensional soundfield acquisition and representation like Ambisonics (also known as B-format).

    Ref :
    wikipedia – https://en.wikipedia.org/wiki/Audio_signal_processing
    IEEE – https://signalprocessingsociety.org/publications-resources/blog/audio-and-acoustic-signal-processing%E2%80%99s-major-impact-smartphones

    JavaScript Session Establishment Protocol (JSEP) in WebRTC handshake

    This article is aimed at explaning the intricacies and detailed offer answer flow in webrtc handshake and JSEP . You can read the following artciles on WebRTC as prereq before reading through this one

    WebRTC API – Peerconnection , getUserMedia , Datachannel , DataStaats

    JSEP (JavaScript Session Establishment Protocol)

    JSEP (JavaScript Session Establishment Protocol) is used during signalling via w3c RTCPeerConnectionAPI interface to set up a multimedia session. The multimedia session description specfies the crtical components of setting up a session between local and remote such as transport ports , protcol , profiles . It also handles the intercation with ICE state machine

    Offer/Answer Excahange Flow

    prereq : Setup Client side for the caller
    PeerConnectionFactory to generate PeerConnections
    PeerConnection for every connection to remote peer
    MediaStream audio and video from client device

    1. Side initiating the session creates a offer by CreateOffer() API
    aPromise = myPeerConnection.createOffer([options]);

    options is type of RTC Offer Options

    • iceRestart
    • offerToReceiveAudio ( legacy)
    • offerToReceiveVideo ( legacy)
    • voiceActivityDetection

    2. The application then stores the offer in local config as setLocalDescriptionAPI()

     myPeerConnection.createOffer().then(function(offer) {
    return myPeerConnection.setLocalDescription(offer);
})

    3. Offer is sent to remote side using its choice of signalling ( SIP , WS , HTTP, XMPP .. )

    4. Remote party stores it use setRemoteDescription() API

    myPeerConnection.setRemoteDescription(sdp)
.then(function () {
  return createMyStream();
})

    4. Remote part generates an answer using createAnswer() API

    aPromise = RTCPeerConnection.createAnswer([options]);

    5. Remote party stores the answer in its local config using setLocalDescription() API

    6. Answer is transferred to Initiator side using choice of signalling ( SIP , WS , HTTP, XMPP .. ) again

    7. Initiating side stores it use setRemoteDescription() API

    Interfaces of webrtc and tracks to stream addition

    Process to perform webrtc handshake

    Webrtc call setup and incoming call callflow between remote peer , peerconnection actory , peerconnection and application

    setup a call
    receive a call

    Signalling state Transitions on PeerConnection

    As the caller initiates a new RTCPeerConnection() , the RTCSignalingState state is “stable” as remote and local descriptions are empty

    As the caller initiates call and calls createOffer() , he now has offer SDP and procced to store offer locally with setLocalDescription(offer) the RTCSignalingState state is “have-local-offer” . After than caller send the offer to callee over signalling channel

    Simillarily as the calle recives the offer , it starts with RTCSignalingState stable and then proceeds to store the Remote’s offer using setRemoteDescription(offer) , its state is now “have-remote-offer”

    The callee generates a provsional answer and for caller and stores it locally , state transitiosn to “have-local-pranswer” . the pranswer SDP is send to caller over signalling channel again .

    Caller stores the callee’s pr answer SDP and state updates to “have-remote-pranswer”

    Once there is no offer/answer exchange in progress the state again changes to ” stable “.

    State schanges to “closed” if RTCpeerConnection is closed

    img : https://w3c.github.io/webrtc-pc

    Detailed Offer / Answer SDP

    Local Offer created by side initiating the session / Caller

    The first offfer called initial offer can have dummy date for contact line such as 0.0.0.0 to prevent leaking a local Ip address

    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0

    “o=” line contains <username> <sess-id> <sess-version> <nettype> <addrtype> <unicast-address>

    o=- 4445251981417004127 2 IN IP4 127.0.0.

    shows username – and 4445251981417004127 as session id. Same username “-” is specified in “s=” line

    “t=” line shows <start time> <stop time>

    t=0 0

    Full session Block example

    type: offer, sdp: v=0
    o=- 4445251981417004127 2 IN IP4 127.0.0.1
    s=-
    t=0 0
    a=group:BUNDLE 0 1 2
    a=msid-semantic: WMS DYVK4IA4kA8LvnIYWjXhRzMgSGicnwVutWE2

    Media Section : An m= section is generated for each RtpTransceiver that has been added to the PeerConnection. For the initial offer since no ports are available yet , dummy port 9 can be sadded. However if it is bundle only then port value is set to 0. Later the port value will be set to the port value of default ICE candidate.

    DTLS filed “UDP/TLS/RTP/SAVPF” is followed by the list of codecs in order of priority.

    “c=” line in msection too must be filled with dummy values if IP 0.0.0.0 as no candidates are available yet .

    ICE 

    a=ice-options:trickle

    Transport

    “a=ice-ufrag” , “a=ice-pwd” , “a=fingerprint” , “a=setup” , “a=tls-id”

    Media Stream Identification attribute “a-mid:”

    For each media format on the m= line, “a=rtpmap” for “rtx” with the clock rate of codec and “a=fmtp” to reference the payload type of the primary codec.  “a=rtcp-fb” specified RTCP feedback

    a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1

    Audio Block exmaple 

    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:JDMg
a=ice-pwd:6OARDQ8U/orhtXZbfN+ars37
a=ice-options:trickle
a=fingerprint:sha-256 1D:C8:1F:18:D2:AB:B7:68:CC:DC:A8:8D:6B:1D:70:11:06:E9:19:D2:22:CE:A5:F3:BE:82:00:ED:99:58:20:4A
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:DYVK4IA4kA8LvnIYWjXhRzMgSGicnwVutWE2 7525d75c-ffe7-4038-8b71-653d249e63bb
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:3968544080 cname:da0nYe1oYR8AvVNp
a=ssrc:3968544080 msid:DYVK4IA4kA8LvnIYWjXhRzMgSGicnwVutWE2 7525d75c-ffe7-4038-8b71-653d249e63bb
a=ssrc:3968544080 mslabel:DYVK4IA4kA8LvnIYWjXhRzMgSGicnwVutWE2
a=ssrc:3968544080 label:7525d75c-ffe7-4038-8b71-653d249e63bb

    // remove video section for simplicity

    Data Block is created if data channle has been created with m= section for data.

    “a=sctp-port” line referencing the SCTP port number set to 5000

     “a=max-message-size”  set to 262144 here

    Data Block example

    m=application 9 UDP/DTLS/SCTP webrtc-datachannel
    c=IN IP4 0.0.0.0
    a=ice-ufrag:JDMg
    a=ice-pwd:6OARDQ8U/orhtXZbfN+ars37
    a=ice-options:trickle
    a=fingerprint:sha-256 1D:C8:1F:18:D2:AB:B7:68:CC:DC:A8:8D:6B:1D:70:11:06:E9:19:D2:22:CE:A5:F3:BE:82:00:ED:99:58:20:4A
    a=setup:actpass
    a=mid:2
    a=sctp-port:5000
    a=max-message-size:262144

    Subsequent Offers

    When createOffer is called a second (or later) time, or is called after a local description has already been installed, the processig is different due to gathered ICE candidates . However the <session-version> is not changed .

    Additionally m section is updated if RtpTransceiver is added or removed

    Each “m=” and c=” line MUST be filled in with the port, relevant RTP profile, and address of the default candidate for the m= section

    If the m= section is not bundled into another m= section, update the “a=rtcp” with port and address of RTCP camdidate and add “a=camdidate” with  “a=end-of-candidates” 

    Local Answer created by side receiving the session/ Callee

    When createAnswer is called for the first time after a remote description has been provided, the result is known as the initial answer. 

     Each offered m= section will have an associated RtpTransceiver

    Remote Destination / Callee can reject the m section by setting port in m line to 0 . It can reject msection if neither of the offered media format are supported , RtpTransceiver is stoopped etc.

    For the initial offer the dummy port value of 9 is set as no ICE candudate is avaible yet . Simillarly  “c=” line must contain the “dummy” value “IN IP4 0.0.0.0” too.

    The <proto> field MUST be set to exactly match the <proto> field for the corresponding m= line in the offer.

    type: answer, sdp: v=0
    o=- 5730481682283561642 3 IN IP4 127.0.0.1
    s=-
    t=0 0
    a=group:BUNDLE 0 1 2
    a=msid-semantic: WMS KGmQ9mTmvTaWlHTQ0B0YP36QIxOYNeB3i2nT

    Audio section

    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    a=ice-ufrag:MgKS
    a=ice-pwd:X3oTkKO/v7GVgd/CDC3e9B7c
    a=ice-options:trickle
    a=fingerprint:sha-256 B9:9C:8A:A9:E9:09:0C:FB:52:2A:D3:18:7B:A9:D4:EC:B3:00:77:72:27:51:EC:5F:82:BE:11:7F:C7:CF:43:43
    a=setup:active
    a=mid:0
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
    a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
    a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
    a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
    a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
    a=sendrecv
    a=msid:KGmQ9mTmvTaWlHTQ0B0YP36QIxOYNeB3i2nT e817fe0f-1cc0-4901-9fd9-e810289cc85d
    a=rtcp-mux
    a=rtpmap:111 opus/48000/2
    a=rtcp-fb:111 transport-cc
    a=fmtp:111 minptime=10;useinbandfec=1
    a=rtpmap:103 ISAC/16000
    a=rtpmap:104 ISAC/32000
    a=rtpmap:9 G722/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=rtpmap:106 CN/32000
    a=rtpmap:105 CN/16000
    a=rtpmap:13 CN/8000
    a=rtpmap:110 telephone-event/48000
    a=rtpmap:112 telephone-event/32000
    a=rtpmap:113 telephone-event/16000
    a=rtpmap:126 telephone-event/8000
    a=ssrc:3260997313 cname:FxLUKuXrLQe0r1rn

    Video section removed for simplicity

    Data stream

    m=application 9 UDP/DTLS/SCTP webrtc-datachannel
    c=IN IP4 0.0.0.0
    b=AS:30
    a=ice-ufrag:MgKS
    a=ice-pwd:X3oTkKO/v7GVgd/CDC3e9B7c
    a=ice-options:trickle
    a=fingerprint:sha-256 B9:9C:8A:A9:E9:09:0C:FB:52:2A:D3:18:7B:A9:D4:EC:B3:00:77:72:27:51:EC:5F:82:BE:11:7F:C7:CF:43:43
    a=setup:active
    a=mid:2
    a=sctp-port:5000
    a=max-message-size:262144

    Subsequent Answers

     Port value would normally be set to the port of the default ICE candidate for this m= section. For the exmaple above

    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126

    will be changes with relevant port adress such as

    type: offer, sdp: v=0
    o=- 6407282338169184323 3 IN IP4 54.190.54.190
    s=-
    t=0 0
    a=group:BUNDLE 0 1 2
    a=msid-semantic: WMS bSrCUCFybGovIy0FUhPTZAr9ToRmx8I09nEj
    m=audio 55375 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    c=IN IP4 54.190.54.190
    a=rtcp:9 IN IP4 0.0.0.0
    a=candidate:2880323124 1 udp 2122260223 54.190.54.190 55375 typ host generation 0 network-id 1 network-c

    Simillarly m video and data line will also get ports

    m=video 53877 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116
    c=IN IP4 54.190.54.190
    a=rtcp:9 IN IP4 0.0.0.0
    a=candidate:2880323124 1 udp 2122260223 54.190.54.190 53877 typ host generation 0 network-id 1 network-cost 10
    ..
    m=application 57991 UDP/DTLS/SCTP webrtc-datachannel
    c=IN IP4 54.190.54.190
    a=candidate:2880323124 1 udp 2122260223 54.190.54.190 57991 typ host generation 0 network-id 1 network-cost 10

    If the answer contains any “a=ice-options” attributes where “trickle” is listed as an attribute, update the PeerConnection canTrickle property to be true. 

    Modifying Offer/answer SDP

    SDP returned from createOffer or createAnswer MUST NOT be changed before passing it to setLocalDescription.
    After calling setLocalDescription with an offer or answer, the application MAY modify the SDP to reduce its capabilities before sending it to the far side

    Assume we have a MCU at location and want the video stream to relay via a Media Server.

    SDP Parsing

    SDP is used for session parsing and contians sequence of line with key value pairs. SDP is read, line-by-line, and converted to a data structure that contains the deserialized information.

    JSEP SDP bears a lot of simillarity to SIP SDP explained here : SIP and SDP Messages Explained

    Session-Level Parsing

    Line “v=” , “o=”,”b=” and “a=” are processed . The “i=”, “u=”, “e=”, “p=”, “t=”, “r=”, “z=”, and “k=” lines are not used by this specification; they MUST be checked for syntax but their values are not used. Line “c=” is checked for syntax and ICE mismatch detection

    “a= ” attribute could be : “a=group” , “s=”ice-lite” , “a=ice-pwd”, “a=ice-options” , “a=fingerprint”, “a=setup” , a=tls-id”, “a=identity” , “a=extmap”

    Media Section Parsing

    Line “m=” for media , proto , port , fmt in RTP

    Attributes “a=” can be

    “a=rtpmap” or “a=fmtp”

    map from an RTP payload type number to a media encoding name that identifies the payload format.

    a=rtpmap:<payload type> <encoding name>/<clock rate> [/<encoding parameters>]
    m=audio 49230 RTP/AVP 96 97 98
a=rtpmap:96 L8/8000
a=rtpmap:97 L16/8000
a=rtpmap:98 L16/11025/2

    “a=ptime” , “a=maxptime”

    dierction as  “a=sendrecv” , a=recvonly , a=sendonly , a=inactive

    Muxing as “a=rtcp-mux” ,

    “a=rtcp-mux-only”

    RTCP attributes “a=rtcp” , “a=rtcp-rsize”

    Line “c=” is checked .

    Line “b=” for bandiwtdh , bwtype

    Attribites for “a=” could be “a=ice-ufrag”, “a=”ice-pwd”, “a=ice-options” , “a=candidate”, “a=remote-candidate” , a=end-of-candidates” and “a=fingerprint”

    Semantics Verification

    Interactive Connectivity Establishment (ICE) for NAT traversal

    Protocols using offer/answer are difficult to operate through Network Address Translators (NATs) since flow of media packets require IP addresses and ports of media sources and sinks within their messages. Also realtime media emphasises on reduced latency and decreased packet loss .

    An extension to the offer/answer model, and works by including a multiplicity of IP addresses and ports in SDP offers and answers, which are then tested for connectivity by peer-to-peer connectivity checks.
    Checks done by STUN and TURN
    also allows for address selection for multi-homed and dual-stack hosts

    ICE allows the agents to discover enough information about their topologies to potentially find one or more paths by which they can communicate. Then it systematically tries all possible pairs (in a carefully sorted order) until it finds one or more that work.

    ICE Gathering

    Caller and callee performs checks to finalize the protocol and routing needed to establish a peer connection . Number of candudates are proposed till they mutually agree upon one . Peerconnection then uses that candiadte detaisl to initiate the connection .

    While Applying a Local Description at the media engine level if m= section is new, WebRTC media stacks begins gathering candidates for it.

    RTCPeerconnection specified canTrickleIceCandidates . ICE trickling is the process of continuing to send candidates after the initial offer or answer has already been sent to the other peer.

    ICE TransportRole is responsible for Choosing a candidate pair

    ICE layer sets one peer as controlling and other as controlled agent . The controling agent makes the final decision as to which candidate pair to choose.

    Final selected canduadte in SDP

    a=group:BUNDLE 0 1 2
    a=msid-semantic: WMS 9Cv3eIelHVuhxrGfxSvUsfokNu4eb4R9PYw2
    m=audio 59937 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
    c=IN IP4 x.x.x.x
    a=rtcp:9 IN IP4 0.0.0.0
    a=candidate:2880323124 1 udp 2122260223 x.x.x.x 59937 typ host generation 0 network-id 1 network-cost 10
    a=candidate:3844981444 1 tcp 1518280447 x.x.x.x 9 typ host tcptype active generation 0 network-id 1 network-cost 10

    An agent identifies all CANDIDATE whic is a transport address. Types:

    • HOST CANDIDATE – directly from a local interface which could be Wifi, Virtual Private Network (VPN) or Mobile IP (MIP)
      if an agent is multihomed ( private and public networks) , it obtains a candidate from each IP address and includes all candidates in its offer.
    • STUN or TURN to obtain additional candidates. Types
      • translated addresses on the public side of a NAT (SERVER REFLEXIVE CANDIDATES)
      • addresses on TURN servers (RELAYED CANDIDATES)

    Mapping Server Reflexive address

    Agent sends the TURN Allocate request from IP address and port X:x,
    NAT will create a binding X1′:x1′, mapping this server reflexive candidate to the host candidate X:x ( BASE).
    Outgoing packets sent from the host candidate will be translated by the NAT to the server reflexive candidate.
    Incoming packets sent to the server reflexive candidate will be translated by the NAT to the host candidate and forwarded to the agent.

    Allocate Request and response fom TURN – Informing the agent of this relayed candidate

    Only STUN based Binding

    agent sends a STUN Binding request to its STUN server which will get server reflexive candidate and send back Binding response.

    STUN Binding request for connectivity checks on CANDIDATE PAIRS

    The candidates are carried in attributes in the SDP offer . The remote peer also follows this process and gather and send lits own sorted list of candidates. Hence CANDIDATE PAIRS from both sides are formed.

    PEER REFLEXIVE CANDIDATES – connectivity checks can produce aditional candidates espceialy around symmetric NAT

    Since the same address is used for STUN. and media ( RTP/RTCP) Demultiplexing based on packet contents helps to identify which one is which.

    Checks : ICE checks are performed in a specific sequence, so that high-priority candidate pairs are checked first.

    TRIGGERED CHECKS – accelerates the process of finding a valid candidate
    ORDINARY CHECKS – agent works through ordered prioritised check list by sending a STUN request for the next candidate pair on the list periodically.

    Checks ensure maintaining frozen candidates and pairs with some foundation for media stream

    Each candidate pair in the check list has a foundation and a state. States for candidates pairs

    1.Waiting: A check has not been performed for this pair, and can be performed as soon as it is the highest-priority Waiting pair onthe check list.

    2. In-Progress: A check has been sent for this pair, but the transaction is in progress.

    3. Succeeded: A check for this pair was already done and produced a successful result.

    4. Failed: A check for this pair was already done and failed, either never producing any response or producing an unrecoverable failure response.

    5. Frozen: A check for this pair hasn’t been performed, and it can’t yet be performed until some other check succeeds, allowing this pair to unfreeze and move into the Waiting state.

    Example of ICE gather state

    icegatheringstatechange – gathering

    icecandidate (host)
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:1511920713 1 udp 2122260223 192.168.0.2 58122 typ host generation 0 ufrag vzpn network-id 1 network-cost 10

    icecandidate (srflx)
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:4081163164 1 udp 1686052607 106.51.26.168 37542 typ srflx raddr 192.168.0.2 rport 58122 generation 0 ufrag vzpn network-id 1 network-cost 10

    icecandidate (host)
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:345893049 1 tcp 1518280447 192.168.0.2 9 typ host tcptype active generation 0 ufrag vzpn network-id 1 network-cost 10

    icecandidate (relay)
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:2130406062 1 udp 41886207 74.125.39.44 27190 typ relay raddr 106.51.26.168 rport 37542 generation 0 ufrag vzpn network-id 1 network-cost 10

    icecandidate (relay)
    sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:3052096874 1 udp 25108479 172.217.163.158 28049 typ relay raddr 106.51.26.168 rport 37543 generation 0 ufrag vzpn network-id 1 network-cost 10

    icegatheringstatechange – complete

    Examaple Candidate Checking

    iceconnectionstatechange : checking

    setRemoteDescription L type: answer, sdp: v=0
    …
    m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 110 112 113 126
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    a=ice-ufrag:ydvf
    a=ice-pwd:mb4ousBoT6B0l//ljjD/9Z/M
    a=ice-options:trickle
    …
    m=video 9 UDP/TLS/RTP/SAVPF 98 100 96 97 99 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116
    c=IN IP4 0.0.0.0
    a=rtcp:9 IN IP4 0.0.0.0
    a=ice-ufrag:ydvf
    a=ice-pwd:mb4ousBoT6B0l//ljjD/9Z/M
    a=ice-options:trickle

    addIceCandidate (host)
    sdpMid: , sdpMLineIndex: 0, candidate: candidate:1511920713 1 udp 2122260223 192.168.0.2 56060 typ host generation 0 ufrag ydvf network-id 1 network-cost 10

    iceconnectionstatechange : connected

    Candidate Nomination for Media Path

    selecting low-latency media paths can use various techniques such as actual round-trip time (RTT) measurement
    controlling agent gets to nominate which candidate pairs will get used for media amongst the ones that are valid. Ways
    regular nomination and aggressive nomination

    TBD

    To read More on WebRTC Communication as a platform

    WebRTC Media Stack

    WebRTC service’s

    References :

    http://w3c.github.io/webrtc-pc/ WebRTC 1.0: Real-time Communication Between Browsers – W3C Editor’s Draft 31 August 2019
    RFC 5245 Inter