WebRTC Audio/Video Codecs


Codecs signifies the media stream’s compession and decompression. For peers to have suceesfull excchange of media, they need a common set of codecs to agree upon for the session. The list codecs are sent  between each other as part of offeer and answer or SDP in SIP.

As WebRTC provides containerless bare mediastreamgtrackobjects. Codecs for these tracks is not mandated by webRTC . Yet the codecs are specified by two seprate RFCs

  1. RFC 7878 WebRTC Audio Codec and Processing Requirements specifies least the Opus codec as well as G.711’s PCMA and PCMU formats.
  2. RFC 7742 WebRTC Video Processing and Codec Requirnments specifies support for  VP8 and H.264’s Constrained Baseline profile for video .

In WebRTC video is protected using Datagram Transport Layer Security (DTLS) / Secure Real-time Transport Protocol (SRTP). In this article we are going to dicuss Audio/Video Codecs processing requirnments only.

WebRTC is free and opensource and its woring bodies promote royality free codecs too. The working groups RTCWEB and IETF make the sure of the fact that non-royality beraning codec are mandatory while other codecs can be optional in WebRTC non browsers .

WebRTC Browsers MUST implement the VP8 video codec as described in RFC6386 and H.264 Constrained Baseline described in RFC 7442.

WebRTC Video Codec and Processing Requirements
Media Flow in WebRTC Call

WebRTC Video Codecs

Most of the codesc below follow Lossy DCT(discrete cosine transform (DCT) based algorithm for encoding. Sample SDP from offer in Chrome browser v80 for Linux incliudes these profile :

m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123
a=rtcp-mux
a=rtcp-rsize

a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98

a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100

a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=102

a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=127

a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125

a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 red/90000
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=124

VP8

Developed by on2 and then acquired and opensource by google.

libvpx encoder library.

  • Supported conatiner – 3GP, Ogg, WebM
  • (+) supported simulcast
  • (+) Now free of royality fees.
  • (+) No limit on frame rate or data rate

Maximum resolution of 16384×16384 pixels.

VP8 encoders must limit the streams they send to conform to the values indicated by receivers in the corresponding max-fr and max-fs SDP attributes.

Encode and decode pixels with an implied 1:1 (square) aspect ratio.

VP9

Video Processor 9 (VP9) is the successor to the older VP8 and comparable to HEVC as they both have simillar bit rates.

  • supported Containers are – MP4, Ogg, WebM
  • (+) Open and free of royalties and any other licensing requirements

H264/AVC constrained

AVC’s Constrained Baseline (CBP ) profile compliant with WebRTC.

  • propertiary, patented codec, mianted by MPEG / ITU

Constrained Baseline Profile Level 1.2 and H.264 Constrained High Profile Level 1.3 . Contrained baseline is a submet of the main profile , suited to low dealy , low complexity. suited to lower processing device like mobile videos

Multiview Video Coding – can have multiple views of the same scene ,such as stereoscopic video.

Other profiles , which are not supporedt are Baseline(BP), Extended(XP), Main(MP) , High(HiP) , Progressive High(ProHiP) , High 10(Hi10P), High 4:2:2 (Hi422P) and High 4:4:4 Predictive

  • supported containers are 3GP, MP4, WebM

Parameter settings:

  • packetization-mode
  • max-mbps, max-smbps, max-fs, max-cpb, max-dpb, and max-br
  • sprop-parameter-sets: H.264 allows sequence and picture information to be sent both in-band and out-of-band. WebRTC implementations must signal this information in-band.
  • Supplemental Enhancement Information (SEI) “filler payload” and “full frame freeze” messages( used while video switching in MCU streams )

AV1 (AOMedia Video 1)

open format designed by the Alliance for Open Media. It is royality free and especially designed for internet video HTML element and WebRTC.

  • higher data compression rates than VP9 and H.265/HEVC

offers 3 profiles in increasing support for color depths and chroma subsampling.
1. main,
2. high, and
3. professional

  • supports HDR
  • supports Varible Frame Rate
  • Supported container are ISOBMFF, MPEG-TS, MP4, WebM

Stats for Video based media stream track

timestamp 04/05/2020, 14:25:59
ssrc 3929649593
isRemote false
mediaType video
kind video
trackId RTCMediaStreamTrack_sender_2
transportId RTCTransport_0_1
codecId RTCCodec_1_Outbound_96
[codec] VP8 (payloadType: 96)
firCount 0
pliCount 9
nackCount 476
qpSum 912936
[qpSum/framesEncoded] 32.86666666666667
mediaSourceId RTCVideoSource_2
packetsSent 333664
[packetsSent/s] 29.021823604499957
retransmittedPacketsSent 0
bytesSent 342640589
[bytesSent/s] 3685.7715977714947
headerBytesSent 8157584
retransmittedBytesSent 0
framesEncoded 52837
[framesEncoded/s] 30.022576142586164
keyFramesEncoded 31
totalEncodeTime 438.752
[totalEncodeTime/framesEncoded_in_ms] 3.5333333333331516
totalEncodedBytesTarget 335009905
[totalEncodedBytesTarget/s] 3602.7091371103397
totalPacketSendDelay 20872.8
[totalPacketSendDelay/packetsSent_in_ms] 6.89655172416302
qualityLimitationReason bandwidth
qualityLimitationResolutionChanges 20
encoderImplementation libvpx
Graph for Video Track in chrome://webrtc-internals

Non WebRTC supported Video codecs

Need active realtime media transcoding

H.263

Already used for video conferencing on PSTN (Public Switched Telephone Networks), RTSP, and SIP (IP-based videoconferencing) systems.

  • suited for low bandwidth networks
  • (-) not comaptible with WebRTC
    • but many media gateways incldue realtime transcoding existed between H263 based SIP systems and vp8 based webrtc ones to enable video communication between them

H.265 / HEVC

proprietary format and is covered by a number of patents. Licensing is managed by MPEG LA .

  • Container – Mp4

Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints

With the rise of Internet of Things many Endpoints especially IP cameras connected to Raspberry Pi like SOC( system on chiops )n wanted to stream directly to the browser within theor own provate network or even on public network using TURN / STUN.

The figure below shows how such a call flow is possible between an IP cemera ( such as Baby Cam ) and its parent monitoring it over a WebRTC suppported mobile phone browser . The process includes streaming teh content from IOT device on RTSP stream and using realtime trans-coding between H264 and VP8

Interoprabiloity between non WebRT Compatible and WebRTC compatible endpoints

WebRTC Audio Codecs

source : unknown

WebRTC endpoints are should implement audio codecs: OPUS and PCMA / PCMU, along with Comforrt Noise and DTMF events.

Trace for audio codecs supported in chrome (Version 80.0.3987.149 (Official Build) (64-bit) on ubuntu)

m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126

a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000

Opus

Opus is a lossy audio compression format developed by the Internet Engineering Task Force (IETF) targeting a broad range of interactive real-time applications over the Internet, from speech to music and supportes multiple compression algorithms

  • Constant and variable bitrate encoding – 6 kbit/s to 510 kbit/s
  • frame sizes – 2.5 ms to 60 ms
  • sampling rates – 8 kHz (with 4 kHz bandwidth) to 48 kHz (with 20 kHz bandwidth, where the entire hearing range of the human auditory system can be reproduced).
  • container- Ogg, WebM, MPEG-TS, MP4

As an open format standardized through RFC 6716, a reference implementation is provided under the 3-clause BSD license. All known software patents which cover Opus are licensed under royalty-free terms.

  • (+ ) flexible, suited for speech ( by SILK) and music ( CELT)
  • (+) support for mono and stereo
  • (+) inbuild FEC( Forward Error Correction) thus resilient to packet loss
  • (+) compression adjustability\ for unpredictable networks
  • (-) Highly CPU intensive ( unsuitable for embedded devices like rpi)
  • (-) processing and memory intensive

For all cases where the endpoint is able to process audio at a sampling rate higher than 8 kHz, it is w3C recommends that Opus be offered before PCMA/PCMU.

AAC (Advanvced Audio Encoding)

part of the MPEG-4 (H.264) standard. Lossy compression but has number pf profiles suiting each usecase like high quality surround sound to low-fidelity audio for speech-only use.

  • supported containers – MP4, ADTS, 3GP

G.711 (PCMA and PCMU)

G.711 is an ITU standard (1972) for audio compression. It is primarily used in telephony.

ITU published Pulse Code Modulation (PCM) with either µ-law or A-law encoding.
vital to interface with the standard telecom network and carriers. G.711 PCM (A-law) is known as PCMA and G.711 PCM (µ-law) is known as PCMU

It is the required standard in many voice-based systems and technologies, for example in H.320 and H.323 specifications.

  • Fixed 64Kbpd bit rate
  • supports 3GP container formats

G.722

ITU standard (1988) Encoded using Adaptive Differential Pulse Code Modulation (ADPCM) which is suited for voice compression

  • 7 kHz Wideband audio codec operating
  • Bitrate 48, 56 and 64 kbit/s.
  • containers used 3GP, AMR-WB

G722 improved speech quality due to a wider speech bandwidth of up to 50-7000 Hz compared to G.711 of 300–3400 Hz.

Comfort noise (CN)

artificial background noise which is used to fill gaps in a transmission instead of using pure silence. It prevents – jarring or RTP Timeout.

Should be used for streams encoded with G.711 or any other supported codec that does not provide its own CN. Use of Discontinuous Transmission (DTX) / CN by senders is optional

Internet Low Bitrate Codec (iLBC)

A opensource narrowband speech codec for VoIP and streaming audio.

  • 8 kHz sampling frequency with a bitrate of 15.2 kbps for 20ms frames and 13.33 kbps for 30ms frames.
  • Defined by IETF RFCs 3951 and 3952.

Internet Speech Audio Codec (iSAC)

iSAC: A wideband and super wideband audio codec for VoIP and streaming audio. It is designed for voice transmissions which are encapsulated within an RTP stream.

  • 16 kHz or 32 kHz sampling frequency
  • adaptive and variable bit rate of 12 to 52 kbps.

Speex

patent-free audio compression format designed for speech and also a free software speech codec that is used in VoIP applications and podcasts. May be obsolete, with Opus as its official successor.

AMR-WB Adaptive Multi-rate Wideband is a patented wideband speech coding standard that provides improved speech quality. This is codec is generally available on mobile phones.

  • wider speech bandwidth of 50–7000 Hz.
  • data rate is between 6-12 kbit/s, and the

DTMF and ‘audio/telephone-event’ media type

endpoints may send DTMF events at any time and should suppress in-band dual-tone multi-frequency (DTMF) tones, if any.

DTMF events list

| 0 | DTMF digit "0"
| 1 | DTMF digit "1"
| 2 | DTMF digit "2"
| 3 | DTMF digit "3"
| 4 | DTMF digit "4"
| 5 | DTMF digit "5"
| 6 | DTMF digit "6"
| 7 | DTMF digit "7"
| 8 | DTMF digit "8"
| 9 | DTMF digit "9"
| 10 | DTMF digit "*"
| 11 | DTMF digit "#"
| 12 | DTMF digit "A"
| 13 | DTMF digit "B"
| 14 | DTMF digit "C"
| 15 | DTMF digit "D"

Stats for Audio Media track

Stats for Audio Media include

  • headerBytesSent
  • packetsSent
  • bytesSent
timestamp 04/05/2020, 14:25:59
ssrc 3005719707
isRemote fals
mediaType audio
kind audio
trackId RTCMediaStreamTrack_sender_1
transportId RTCTransport_0_1
codecId RTCCodec_0_Outbound_111
[codec] opus (payloadType: 111)
mediaSourceId RTCAudioSource_1
packetsSent 88277
[packetsSent/s] 50.03762690431027
retransmittedPacketsSent 0
bytesSent 1977974
[bytesSent/s] 150.11288071293083
headerBytesSent 2118648
retransmittedBytesSent 0
Graphs in chrome://webrtc-internals for Audio

DataChannel

m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:blj+
a=ice-pwd:Ytdofc24WZYWRAnyNSNhuF4F
a=ice-options:trickle
a=fingerprint:sha-256 18:2F:B9:13:A1:BA:33:0C:D0:59:DB:83:9A:EA:38:0B:D7:DC:EC:50:20:6E:89:54:CC:E8:70:10:80:2B:8C:EE
a=setup:active
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

Stats for Datachannel

Statistics RTCDataChannel_1
timestamp 04/05/2020, 14:25:59
label sctp
protocol
datachannelid 1
state open
messagesSent 1
[messagesSent/s] 0
bytesSent 228
[bytesSent/s] 0
messagesReceived 1
[messagesReceived/s] 0
bytesReceived 228
[bytesReceived/s] 0

Refrenecs :

Quick links : If you are new to WebRTC read : Introduction to WebRTC is at https://telecom.altanai.com/2013/08/02/what-is-webrtc/

Layers of WebRTC at https://telecom.altanai.com/2013/07/31/webrtc/

RTCP Reports and QoE metric calculation

RTCP works alongside RTP to monitor and control media streams with QoS feedback, synchronization and session management . This writeup describes the key format and functions of this protocol

  1. RTCP (Real-Time Transport Control Protocol )
    1. RTCP Control and Management
    2. Gathers statistics on media connection
    3. SR: Sender Report RTCP Packet
    4. RR: Receiver Report RTCP Packet
    5. SDES: Source Description RTCP Packet
    6. BYE: Goodbye RTCP Packet
    7. APP: Application-Defined RTCP Packet
  2. RTCP XR (Extended Reports) 
  3. Extended RTP Profile for RTCP Based Feedback (RTP/AVPF)
  4. RTCP operation modes
  5.  RTCP for multicast sessions with unicast feedback
  6. RTCP Extensions for Multiplexed Media Streams

    RTCP (Real-Time Transport Control Protocol )

    Real-time Transport Control Protocol (RTCP) defined in RFC 3550, is used to send control packets and feedback on QoS to participants in a call along with RTP which sends actual media packets. RTCP provides monitoring of the data delivery, qos in a manner scalable to large multicast networks, and to provide minimal control and identification functionality.

    RTCP is typically on port RTP+1, e.g., RTP=5004 → RTCP=5005.
    Also RTCP uses 5% of total session bandwidth (RTP uses 95%). One can also adjust RTCP report intervals to avoid congestion.

    RTCP Control and Management

    RTCP provides feedback on the quality of the data distribution, congestion control, fault diagnosis, control of adaptive encoding. It is a periodic transmission of control packets. Since the control traffic is not self-limiting, the RR (Receiver Reports) from participants should be rate adjusted to limit traffic to the sender thus we observe for number of participants to estimate RTCP Transmission Interval for scaling up. This allows the communication system to monitor multimedia delivered on large multicast networks with hundreds of receivers. The underlying protocol must provide multiplexing of the data and control packets to convey minimal session control information such as Bytes sent, packets sent, lost packets, jitter, feedback and round trip delay.

    Gathers statistics on media connection

    Some metrics gathered by RTCP reports are :

    • timestamps
      • tp : last RTCP transmit time 
      • tr : curr time
      • tn : next schedule RTCP transmission time 
      • pmembers : last estimated count of members 
      • members: a current estimate of the number of session members 
      • senders 
      • rtcp_bw
      • avg_rtcp_size
    • flags as 
      • initial 
      • we_sent if the participant is sender ie has sent an RTP packet 
    • constants 
      • n is set to the number of receivers = members – senders
      • C : If sender then C =avg_rtcp_size / 25% of rtcp_bw else C=avg_rtcp_size / 75% of rtcp_bw

    Time Intervals calculation should be random but should give at least 25 of bw to senders and if sender >25% of members then split equally. This is done so that it is uniformly distributed and should avoid unintended synchronization or burst of RTCP packets to the sender. 

    • step 1 : Tmin=2.5 seconds if not yet sent an RTCP packet, else Tmin=5 seconds.
    • step 2 : Td = max(Tmin, n*C)
    • step 3 : T = 0.5 or 1.5 times Td.  
    • step 4 : resulting T is divided by e-3/2=1.21828 to compensate for the fact that the timer reconsideration algorithm converges to a value of the RTCP bandwidth below the intended average.

    Application may use this information to increase the quality of service, perhaps by limiting flow or using a different codec. 

    RTCP often uses the next consecutive port( odd number) as RTP( even number). Example screenshot shows port 20720 for RTP

    And next consecutive port 20721 for RTCP

    When RTCP is not being used or the CNAME identifier corresponding to a synchronization source has not been received yet, the participant associated with a synchronization source is not known.

    • (+) RTCP helps in monitoring the quality of service for every session
    • (+) RTCP sender and receiver reports allow the implementation of adaptive streaming where senders scale their bandwidth consumption based on network load.
    • (+) RCP SDES contains additional information like CNAME which helps in tracing of troublesome multimedia sources.( via email , phone number etc )

    Types of RTCP packet

    1. SR: Sender report, for transmission and reception statistics from participants that are active senders
    2. RR: Receiver report, for reception statistics from participants that are not active senders and in combination with SR for active senders reporting on more than 31 sources
    3. SDES: Source description items, including CNAME,email or phone
    4. BYE: Indicates end of participation
    5. APP: Application-specific functions

    SR is issued if a site has sent any data packets during the interval since issuing the last report or the previous one, otherwise the RR is issued.

    SR: Sender Report RTCP Packet

    Sender Report RTCP Packet.

    Expanded Sender Report RTCP Packet has sender information is 20 octets long and is present in every sender report packet. It summarizes the data transmissions from this sender.

    • NTP timestamp 64 bit
    • RTP Timestamp 32 bit
    • sender’s packet count: 32 bits, total number of RTP data packets transmitted by the sender since starting transmission up until the time this SR packet was generated.
    • sender’s octet count: 32 bits
    SR Report in RTCP

    Explanation for some attributes

    • highest sequence number received: 32 bits
    • fraction lost: 8 bits, fraction of RTP data packets from source SSRC_n lost since the previous SR or RR packet was sent
    • cumulative number of packets lost: 24 bits size, total number of RTP data packets from source SSRC_n that have been lost since the beginning of reception.
    • interarrival jitter: 32 bits, estimate of the statistical variance of the RTP data packet interarrival time, measured in timestamp unit. Jitter J is mean deviation (smoothed absolute value) of the difference D is packet spacing at the receiver compared to the sender for a pair of packets.
    RTCP SR ( Senders Report)

    Synchronization and exposing delays using RTCP : For multimedia conferences the NTP timestamp from RTCP SR is used to give a common time reference that can associate these independent timestamps with a wall clock shared time. the NTP timestamps also help the endpoints measure their delays.

    RR: Receiver Report RTCP Packet

    Snapshot

    SDES: Source Description RTCP Packet

    abbrev. name value

    • END end of SDES list 0
    • CNAME canonical name 1
    • NAME user name 2
    • EMAIL user’s electronic mail address 3
    • PHONE user’s phone number 4
    • LOC geographic user location 5
    • TOOL name of application or tool 6
    • NOTE notice about the source 7
    • PRIV private extensions 8

    BYE: Goodbye RTCP Packet

    APP: Application-Defined RTCP Packet

    Intended for experimental use

    Instance of RTCP sender and receiver reports on transmission and reception statistics

    Real-time Transport Control Protocol (Receiver Report)
        [Stream setup by SDP (frame 4)]
            [Setup frame: 4]
            [Setup Method: SDP]
        10.. .... = Version: RFC 1889 Version (2)
        ..0. .... = Padding: False
        ...0 0001 = Reception report count: 1
        Packet type: Receiver Report (201)
        Length: 7 (32 bytes)
        Sender SSRC: 0x796dd0d6 (2037240022)
        Source 1
            Identifier: 0x00000000 (0)
            SSRC contents
                Fraction lost: 0 / 256
                Cumulative number of packets lost: 1
            Extended highest sequence number received: 6534
                Sequence number cycles count: 0
                Highest sequence number received: 6534
            Interarrival jitter: 0
            Last SR timestamp: 0 (0x00000000)
            Delay since last SR timestamp: 0 (0 milliseconds)
    Real-time Transport Control Protocol (Source description)
        [Stream setup by SDP (frame 4)]
            [Setup frame: 4]
            [Setup Method: SDP]
        10.. .... = Version: RFC 1889 Version (2)
        ..0. .... = Padding: False
        ...0 0001 = Source count: 1
        Packet type: Source description (202)
        Length: 6 (28 bytes)
        Chunk 1, SSRC/CSRC 0x796DD0D6
            Identifier: 0x796dd0d6 (2037240022)
            SDES items
                Type: CNAME (user and domain) (1)
                Length: 8
                Text: 796dd0d6
                Type: NOTE (note about source) (7)
                Length: 5
                Text: telecomorg
                Type: END (0)

    Negative Acknowledgment (NACK) packets can be used to explicitly indicate that packets have not been received.

    Full Intra Request (FIR) and Picture Loss Indication (PLI) packets are used for video to indicate that there is a need for the sender to produce a refresh point( key frame) in the stream.

    Receiver-Estimated Maximum Bitrate (REMB) feedback packets signal to a sender the maximum bitrate a receiver wishes to receive.

    Transport-wide Congestion Control (TCC) feedback packets are used to provide detailed packet-by-packet reception information from a receiver to the sender.

    RTCP XR (Extended Reports) 

    The purpose of the extended reporting format is to convey information that supplements the six statistics that are contained in the report blocks used by RTCP’s Sender Report (SR) and Receiver Report (RR) packets.

        0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | BT | type-specific | block length |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    : type-specific block contents :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Categories

    1. packet-by-packet reports on received or lost RTP packets

    Loss RLE Report Block (1)

        0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | BT=1 | rsvd. | T | block length |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | SSRC of source |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | begin_seq | end_seq |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | chunk 1 | chunk 2 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    : ... :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | chunk n-1 | chunk n |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Duplicate RLE Report Block ( 2)

    Packet Receipt Times Report Block (3)

        0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | BT=3 | rsvd. | T | block length |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | SSRC of source |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | begin_seq | end_seq |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Receipt time of packet begin_seq |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Receipt time of packet (begin_seq + 1) mod 65536 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    : ... :
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Receipt time of packet (end_seq - 1) mod 65536 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    1. reference time information between RTP participants

    Receiver Reference Time Report Block : Receiver-end wallclock timestamps(4)
    DLRR Report Block : delay since the last Receiver Reference Time Report Block was received (5)

    3. metrics relating to packet receipts, that are summary in nature

    Statistics summary block (6)

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | BT=6 |L|D|J|ToH|rsvd.| block length = 9 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | SSRC of source |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | begin_seq | end_seq |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | lost_packets |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | dup_packets |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | min_jitter |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | max_jitter |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | mean_jitter |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | dev_jitter |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | min_ttl_or_hl | max_ttl_or_hl |mean_ttl_or_hl | dev_ttl_or_hl |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    VOIP metric report block (7)

          0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | BT=7 | reserved | block length = 8 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | SSRC of source |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | loss rate | discard rate | burst density | gap density |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | burst duration | gap duration |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | round trip delay | end system delay |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | signal level | noise level | RERL | Gmin |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | R factor | ext. R factor | MOS-LQ | MOS-CQ |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | RX config | reserved | JB nominal |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | JB maximum | JB abs max |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      Extended RTP Profile for RTCP Based Feedback (RTP/AVPF)

      RTP provides continuous feedback about the overall reception quality from all receivers — thereby allowing the sender(s) in the mid-term to adapt their coding scheme and transmission behavior to the observed network quality of service (QoS).

      RTP makes no provision for timely feedback that would allow a sender to repair the media stream immediately: through retransmissions, retroactive Forward Error Correction (FEC) control, or media-specific mechanisms for some video codecs, such as reference picture selection.

      Components of RTCP based feedback

      • Status reports contained in sender report (SR)/received report (RR) packet transmitted at regular intervals . Can also contain SDES
      • FB ( Feedback ) messages . Indicate loss or reception of particular pieces of a media stream

      Types of RTCP Feedback packet

      Minimal compound RTCP feedback packet

      This is to minimize the size of the RTCP packet transmitted to convey feedback and maximize the frequency at which feedback can be provided. MUST contain only the mandatory information :

      • encryption prefix if necessary,
      • exactly one RR or SR,
      • exactly one SDES with only the CNAME item present, and
      • FB message(s)

      Full compound RTCP feedback packet

      MAY contain any additional number of RTCP packet

      RTCP operation modes

      1. Immediate Feedback mode
      2. Early RTCP mode
      3. Regular RTCP Mode

      The Application specific feedback threshold is a function of a number of parameters including (but not necessarily limited to):

      • type of feedback used (e.g., ACK vs. NACK),
      • bandwidth,
      • packet rate,
      • packet loss
      • probability and distribution,
      • media type,
      • codec,
      • (worst case or observed) frequency of events to report (e.g., frame received, packet lost).

      Payload specific Feedback messages

      Three payload-specific FB messages are defined so far plus an application layer FB message. They are identified by means of the FMT parameter as follows:

      • 0: unassigned
      • 1: Picture Loss Indication (PLI)
      • 2: Slice Loss Indication (SLI)
      • 3: Reference Picture Selection Indication (RPSI)
      • 4-14: unassigned
      • 15: Application layer FB (AFB) message
      • 16-30: unassigned
      • 31: reserved for future expansion of the sequence number space

       RTCP for multicast sessions with unicast feedback

      Single-source multicast sessions (e.g., IPTV, live streaming) where receivers provide feedback via unicast RTCP. These feedbacks can quickly overwhelm the sender as it will receive as many feedback as the number of viewers which is many folds in case of large scale deployments for example webinars.

      To mitigate this feedback implosion, sender aggregates reports instead of processing individual multicast feedback in Multicast Acquisition Report (MAR)

      a=rtcp-fb:* x-mar unicast   // Supports MAR over unicast
      a=rtcp-fb:* nack unicast // NACKs sent via unicast

      RTCP Extensions for Multiplexed Media Streams

      For multiplexed media streams , where different kinds of media share a common port, we use payload type and SSRC to distinguish streams. we can mux the rtcp too. The Offer/Answer Negotiation has the following attribute in SDP

      a=rtcp-mux
      FeaturesRTP and RTCP on different port RTP , RTCP mux
      Number of ports21
      NAT simplicity complex with per stream overhead for ICE candidates gatheringrelatively simple , only 1 pinhole

      References :

      • RFC 3611 RTP Control Protocol Extended Reports (RTCP XR)
      • RFC 4585 Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)
      • RFC 7002 RTP Control Protocol (RTCP) Extended Report (XR) Block for Discard Count Metric Reporting
      • RFC 7003 RTP Control Protocol (RTCP) Extended Report (XR) Block for Burst/Gap Discard Metric Reporting

      RTPengine on kamailio SIP server


      RTPengine is a proxy for RTP traffic and other UDP based media traffic over either IPv4 or IPv6. It can even bridge between diff IP networks and interfaces. It can do TOS/QoS field setting. It is Multi-threaded, can advertise different addresses for operation behind NAT.

      This article focuses on setting up sipwise rtpegine to proxy RTP traffic from the Kamailio app server. This is an updated version of the old article on RTPEngine, since then there have many many updates on the software. I also wrote an article covering all relevant and important Kamailio modules earlier including RTPProxy and RTP engine https://telecom.altanai.com/2014/11/18/kamailio-modules/

      It bears in-kernel packet forwarding for low-latency and low-CPU performance. When used with the Kamailio, the RTP engine module adds more features to media stream routing and management, especially around RTP proxy and Mos scores.

      Features

      • Full SDP parsing and rewriting
      • Supports non-standard RTCP ports (RFC 3605)
      • ICE (RFC 5245):
        • Bridging between ICE-enabled and ICE-unaware user agents
        • Optionally acting only as additional ICE relay/candidate
        • Optionally forcing relay of media streams by removing other ICE candidates
      • SRTP (RFC 3711):
        • Support for SDES (RFC 4568) and DTLS-SRTP (RFC 5764)
        • AES-CM and AES-F8 ciphers, both in userspace and in kernel
        • HMAC-SHA1 packet authentication
        • Bridging between RTP and SRTP user agents
      • RTCP profile with feedback extensions (RTP/AVPF, RFC 4585 and 5124)
      • Arbitrary bridging between any of the supported RTP profiles (RTP/AVP, RTP/AVPF, RTP/SAVP, RTP/SAVPF)
      • RTP/RTCP multiplexing (RFC 5761) and demultiplexing
      • Breaking of BUNDLE’d media streams (draft-ietf-mmusic-sdp-bundle-negotiation)
      • Recording of media streams, decrypted if possible
      • Transcoding and repacketization
      • Playback of pre-recorded streams/announcements

      Sipwise NGCP RTP Engine Source Code

      There are 3 parts of the source structure in sipwise NGCP ( Next Generation communication Platform) rtpengine :

      1.daemon

      The userspace daemon and workhorse, minimum requirement for anything to work. Running make will compile the binary, which will be called rtpengine.

      Required packages including their development headers are required to compile the daemon:

      • pkg-config
      • GLib including GThread and GLib-JSON version 2.x
      • zlib
      • OpenSSL
      • PCRE library
      • XMLRPC-C version 1.16.08 or higher
      • hiredis library
      • gperf
      • libcurl version 3.x or 4.x
      • libevent version 2.x
      • libpcap
      • libsystemd
      • MySQL or MariaDB client library (optional for media playback and call recording daemon)
      • libiptc library for iptables management (optional)
      • ffmpeg codec libraries for transcoding (optional) such as libavcodec, libavfilter, libswresample
      • bcg729 for full G.729 transcoding support (optional)

      options for make – with_iptables_option , with_transcoding

       with_transcoding=no make 

      2.iptables-extension

      Required for in-kernel packet forwarding. With the iptables development headers installed, issuing make will compile the plugin for iptables and ip6tables. The file will be called libxt_RTPENGINE.so and needs to be copied into the xtables module directory. The location of this directory can be determined through pkg-config xtables –variable=xtlibdir on newer systems, and/or is usually either /lib/xtables/ or /usr/lib/x86_64-linux-gnu/xtables/.

      3.kernel-module

      Required for in-kernel packet forwarding. Compilation of the kernel module requires the kernel development headers to be installed in/lib/modules/$VERSION/build/, where $VERSION is the output of the command uname -r.

      Successful compilation of the module will produce the file xt_RTPENGINE.ko. The module can be inserted into the running kernel manually through insmod xt_RTPENGINE.ko

      It is recommended to copy the module into /lib/modules/$VERSION/updates/, followed by running depmod -a.

      After this, the module can be loaded by issuing modprobe xt_RTPENGINE.

      Installation

      Follow instructions on https://gist.github.com/altanai/0d8cadbe6876d545fd63d6b3e79dcf73

      Requirements

      sudo su apt-get install debhelper iptables-dev libcurl4-openssl-dev libglib2.0-dev  libjson-glib-dev libxmlrpc-core-c3-dev libhiredis-dev build-essential:native

      For Pcap

      apt install ibpcap-dev

      Also instal ffmpeg pakages

      apt install libavcodec-dev libavfilter-dev libavformat-dev libavresample-dev  libavutil-dev

      Use dpkg

      libcrypt-openssl-rsa-perl libdigest-crc-perl libio-multiplex-perl libnet-interface-perl libsystemd-dev markdown

      For debhelper>10

      vi /etc/apt/sources.list

      add line

      deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse
      sudo apt update

      check version

      apt-cache policy debhelper dh-autoreconf
      debhelper:
      Installed: 9.20160115ubuntu3
      Candidate: 9.20160115ubuntu3
      Version table:
      10.2.2ubuntu1~ubuntu16.04.1 100
      100 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
      100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
      *** 9.20160115ubuntu3 500
      500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
      100 /var/lib/dpkg/status
      dh-autoreconf:
      Installed: (none)
      Candidate: 11
      Version table:
      12~ubuntu16.04.1 100
      100 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
      100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
      11 500
      500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

      Force installing the version from backports repo as it have low priority.

      sudo apt install dh-autoreconf=12~ubuntu16.04.1 debhelper=10.2.2ubuntu1~ubuntu16.04.1

      so now new priority will be

      debhelper:
      
        Installed: 10.2.2ubuntu1~ubuntu16.04.1
        Candidate: 10.2.2ubuntu1~ubuntu16.04.1
        Version table:
       *** 10.2.2ubuntu1~ubuntu16.04.1 100
              100 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
              100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
              100 /var/lib/dpkg/status
           9.20160115ubuntu3 500
              500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
      dh-autoreconf:
        Installed: 12~ubuntu16.04.1
        Candidate: 12~ubuntu16.04.1
        Version table:
       *** 12~ubuntu16.04.1 100
              100 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
              100 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages
              100 /var/lib/dpkg/status
           11 500
              500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
      
      ref :https://askubuntu.com/questions/863221/need-help-building-debhelper-10-2-2-bpo8-from-source

      Get sourcecode

      cd /usr/local/src
      git clone https://github.com/sipwise/rtpengine.git
      cd rtpengine
       ./debian/flavors/no_ngcp

      use dpkg-checkbuilddeps to find any missing dependices

      For missing dependencies

      dpkg-checkbuilddeps: error: Unmet build dependencies: libbcg729-dev
      remove the encoder for G.729 which is not supported by ffmoeg by exporting varible

      export DEB_BUILD_PROFILES="pkg.ngcp-rtpengine.nobcg729"

      Ref : https://github.com/sipwise/rtpengine#g729-support

      for defaultlibmysqlclient-dev and libiptc-dev

      vi debian/control
      change from default-libmysqlclient-dev to libmysqlclient-dev, change from libiptcdata-dev to libiptc-dev and install the alternatives such as

      apt install libmysqlclient-dev libiptcdata-dev 

      Generated deb files should be outside the rtpegine home folder

      generated ngcp-rtpegine deb files
      cd ..
      dpkg -i ngcp-rtpengine-daemon_7.3.0.0+0~mr7.3.0.0_amd64.deb
      dpkg -i ngcp-rtpengine-iptables_7.3.0.0+0~mr7.3.0.0_amd64.deb
      dpkg -i ngcp-rtpengine-kernel-dkms_7.3.0.0+0~mr7.3.0.0_all.deb
      dpkg -i ngcp-rtpengine-kernel-source_7.3.0.0+0~mr7.3.0.0_all.deb
      dpkg -i ngcp-rtpengine-recording-daemon_7.3.0.0+0~mr7.3.0.0_amd64.deb
      dpkg -i ngcp-rtpengine-utils_7.3.0.0+0~mr7.3.0.0_all.deb
      dpkg -i ngcp-rtpengine_7.3.0.0+0~mr7.3.0.0_all.deb
      After depackaging

      Manual installation and running all test cases

      cd rtpengine
      make check

      If you dont find a package you are looking for , some alternatives are to do apt-cache search like

      apt-cache search libavfilter
      libavfilter-dev - FFmpeg library containing media filters - development files
      libavfilter-ffmpeg5 - FFmpeg library containing media filters - runtime files

      or to search in ubuntu packages web https://packages.ubuntu.com/

      Running RTPEngine

      rtpegine application options

      • -v, –version Print build time and exit
      • –config-file=FILE Load config from this file
      • –config-section=STRING Config file section to use
      • –log-facility=daemon|local0|…|local7 Syslog facility to use for logging
      • -L, –log-level=INT Mask log priorities above this level
      • -E, –log-stderr Log on stderr instead of syslog
      • –no-log-timestamps Drop timestamps from log lines to stderr
      • –log-mark-prefix Prefix for sensitive log info
      • –log-mark-suffix Suffix for sensitive log info
      • -p, –pidfile=FILE Write PID to file
      • -f, –foreground Don’t fork to background
      • -t, –table=INT Kernel table to use
      • -F, –no-fallback Only start when kernel module is available
      • -i, –interface=[NAME/]IP[!IP] Local interface for RTP
      • -k, –subscribe-keyspace=INT INT … Subscription keyspace list
      • -l, –listen-tcp=[IP:]PORT TCP port to listen on
      • -u, –listen-udp=[IP46|HOSTNAME:]PORT UDP port to listen on
      • -n, –listen-ng=[IP46|HOSTNAME:]PORT UDP port to listen on, NG protocol
      • -c, –listen-cli=[IP46|HOSTNAME:]PORT UDP port to listen on, CLI
      • -g, –graphite=IP46|HOSTNAME:PORT Address of the graphite server
      • -G, –graphite-interval=INT Graphite send interval in seconds
      • –graphite-prefix=STRING Prefix for graphite line
      • -T, –tos=INT Default TOS value to set on streams
      • –control-tos=INT Default TOS value to set on control-ng
      • -o, –timeout=SECS RTP timeout
      • -s, –silent-timeout=SECS RTP timeout for muted
      • -a, –final-timeout=SECS Call timeout
      • –offer-timeout=SECS Timeout for incomplete one-sided calls
      • -m, –port-min=INT Lowest port to use for RTP
      • -M, –port-max=INT Highest port to use for RTP
      • -r, –redis=[PW@]IP:PORT/INT Connect to Redis database
      • -w, –redis-write=[PW@]IP:PORT/INT Connect to Redis write database
      • –redis-num-threads=INT Number of Redis restore threads
      • –redis-expires=INT Expire time in seconds for redis keys
      • -q, –no-redis-required Start no matter of redis connection state
      • –redis-allowed-errors=INT Number of allowed errors before redis is temporarily disabled
      • –redis-disable-time=INT Number of seconds redis communication is disabled because of errors
      • –redis-cmd-timeout=INT Sets a timeout in milliseconds for redis commands
      • –redis-connect-timeout=INT Sets a timeout in milliseconds for redis connections
      • -b, –b2b-url=STRING XMLRPC URL of B2B UA
      • –log-facility-cdr=daemon|local0|…|local7 Syslog facility to use for logging CDRs
      • –log-facility-rtcp=daemon|local0|…|local7 Syslog facility to use for logging RTCP
      • –log-facility-dtmf=daemon|local0|…|local7 Syslog facility to use for logging DTMF
      • –log-format=default|parsable Log prefix format
      • -x, –xmlrpc-format=INT XMLRPC timeout request format to use. 0: SEMS DI, 1: call-id only, 2: Kamailio
      • –num-threads=INT Number of worker threads to create
      • –media-num-threads=INT Number of worker threads for media playback
      • -d, –delete-delay=INT Delay for deleting a session from memory.
      • –sip-source Use SIP source address by default
      • –dtls-passive Always prefer DTLS passive role
      • –max-sessions=INT Limit of maximum number of sessions
      • –max-load=FLOAT Reject new sessions if load averages exceeds this value
      • –max-cpu=FLOAT Reject new sessions if CPU usage (in percent) exceeds this value
      • –max-bandwidth=INT Reject new sessions if bandwidth usage (in bytes per second) exceeds this value
      • –homer=IP46|HOSTNAME:PORT Address of Homer server for RTCP stats
      • –homer-protocol=udp|tcp Transport protocol for Homer (default udp)
      • –homer-id=INT ‘Capture ID’ to use within the HEP protocol
      • –recording-dir=FILE Directory for storing pcap and metadata files
      • –recording-method=pcap|proc Strategy for call recording
      • –recording-format=raw|eth File format for stored pcap files
      • –iptables-chain=STRING Add explicit firewall rules to this iptables chain
      • –codecs Print a list of supported codecs and exit
      • –scheduling=default|none|fifo|rr|other|batch|idle Thread scheduling policy
      • –priority=INT Thread scheduling priority
      • –idle-scheduling=default|none|fifo|rr|other|batch|idle Idle thread scheduling policy
      • –idle-priority=INT Idle thread scheduling priority
      • –log-srtp-keys Log SRTP keys to error log
      • –mysql-host=HOST|IP MySQL host for stored media files
      • –mysql-port=INT MySQL port
      • –mysql-user=USERNAME MySQL connection credentials
      • –mysql-pass=PASSWORD MySQL connection credentials
      • –mysql-query=STRING MySQL select query

      Run command

      rtpengine --interface="10.10.10.10" --listen-ng=25061 --listen-cli=25062 --foreground --log-stderr --listen-udp=25060 --listen-tcp=25060

      In-Kernal Packet Forwarding

      To avoid the overhead involved in processing each individual RTP packet in userspace-only operation, especially as RTP traffic consists of many small packets at high rates, rtpengine provides a kernel module to offload the bulk of the packet forwarding duties from user space to kernel space. This also results in increasing the number of concurrent calls as CPU usage decreases.In-kernel packet forwarding is implemented as an iptables module (x_tables) and has 2 parts – xt_RTPENGINE and plugin to the iptables and ip6tables command-line utilities

      Sequence of events for a newly established media stream is then:

      1. Kamailio as SIP proxy controls rtpengine and signals it about a newly established call.
      2. Rtpengine daemon allocates local UDP ports and sets up preliminary forward rules based on the info received from the SIP proxy.
      3. An RTP packet is received on the local port.
      4. It traverses the iptables chains and gets passed to the xt_RTPENGINE module.
      5. The module doesn’t recognize it as belonging to an established stream and thus ignores it.
      6. The packet continues normal processing and eventually ends up in the daemon’s receive queue.
      7. The daemon reads it, processes it and forwards it. It also updates some internal data.
      8. This userspace-only processing and forwarding continues for a little while, during which time information about additional streams and/or endpoints may be obtained from the SIP proxy.
      9. After a few seconds, when the daemon is satisfied with what it has learned about the media endpoints, it pushes the forwarding rules to the kernel.
      10. From this moment on, the kernel module will recognize incoming packets belonging to those streams and will forward them on its own. It will stop those packets from traversing the network stacks any further, so the daemon will not see them any more on its receive queues.
      11. In-kernel forwarding is allowed to cease to work at any given time, either accidentally (e.g. by removal of the iptablesrule) or deliberatly (the daemon will do so in case of a re-invite), in which case forwarding falls back to userspace-only operation.

      Kernel Module

      The kernel module supports multiple forwarding tables, identified through their ID number, bydefault 0 to 63. Each running instance of the rtpengine daemon controls one such table.

      To load use modprobe xt_RTPENGINE and to unload rmmod xt_RTPENGINE. With the module loaded, a new directory will appear in /proc/, namely /proc/rtpengine/, containing pseudo-files, control ( to create and delete forwarding tables) and list ( list of currently active forwarding tables)

      To manually create a forwarding table with ID 33, the following command can be used:

      echo 'add 43' > /proc/rtpengine/control

      iptables module

      In order for the kernel module to be able to actually forward packets, an iptables rule must be set up to send packets into the module. Each such rule is associated with one forwarding table. In the simplest case, for forwarding table 33, this can be done through:

      iptables -I INPUT -p udp -j RTPENGINE --id 33

      To restrict the rules to the UDP port range used by rtpengine, e.g. by supplying a parameter like –dport 30000:40000. If the kernel module receives a packet that it doesn’t recognize as belonging to an active media stream, it will simply ignore it and hand it back to the network stack for normal processing.

      A typical start-up sequence including in-kernel forwarding might look like this:

      modprobe xt_RTPENGINE
      iptables -I INPUT -p udp -j RTPENGINE --id 0
      ip6tables -I INPUT -p udp -j RTPENGINE --id 0

      ensure that the table we want to use doesn’t exist – usually needed after a daemon restart, otherwise will error

      echo 'del 0' > /proc/rtpengine/control

      start daemon

      /usr/sbin/rtpengine --table=0 --interface=10.64.73.31 --interface=2001:db8::4f3:3d \
      --listen-ng=127.0.0.1:2223 --tos=184 --pidfile=/run/rtpengine.pid --no-fallback

      Running Multiple Instances

      To run multiple instances of rtpengine on the same machine run multiple instances of the daemon using different command-line options ( local addresses and listening ports), together with multiple different kernel forwarding tables.

      For example, if one local network interface has address 10.64.73.31 and another has address 192.168.65.73, then the start-up sequence might look like this:

      modprobe xt_RTPENGINE
      
      iptables -I INPUT -p udp -d y.y.y.y -j RTPENGINE --id 0
      iptables -I INPUT -p udp -d x.x.x.x -j RTPENGINE --id 1
      echo 'del 0' > /proc/rtpengine/control
      echo 'del 1' > /proc/rtpengine/control
      /usr/sbin/rtpengine --table=0 --interface=<ip> \
      --listen-ng=127.0.0.1:2223 --tos=184 --pidfile=/run/rtpengine-10.pid --no-fallback
      /usr/sbin/rtpengine --table=1 --interface=<ip_pvy>\
      --listen-ng=127.0.0.1:2224 --tos=184 --pidfile=/run/rtpengine-192.pid --no-fallback

      With this setup, the SIP proxy can choose which instance of rtpengine to talk to and thus which local interface to use by sending its control messages to either port 2223 or port 2224.

      Transcoding

      Currently transcoding is supported for audio streams. Can be turned off with with_transcoding=no option in makeFile.

      Normally rtpengine leaves codec negotiation up to the clients involved in the call and does not interfere. In this case, if the clients fail to agree on a codec, the call will fail.

      Transcoding options in the ng control protocol,  transcode or ptime. If a codec is requested via the transcode option that was not originally offered, transcoding will be engaged for that call. With transcoding active for a call, all unsupported codecs will be removed from the SDP.

      Transcoding happens in userspace only, so in-kernel packet forwarding will not be available for transcoded codecs. Codecs that are supported by both sides will simply be passed through transparently (unless repacketization is active). In-kernel packet forwarding will still be available for these codecs.

      Codecs supported by rtpengine can be shown with –codecs options

      • rtpengine –codecs
      • PCMA: fully supported
      • PCMU: fully supported
      • G723: fully supported
      • G722: fully supported
      • QCELP: supported for decoding only
      • G729: supported for decoding only
      • speex: fully supported
      • GSM: fully supported
      • iLBC: not supported
      • opus: fully supported
      • vorbis: codec supported but lacks RTP definition
      • ac3: codec supported but lacks RTP definition
      • eac3: codec supported but lacks RTP definition
      • ATRAC3: supported for decoding only
      • ATRAC-X: supported for decoding only
      • AMR: supported for decoding only
      • AMR-WB: supported for decoding only
      • PCM-S16LE: codec supported but lacks RTP definition
      • PCM-U8: codec supported but lacks RTP definition
      • MP3: codec supported but lacks RTP definition

      ng Control Protocol

      Advanced control protocol to pass SDP body from the SIP proxy to the rtpengine daemon, has the body rewritten in the daemon, and then pas back to the SIP proxy to embed into the SIP message. It is  based on the bencode standard and runs over UDP transport.

      Each message passed between the SIP proxy and the media proxy contains of two parts:

      1. message cookie ( to match requests to responses, and retransmission detection) and
      2. bencoded dictionary

      The dictionary of each request must contain at least one key called command and corresponding value must be a string and determines the type of message. Currently the following commands are defined:

      • ping
      • offer
      • answer
      • delete
      • query
      • start recording
      • stop recording
      • block DTMF
      • unblock DTMF
      • block media
      • unblock media
      • start forwarding
      • stop forwarding
      • play media
      • stop media

      The response dictionary must contain at least one key called result. The value can be either ok (optional key warning) or error( to be accompanied by error-reason). For the ping command, the additional value pong is allowed.

      rtpengine.sample.conf

      [rtpengine]
      
      table = 0
      no-fallback = false
      for userspace forwarding only:
      table = -1
      
      // separate multiple interfaces with semicolons:
      interface = internal/12.23.34.45;external/23.34.45.54
      
      listen-ng = 127.0.0.1:2223
      listen-tcp = 25060
      listen-udp = 12222
      
      timeout = 60
      silent-timeout = 3600
      tos = 184
      control-tos = 184
      delete-delay = 30
      final-timeout = 10800
      
      foreground = false
      pidfile = /run/ngcp-rtpengine-daemon.pid
      num-threads = 16
      
      port-min = 30000
      port-max = 40000
      max-sessions = 5000
      
      recording-dir = /var/spool/rtpengine
      recording-method = proc
      recording-format = raw
      
      redis = 127.0.0.1:6379/5
      redis-write = password@x.x.x.x:6379/42
      redis-num-threads = 8
      no-redis-required = false
      redis-expires = 86400
      redis-allowed-errors = -1
      redis-disable-time = 10
      redis-cmd-timeout = 0
      redis-connect-timeout = 1000
      
      b2b-url = http://127.0.0.1:8090/
      xmlrpc-format = 0
      
      log-level = 6
      log-stderr = false
      log-facility = daemon
      log-facility-cdr = local0
      log-facility-rtcp = local1
      
      graphite = 127.0.0.1:9006
      graphite-interval = 60
      graphite-prefix = foobar.
      
      homer = 123.234.345.456:65432
      homer-protocol = udp
      homer-id = 2001
      
      sip-source = false
      dtls-passive = false

      ngcp-rtpengine-daemon Service

      To start the ngcp-rtpengine-daemon service

      /etc/init.d/ngcp-rtpengine-daemon start
      [ ok ] Starting ngcp-rtpengine-daemon (via systemctl): ngcp-rtpengine-daemon.service.
      

      Checking status ngcp-rtpengine-daemonservice

      # systemctl status ngcp-rtpengine-daemon.service
      
      ● ngcp-rtpengine-daemon.service - NGCP RTP/media Proxy Daemon
         Loaded: loaded (/lib/systemd/system/ngcp-rtpengine-daemon.service; disabled; vendor preset: enabled)
         Active: active (running) since Thu 2019-04-11 10:16:20 UTC; 24s ago
        Process: 13751 ExecStopPost=/usr/sbin/ngcp-rtpengine-iptables-setup stop (code=exited, status=0/SUCCESS)
        Process: 13797 ExecStartPre=/usr/sbin/ngcp-rtpengine-iptables-setup start (code=exited, status=0/SUCCESS)
       Main PID: 13814 (rtpengine)
          Tasks: 19
         Memory: 10.5M
            CPU: 102ms
         CGroup: /system.slice/ngcp-rtpengine-daemon.service
                 └─13814 /usr/sbin/rtpengine -f -E --no-log-timestamps --pidfile /run/ngcp-rtpengine-daemon.pid --config-file /etc/rtpengine/rtpengine.conf --table 0

      To start recording service

      /etc/init.d/ngcp-rtpengine-recording-daemon start
      

      RTP engine receives command offer

      Received command 'offer' from :53888
      Dump for 'offer' from :53888: {  
      "sdp":"v=0 
       o=- 1554978148897419 1 IN IP4 pvt_ip 
       s=Bria 3 release 3.5.5 stamp 71243 
       c=IN IP4 192.168.1.23 
       t=0 0 
       m=audio 50754 RTP/AVP 0 98 101 
       a=rtpmap:98 ILBC/8000 
       a=rtpmap:101 telephone-event/8000 
       a=fmtp:101 0-15 
       a=sendrecv 
       ",
          "ICE":"remove",
          "record-call":"yes",
          "direction":[  
             "internal",
             "internal"
          ],
          "flags":[  
             "no-rtcp-attribute"
          ],
          "replace":[  
             "origin",
             "session-connection"
          ],
          "transport-protocol":"RTP/AVP",
          "call-id":"732597d6-6d96-485b-b6dc-7d93703c1405",
          "received-from":[  
             "IP4",
             ""
      Creating new call
      Turning on call recording.
      Wrote metadata file to temporary path: /var/spool/rtpengine/tmp/
      ...

      RTP engine receives command delete

      Received command 'delete' from :57304
      Dump for 'delete' from :57304: { "call-id": "732597d6-6d96-485b-b6dc-7d93703c1405", "received-from": [ "IP4", "" ], "from-tag": "cb8a1e30", "command": "delete" }
      Deleting call branch 'cb8a1e30' (via-branch '')
      Call branch 'cb8a1e30' (via-branch '') deleted, no more branches remaining
        Deleting entire call
       INFO: [ID="732597d6-6d96-485b-b6dc-7d93703c1405"]: Final packet stats:
       --- Tag 'cb8a1e30', created 0:05 ago for branch '', in dialogue with ''
       ------ Media #1 (audio over RTP/AVP) using unknown codec
       --------- Port   :10044 <>    :50754, SSRC 0, 0 p, 0 b, 0 e, 5 ts
       freeing send_timer
       --------- Port   :10045 <>    :50755 (RTCP), SSRC 0, 0 p, 0 b, 0 e, 5 ts
       freeing send_timer
       --- Tag '', created 0:05 ago for branch '', in dialogue with 'cb8a1e30'
       ------ Media #1 (audio over RTP/AVP) using unknown codec
      --------- Port   :10032 <>          (null):0    , SSRC 0, 0 p, 0 b, 0 e, 5 ts
      freeing send_timer
      --------- Port   :10033 <>          (null):0     (RTCP), SSRC 0, 0 p, 0 b, 0 e, 5 ts
      freeing send_timer
       rtpengine: ci=732597d6-6d96-485b-b6dc-7d93703c1405, created_from=:53888, 
       last_signal=1554978149, 
       tos=0, 
       ml0_start_time=1554978149.645290, 
       ml0_end_time=1554978154.822680, 
       ml0_duration=5.177390, 
       ml0_termination=REGULAR, 
       ml0_local_tag=cb8a1e30, 
       ml0_local_tag_type=FROM_TAG, 
      ...



      continue : Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

      This blog is in continuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC ).


      Attempt 4: Stream the content to a WebRTC endpoint which is hidden in a video call . Pick the stream from vp8 object URL send to a streaming server

      This process involved the following components :

      • WebRTC API : simplewebrtc on Chrome
      • Transfer mechanism from client to Streaming server:  webrtc media channel

      Problems : No streaming server is qualified to handle a direct webrtc input and stream it on network .


      Attempt 4.1 : Stream the content to a WebRTC endpoint . Do WebRTC Endpoint to RTP Endpoint bridge using Kurento APIs. 

      Use the RTP port and ip address to input into a ffmpeg or gstreamer or VLC terminal command and out put a live H264 stream on another ip and port address .  

      This process involved the following components :

      • API : Kurento
      • Transfer mechanism : HTML5 webrtc client -> application server hosting java -> media server -> application for webrtc media to RTP media conversation -> RTP player

      Screenshots of attempts with Wowza to stream RTP from a IP and port

      kurentowowoza

      Problems : The stream was black which means 100% loss.

      Lesson learned : RTP is not suitable for over the intgernet transmission especially with firewalls


      Attempt 4.2 : Build a WebRTC Endpoint to Http endpoint in kurento and force the video audio encoding to be that of H264 and PCMU.

      Code snippet for adding constraints to output media via pipeline and forcing choice of codecs( H264 for video and PCMU for audio ).

      MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      HttpGetEndpoint httpEndpoint=new HttpGetEndpoint.Builder(pipeline).build();
      
      org.kurento.client.Fraction fr= new org.kurento.client.Fraction(1, 30);
      VideoCaps vc= new VideoCaps(VideoCodec.H264,fr);
      httpEndpoint.setVideoFormat(vc);
      
      AudioCaps ac= new AudioCaps(AudioCodec.PCMU, 65536);
      httpEndpoint.setAudioFormat(ac);
      
      webRtcEndpoint.connect(httpEndpoint);
      

      Alternatively one can opt to use gstreamer filter to force the output in raw format.

      // basic media operation of 1 pipeline and 2 endpoints
      MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
      
      // adding Gstream filters
      GStreamerFilter filter1 = new GStreamerFilter.Builder(pipeline, &quot;videorate max-rate=30&quot;).withFilterType(FilterType.VIDEO).build();
      GStreamerFilter filter2 = new GStreamerFilter.Builder(pipeline, &quot;capsfilter caps=video/x-h264,width=1280,height=720,framerate=30/1&quot;).withFilterType(FilterType.VIDEO).build();
      GStreamerFilter filter3 = new GStreamerFilter.Builder(pipeline, &quot;capsfilter caps=audio/x-mpeg,layer=3,rate=48000&quot;).withFilterType(FilterType.AUDIO).build();
      
      // connecting all poin ts to one another
      webRtcEndpoint.connect (filter1);
      filter1.connect (filter2);
      filter2.connect (filter3);
      filter3.connect (rtpEndpoint);
      
      // RTP SDP offer and answer
      String requestRTPsdp = rtpEndpoint.generateOffer();
      rtpEndpoint.processAnswer(requestRTPsdp);
      

      End result : The output is still webm based and doesnt work on h264 clients.


      Attempt 5  : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Wowza streaming server

      This process involved the following components

      1. WebRTC Stream and object URL of the blob containing VP8 media
      2. Kurento  WebRTC Endpoint  bridge to generate SDP
      3. Wowza Streaming server

      Snippet used for kurento to generate a SDP file from WebRTC to RTP bridge

      @RequestMapping(value = &quot;/rtpsdp&quot;, method = RequestMethod.POST)
      private String processRequestrtpsdp(@RequestBody String sdpOffer)
      throws IOException, URISyntaxException, InterruptedException {
      
      //basic media operation of 1 pipeline and 2 endpoinst
      MediaPipeline pipeline = kurento.createMediaPipeline();
      WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
      RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();
      
      //connecting all poin ts to one another
      webRtcEndpoint.connect (rtpEndpoint);
      
      // RTP SDP offer and answer
      String requestRTPsdp = rtpEndpoint.generateOffer();
      rtpEndpoint.processAnswer(requestRTPsdp);
      
      // write the SDP conector to an external file
      PrintWriter out = new PrintWriter(&quot;/tmp/test.sdp&quot;);
      out.println(requestRTPsdp);
      out.close();
      
      HttpGetEndpoint httpEndpoint = new HttpGetEndpoint.Builder(pipeline).build();
      PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline, requestRTPsdp).build();
      httpEndpoint.connect(rtpEndpoint);
      player.connect(httpEndpoint);
      
      // Playing media and opening the default desktop browser
      player.play();
      String videoUrl = httpEndpoint.getUrl();
      System.out.println(&quot; ------- video URL -------------&quot;+ videoUrl);
      
      // send the response to front client
      String responseSdp = webRtcEndpoint.processOffer(sdpOffer);
      
      return responseSdp;
      }
      

      End result : wowza doesnt not recognize the WebRTC SDP and play the video

      screenshot of wowza with SDP input

      Screenshot from 2015-01-30 15:28:59

      Attempt 5.1 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Default Ubuntu media player 

      SDP file formed contains contents such as :

      v=0
      o=- 3631611195 3631611195 IN IP4 192.168.0.119
      s=Kurento Media Server
      c=IN IP4 192.168.0.119
      t=0 0
      m=audio 42802 RTP/AVP 98 99 0
      a=rtpmap:98 OPUS/48000/2
      a=rtpmap:99 AMR/8000/1
      a=rtpmap:0 PCMU/8000
      a=ssrc:2713728673 cname:user59375791@host-ad1117df
      m=video 35946 RTP/AVP 96 97 100 101
      a=rtpmap:96 H263-1998/90000
      a=rtpmap:97 VP8/90000
      a=rtpmap:100 MP4V-ES/90000
      a=rtpmap:101 H264/90000
      a=ssrc:93449274 cname:user59375791@host-ad1117df

      End result : wowza doesnt not recognize the WebRTC SDP and play the video : deformed media

      screenshot of playing from a SDP file

      Screenshot from 2015-01-29 17:42:21

      Attempt 5.2 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over VLC using socket input

      End result : nothing plays

      screenshot of VLC connected to play from socket and failure to play anything

      Screenshot from 2015-01-21 17:49:52

      Attempt 5.3: Create a WebRTC endpoint and connected it to RTP endpoint via media pipelines . Also make the RTP SDP offer and answering the same . Play with ffnpeg / ffplay / gst playbin

      String requestRTPsdp = rtpEndpoint.generateOffer();
      rtpEndpoint.processAnswer(requestRTPsdp);
      

      Write the requestRTPsdp to a file and obtain a RTP connector endpoint with Application/SDP .It plays okay with gst playbin ( 10 secs without audio ). Successful attempt to play from a gst playbin

      gst-launch -vvv playbin uri=file:///tmp/test.sdp 
      donekurento streaming

      but refuses to be played by VLC , ffplay and even wowza . The error generated with

      ffmpeg -i test.sdp -vcodec copy -acodec copy -f mpegts output-file.ts

      or

      ffmpeg -re -i test.sdp -vcodec h264 -acodec mp3 -f mpegts "udp://192.168.4.26:5000"

      End result : This results in “Could not find codec parameter for stream1 ( video:h263, none ) .Other errors types are , Could not write header for output file output file is empty nothing was encoded”

      Error screenshots of trying to play the RTP SDP file with ffmpeg

      ffmpeg error kurebto1
      ffmpeg error kurebto2

      Attempt 6 : Use a WebRTC capable media and streaming server ( eg Kurento )  to pick a live stream of VP8 .

      Convert the VP8 to H264  ( ffmpeg / RTP endpoint )

      Convert H264 to Mp4 using MP4 parser and pass to a streaming server  ( wowza)

      End Result : yes it did work on mozilla but with considerable lag


      Update : Thankfully the updates to WebRTC standards mandated the support for PCMU and AVC/H264 CB profile in the media stack of the browser thus solving the “from scratch buildup of transcoder between webrtc and non webrtc endpoints”.

      • Video Codecs : RFC 7742 specifies that all WebRTC-compatible browsers must support VP8 and H.264’s Constrained Baseline profile for video.
      • Audio Codecs : RFC 7874 specifies that browsers must support at least the Opus codec as well as G.711’s PCMA and PCMU formats.

      The latest Webrtc specification lists a set of codecs which all compliant browsers are required to support which includes chrome 52 , Firefox , safari , edge.

      References :

      1. RFC7742: WebRTC Video Processing and Codec Requirements
      2. RFC 7874: WebRTC Audio Codec and Processing Requirements

      Read more about Webrtc Audio Video Codecs