• MediaDevices
  • getUserMedia()
  • getDisplayMedia()
  • enumerateDevices()
• Peer to peer Connection
  • RTCPeerConnection()
  • RTCConfiguration, ICE
  • CreateOffer()
  • CreateAnswer()
  • Offer/Answer Options – VoiceActivityDetection
  • states
• RTP Media API
  • RTCRtpSender
  • RTCRtpReceiver
  • RTCRtpTransreciver
  • setCodecPreferences()
  • SDP Semantics
• RTCDTLSTransport
• RTCIceCandidate
  • ICE gathering
• RTCIceTransport Interface
• Peer-to-peer Data API
• Peer-to-peer DTMF
• statistics
  • getStats()

MediaDevices

getUserMedia()

Media Capture and streams

const stream = await navigator.mediaDevices.getUserMedia({video: true});

getDisplayMedia()

used for screenshare access which could be full csreen a specific appliction or a browser tab

Other APIs in pipeline

• preferCurrentTab – formerly known as getCurrentBrowsingContextMedia() API

navigator.mediaDevices.getDisplayMedia(
    {video: true, audio: true, preferCurrentTab: true});

enumerateDevices()

Enumerate Devices

Supported constraints

> navigator.mediaDevices.getSupportedConstraints();

aspectRatio: true
autoGainControl: true
brightness: true
channelCount: true
colorTemperature: true
contrast: true
deviceId: true
echoCancellation: true
exposureCompensation: true
exposureMode: true
exposureTime: true
facingMode: true
focusDistance: true
focusMode: true
frameRate: true
groupId: true
height: true
iso: true
latency: true
noiseSuppression: true
pan: true
pointsOfInterest: true
resizeMode: true
sampleRate: true
sampleSize: true
saturation: true
sharpness: true
tilt: true
torch: true
whiteBalanceMode: true
width: true
zoom: true

Peer-to-peer connections

RTCPeerConnection Interface

RTCPeerConnection creates , manages , monitors as well as tears down p2p communication channel

new RTCPeerConnection(servers, pcConstraint);

Deatiled Interface defination

interface RTCPeerConnection : EventTarget {

	RTCConfiguration getConfiguration();
	void setConfiguration(RTCConfiguration configuration);
	void close();

	Promise<RTCSessionDescriptionInit> createOffer(optional RTCOfferOptions options);
	Promise<RTCSessionDescriptionInit> createAnswer(optional RTCAnswerOptions options);
	Promise<void> setLocalDescription(optional RTCSessionDescriptionInit description);
	Promise<void> setRemoteDescription(optional RTCSessionDescriptionInit description);
	Promise<void> addIceCandidate(optional RTCIceCandidateInit candidate);

	readonly attribute RTCSessionDescription? localDescription;
	readonly attribute RTCSessionDescription? currentLocalDescription;
	readonly attribute RTCSessionDescription? pendingLocalDescription;  
	readonly attribute RTCSessionDescription? remoteDescription;
	readonly attribute RTCSessionDescription? currentRemoteDescription;
	readonly attribute RTCSessionDescription? pendingRemoteDescription;
	readonly attribute RTCSignalingState signalingState;
	readonly attribute RTCIceGatheringState iceGatheringState;
	readonly attribute RTCIceConnectionState iceConnectionState;
	readonly attribute RTCPeerConnectionState connectionState;
	readonly attribute boolean? canTrickleIceCandidates;

	void restartIce();
	static sequence<RTCIceServer> getDefaultIceServers();

	attribute EventHandler onnegotiationneeded;
	attribute EventHandler onicecandidate;
	attribute EventHandler onicecandidateerror;
	attribute EventHandler onsignalingstatechange;
	attribute EventHandler oniceconnectionstatechange;
	attribute EventHandler onicegatheringstatechange;
	attribute EventHandler onconnectionstatechange;
};

RTCConfiguration Dictionary

dictionary RTCConfiguration {
  sequence<RTCIceServer> iceServers;
  RTCIceTransportPolicy iceTransportPolicy;
  RTCBundlePolicy bundlePolicy;
  RTCRtcpMuxPolicy rtcpMuxPolicy;
  DOMString peerIdentity;
  sequence<RTCCertificate> certificates;
  [EnforceRange] octet iceCandidatePoolSize = 0;
};

RTCOAuthCredential Dictionary

Describes the OAuth auth credential information which is used by the STUN/TURN client (inside the ICE Agent) to authenticate against a STUN/TURN server

dictionary RTCOAuthCredential {
  required DOMString macKey;
  required DOMString accessToken;
};

RTCRtcpMuxPolicy Enum

what ICE candidates are gathered to support non-multiplexed RTCP.

• negotiate – Gather ICE candidates for both RTP and RTCP candidates. If the remote-endpoint is capable of multiplexing RTCP, multiplex RTCP on the RTP candidates. If it is not, use both the RTP and RTCP candidates separately.

• require – Gather ICE candidates only for RTP and multiplex RTCP on the RTP candidates. If the remote endpoint is not capable of rtcp-mux, session negotiation will fail.

If the value of configuration.rtcpMuxPolicy is set and its value differs from the connection’s rtcpMux policy, throw an InvalidModificationError. If the value is “negotiate” and the user agent does not implement non-muxed RTCP, throw a NotSupportedError.

An RTCPeerConnection object has a signaling state, a connection state, an ICE gathering state, and an ICE connection state.

RTCSignalingState Enum

“stable”,
“have-local-offer”,
“have-remote-offer”,
“have-local-pranswer”,
“have-remote-pranswer”,
“closed”

An RTCPeerConnection object has an operations chain which ensures that only one asynchronous operation in the chain executes concurrently.

Also an RTCPeerConnection object MUST not be garbage collected as long as any event can cause an event handler to be triggered on the object. When the object’s internal slot is true ie closed, no such event handler can be triggered and it is therefore safe to garbage collect the object.

CreateOffer()

generates a blob of SDP that contains an RFC 3264 offer with the supported configurations for the session, including

• descriptions of the local MediaStreamTracks attached to this RTCPeerConnection,
• codec/RTP/RTCP capabilities
• ICE agent (usernameFragment, password , local candiadtes etc )
• DTLS connection

const pc = new RTCPeerConnection();
pc.createOffer()
.then(desc => pc.setLocalDescription(desc));

With more attributes

var pc = new RTCPeerConnection();
pc.createOffer({
     mandatory: {
        OfferToReceiveAudio: true,
        OfferToReceiveVideo: true
    },
    optional: [{
        VoiceActivityDetection: false
    }]
}).then(function(offer) {
	return pc.setLocalDescription(offer);
})
.then(function() {
// Send the offer to the remote through signaling server
})
.catch(handleError);

CreateAnswer()

generates an SDPanswer with the supported configuration for the session that is compatible with the parameters in the remote configuration

var pc = new RTCPeerConnection();
pc.createAnswer({
  OfferToReceiveAudio: true
  OfferToReceiveVideo: true
})
.then(function(answer) {
  return pc.setLocalDescription(answer);
})
.then(function() {
  // Send the answer to the remote through signaling server
})
.catch(handleError);

Offer/Answer Options – VoiceActivityDetection

dictionary RTCOfferAnswerOptions {
  boolean voiceActivityDetection = true;
};

capable of detecting “silence”

dictionary RTCOfferOptions : RTCOfferAnswerOptions {
  boolean iceRestart = false;
};

dictionary RTCAnswerOptions : RTCOfferAnswerOptions {};

Codec preferences of an m= section’s associated transceiver is said to be the value of the RTCRtpTranceiver with the following filtering applied

• If direction is “sendrecv”, exclude any codecs not included in the intersection of RTCRtpSender.getCapabilities(kind).codecs and RTCRtpReceiver.getCapabilities(kind).codecs.
• If direction is “sendonly”, exclude any codecs not included in RTCRtpSender.getCapabilities(kind).codecs.
• If direction is “recvonly”, exclude any codecs not included in RTCRtpReceiver.getCapabilities(kind).codecs.

Legacy Interface Extensions

partial interface RTCPeerConnection {
 
Promise<void> createOffer(
 RTCSessionDescriptionCallback successCallback,
 RTCPeerConnectionErrorCallback failureCallback,
 optional RTCOfferOptions options);
 
Promise<void> setLocalDescription(
optional RTCSessionDescriptionInit description,
VoidFunction successCallback,                                  
RTCPeerConnectionErrorCallback failureCallback);

Promise<void> createAnswer(
RTCSessionDescriptionCallback successCallback,
RTCPeerConnectionErrorCallback failureCallback);  

Promise<void> setRemoteDescription(
optional RTCSessionDescriptionInit description,
VoidFunction successCallback,                    
RTCPeerConnectionErrorCallback failureCallback);

Promise<void> addIceCandidate(
RTCIceCandidateInit candidate,
VoidFunction successCallback,
RTCPeerConnectionErrorCallback failureCallback);
};

Session Description Model

enum RTCSdpType {
  "offer",
  "pranswer",
  "answer",
  "rollback"
};

interface RTCSessionDescription {
  readonly attribute RTCSdpType type;
  readonly attribute DOMString sdp;
  [Default] object toJSON();
};

dictionary RTCSessionDescriptionInit {
  RTCSdpType type;
  DOMString sdp = "";
};

RTCPriorityType
Priority and QoS Model which can be : “very-low”, “low”, “medium”, “high”

RTP Media API

Send and receive MediaStreamTracks over a peer-to-peer connection.
Tracks, when added to an RTCPeerConnection, result in signaling; when this signaling is forwarded to a remote peer, it causes corresponding tracks to be created on the remote side.

getTracks()

const pc = new RTCPeerConnection();
let stream = await navigator.mediaDevices.getUserMedia({
video: true});
...
stream.getTracks()

[MediaStreamTrack]
0: MediaStreamTrack
contentHint: ""
enabled: true
id: "d5b10d39-cf26-4705-8831-6a0fa89546a0"
kind: "video"
label: "Integrated Webcam (0bda:58c2)"
muted: false
oncapturehandlechange: null
onended: null
onmute: null
onunmute: null
readyState: "live"

RTCRtpSenders objects manages encoding and transmission of MediaStreamTracks .

RTCRtpReceivers objects manage the reception and decoding of MediaStreamTracks. These are associated with one track.

• RTCRtpReceiver.getContributingSources()
• RTCRtpReceiver.getParameters()
• RTCRtpReceiver.getStats()
• RTCRtpReceiver.getSynchronizationSources()
  • returns audio level, rtptimestamp, source , timesatmp
• RTCRtpReceiver.getCapabilities()

RTCRtpTransceivers interface describes a permanent pairing of an RTCRtpSender and an RTCRtpReceiver. Each transceiver is uniquely identified using its mid ( media id) property from the corresponding m-line.

They are created implicitly when the application attaches a MediaStreamTrack to an RTCPeerConnection via the addTrack(), or explicitly when the application uses the addTransceiver(). They are also created when a remote description is applied that includes a new media description.

rtpTransceiver = RTCPeerConnection.addTransceiver(trackOrKind, init);

trackOrKind should be either audio or video othereise a TypeError is thrown. init is optional – can contain direction , sendEncodings , streams.

var tracks = stream.getTracks();
tracks.forEach(function(track){ 
    pc.addTrack(track, stream)
 });

For example if the stream’s track is

stream.getTracks()

[MediaStreamTrack]
0: MediaStreamTrack
    contentHint: ""
    enabled: true
    id: "d5b10d39-cf26-4705-8831-6a0fa89546a0"
    kind: "video"
    label: "Integrated Webcam (0bda:58c2)"
    muted: false
    oncapturehandlechange: null
    onended: null
    onmute: null
    onunmute: null
    readyState: "live"
[[Prototype]]: MediaStreamTrack
length: 1

the the sender ( fetched from getTransceivers() API) is

pc.getTransceivers()

0: RTCRtpTransceiver
	currentDirection: null
	direction: "sendrecv"
	headerExtensionsNegotiated: []
	headerExtensionsToOffer: (15) [{…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}, {…}]
	mid: null
	receiver: RTCRtpReceiver {track: MediaStreamTrack, transport: null, rtcpTransport: null, playoutDelayHint: null}
	sender: RTCRtpSender
	    dtmf: null
	    rtcpTransport: null
	    track: MediaStreamTrack {kind: 'video', id: 'd5b10d39-cf26-4705-8831-6a0fa89546a0', label: 'Integrated Webcam (0bda:58c2)', enabled: true, muted: false, …}
	    transport: null
	    [[Prototype]]: RTCRtpSender
	    stopped: false
	[[Prototype]]: RTCRtpTransceiver
length: 1

RTCPeerConnection Interface

partial interface RTCPeerConnection {
  
sequence<RTCRtpSender> getSenders();
sequence<RTCRtpReceiver> getReceivers();
sequence<RTCRtpTransceiver> getTransceivers();

RTCRtpSender addTrack(MediaStreamTrack track, MediaStream... streams);
void removeTrack(RTCRtpSender sender);
RTCRtpTransceiver addTransceiver((MediaStreamTrack or DOMString) trackOrKind, optional RTCRtpTransceiverInit init);
attribute EventHandler ontrack;
};

RTCRtpTransceiverInit

dictionary RTCRtpTransceiverInit {
  RTCRtpTransceiverDirection direction = "sendrecv";
  sequence<MediaStream> streams = [];
  sequence<RTCRtpEncodingParameters> sendEncodings = [];
};

RTCRtpTransceiverDirection can be either of : “sendrecv”, “sendonly”, “recvonly”, “inactive”, “stopped”

RTCRtpSender Interface

Allows an application to control how a given MediaStreamTrack is encoded and transmitted to a remote peer.

interface RTCRtpSender {
readonly attribute MediaStreamTrack? track;
readonly attribute RTCDtlsTransport? transport;
readonly attribute RTCDtlsTransport? rtcpTransport;
static RTCRtpCapabilities? getCapabilities(DOMString kind);
Promise<void> setParameters(RTCRtpSendParameters parameters);
RTCRtpSendParameters getParameters();
Promise<void> replaceTrack(MediaStreamTrack? withTrack);
void setStreams(MediaStream... streams);
Promise<RTCStatsReport> getStats();
};

RTCRtpParameters Dictionary

dictionary RTCRtpParameters {
  required sequence<RTCRtpHeaderExtensionParameters> headerExtensions;
  required RTCRtcpParameters rtcp;
  required sequence<RTCRtpCodecParameters> codecs;
};

RTCRtpSendParameters Dictionary

dictionary RTCRtpSendParameters : RTCRtpParameters {
  required DOMString transactionId;
  required sequence<RTCRtpEncodingParameters> encodings;
  RTCDegradationPreference degradationPreference = "balanced";
  RTCPriorityType priority = "low";
};

RTCRtpReceiveParameters Dictionary

dictionary RTCRtpReceiveParameters : RTCRtpParameters {
  required sequence<RTCRtpDecodingParameters> encodings;
};

RTCRtpCodingParameters Dictionary

dictionary RTCRtpCodingParameters {
  DOMString rid;
};

RTCRtpDecodingParameters Dictionary

dictionary RTCRtpDecodingParameters : RTCRtpCodingParameters {};

RTCRtpEncodingParameters Dictionary

dictionary RTCRtpEncodingParameters : RTCRtpCodingParameters {
  octet codecPayloadType;
  RTCDtxStatus dtx;
  boolean active = true;
  unsigned long ptime;
  unsigned long maxBitrate;
  double maxFramerate;
  double scaleResolutionDownBy;
};

RTCDtxStatus Enum

disabled- Discontinuous transmission is disabled.
enabled- Discontinuous transmission is enabled if negotiated.

RTCDegradationPreference Enum

enum RTCDegradationPreference {
  "maintain-framerate",
  "maintain-resolution",
  "balanced"
};

RTCRtcpParameters Dictionary

dictionary RTCRtcpParameters {
  DOMString cname;
  boolean reducedSize;
};

RTCRtpHeaderExtensionParameters Dictionary

dictionary RTCRtpHeaderExtensionParameters {
  required DOMString uri;
  required unsigned short id;
  boolean encrypted = false;
};

RTCRtpCodecParameters Dictionary

dictionary RTCRtpCodecParameters {
  required octet payloadType;
  required DOMString mimeType;
  required unsigned long clockRate;
  unsigned short channels;
  DOMString sdpFmtpLine;
};

payloadType – identify this codec.
mimeType – codec MIME media type/subtype. Valid media types and subtypes are listed in [IANA-RTP-2]
clockRate – expressed in Hertz
channels – number of channels (mono=1, stereo=2).
sdpFmtpLine – “format specific parameters” field from the “a=fmtp” line in the SDP corresponding to the codec

RTCRtpCapabilities Dictionary

dictionary RTCRtpCapabilities {
  required sequence<RTCRtpCodecCapability> codecs;
  required sequence<RTCRtpHeaderExtensionCapability> headerExtensions;
};

RTCRtpCodecCapability Dictionary

dictionary RTCRtpCodecCapability {
  required DOMString mimeType;
  required unsigned long clockRate;
  unsigned short channels;
  DOMString sdpFmtpLine;
};

RTCRtpHeaderExtensionCapability Dictionary

dictionary RTCRtpHeaderExtensionCapability {
  DOMString uri;
};

Example JS code to RTCRtpCapabilities

const pc = new RTCPeerConnection();
const transceiver = pc.addTransceiver('audio');
const capabilities = RTCRtpSender.getCapabilities('audio');

Output :

codecs: Array(13)
0: {channels: 2, clockRate: 48000, mimeType: "audio/opus", sdpFmtpLine: "minptime=10;useinbandfec=1"}
1: {channels: 1, clockRate: 16000, mimeType: "audio/ISAC"}
2: {channels: 1, clockRate: 32000, mimeType: "audio/ISAC"}
3: {channels: 1, clockRate: 8000, mimeType: "audio/G722"}
4: {channels: 1, clockRate: 8000, mimeType: "audio/PCMU"}
5: {channels: 1, clockRate: 8000, mimeType: "audio/PCMA"}
6: {channels: 1, clockRate: 32000, mimeType: "audio/CN"}
7: {channels: 1, clockRate: 16000, mimeType: "audio/CN"}
8: {channels: 1, clockRate: 8000, mimeType: "audio/CN"}
9: {channels: 1, clockRate: 48000, mimeType: "audio/telephone-event"}
10: {channels: 1, clockRate: 32000, mimeType: "audio/telephone-event"}
11: {channels: 1, clockRate: 16000, mimeType: "audio/telephone-event"}
12: {channels: 1, clockRate: 8000, mimeType: "audio/telephone-event"}
length: 13

headerExtensions: Array(6)
0: {uri: "urn:ietf:params:rtp-hdrext:ssrc-audio-level"}
1: {uri: "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time"}
2: {uri: "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01"}
3: {uri: "urn:ietf:params:rtp-hdrext:sdes:mid"}
4: {uri: "urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id"}
5: {uri: "urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id"}
length: 6

RTCRtpReceiver Interface

allows an application to inspect the receipt of a MediaStreamTrack.

interface RTCRtpReceiver {
  readonly attribute MediaStreamTrack track;
  readonly attribute RTCDtlsTransport? transport;
  readonly attribute RTCDtlsTransport? rtcpTransport;
  static RTCRtpCapabilities? getCapabilities(DOMString kind);
  RTCRtpReceiveParameters getParameters();
  sequence<RTCRtpContributingSource> getContributingSources();
  sequence<RTCRtpSynchronizationSource> getSynchronizationSources();
  Promise<RTCStatsReport> getStats();
};

dictionary RTCRtpContributingSource

dictionary RTCRtpContributingSource {
  required DOMHighResTimeStamp timestamp;
  required unsigned long source;
  double audioLevel;
  required unsigned long rtpTimestamp;
};

dictionary RTCRtpSynchronizationSource

dictionary RTCRtpSynchronizationSource : RTCRtpContributingSource {
  boolean voiceActivityFlag;
};

voiceActivityFlag of type boolean – Only present for audio receivers. Whether the last RTP packet, delivered from this source, contains voice activity (true) or not (false).

RTCRtpTransceiver Interface

Each SDP media section describes one bidirectional SRTP (“Secure Real Time Protocol”) stream. RTCRtpTransceiver describes this permanent pairing of an RTCRtpSender and an RTCRtpReceiver, along with some shared state. It is uniquely identified using its mid property.

Thus it is combination of an RTCRtpSender and an RTCRtpReceiver that share a common mid. An associated transceiver( with mid) is one that’s represented in the last applied session description.

interface RTCRtpTransceiver {
    readonly attribute DOMString? mid;
    [SameObject] readonly attribute RTCRtpSender sender;
    [SameObject] readonly attribute RTCRtpReceiver receiver;
    attribute RTCRtpTransceiverDirection direction;
    readonly attribute RTCRtpTransceiverDirection? currentDirection;
    void stop();
    void setCodecPreferences(sequence<RTCRtpCodecCapability> codecs);
};

Method stop() – Irreversibly marks the transceiver as stopping, unless it is already stopped. This will immediately cause the transceiver’s sender to no longer send, and its receiver to no longer receive.
stopping transceiver will cause future calls to createOffer to generate a zero port in the media description for the corresponding transceiver and stopped transceiver will cause future calls to createOffer or createAnswer to generate a zero port in the media description for the corresponding transceiver

setCodecPreferences()

This method overrides the default codec preferences used by the user agent.

Example setting codec Preference for OPUS in audio

peer = new RTCPeerConnection();    
const transceiver = peer.addTransceiver('audio');
const audiocapabilities = RTCRtpSender.getCapabilities('audio');
let codec = [];
codec.push(audiocapabilities.codecs[0]); // opus 
transceiver.setCodecPreferences(codec);

Before setting codec preference for OPUS

m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
…
a=recvonly
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000

After setting codec preference for OPUS audio

m=audio 9 UDP/TLS/RTP/SAVPF 111
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
…
a=sendrecv
a=msid:hcgvWcGG7WhdzboWk79q39NiO8xkh4ArWhbM f15d77bb-7a6f-4f41-80cd-51a3c40de7b7
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1

RTCDtlsTransport

Access to information about the Datagram Transport Layer Security (DTLS) transport over which RTP and RTCP packets are sent and received by RTCRtpSender and RTCRtpReceiver objects, as well other data such as SCTP packets sent and received by data channels.
Each RTCDtlsTransport object represents the DTLS transport layer for the RTP or RTCP component of a specific RTCRtpTransceiver, or a group of RTCRtpTransceivers if such a group has been negotiated via [BUNDLE].

interface RTCDtlsTransport : EventTarget {
  [SameObject] readonly attribute RTCIceTransport iceTransport;
  readonly attribute RTCDtlsTransportState state;
  sequence<ArrayBuffer> getRemoteCertificates();
  attribute EventHandler onstatechange;
  attribute EventHandler onerror;
};

RTCDtlsTransportState Enum

• “new”- DTLS has not started negotiating yet.
• “connecting” – DTLS is in the process of negotiating a secure connection and verifying the remote fingerprint.
• “connected”- DTLS has completed negotiation of a secure connection and verified the remote fingerprint.
• “closed” – transport has been closed intentionally like close_notify alert, or calling close().
• “failed” – transport has failed as the result of an error like failure to validate the remote fingerprint

RTCDtlsFingerprint dictionary

dictionary RTCDtlsFingerprint {
  DOMString algorithm;
  DOMString value;
};

Protocols multiplexed with RTP (e.g. data channel) share its component ID. This represents the component-id value 1 when encoded in candidate-attribute while ICE candadte for RTCP has component-id value 2 when encoded in candidate-attribute.

RTCTrackEvent

The track event uses the RTCTrackEvent interface.

interface RTCTrackEvent : Event {
  readonly attribute RTCRtpReceiver receiver;
  readonly attribute MediaStreamTrack track;
  [SameObject] readonly attribute FrozenArray<MediaStream> streams;
  readonly attribute RTCRtpTransceiver transceiver;
};

dictionary RTCTrackEventInit

dictionary RTCTrackEventInit : EventInit {
  required RTCRtpReceiver receiver;
  required MediaStreamTrack track;
  sequence<MediaStream> streams = [];
  required RTCRtpTransceiver transceiver;
};

RTCIceCandidate

This interface candidate Internet Connectivity Establishment (ICE) configuration used to setup RTCPeerconnection. To facilitate routing of media on given peer connection, both endpoints exchange several candidates and then one candidate out of the lot is chosen which will be then used to initiate the connection.

const pc = new RTCPeerConnection();
pc.addIceCandidate({candidate:''});

• candidate – transport address for the candidate that can be used for connectivity checks.
• component – candidate is an RTP or an RTCP candidate
• foundation – unique identifier that is the same for any candidates of the same type , helps optimize ICE performance while prioritizing and correlating candidates that appear on multiple RTCIceTransport objects.
• ip , port
• priority
• protocol – tcp/udp
• relatedAddress , relatedPort
• sdpMid – candidate’s media stream identification tag
• sdpMLineIndex

usernameFragment – randomly-generated username fragment (“ice-ufrag”) which ICE uses for message integrity along with a randomly-generated password (“ice-pwd”).

RTCIceCredentialType Enum : supports OAuth 2.0 based authentication. The application, acting as the OAuth Client, is responsible for refreshing the credential information and updating the ICE Agent with fresh new credentials before the accessToken expires. The OAuth Client can use the RTCPeerConnection setConfiguration method to periodically refresh the TURN credentials.

enum RTCIceCredentialType {
  "password",
  "oauth"
};

RTCIceServer Dictionary : Describes the STUN and TURN servers that can be used by the ICE Agent to establish a connection with a peer.

dictionary RTCIceServer {    
   required (DOMString or sequence<DOMString>) urls; 
   DOMString username;      
   (DOMString or RTCOAuthCredential) credential; 
   RTCIceCredentialType credentialType = "password";  
};   

 [{urls: 'stun:stun1.example.net'},
  {urls: ['turns:turn.example.org', 'turn:turn.example.net'],
    username: 'user',
    credential: 'myPassword',
    credentialType: 'password'},
  {urls: 'turns:turn2.example.net',
    username: 'xxx',
    credential: {
      macKey: 'xyz=',
      accessToken: 'xxx+yyy=='
    },
    credentialType: 'oauth'}
];

RTCIceTransportPolicy Enum

ICE candidate policy [JSEP] to select candidates for the ICE connectivity checks

• relay – use only media relay candidates such as candidates passing through a TURN server. It prevents the remote endpoint/unknown caller from learning the user’s IP addresses

• all – ICE Agent can use any type of candidate when this value is specified.

RTCBundlePolicy Enum

• balanced – Gather ICE candidates for each media type (audio, video, and data). If the remote endpoint is not bundle-aware, negotiate only one audio and video track on separate transports.

• max-compat – Gather ICE candidates for each track. If the remote endpoint is not bundle-aware, negotiate all media tracks on separate transports.

• max-bundle – Gather ICE candidates for only one track. If the remote endpoint is not bundle-aware, negotiate only one media track. If the remote endpoint is bundle-aware, all media tracks and data channels are bundled onto the same transport.

If the value of configuration.bundlePolicy is set and its value differs from the connection’s bundle policy, throw an InvalidModificationError.

Interfaces for Connectivity Establishment

describes ICE candidates

interface RTCIceCandidate {
  DOMString candidate;
  DOMString sdpMid;
  unsigned short sdpMLineIndex;
  DOMString foundation;
  RTCIceComponent component;
  unsigned long priority;
  DOMString address;
  RTCIceProtocol protocol;
  unsigned short port;
  RTCIceCandidateType type;
  RTCIceTcpCandidateType tcpType;
  DOMString relatedAddress;
  unsigned short relatedPort;
  DOMString usernameFragment;
  RTCIceCandidateInit toJSON();
};

RTCIceProtocol can be either tcp or udp

TCP candidate type which can be either of

• active – An active TCP candidate is one for which the transport will attempt to open an outbound connection but will not receive incoming connection requests.
• passive – A passive TCP candidate is one for which the transport will receive incoming connection attempts but not attempt a connection.
• so – An so candidate is one for which the transport will attempt to open a connection simultaneously with its peer.

UDP candidate type

• host – actual direct IP address of the remote peer
• srflx – server reflexive ,  generated by a STUN/TURN server
• prflx – peer reflexive ,IP address comes from a symmetric NAT between the two peers, usually as an additional candidate during trickle ICE
• relay – generated using TURN

ICE Candidate UDP Host

sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 51577 typ host generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 51382 typ host generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:27784895 1 udp 2122260223 192.168.1.114 53600 typ host generation 0 ufrag muSq network-id 1 network-cost 10

ICE Candidate TCP Host

Notice TCP host candidates for mid 0 , 1 and 3 for video , audio and data media types

sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:1327761999 1 tcp 1518280447 192.168.1.114 9 typ host tcptype active generation 0 ufrag muSq network-id 1 network-cost 10

ICE Candidate UDP Srflx

Notice 3 candidates for 3 streams sdpMid 0,1 and 2

sdpMid: 2, sdpMLineIndex: 2, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27177 typ srflx raddr 192.168.1.114 rport 53600 generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 1, sdpMLineIndex: 1, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27176 typ srflx raddr 192.168.1.114 rport 51382 generation 0 ufrag muSq network-id 1 network-cost 10

sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:2163208203 1 udp 1686052607 117.201.90.218 27175 typ srflx raddr 192.168.1.114 rport 51577 generation 0 ufrag muSq network-id 1 network-cost 10

ICE Candidate (host)

sdpMid: 0, sdpMLineIndex: 0, candidate: candidate:2880323124 1 udp 2122260223 192.168.1.116 61622 typ host generation 0 ufrag jsPO network-id 1 network-cost 10

usernameFragment – randomly-generated username fragment (“ice-ufrag”) which ICE uses for message integrity along with a randomly-generated password (“ice-pwd”).

a=ice-ufrag:D+yg
a=ice-pwd:2ep6CXO0FP/JfS4ue/dfjeQM
a=ice-options:trickle

RTCPeerConnectionIceEvent

interface RTCPeerConnectionIceEvent : Event {
  readonly attribute RTCIceCandidate? candidate;
  readonly attribute DOMString? url;
};

RTCPeerConnectionIceErrorEvent

interface RTCPeerConnectionIceErrorEvent : Event {
  readonly attribute DOMString hostCandidate;
  readonly attribute DOMString url;
  readonly attribute unsigned short errorCode;
  readonly attribute USVString errorText;
};

RTCIceTransport Interface

Access to information about the ICE transport over which packets are sent and received. Each RTCIceTransport object represents the ICE transport layer for the RTP or RTCP component of a specific RTCRtpTransceiver, or a group of RTCRtpTransceivers if such a group has been negotiated via [BUNDLE].

interface RTCIceTransport : EventTarget {
    readonly attribute RTCIceRole role;
    readonly attribute RTCIceComponent component;
    readonly attribute RTCIceTransportState state;
    readonly attribute RTCIceGathererState gatheringState;
    sequence<RTCIceCandidate> getLocalCandidates();
    sequence<RTCIceCandidate> getRemoteCandidates();
    RTCIceCandidatePair? getSelectedCandidatePair();
    RTCIceParameters? getLocalParameters();
    RTCIceParameters? getRemoteParameters();
    attribute EventHandler onstatechange;
    attribute EventHandler ongatheringstatechange;
    attribute EventHandler onselectedcandidatepairchange;
};

RTCIceParameters Dictionary

dictionary RTCIceParameters {
  DOMString usernameFragment;
  DOMString password;
};

RTCIceCandidatePair Dictionary

dictionary RTCIceCandidatePair {
  RTCIceCandidate local;
  RTCIceCandidate remote;
};

RTCIceTransportState Enum

1. new – ICE agent is gathering addresses or is waiting to be given remote candidates 
2. checking –
3. connected – Found a working candidate pair, but still performing connectivity checks to find a better one.
4. completed – Found a working candidate pair and done performing connectivity checks.
5. disconnected
6. failed
7. closed

RTCIceRole Enum

unknown : agent who role is not yet defined
controlling : controlling agent
controlled : controlled agent

RTCIceComponent Enum

rtp : ICE Transport is used for RTP (or RTCP multiplexing)
rtcp : ICE Transport is used for RTCP

Peer-to-peer Data API

Connection.createDataChannel('sendDataChannel', dataConstraint);

With SCTP, the protocol used by WebRTC data channels, reliable and ordered data delivery is on by default.

Sending large files

 Split data channel message in chunks 

var CHUNK_LEN = 64000; // 64 Kb
var img = photoContext.getImageData(0, 0, photoContextW, photoContextH),
len = img.data.byteLength,
n = len / CHUNK_LEN | 0;

for (var i = 0; i < n; i++) {
    var start = i * CHUNK_LEN, end = (i + 1) * CHUNK_LEN;
    dataChannel.send(img.data.subarray(start, end));
}
// last chunk
if (len % CHUNK_LEN) {
   dataChannel.send(img.data.subarray(n * CHUNK_LEN));
}

Statistics

The browser maintains a set of statistics for monitored objects, in the form of stats objects. A group of related objects may be referenced by a selector( like MediaStreamTrack that is sent or received by the RTCPeerConnection).

Statistics API extends the RTCPeerConnection interface

partial interface RTCPeerConnection {
  Promise<RTCStatsReport> getStats(optional MediaStreamTrack? selector = null);
  attribute EventHandler onstatsended;
};

getStats()

Method getStats() Gathers stats for the given selector and reports the result asynchronously.

{
  clockRate: 90000,
  id: "RTCCodec_0_Inbound_96",
  mimeType: "video/VP8",
  payloadType: 96,
  timestamp: 1644263636999.506,
  transportId: "RTCTransport_0_1",
  type: "codec"
}
{
  clockRate: 90000,
  id: "RTCCodec_0_Inbound_97",
  mimeType: "video/rtx",
  payloadType: 97,
  sdpFmtpLine: "apt=96",
  timestamp: 1644263636999.506,
  transportId: "RTCTransport_0_1",
  type: "codec"
}
{
  clockRate: 90000,
  id: "RTCCodec_0_Inbound_98",
  mimeType: "video/VP9",
  payloadType: 98,
  sdpFmtpLine: "profile-id=0",
  timestamp: 1644263636999.506,
  transportId: "RTCTransport_0_1",
  type: "codec"
}
{
  clockRate: 90000,
  id: "RTCCodec_0_Inbound_99",
  mimeType: "video/rtx",
  payloadType: 99,
  sdpFmtpLine: "apt=98",
  timestamp: 1644263636999.506,
  transportId: "RTCTransport_0_1",
  type: "codec"
}

Quality limitation in pc.getstats()

{
  bytesSent: 2646771,
  codecId: "RTCCodec_0_Outbound_96",
  encoderImplementation: "libvpx",
  firCount: 0,
  frameHeight: 540,
  framesEncoded: 471,
  framesPerSecond: 31,
  framesSent: 471,
  frameWidth: 960,
  headerBytesSent: 67900,
  hugeFramesSent: 0,
  id: "RTCOutboundRTPVideoStream_971979131",
  keyFramesEncoded: 4,
  kind: "video",
  mediaSourceId: "RTCVideoSource_48",
  mediaType: "video",
  nackCount: 0,
  packetsSent: 2657,
  pliCount: 0,
  qpSum: 6598,
  qualityLimitationDurations: {
    bandwidth: 15882,
    cpu: 0,
    none: 36,
    other: 0
  },
  qualityLimitationReason: "bandwidth",
  qualityLimitationResolutionChanges: 0,
  remoteId: "RTCRemoteInboundRtpVideoStream_971979131",
  retransmittedBytesSent: 0,
  retransmittedPacketsSent: 0,
  ssrc: 971979131,
  timestamp: 1644263635999.547,
  totalEncodedBytesTarget: 0,
  totalEncodeTime: 2.866,
  totalPacketSendDelay: 47.68,
  trackId: "RTCMediaStreamTrack_sender_48",
  transportId: "RTCTransport_0_1",
  type: "outbound-rtp"
}

RTCStatsReport Object

map between strings that identify the inspected objects (id attribute in RTCStats instances), and their corresponding RTCStats-derived dictionaries.

interface RTCStatsReport {
  readonly maplike<DOMString, object>;
};

RTCStats Dictionary

stats object constructed by inspecting a specific monitored object.

dictionary RTCStats {
  required DOMHighResTimeStamp timestamp;
  required RTCStatsType type;
  required DOMString id;
};

RTCStatsEvent

Constructor (DOMString type, RTCStatsEventInit eventInitDict)]

interface RTCStatsEvent : Event {
  readonly attribute RTCStatsReport report;
};

dictionary RTCStatsEventInit

dictionary RTCStatsEventInit : EventInit {
  required RTCStatsReport report;
};

Stats

• RTCSenderVideoTrackAttachmentStats
• RTCSenderAudioTrackAttachmentStats

Stream stats

• RTCRTPStreamStats : ssrc, kind, transportId, codecId + attr
• RTCReceivedRTPStreamStats : packetsReceived, packetsLost, jitter, packetsDiscarded + attr
• RTCSentRTPStreamStats : packetsSent, bytesSent + attr
• RTCPeerConnectionStats, with attributes dataChannelsOpened, dataChannelsClosed
• RTCDataChannelStats : label, protocol, dataChannelIdentifier, state, messagesSent, bytesSent, messagesReceived, bytesReceived + attr
• RTCMediaStreamStats : streamIdentifer, trackIds
• RTCVideoSenderStats : framesSent
• RTCVideoReceiverStats : framesReceived, framesDecoded, framesDropped, partialFramesLost + attr
• RTCCodecStats : payloadType, codecType, mimeType, clockRate, channels, sdpFmtpLine
• RTCTransportStats : bytesSent, bytesReceived, rtcpTransportStatsId, selectedCandidatePairId, localCertificateId, remoteCertificateId
• RTCCertificateStats : fingerprint, fingerprintAlgorithm, base64Certificate, issuerCertificateId + attr

OutboundInbound stats

• RTCOutboundRTPStreamStats : trackId, senderId, remoteId, framesEncoded, nackCount + attr
• RTCRemoteOutboundRTPStreamStats : localId, remoteTimestamp + attr
• RTCInboundRTPStreamStats : trackId, receiverId, remoteId, framesDecoded, nackCount + attr
• RTCRemoteInboundRTPStreamStats : localId, bytesReceived, roundTripTime + attr

Handler Stats

• RTCMediaHandlerStats : trackIdentifier, ended + attr
• RTCAudioHandlerStats : audioLevel + attr
• RTCVideoHandlerStats : frameWidth, frameHeight, framesPerSecond + attr

RTCICE stats

• RTCIceCandidatePairStats : transportId, localCandidateId, remoteCandidateId, state, priority, nominated, bytesSent, bytesReceived, totalRoundTripTime, currentRoundTripTime
• RTCIceCandidateStats : address, port, protocol, candidateType, url

References :

• W3C https://w3c.github.io/mediacapture-main
• W3C Webrtc peerconnection : http://w3c.github.io/webrtc-pc
• W3C Webrtc stats : https://www.w3.org/TR/webrtc-stats/
• w3c test apis https://w3c-test.org/webrtc/
• IETF – Trickle ICE: Incremental Provisioning of Candidates for the Interactive Connectivity Establishment (ICE) Protocol draft-ietf-mmusic-trickle-ice-02
• developer mozilla – https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver