• Load Balancers
  • MPLS
• Service-discovery
  • Keepalive, unregistering unhealthy nodes
• Replication
  • Data Store Replication
• Quick Response / Low latency
• Scalability
  • autoscalling
  • Partitioining
• Multiple PoPs (point of presence)
  • Minimal Latency and lowest amount of tarffic via public internet
• High availiability (HA)
  • 5 9’s in aggregate failures
  • HA for Load balancer (LB)
  • HA for Call Control app server
  • Media Server HA
• Security against malicious attacks
  • MITM
  • DDOS
• Identifying outages , logs and pcap analysis + alerting
• Bottlenecks
  • Performance testing
  • Robust QA for potential bottlenecks before going live
• Distributed Data Store
  • message-queues-vs-batch-processing
• Distributed-event-management-and-event-driven-architecture-using-streams
• Distributed cache for call control Servers
  • Choosing cache host
  • cache-replacement
  • Consistency and High Availiability in Cache setup
• Circuits – fail fast , wait for circuit to recover before calling again

Load Balancers

Load Balancer(LB) is the initial point of interaction between the client application and the core system. It is pivotal in the distribution of the load across multiple servers and ensuring the client is connected to the nearest VoIP/SIP application server to minimize latency. However, the load balancers are also susceptible to security breaches and DOS attacks as they have a public-facing interface. This section lists the protocols, types and algorithms used popularly in Load balancers of VoIP systems.

Load Balancers(LB) ping each server for health status and greylists servers that are unhealthy( respond late) as they may be overloaded or experiencing congestion. The LB monitors it rechecks after a while and if a server is healthy ( ie if a server responds with responds with status update) it can resume sending traffic to it. LB should also be distributed to different data centres in primary-secondary setup for HA.

Networking protocol

Load balancing algorithms

• Weighted Scheduling Algorithm
• Round Robin Algorithm
• Least Connection First Scheduling
• Lest response time algorithms
• Hash based algorithm ( send req based on hashed value such as suing IP address of request URL)

Service Discovery

Client-side or even backend service discovery uses a broadcasting or heartbeat mechanism to keep track of active servers and deactivates unresponsive or failed servers. This process of maintaining active servers helps in faster connection time. Some approaches to Service Discovery

1. Mesh
   1. (-) exponentially incresing network traffic
2. Gossip
3. Distributed cache
4. Coordination service with Service
   • (-) requesres coordination service for leader selection
   • (-) needs consensus
   • (-) RAFT and pbFT for mnaging failures
5. Random leader selection
   • (+) quicker
   • (-) may not gurantee one leader
   • (-)split brain problem

Keepalive, unregistering unhealthy nodes

Systems such as Consul, Etcd, and Zookeeper can help services find each other by keeping track of registered names, addresses, and ports. Health checks help verify service integrity and are often done using an HTTP endpoint.

Replication

Usuallay there is a tradeoff between liveness and safety.

1. Single leader replication
   • (-) vulnerable to loss of data is leader goes down before replication completes
   • used to in sql
2. multileader replication and
3. leaderless replication
   • (-) increases latencies
   • (-) quorem based on majority , cannot function is majority node are not down
   • used in cassandra

Data Store Replication

For Relatonal Dataabase

For NoSQL databse replication and HA

Quick Response / Low latency

Message format

Gateways for faster routing and caching to services

gateways are single entry point to route user requests to backend services .

Separate hot storage from cold storage

hot storage is frquently accessed data which must be near to server

cold storage is less frequently accessed data such as archives

• object storage
• slow access

Scalability

To make a system :-

• scalable : use partitioing
• reliable : use replication and checkpointing to not loose data in failures
• fast : use in -memory usage

According to CAP theorem Consistency and Availiability are difficult to achieve together and there has to be a tradeof acc to requirnments.

Partitioining

Partition strategy can be based on various ways such as :-

• Name based partition
• geographic partition
• names’ hashed value based on identifier
  • (-) can lead to hot partitions ( high density in areas of freq accessible identioers )
  • (-) high density spots for example all messages with a null key to go to the same partition
  • (-) doesnt scale
• event time based hash
  • (+) data is spread evenly over time

To create a well distributed partition we could spread hot partition into 2 partitions or dedicate partitions for freq accessible items. An effective partitioning keys uses

• Cardinality : total num of unique keys for a usecase. High cardinality leads to better distribution.
  • high cardinatility keys : names , email address , url since they have high variatioln
  • low cardinatlity keys : boolean flags such as gender M/F
• Selectivity : number of message with each key. High selectivity leads to hotspots and hence low selectivity is better for even distribution.

Autoscalling

Scale Out not Up !

Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management used in DevOps. I have mentioned this in detail on the article on VoIP and DevOps below.

Multiple PoPs (point of presence)

for a VOIP system catering to many clients accross the globe or accessing multiple carriers meant for different counteries based on Prefix matching , there should be alocal PoP in most used regions . typically these regions include – US east – west coasts, UK – germany of London , Asia Pacific – Mumbai ,Hong Kong and Australia.

Minimal Latency and lowest amount of tarffic via public internet

Creating multiple POPs and enabling private traffic via VPN in between them ensures that we use the backbone of our cloud provider such as AWS or datacentre instead of traversing via public internet which is slower and more insecure. Hopping on a private interface between the cloud server and maintaining a private connection and keepalive between them helps optimize the traffic flow while keeping the RTT and latency low.

HA ( High-availability )

Some factors affecting Dependability are

• Eventual Consistency
• MultiRegion failover
• Disaster Recovery

A high-availability (HA) architecture implies Dependability.Usually via existence of redundant applications servers for backups: a primary and a standby. These applications are configured so that if primary fails, the other can take over its operations without significant loss of data or impact to business operations.

Downtime / SLA of 5 9’s in aggregate failures

4 9’s of availiability on each service components gives a downtime of 53 mins per service each year. However in aggregate failure this could amlount to (99.99)¹⁰ = 99.9 downtime which is 8-10 hours each year.

Thus, aggregate failure should be taken into consideration while designing reliable systems.

HA for Proxy / Load balancer (LB)

A LB is the first point of contact for outbound calls and usually does not save the dialogue information into memory or database but still contain the transaction information in memory. In case the LB crashes and has to restart, it should

• have a quick uptime
• be able to handle in dialogue requests
• handle new incoming dialogue requests in a stateless manner
• verify auth/authorization details from requests even after restart

HA for Call Control app server

App server is where all the business logic for call flow management resides and it maintains the dialog information in memory.

Issues with in-memory call states : If the VM or server hosting the call control app server is down or disconnected, then live calls are affected, this, in turn, causes revenue loss. Primarily since the state variable holding the call duration would be able to pass onto the CDR/ billing service upon the termination of the call. For long-distance, multi telco endpoint calls running hours this could be a significant loss.

• Standby app server configuration and shared memory : If the primary app server crashes the standby app server should be ready to take its place and reads the dialog states from the shared memory.
• Live load balanced secondary app server + external cache for state varaibles : External cache for state variables: a cluster of master-slave caches like Redis is a good way of maintaining the dialogue state and reading from it once the app server recovers from a failed state or when a secondary server figures it has a missing variable in local memory.

Media Server HA

Assuming the kamailio-RTPengine duo as App server and Media Server. These components can reside in same or different VMs. Incase of media server crash, during the process of restoring restarted RTpengine or assigning a secondary backup RTpengine , it should load the state of all live calls without dropping any and causing loss of revenue . This is achived by

• external cache such as Redis ,
• quick switchover from primary to secondary/fallback media server and
• floating IPs for media servers that ensures call continuity inspite of failure on active media server.

Architecturally it looks the same as fig above on HA for the SIP app server.

Security against malicious attacks

Attacks and security compromisation pose a very signficant threat to a VoIP platform.

MITM attacks

Man in midddle attacks can be counetred by

• End to end encryption of media using SRTP and signals using TLS
• Strong SIP auth mechanism using challenges and creds where password is composed of mixed alphanumeric charecters and atleast 12 digits long
• Authorization / whitelisting based on IP which adheres to CIDR notation

DDOS attacks

DDOS renders a particular network element unavailable, usually by directing an excessive amount of network traffic at its interfaces.

dDOS – multiple network hosts to flood a target host with a large amount of network traffic. Can be created by sending falsified sip requests to other parties such that numerous transactions originating in the backwards direction comes to the target server created congestion.

Can be counetred by

• detect flooding and q in traffic and use Fail2ban to block
• challenge questionable requests with only a single 401 (Unauthorized) or 407 (Proxy Authentication Required)

Read about SIP security practices in deatils https://telecom.altanai.com/2020/04/12/sip-security/

Other important factors leading to security

• Keystores and certificate expiry tracker
• priveligges and roles
• Test cases and code coverage
• Reviewers approval before code merge
• Window for QA setup and testing , to give go ahead before deployment

Identifying outages and Alerting

Raise Event notification alerts to designated developers for any anolous behavior. It could be call based or SMS basef alert based on the sevirity of the situtaion .

Sources for alert manager

• Build failed ( code crashes, Jenkins error)
• Deployment failed ( from Kubernetes , codechef, docker ..)
• configuration errors ( setting VPN etc )
• Server logs
• Server health
• homer alerts ( SIP calls responses 4xx,5xx,6xx)
• PCAP alerts ( Malformed SIP SDP ..)
• Internal Smoke test ( auto testing procedure done routinely to check live systems )
• Support tickets from customer complaints ( treat these as high priority since they are directly impacting customers)

Bottlenecks

The test bed and QA framework play a very crticial role in final product’s credibility and quality.

Performance Testing

• Stress Testing : take to breaking
• Load Testing : 2x to 3x testing
• Soak Testing : typical network load to long time ( identify leaks )

Robust QA framework( stress and monkey testing) to identify potential bottlenecks before going live

A QA framework basically validates the services and callflows on staging envrionment before pushing changes to production. Any architectural changes should especially be validated throughly on staginng QA framework befire making the cut. The qualities of an efficient QA platform are :

Genric nature – QA framework should be adatable to different envrionments such as dev , staging , prod

Containerized – it should be easy to spn the QA env to do large scale or small scale testing and hence it should be dockerized

CICD Integration and Automation – integrate the testcases tightly with gt post push and pull request creation . Minimal Latency and lowest amount of tarffic via public internet

Keep as less external dependecies as possible for exmaple a telecom carrier can be simulated by using an PBX like freeswitch or asterix

Asynchronous Run – Test cases should be able to run asynchronously. Such as seprate sipp xml script for reach usecase

Sample Testcases for VoIP

• Authentication before establish a session
• Balance and account check before establishing a session like whitelisting , blacklisting , restricted permission in a particular geography
• Transport security and adaptibility checks , TLS , UDP , TCP
• codec support validation
• DTMF and detection
• Cross checking CDR values with actual call initiator and terminator party
• cross checking call uuid and stats
• Validating for media and related timeouts

QA frameworks tools – Robot framework

traffic monitor – VOIP monitor

customer simulator – sipP scripts

network traffic analyser – wireshark

pcap collevcter – tcpdump , sngrep

Distributed Data Store

A Distributed Database Design could have many components. It could work on static datastore like

• SQL DB where schema is important
  • MySQL
  • postgress
  • Spanner – Globally-distributed database from Google
• NoSQL DB for to store records in json
  • Cassandra – Distributed column-oriented database
• Cache for low latency retrivals
  • Memcached – Distributed memory caching system
  • Redis – Distributed memory caching system with persistence and value types
• Data lakes for heavy sized data
  • AWS s3 object storage
  • blob storage
• File System
  • Google File System (GFS) – Distributed file system
  • Hadoop File System (HDFS) – Open source Distributed file system

or work on realtime data streams

• Batch processing ( Hadoop Mapreduce)
• Stream processing ( Kafka + spark)
  • Kafka – Pub/sub message queue
• Cloud native stream processing ( kinesis)

Each component has its own pros and cons. The choice depends on requirnments and scope for system behaviour like

• users/customer usuage and expectation ,
• Scale ( read and write )
• Performnace
• Cost

Some fundamental constrains while design distributed data structure :-

p99 latency : 99% of the requests should be faster than given latency. In other words only 1% of the requests are allowed to be slower.

Request latency:
    min: 0.1
    max: 7.2
    median: 0.2
    p95: 0.5
    p99: 1.3

Inidiviual Events vs Aggregate Data

Push vs Pull Architecture

Push : A processing server manages state of varaible in memory and pushes them to data store.

• (-) crashed processingserver means all data is lost

Pull : A temporary data strcyture such as a queue manages the stream of data and processing service pull from it to process before pusging to data stoore.

• (+) a crashed server has to effect on temporarily queue held data and new server can simply take on where previous processing server left.
• (+) can use checkpointing

Popular DB storage technologies

A NoSQL databse can be of type

• Quorem
• Document
• Key value
• Graph

Cassandra is wide column supports asyn master less replication

Hinge base also a quorem based db also has master based preplication

MongoDB documente orientd DB used leacder based replication

SQL scaling patterns include:

• Federation/ federated database system : transparently maps multiple autonomous database systems into a single virtual/federated database.
  • (-) slow since it access multiple data storages to get the value
• Sharding / horizontal partition
• Denormalization : Even though normalization is more memory efficient denormalization can enhance read performance by additing redundant pre computed data in db or grouping related data.
  • Normalizing data reduces data warehouse disk space by reducing data duplication and dimension cardinality. In its full definition, normalization is the process of discarding repeating groups, minimizing redundancy, eliminating composite keys for partial dependency and separating non-key attributes.

• SQL Tuning : “iterative process of improving SQL statement performance to meet specific, measurable, and achievable goals”

Influx DB : to store time series data

AWS Redshift

Apache Hadoop

Redis

Embeed Data : RocksDB

Message Queues(Buffering) vs Batch Processing

Distributed event management, monitoring and working on incoming realtime data instead of stored Database is the preferred way to churn realtime analysis and updates. The multiple ways to handle incoming data are

1. Batch processing – has lags to produce results, not time crtical
2. Data stream – realtime response
3. Message Queues – ensures timely sequence and order

Timeout

• Connection timeout : use latency percentiles to calculate this
• Request timeout

Retries

• exponential backoff : increase waiting time each try
• jitter : adds rabdomness to retry intervals to spread out the load.

Grouping events into object storage and Message Brokers

slower than stream processing but faster than batch processing.

Distributed Event management and Event Driven architecture using streams

In event driven archietcture a produce components performs and action which creates an event thata consumer/listener would subscribes to consume.

• (+) time sensitive
• (+)Asynch
• (+) Decoupled
• (+) Easy scaling and Elasticity
• (+) Heterogeneous
• (+) contginious

Expanding the stream pipeline

Event Streams decouple the source and sink applications. The event source and event sinks (such as webhooks) can asynchronously communicate with each other through events.

Options for stream processing architectures

• Apache Kafka
• Apache Spark
• Amazon kinesis
• Google Cloud Data Flow
• Spring Cloud Data Flow

Here is a post from earlier which discusses – Scalable and Flexible SIP platform building, Multi geography Scaled via Universal Router, Cluster SIP telephony Server for High Availability, Failure Recovery, Multi-tier cluster architecture, Role Abstraction / Micro-Service based architecture, Load Balancer / Message Dispatcher, Back end Dynamic Routing and REST API services, Containerization and Auto Deployment, Auto scaling Cloud Servers using containerized images.

VoIP/ OTT / Telecom Solution startup’s strategy for building a scalable flexible SIP platform

Lambda Architecture

Stream processing on top of map reduce and stream processing engine. In lambda architecture we can send events to batch system and stream processing system in parallel. The results are stiched together at query time.

Apache Kafka is used as source which is a framework implementation of a software bus using stream-processing. “.. high-throughput, low-latency platform for handling real-time data feeds”.

Apache Spark : Data partitioning and in memory aggregation.

Distributed cache for call control Servers

Choosing cache host

• Mod function
  • (-) behaves differently when a new client is added or one is removed , unsuitable for prod
• Consistent hashing ( chord)
  • maps each value to a point on circle

Cache Replacement

Least Recently Used Cache Replacement

Consistency and High Availiability in Cache setup

ReadReplicas live in differenet data centre for disaster recovery.

Strong consistency using Master Slave

Circuits – fail fast, wait for circuit to recover before using again

Design patterns for a circuit base setup to gracefully handle exceptions using fallback.

Circuit breaker : stops client from repeatedly trying to exceute by calculate the error threshold.

Isolated thread pool in circuits and ensure full recovery before calling the service again.

(+) Circuit breaker event causes the entire circuit to repair itself before attempting operations.

References :

• [1] http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html
• [2] http://engineering.hackerearth.com/2013/10/07/scaling-database-with-django-and-haproxy/
• [3] Mastering Chaos – A Netflix Guide to Microservices , QCon London
• [4] https://dzone.com/refcardz/event-stream-processing-essentials
• [5] p99 latency https://stackoverflow.com/questions/12808934/what-is-p99-latency
• [6] Oracle site on partitioing- https://docs.oracle.com/en-us/iaas/Content/Streaming/Concepts/partitioningastream.htm
• [7] Mastering Chaos – A Netflix Guide to Microservices https://www.youtube.com/watch?v=CZ3wIuvmHeM&ab_channel=InfoQ
• [8] Netflix cloud architecture https://netflixtechblog.com/tagged/cloud-architecture

SIPp is an opensource (GNU GPL license) performance testing tool for the SIP protocol and is widely used for Quality assurabce of callflows in voip applications for UAC / UASs cenarios.

It can emulate functioing of a sip phone such as REGISTER , establishes and releases multiple calls with the INVITE and BYE methods , send other SIP requests and wait for reponses based on dafult of custom xml scenario files.

Plus factor is the dynamic display of statistics about running tests (call rate, round trip delay, and message statistics), periodic CSV statistics dumps, TCP and UDP over multiple sockets or multiplexed with retransmission management, regular expressions and variables in scenario files, and dynamically adjustable call rates.

sipp -sn uac -d 10000 -s 9876543210 127.0.0.1:5060  -l 10

It is widley used as aperformnace and load testing tool since it can test SIP equipements like SIP proxies, B2BUAs, SIP media servers, SIP/x gateways, and SIP PBXes and can also emulate thousands of user agents calling your SIP system.

More on SIPp scripts and various exmaples can be read from

https://github.com/altanai/kamailioexamples/tree/master/sipp

Installation

Pre-requisites to compile SIPp are:
– C++ Compiler
– curses or ncurses library
– For TLS support: OpenSSL >= 0.9.8
– For pcap play support: libpcap and libnet
– For SCTP support: lksctp-tools
– For distributed pauses: Gnu Scientific Libraries

sudo apt-get install dh-autoreconf ncurses-dev libssl-dev libpcap-dev libncurses5-dev libsctp-dev lksctp-tools

Either get source code from git

git clone https://github.com/SIPp/sipp.git
cd sipp
cmake . -DUSE_SSL=1 -DUSE_SCTP=1 -DUSE_PCAP=1 -DUSE_GSL=1
make

or download readymade tar , then extract and build with options like

tar -xvzf sipp-xxx.tar.gz
cd sipp
./configure --with-sctp --with-pcap --with-openssl
make

Building certs for TLS based sipp UAS server

make master dir for all certs

mkdir certs 
chmod 0700 certs
cd certs

Make CA folder, create cert and check

mkdir demoCA
cd demoCA
mkdir newcerts
echo '01' > serial
touch index.txt
openssl req -new -x509 -extensions v3_ca -keyout key.pem -out cert.pem -days 3650

Validation of the contents of certs ( optional )

openssl x509 -in cert.pem -noout -text
openssl x509 -in cert.pem -noout -dates
openssl x509 -in cert.pem -noout -purpose

Make domain folder and create the certs for the sip domain name from parent and check

cd ..
mkdir 10.10.10.10
openssl req -new -nodes -keyout key.pem -out req.pem
cd ..
openssl ca -days 730 -out 10.10.10.10/cert.pem -keyfile demoCA/key.pem -cert demoCA/cert.pem -infiles 10.10.10.10/req.pem

Verify the generated certificate for for SIP domain

openssl x509 -in 10.10.10.10/cert.pem -noout -text

Run sipp

sipp -sn uas -p 5077 -t l1 -tls_key /home/ubuntu/certs/10.10.10.10/key.pem  -tls_cert /home/ubuntu/certs/10.10.10.10/cert.pem  -i 10.10.10.10

Verify installation

Run sipp with embedded server (uas) scenario:

sipp -sn uas

On the same host, run sipp with embedded client (uac) scenario:

sipp -sn uac 127.0.0.1 -trace_msg -trace_err

output for server 

 # sipp -sn uas

------------------------------ Scenario Screen -------- [1-9]: Change Screen --

  Port   Total-time  Total-calls  Transport
  5060      32.95 s           61  UDP

0 new calls during 0.874 s period      1 ms scheduler resolution
  19 calls                               Peak was 41 calls, after 28 s
  0 Running, 63 Paused, 12 Woken up
  0 dead call msg (discarded)          
  3 open sockets                        

                             Messages  Retrans   Timeout   Unexpected-Msg

----------> INVITE             61        0         0         0 
<---------- 180                61        0                               <---------- 200                61        0         0                     ----------> ACK         E-RTD1 61        0         0         0               

----------> BYE                61        0         0         0        
   <---------- 200                61        0                            
   [   4000ms] Pause              61                            0        
 ------------------------------ Test Terminated --------------------------------

----------------------------- Statistics Screen ------- [1-9]: Change Screen --

  Start Time             | 2019-02-04    13:04:32.108663 1549265672.108663         
  Last Reset Time        | 2019-02-04    13:05:04.189720 1549265704.189720         
  Current Time           | 2019-02-04    13:05:05.065119 1549265705.065119         
-------------------------+---------------------------+--------------------------
  Counter Name           | Periodic value            | Cumulative value
-------------------------+---------------------------+--------------------------
  Elapsed Time           | 00:00:00:875000           | 00:00:32:956000          
  Call Rate              |    0.000 cps              |    1.851 cps             
-------------------------+---------------------------+--------------------------

  Incoming call created  |        0                  |       61                 

  OutGoi traceings 

———————————————– 2019-02-04 13:08:13.939148
UDP message sent (530 bytes):

INVITE sip:service@127.0.0.1:5060 SIP/2.0
 Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-25-0
 From: sipp ;tag=52422SIPpTag0025
 To: service 
 Call-ID: 25-52422@192.x.x.x
 CSeq: 1 INVITE
 Contact: sip:sipp@192.x.x.x:5061
 Max-Forwards: 70
 Subject: Performance Test
 Content-Type: application/sdp
 Content-Length:   135

v=0
o=user1 53655765 2353687637 IN IP4 192.x.x.x
s=-
c=IN IP4 192.x.x.x
t=0 0
m=audio 6004 RTP/AVP 0
a=rtpmap:0 PCMU/8000

———————————————– 2019-02-04 13:08:13.939310
UDP message received [321] bytes :

SIP/2.0 180 Ringing
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-0
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 INVITE
Contact: 
Content-Length: 0

———————————————– 2019-02-04 13:08:13.939905
UDP message received [486] bytes :

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-0
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 INVITE
Contact: 
Content-Type: application/sdp
Content-Length:   135

v=0
o=user1 53655765 2353687637 IN IP4 192.x.x.x
s=-
c=IN IP4 192.x.x.x
t=0 0
m=audio 6000 RTP/AVP 0
a=rtpmap:0 PCMU/8000

———————————————– 2019-02-04 13:08:13.940159
UDP message sent (371 bytes):

ACK sip:service@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-5
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 ACK
Contact: sip:sipp@192.x.x.x:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

~ RTP

———————————————– 2019-02-04 13:08:13.941658
UDP message sent (371 bytes):

BYE sip:service@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-7
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 2 BYE
Contact: sip:sipp@192.x.x.x:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

———————————————– 2019-02-04 13:08:13.952888
UDP message received [313] bytes :

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-7
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 2 BYE
Contact: 
Content-Length: 0

Time

---------------------------- Repartition Screen ------- [1-9]: Change Screen --

  Average Response Time Repartition 1                                           

           0 ms <= n <        10 ms :        293                                
          10 ms <= n <        20 ms :          9                                
          20 ms <= n <        30 ms :          0                                
          30 ms <= n <        40 ms :          0                                
          40 ms <= n <        50 ms :          0                                
          50 ms <= n <       100 ms :          0                                
         100 ms <= n <       150 ms :          0                                
         150 ms <= n <       200 ms :          0                                
                   n >=      200 ms :          0                                

  Average Call Length Repartition                                               

           0 ms <= n <        10 ms :          0                                
          10 ms <= n <        50 ms :          0                                
          50 ms <= n <       100 ms :          0                                
         100 ms <= n <       500 ms :          0                                
         500 ms <= n <      1000 ms :          0                                
        1000 ms <= n <      5000 ms :        262                                
        5000 ms <= n <     10000 ms :          0                                
                   n >=    10000 ms :          0                                

------------------------------ Sipp Server Mode -------------------------------

Output for client

uac.xml
 
SIPp UAC Remote
 |(1) INVITE |
 |------------------>|
 |(2) 100 (optional) |
 |<------------------| 
 |(3) 180 (optional) | 
  |<------------------| 
|(4) 200             | 
|<------------------| 
|(5) ACK             | 
|------------------>|
 |                     |
 |(6) PAUSE             |
 |                     |
 |(7) BYE             |
 |------------------>|
 |(8) 200             |
 |<------------------|

sipp -sn uac 127.0.0.1 -trace_msg -trace_err
Resolving remote host ‘127.0.0.1’… Done.
—————————— Scenario Screen ——– [1-9]: Change Screen —
Call-rate(length) Port Total-time Total-calls Remote-host
10.0(0 ms)/1.000s 5061 17.32 s 98 127.0.0.1:5060(UDP)

3 new calls during 0.286 s period 1 ms scheduler resolution
0 calls (limit 30) Peak was 25 calls, after 10 s
0 Running, 101 Paused, 7 Woken up
0 dead call msg (discarded) 0 out-of-call msg (discarded)
3 open sockets

                             Messages  Retrans   Timeout   Unexpected-Msg
  INVITE ---------->         98        0         0                  
     100 <----------         0         0         0         0        
     180 <----------         98        0         0         0        
     183 <----------         0         0         0         0        
     200          98        0                            
   Pause [      0ms]         98                            0        
     BYE ---------->         98        0         0                  
     200 <----------         98        0         0         0        

—————————— Test Terminated ——————————–

----------------------------- Statistics Screen ------- [1-9]: Change Screen --

  Start Time             | 2019-02-04    13:08:03.908208 1549265883.908208         
  Last Reset Time        | 2019-02-04    13:08:20.954289 1549265900.954289         
  Current Time           | 2019-02-04    13:08:21.241152 1549265901.241152         

-------------------------+---------------------------+--------------------------
  Counter Name           | Periodic value            | Cumulative value

-------------------------+---------------------------+--------------------------
  Elapsed Time           | 00:00:00:286000           | 00:00:17:332000          

  Call Rate  

Tracings

———————————————– 2019-02-04 13:08:13.934840
UDP message received [527] bytes :

INVITE sip:service@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-0
From: sipp ;tag=52422SIPpTag001
To: service 
Call-ID: 1-52422@192.x.x.x
CSeq: 1 INVITE
Contact: sip:sipp@192.x.x.x:5061
Max-Forwards: 70
Subject: Performance Test
Content-Type: application/sdp
Content-Length:   135

v=0
o=user1 53655765 2353687637 IN IP4 192.x.x.x
s=-
c=IN IP4 192.x.x.x
t=0 0
m=audio 6004 RTP/AVP 0
a=rtpmap:0 PCMU/8000

———————————————– 2019-02-04 13:08:13.936616
UDP message sent (321 bytes):

SIP/2.0 180 Ringing
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-0
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 INVITE
Contact: 
Content-Length: 0

———————————————– 2019-02-04 13:08:13.937003
UDP message sent (486 bytes):

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-0
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 INVITE
Contact: 
Content-Type: application/sdp
Content-Length:   135

v=0
o=user1 53655765 2353687637 IN IP4 192.x.x.x
s=-
c=IN IP4 192.x.x.x
t=0 0
m=audio 6000 RTP/AVP 0
a=rtpmap:0 PCMU/8000

———————————————– 2019-02-04 13:08:13.948679
UDP message received [371] bytes :

ACK sip:service@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-5
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 1 ACK
Contact: sip:sipp@192.x.x.x:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

~ RTP

———————————————– 2019-02-04 13:08:13.949168
UDP message received [371] bytes :

BYE sip:service@127.0.0.1:5060 SIP/2.0
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-7
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 2 BYE
Contact: sip:sipp@192.x.x.x:5061
Max-Forwards: 70
Subject: Performance Test
Content-Length: 0

———————————————– 2019-02-04 13:08:13.949245
UDP message sent (313 bytes):

SIP/2.0 200 OK
Via: SIP/2.0/UDP 192.x.x.x:5061;branch=z9hG4bK-52422-1-7
From: sipp ;tag=52422SIPpTag001
To: service ;tag=52416SIPpTag011
Call-ID: 1-52422@192.x.x.x
CSeq: 2 BYE
Contact: 
Content-Length: 0

time

---------------------------- Repartition Screen ------- [1-9]: Change Screen --

  Average Response Time Repartition 1                                           

           0 ms <= n <        10 ms :        657                                
          10 ms <= n <        20 ms :         20                                
          20 ms <= n <        30 ms :          0                                
          30 ms <= n <        40 ms :          0                                
          40 ms <= n <        50 ms :          0                                
          50 ms <= n <       100 ms :          0                                
         100 ms <= n <       150 ms :          0                                
         150 ms <= n <       200 ms :          0                                
                   n >=      200 ms :          0                                

  Average Call Length Repartition                                               

           0 ms <= n <        10 ms :        649                                
          10 ms <= n <        50 ms :         28                                
          50 ms <= n <       100 ms :          0                                
         100 ms <= n <       500 ms :          0                                
         500 ms <= n <      1000 ms :          0                                
        1000 ms <= n <      5000 ms :          0                                
        5000 ms <= n <     10000 ms :          0                                
                   n >=    10000 ms :          0                                

------ [+|-|*|/]: Adjust rate ---- [q]: Soft exit ---- [p]: Pause traffic -----

Last Error: Overload warning: the major watchdog timer 3000ms has been t…

UAC with Media

SIPp UAC            Remote
    |(1) INVITE         |
    |------------------>|
    |(2) 100 (optional) |
    |<------------------|
    |(3) 180 (optional) |
    |<------------------|
    |(4) 200            |
    |<------------------|
    |(5) ACK            |
    |------------------>|
    |                   |
    |(6) RTP send (8s)  |
    |==================>|
    |                   |
    |(7) RFC2833 DIGIT 1|
    |==================>|
    |                   |
    |(8) BYE            |
    |------------------>|
    |(9) 200            |
    |<------------------|

sipp Usage:

sipp remote_host[:remote_port] [options]

Run SIPp with embedded server (uas) scenario: ./sipp -sn uas On the same host, run SIPp with embedded client (uac) scenario: ./sipp -sn uac 127.0.0.1

Scenario file options:

• -sd : Dumps a default scenario (embedded in the SIPp executable)
• -sf : Loads an alternate XML scenario file. To learn more about XML scenario syntax, use the -sd option to dump embedded scenarios. They contain all the necessary help.
• -oocsf : Load out-of-call scenario.
• -oocsn : Load out-of-call scenario.
• -sn : Use a default scenario (embedded in the SIPp executable). If this option is omitted, the Standard SipStone UAC scenario is loaded. Available values in this version: 
  • ‘uac’ : Standard SipStone UAC (default).
  • ‘uas’ : Simple UAS responder.
  • ‘regexp’ : Standard SipStone UAC – with regexp and variables.
  • ‘branchc’ : Branching and conditional branching in scenarios – client.
  • ‘branchs’ : Branching and conditional branching in scenarios – server.
  Default 3pcc scenarios (see -3pcc option):
  • ‘3pcc-C-A’ : Controller A side (must be started after all other 3pcc scenarios)
  • ‘3pcc-C-B’ : Controller B side.
  • ‘3pcc-A’ : A side.
  • ‘3pcc-B’ : B side.

IP, port and protocol options

• -t : Set the transport mode:
  • u1: UDP with one socket (default),
  • un: UDP with one socket per call,
  • ui: UDP with one socket per IP address. The IP addresses must be defined in the injection file.
  • t1: TCP with one socket,
  • tn: TCP with one socket per call,
  • l1: TLS with one socket,
  • ln: TLS with one socket per call,
  • c1: u1 + compression (only if compression plugin loaded),
  • cn: un + compression (only if compression plugin loaded). This plugin is not provided with SIPp.
• -i : Set the local IP address for ‘Contact:’,’Via:’, and ‘From:’ headers. Default is primary host IP address.
• -p : Set the local port number. Default is a random free port chosen by the system 
• -bind_local : Bind socket to local IP address, i.e. the local IP address is used as the source IP address. If SIPp runs in server mode it will only listen on the local IP address instead of all IP addresses.
• -ci : Set the local control IP address
• -cp : Set the local control port number. Default is 8888.
• -max_socket : Set the max number of sockets to open simultaneously. This option is significant if you use one socket per call. Once this limit is reached, traffic is distributed over the sockets already opened. Default value is 50000
• -max_reconnect : Set the the maximum number of reconnection.
• -reconnect_close : Should calls be closed on reconnect?
• -reconnect_sleep : How long (in milliseconds) to sleep between the close and reconnect?
• -rsa : Set the remote sending address to host:port for sending the messages.
• -tls_cert : Set the name for TLS Certificate file. Default is ‘cacert.pem
• -tls_key : Set the name for TLS Private Key file. Default is ‘cakey.pem’
• -tls_ca : Set the name for TLS CA file. If not specified, X509 verification is not activated.
• -tls_crl : Set the name for Certificate Revocation List file. If not specified, X509 CRL is not activated.
• -tls_version : Set the TLS protocol version to use (1.0, 1.1, 1.2) — default is autonegotiate

SIPp overall behavior options:

• -v : Display version and copyright information.
• -bg : Launch SIPp in background mode.
• -nostdin : Disable stdin.
• -plugin : Load a plugin.
• -sleep : How long to sleep for at startup. Default unit is seconds.
• -skip_rlimit : Do not perform rlimit tuning of file descriptor limits. Default: false.
• -buff_size : Set the send and receive buffer size.
• -sendbuffer_warn : Produce warnings instead of errors on SendBuffer failures.
• -lost : Set the number of packets to lose by default (scenario specifications override this value).
• -key : keyword value Set the generic parameter named “keyword” to “value”.
• -set : variable value Set the global variable parameter named “variable” to “value”.
• -tdmmap : Generate and handle a table of TDM circuits. A circuit must be available for the call to be placed. Format: -tdmmap {0-3}{99}{5-8}{1-31}
• -dynamicStart : variable value Set the start offset of dynamic_id variable
• -dynamicMax : variable value Set the maximum of dynamic_id variable 
• -dynamicStep : variable value Set the increment of dynamic_id variable

Call behavior options:

• -aa : Enable automatic 200 OK answer for INFO, NOTIFY, OPTIONS and UPDATE.
• -base_cseq : Start value of [cseq] for each call.
• -cid_str : Call ID string (default %u-%p@%s). %u=call_number, %s=ip_address, %p=process_number, %%=% (in any order).
• -d : Controls the length of calls. More precisely, this controls the duration of ‘pause’ instructions in the scenario, if they do not have a ‘milliseconds’ section. Default value is 0 and default unit is milliseconds.
• -deadcall_wait : How long the Call-ID and final status of calls should be kept to improve message and error logs (default unit is ms).
• -auth_uri : Force the value of the URI for authentication. By default, the URI is composed of remote_ip:remote_port.
• -au : Set authorization username for authentication challenges. Default is taken from -s argument
• -ap : Set the password for authentication challenges. Default is ‘password’
• -s : Set the username part of the request URI. Default is ‘service’.
• -default_behaviors: Set the default behaviors that SIPp will use. Possible values are:
  • all Use all default behaviors
  • none Use no default behaviors
  • bye Send byes for aborted calls
  • abortunexp Abort calls on unexpected messages
  • pingreply Reply to ping requests If a behavior is prefaced with a -, then it is turned off. Example: all,-bye
• -nd : No Default. Disable all default behavior of SIPp which are the following:
• On UDP retransmission timeout, abort the call by sending a BYE or a CANCEL
• On receive timeout with no ontimeout attribute, abort the call by sending a BYE or a CANCEL
• On unexpected BYE send a 200 OK and close the call
• On unexpected CANCEL send a 200 OK and close the call
• On unexpected PING send a 200 OK and continue the call
• On any other unexpected message, abort the call by sending a BYE or a CANCEL
• -pause_msg_ign : Ignore the messages received during a pause defined in the scenario 
• -callid_slash_ign: Don’t treat a triple-slash in Call-IDs as indicating an extra SIPp prefix.

Injection file options:

• -inf : Inject values from an external CSV file during calls into the scenarios. First line of this file say whether the data is to be read in sequence (SEQUENTIAL), random (RANDOM), or user (USER) order. Each line corresponds to one call and has one or more ‘;’ delimited data fields. Those fields can be referred as [field0], [field1], … in the xml scenario file. Several CSV files can be used simultaneously (syntax: -inf f1.csv -inf f2.csv …)
• -infindex : file field Create an index of file using field. For example -inf ../path/to/users.csv -infindex users.csv 0 creates an index on the first key.
• -ip_field : Set which field from the injection file contains the IP address from which the client will send its messages. If this option is omitted and the ‘-t ui’ option is present, then field 0 is assumed. Use this option together with ‘-t ui’

RTP behaviour options:

• -mi : Set the local media IP address (default: local primary host IP address)
• -rtp_echo : Enable RTP echo. RTP/UDP packets received on port defined by -mp are echoed to their sender. RTP/UDP packets coming on this port + 2 are also echoed to their sender (used for sound and video echo).
• -mb : Set the RTP echo buffer size (default: 2048).
• -mp : Set the local RTP echo port number. Default is 6000.
• -rtp_payload : RTP default payload type.
• -rtp_threadtasks : RTP number of playback tasks per thread.
• -rtp_buffsize : Set the rtp socket send/receive buffer size.

Call rate options:

• -r : Set the call rate (in calls per seconds). This value can bechanged during test by pressing ‘+’, ‘_’, ‘*’ or ‘/’. Default is 10.
  • pressing ‘+’ key to increase call rate by 1 * rate_scale,
  • pressing ‘-‘ key to decrease call rate by 1 * rate_scale,
  • pressing ‘*’ key to increase call rate by 10 * rate_scale,
  • pressing ‘/’ key to decrease call rate by 10 * rate_scale.
• -rp : Specify the rate period for the call rate. Default is 1 second and default unit is milliseconds. This allows you to have n calls every m milliseconds(by using -r n -rp m). Example: -r 7 -rp 2000 ==> 7 calls every 2 seconds. -r 10 -rp 5s => 10 calls every 5 seconds.
• -rate_scale : Control the units for the ‘+’, ‘-‘, ‘*’, and ‘/’ keys.
• -rate_increase : Specify the rate increase every -rate_interval units (default is seconds). This allows you to increase the load for each independent logging period. Example: -rate_increase 10 -rate_interval 10s ==> increase calls by 10 every 10 seconds.
• -rate_max : 

If -rate_increase is set, then quit after the rate reaches this value. Example: -rate_increase 10 -rate_max 100 ==> increase calls by 10 until 100 cps is hit.

• -rate_interval : Set the interval by which the call rate is increased. Defaults to the value of -fd.
• -no_rate_quit : If -rate_increase is set, do not quit after the rate reaches -rate_max.
• -l :  Set the maximum number of simultaneous calls. Once this limit is reached, traffic is decreased until the number of open calls goes down. Default: (3 * call_duration (s) * rate).

• -m : Stop the test and exit when ‘calls’ calls are processed
• -users : Instead of starting calls at a fixed rate, begin ‘users’ calls at startup, and keep the number of calls constant.

Retransmission and timeout options:

• -recv_timeout : Global receive timeout. Default unit is milliseconds. If the expected message is not received, the call times out and is aborted.
• -send_timeout : Global send timeout. Default unit is milliseconds. If a message is not sent (due to congestion), the call times out and is aborted.
• -timeout : Global timeout. Default unit is seconds. If this option is set, SIPp quits after nb units (-timeout 20s quits after 20 seconds).
• -timeout_error : SIPp fails if the global timeout is reached is set (-timeout option required).
• -max_retrans : Maximum number of UDP retransmissions before call ends on timeout. Default is 5 for INVITE transactions and 7 for others.
• -max_invite_retrans: Maximum number of UDP retransmissions for invite transactions before call ends on timeout.
• -max_non_invite_retrans: Maximum number of UDP retransmissions for non-invite transactions before call ends on timeout.
• -nr : Disable retransmission in UDP mode.
• -rtcheck : Select the retransmission detection method: full (default) or loose.
• -T2 : Global T2-timer in milli seconds

Third-party call control options:

• -3pcc : Launch the tool in 3pcc mode (“Third Party call control”). The passed IP address depends on the 3PCC role.
  • When the first twin command is ‘sendCmd’ then this is the address of the remote twin socket. SIPp will try to connect to this address:port to send the twin command (This instance must be started after all other 3PCC scenarios). Example: 3PCC-C-A scenario.
  • When the first twin command is ‘recvCmd’ then this is the address of the local twin socket. SIPp will open this address:port to listen for twin command. Example: 3PCC-C-B scenario.
• -master : 3pcc extended mode: indicates the master number
• -slave : 3pcc extended mode: indicates the slave number
• -slave_cfg : 3pcc extended mode: indicates the file where the master and slave addresses are stored

Performance and watchdog options:

• -timer_resol
  Set the timer resolution. Default unit is milliseconds. This option has an impact on timers precision.Small values allow more precise scheduling but impacts CPU usage.If the compression is on, the value is set to 50ms. The default value is 10ms.
• -max_recv_loops Set the maximum number of messages received read per cycle. Increase this value for high traffic level. The default value is 1000.
• -max_sched_loops Set the maximum number of calls run per event loop. Increase this value for high traffic level. The default value is 1000.
• -watchdog_interval : Set gap between watchdog timer firings. Default is 400.
• -watchdog_reset : If the watchdog timer has not fired in more than this time period, then reset the max triggers counters. Default is 10 minutes.
• -watchdog_minor_threshold: If it has been longer than this period between watchdog executions count a minor trip. Default is 500.
• -watchdog_major_threshold: If it has been longer than this period between watchdog executions count a major trip. Default is 3000.
• -watchdog_major_maxtriggers : How many times the major watchdog timer can be tripped before the test is terminated. Default is 10.
• -watchdog_minor_maxtriggers: How many times the minor watchdog timer can be tripped before the test is terminated. Default is 120.

Tracing, logging and statistics options:

• -f : Set the statistics report frequency on screen. Default is 1 and default unit is seconds.
• -trace_stat : Dumps all statistics in <scenario_name>_.csv file. Use the ‘-h stat’ option for a detailed description of the statistics file content.
• -stat_delimiter : Set the delimiter for the statistics file
• -stf : Set the file name to use to dump statistics
• -fd : Set the statistics dump log report frequency. Default is 60 and default unit is seconds.
• -periodic_rtd : Reset response time partition counters each logging interval.
• -trace_msg : Displays sent and received SIP messages in __messages.log
• -message_file : Set the name of the message log file.
• -message_overwrite: Overwrite the message log file (default true).
• -trace_shortmsg : Displays sent and received SIP messages as CSV in <scenario file name>__shortmessages.log
• -shortmessage_file: Set the name of the short message log file.
• -shortmessage_overwrite: Overwrite the short message log file (default true).
• -trace_counts : Dumps individual message counts in a CSV file.
• -trace_err : Trace all unexpected messages in __errors.log.
• -error_file : Set the name of the error log file.
• -error_overwrite : Overwrite the error log file (default true).
• -trace_error_codes: Dumps the SIP response codes of unexpected messages to <scenario file name>__error_codes.log.
• -trace_calldebug : Dumps debugging information about aborted calls to <scenario_name>__calldebug.log file.
• -calldebug_file : Set the name of the call debug file.
• -calldebug_overwrite: Overwrite the call debug file (default true).
• -trace_screen : Dump statistic screens in the <scenario_name>__screens.log file when quitting SIPp. Useful to get a final status report in background mode (-bg option).
• -screen_file : Set the name of the screen file.
• -screen_overwrite: Overwrite the screen file (default true).
• -trace_rtt : Allow tracing of all response times in __rtt.csv.
• -rtt_freq : freq is mandatory. Dump response times every freq calls in the log file defined by -trace_rtt. Default value is 200.
• -trace_logs : Allow tracing of actions in __logs.log.
• -log_file : Set the name of the log actions log file.
• -log_overwrite : Overwrite the log actions log file (default true).
• -ringbuffer_files: How many error, message, shortmessage and calldebug files should be kept after rotation?
• -ringbuffer_size : How large should error, message, shortmessage and calldebug files be before they get rotated?
• -max_log_size : What is the limit for error, message, shortmessage and calldebug file sizes.

Signal handling:

SIPp can be controlled using POSIX signals. The following signals are handled: USR1: Similar to pressing the ‘q’ key. It triggers a soft exit of SIPp. No more new calls are placed and all ongoing calls are finished before SIPp exits. Example: kill -SIGUSR1 732 USR2: Triggers a dump of all statistics screens in <scenario_name>__screens.log file. Especially useful in background mode to know what the current status is. Example: kill -SIGUSR2 732

Exit codes:

Upon exit (on fatal error or when the number of asked calls (-m option) is reached, SIPp exits with one of the following exit code: 0: All calls were successful 1: At least one call failed 97: Exit on internal command. Calls may have been processed 99: Normal exit without calls processed -1: Fatal error -2: Fatal error binding a socket

Debugging

Issue1  The commonName field needed to be supplied and was missing 

Solution Given the common name while generating the certs

Issue2 If cmake error appears such as “command not found: cmake” then 

solutionsudo apt-get install build-essential cmake

References :

• https://media.readthedocs.org/pdf/sipp-wip/latest/sipp-wip.pdf
• https://github.com/SIPp/sipp.git