TIPS Stress Testing Asterisk

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
I'm playing around with SIPp and its various support programs such as SIPGUI and WebFrontEnd. Interestingly, Ubuntu installs SIPp V3.1 under the name sip-tester. You can play around with it but really need the WebFrontEnd GUI to control it. SIPp is a command line tool.

I'm thinking of testing along these lines:

1. Stress test the PBX's hardware capacity by testing directly across the LAN. First testing using a dedicated SIP trunk then using multiple clients with their own IP addresses.

2. Stress test the incoming bandwidth by going across the Internet.

Another option is to use two Cisco routers connected together via a serial link and test by lowering the clock rate (bandwidth) between them.

3. Also I should be able to select the codecs used and stress for the PBXs ability with different codec plus transcoding.

Any suggestions? Or comments?

I've tried V3.2 but couldn't get it working properly. V3.1 seems to be rock stable. These are Unix versions. There is a V3.1 for Windows which works well - (couldn't get V3.2 to work either). The Windows version maxs out on my laptop at 80 calls per sec, after that the program crashes. The docs recommend using Linux for high rates in the 10,000 calls per min range.

They also have an IMS version which seems more developed for testing to TIA standards but it requires some recompiling of the kernel and other issues.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Here's one of the best tutorials on the SIP protocol that I've found. His intro explains why he wrote it - too much technobabble on the Internet so he wrote a simple clear explanation. If you are to use SIPp, then you need to understand the fundamentals of the SIP protocol.

Another handy tool is the VoIP Bandwidth Calculator, it gives you an idea of how many calls you can make over differing bandwidth using different codecs and such...
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Just a note, you don't run SIPp on PiaF, you run it on another server/PC to connect it to PiaF. There's two user agents (UA) that you can run UAC (client) and UAS (server). Just for testing, you can run two instances of SIPp at the same time on the SIPp server - one a UAC who talks to the UAS. They can test each other through the loopback address 127.0.0.1.
 

atsak

Guru
Joined
Sep 7, 2009
Messages
1,854
Reaction score
197
A useful exercise. Mind you I wonder if we could take a summary of data from everyone who has these systems as well as a "real world" starting point for comparative purposes.

Put another way; I can say I can maintain 20 channels of calls on a dual dual core Xeon with 4GB RAM, with mixed SIP and PRI traffic along with conferencing and 72 active extensions (Because that's something that happens on one of PIAF servers regularly).

I wonder if we all compared notes and contrasted that with your lab testing for interests sake to get a serious "feel" for the limitations?
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
As soon as I get it up and run some tests then I'll post my results here for comparison. I'm interested in raw hardware performance. I have several different levels of servers I can test. And I'm interested in bandwidth and codec performance.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
SIPp version 3.2 up and running...

I did a complete CentOS 6.0 install on a spare server to be used as a SIP stress test server only. The server has 2 Quad core Xeon processors (Linux reports them as 3.4 GHZ) and 2 GB ram.

1. Install CentOS 6.0

- use Live CD (smaller download) then boot to the Live CD and install to HD

# yum install make
# yum install gcc
# yum install gcc-c++
# yum install openssl-devel
# yum install libpcap-devel
# yum install ncurses-devel

2. Download SIPp 3.2 (sipp.svn.tar.gz) from

http://sourceforge.net/projects/sipp/files/sipp/3.2/

Copy to /usr/src/ directory

3. Compile SIPp

# gunzip sipp-svn.tar.gz
# tar -xvf sipp-svn.tar
# cd sipp-svn

Choose one of the following:

* Without TLS (Transport Layer Security) and authentication support: This is the setup if you don't need to handle SIP authentication and/or TLS. It is straight forward:

# make

* With TLS and authentication support:

# make ossl

* With PCAP play and without authentication support:

# make pcapplay

* With PCAP play and authentication support:

# make pcapplay_ossl <= this is the one you really need - it has support for RTP streams and authentication

Lastly copy the newly created sipp application to the /usr/bin directory:

# cp sipp /usr/bin

4. Open up two terminal windows to test:

In Terminal 1, start a User Agent Server (UAS) to listen for SIP requests on localhost (127.0.0.1) and port 5060:

# sipp -sn uas localhost:5060

In Terminal 2, start a User Agent Client (UAC) to send SIP requests to localhost (127.0.0.1) and port 5060:

# sipp -sn uac localhost:5060

The "+" key increases the number of calls by 1, "-" key reduces by calls by 1, "*" key increases by 10, "/" decreases by 10

After easily doing 500 test calls per second, I gave up for now... (Compare that to WinXP's 80 calls per second limit)

5. Read lots of docs on using SIPp
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Pictures of SIPp windows

The client generates the calls with an Invite. Here's what the UAC (client) window looks like:



It shows the SIP call flow and the messages that are being sent.

It sends an Invite to the server.
The client expects the server to respond with a SIP 100 Trying (optional), 180 Ringing, 183 Session Progress (optional), 200 Ok
If it is a good response, the client will send an ACK and a Pause, then a BYE
The server should respond with a 200 OK
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
The server replies to the calls (Invites). Here's what the UAS (server) call flow window looks like:



The server will respond to an Invite from the client with the following:
The server responds with a 180 Ringing and an 200 Ok
The server expects the client to reply with an ACK then a BYE
The server then will respond with a 200 OK and a 4 second pause.

The screen captures were taken at different times so the difference in calls. There's also 4 other windows that can be selected to show other data.

This is a very simple test call flow with the bare basics shown. There are several more complex default tests that can be done and custom ones that you can create yourself using authentication.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
In the above examples, it shows a SIP Invite message. This is what the actual message would be like:

Code:
INVITE sip:[email protected] SIP/2.0
 Via: SIP/2.0/UDP 10.10.1.99:5060;branch=z9hG4bK343bf628;rport
 From: "Test 15" <sip:[email protected]>tag=as58f4201b
 To: <sip:[email protected]>
 Contact: <sip:[email protected]>
 Call-ID: [email protected]
 CSeq: 102 INVITE
 User-Agent: Asterisk PBX
 Max-Forwards: 70
 Date: Wed, 06 Dec 2009 14:12:45 GMT
 Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY
 Supported: replaces
 Content-Type: application/sdp
 Content-Length: 258

 v=0
 o=root 1821 1821 IN IP4 10.10.1.99
 s=session
 c=IN IP4 10.10.1.99
 t=0 0
 m=audio 11424 RTP/AVP 0 8 101
 a=rtpmap:0 PCMU/8000
 a=rtpmap:8 PCMA/8000
 a=rtpmap:101 telephone-event/8000
 a=fmtp:101 0-16
 a=silenceSupp:off - - - -
 a=ptime:20
 a=sendrecv
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Preliminary SIP stress testing results.

Got the SIPp server up and running. It is a dual Quad core Xeon 3.4 Ghz 2 GB RAM system. With an internal test UAC to UAS (loopback to loopback) I was getting 1200 calls per second (72000 calls per minute) and with an average of 40% loading across 4 CPUs. I have a screen capture I'll post later. These are sequential calls - one right after the other.

Tested a P4 2.4 GHz (dual internal core) PiaF box with 512 MB RAM. RAM doesn't seem to make too much difference. Sending UAC sequential SIP invites as outlined above, the CPU started to max out at 18 cps (1080 calls per min). The CPU usage as reported in FreePBX stats would be around 65% and bounce between 15% and 95%.

I had the asterisk CLI open with -rvvvvv and there was a lot of logging going on. It seemed to chew up a lot of CPU resources. I'll have to check on that and the logging to the hard-drive to see what performance can be gained by reducing the logging.

I'll have to do the same test with tops to see what the true cpu usage is tomorrow and with an rtp stream rather than just a pause then hangup. Fun times! :D
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,833
Reaction score
71
Hi

I had the asterisk CLI open with -rvvvvv and there was a lot of logging going on. It seemed to chew up a lot of CPU resources. I'll have to check on that and the logging to the hard-drive to see what performance can be gained by reducing the logging.
I'd be interested to see what happens if you turn off FOP. My last test some years ago saw a massive increase in capacity.

Joe
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Hi

I'd be interested to see what happens if you turn off FOP. My last test some years ago saw a massive increase in capacity.

Joe
You're right on the money. With FOP on, the P4 2.4 GHz processor maxed out at about 20 calls per second (cps) with 90%+ cpu utilization. This was directly attributed to FOP - it was bouncing around 90%. Asterisk was around 17% cpu utilization.

Disabling FOP, the number of calls was approximately equivalent to 1% cpu per cps. At 60 cps, asterisk was hitting around 65%. Surprisingly fail2ban was hitting between 20% and 25% utilization! That's another program that may need a second look.

In summary, with FOP the max was about 20 cps (1200 calls per min) sequential. Without FOP, 60 cps (3600 calls per min) sequential.

My recommendation is don't use FOP! I'll install FOP2 and test it next.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
SIPp Internal Tests

Screen capture of the initial SIPp server test running at over 1200 calls per sec (72000 calls per min) as discussed earlier. Sorry for the large image.

 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Testing with FOP enabled

Here's the results of testing with FOP enabled at 20 cps. op_server.pl is the FOP application sitting at 94.3% cpu utilization.

 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Testing with FOP disabled

Here's SIPp testing with FOP disabled and running 50 cps. Asterisk is at 52.9% and fail2ban at 19.5% is now a concern.

 

ukstevef

Guru
Joined
Sep 20, 2009
Messages
134
Reaction score
0
Location
Birmingham, UK
Here's SIPp testing with FOP disabled and running 50 cps. Asterisk is at 52.9% and fail2ban at 19.5% is now a concern.
There are a few posts about fail2ban and the size of /var/log/secure that is read to find peeps to ban.
Thinking aloud, can you move /var/log to ramdisk or lograte it if it gets too huge ?
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
SIPp testing with FOP2

Screwed up my baseline when I went to test with FOP2. I was using FreePBX 2.6 and upgraded to 2.8. Base asterisk cpu utilization now sits at 20-25% without FOP2 running.

Without FOP2 running, each call adds about 1%. So at 50 cps, the cpu utilization is about 75-95%. Fail2ban is definitely an issue.

With FOP2 back at 20 cps, there are some very interesting results. FreePBX reports 99% cpu utilization and top reports anywhere from 100 to 195% cpu utilization for asterisk! fop2_server will pop up once in a while at high cpu utilization from 40 to 90%.


I don't know if this is an unfair evaluation of FOP2 as it polls in real-time but it sure seems to put a load on asterisk.

 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Baseline without FOP2 and Fail2ban

So here's testing without FOP2 and fail2ban.

Without any calls, asterisk uses 0.3% cpu. With 20 cps, asterisk uses approximately 1% per call (bounces between 16 and 22%), with 50 cps, FreePBX reports 30-37% and top about 50%. So with some pretty basic hardware we can get some pretty high numbers. But remember this is no audio or transcoding and just sequential calls consisting of call and hang-up - no authentication either.

With 100 cps (6000 calls per min!), FreePBX reports 75% utilization and top 115%.

 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Asterisk log "full" is full, full, full, full!

Checking the asterisk log file "/var/log/asterisk/full", it is over 300 MB in size! That may be why fail2ban is pulling high cpu utilization numbers! I'll have to see how to reduce the log file and/or rotate it quicker based on /etc/asterisk/logger.conf and logrotate's config file /etc/logrotate.d/asterisk
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
FreePBX Modules...

When I was returning to a baseline, I started disabling the FreePBX 2.8 modules to see if any that were added in the upgrade from 2.6 were loading the system. The modules didn't make any difference.

The ones that I disabled were: conference, queue, queue priority, parking lot, sys info, route congestion messages and fop2admin. I have a pretty basic system for testing.
 

Members online

PIAF 5 - Powered by 3CX

Forum statistics

Threads
22,582
Messages
138,983
Members
14,677
Latest member
cdmobile