1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. If you had a PIAF Forum account in the vBulletin days, log in with your old credentials. Otherwise, sign up again and we'll get you back in business as soon as we can.
  3. A serious FreePBX vulnerability has been reported. Update your Framework Module immediately. Click here for details.
  4. Critical FreePBX vulnerability! Update your server immediately. Details here.

TIPS Stress Testing Asterisk

Discussion in 'Open Discussion' started by blanchae, Dec 7, 2011.

  1. blanchae Guru

    I'm playing around with SIPp and its various support programs such as SIPGUI and WebFrontEnd. Interestingly, Ubuntu installs SIPp V3.1 under the name sip-tester. You can play around with it but really need the WebFrontEnd GUI to control it. SIPp is a command line tool.

    I'm thinking of testing along these lines:

    1. Stress test the PBX's hardware capacity by testing directly across the LAN. First testing using a dedicated SIP trunk then using multiple clients with their own IP addresses.

    2. Stress test the incoming bandwidth by going across the Internet.

    Another option is to use two Cisco routers connected together via a serial link and test by lowering the clock rate (bandwidth) between them.

    3. Also I should be able to select the codecs used and stress for the PBXs ability with different codec plus transcoding.

    Any suggestions? Or comments?

    I've tried V3.2 but couldn't get it working properly. V3.1 seems to be rock stable. These are Unix versions. There is a V3.1 for Windows which works well - (couldn't get V3.2 to work either). The Windows version maxs out on my laptop at 80 calls per sec, after that the program crashes. The docs recommend using Linux for high rates in the 10,000 calls per min range.

    They also have an IMS version which seems more developed for testing to TIA standards but it requires some recompiling of the kernel and other issues.
  2. blanchae Guru

    Here's one of the best tutorials on the SIP protocol that I've found. His intro explains why he wrote it - too much technobabble on the Internet so he wrote a simple clear explanation. If you are to use SIPp, then you need to understand the fundamentals of the SIP protocol.

    Another handy tool is the VoIP Bandwidth Calculator, it gives you an idea of how many calls you can make over differing bandwidth using different codecs and such...
  3. blanchae Guru

    Just a note, you don't run SIPp on PiaF, you run it on another server/PC to connect it to PiaF. There's two user agents (UA) that you can run UAC (client) and UAS (server). Just for testing, you can run two instances of SIPp at the same time on the SIPp server - one a UAC who talks to the UAS. They can test each other through the loopback address 127.0.0.1.
  4. atsak Guru

    A useful exercise. Mind you I wonder if we could take a summary of data from everyone who has these systems as well as a "real world" starting point for comparative purposes.

    Put another way; I can say I can maintain 20 channels of calls on a dual dual core Xeon with 4GB RAM, with mixed SIP and PRI traffic along with conferencing and 72 active extensions (Because that's something that happens on one of PIAF servers regularly).

    I wonder if we all compared notes and contrasted that with your lab testing for interests sake to get a serious "feel" for the limitations?
  5. blanchae Guru

    As soon as I get it up and run some tests then I'll post my results here for comparison. I'm interested in raw hardware performance. I have several different levels of servers I can test. And I'm interested in bandwidth and codec performance.
  6. blanchae Guru

    SIPp version 3.2 up and running...

    I did a complete CentOS 6.0 install on a spare server to be used as a SIP stress test server only. The server has 2 Quad core Xeon processors (Linux reports them as 3.4 GHZ) and 2 GB ram.

    1. Install CentOS 6.0

    - use Live CD (smaller download) then boot to the Live CD and install to HD

    # yum install make
    # yum install gcc
    # yum install gcc-c++
    # yum install openssl-devel
    # yum install libpcap-devel
    # yum install ncurses-devel

    2. Download SIPp 3.2 (sipp.svn.tar.gz) from

    http://sourceforge.net/projects/sipp/files/sipp/3.2/

    Copy to /usr/src/ directory

    3. Compile SIPp

    # gunzip sipp-svn.tar.gz
    # tar -xvf sipp-svn.tar
    # cd sipp-svn

    Choose one of the following:

    * Without TLS (Transport Layer Security) and authentication support: This is the setup if you don't need to handle SIP authentication and/or TLS. It is straight forward:

    # make

    * With TLS and authentication support:

    # make ossl

    * With PCAP play and without authentication support:

    # make pcapplay

    * With PCAP play and authentication support:

    # make pcapplay_ossl <= this is the one you really need - it has support for RTP streams and authentication

    Lastly copy the newly created sipp application to the /usr/bin directory:

    # cp sipp /usr/bin

    4. Open up two terminal windows to test:

    In Terminal 1, start a User Agent Server (UAS) to listen for SIP requests on localhost (127.0.0.1) and port 5060:

    # sipp -sn uas localhost:5060

    In Terminal 2, start a User Agent Client (UAC) to send SIP requests to localhost (127.0.0.1) and port 5060:

    # sipp -sn uac localhost:5060

    The "+" key increases the number of calls by 1, "-" key reduces by calls by 1, "*" key increases by 10, "/" decreases by 10

    After easily doing 500 test calls per second, I gave up for now... (Compare that to WinXP's 80 calls per second limit)

    5. Read lots of docs on using SIPp
  7. blanchae Guru

    Pictures of SIPp windows

    The client generates the calls with an Invite. Here's what the UAC (client) window looks like:

    [IMG]

    It shows the SIP call flow and the messages that are being sent.

    It sends an Invite to the server.
    The client expects the server to respond with a SIP 100 Trying (optional), 180 Ringing, 183 Session Progress (optional), 200 Ok
    If it is a good response, the client will send an ACK and a Pause, then a BYE
    The server should respond with a 200 OK
  8. blanchae Guru

    The server replies to the calls (Invites). Here's what the UAS (server) call flow window looks like:

    [IMG]

    The server will respond to an Invite from the client with the following:
    The server responds with a 180 Ringing and an 200 Ok
    The server expects the client to reply with an ACK then a BYE
    The server then will respond with a 200 OK and a 4 second pause.

    The screen captures were taken at different times so the difference in calls. There's also 4 other windows that can be selected to show other data.

    This is a very simple test call flow with the bare basics shown. There are several more complex default tests that can be done and custom ones that you can create yourself using authentication.
  9. blanchae Guru

    In the above examples, it shows a SIP Invite message. This is what the actual message would be like:

    Code:
    INVITE sip:13@10.10.1.13 SIP/2.0
     Via: SIP/2.0/UDP 10.10.1.99:5060;branch=z9hG4bK343bf628;rport
     From: "Test 15" <sip:15@10.10.1.99>tag=as58f4201b
     To: <sip:13@10.10.1.13>
     Contact: <sip:15@10.10.1.99>
     Call-ID: 326371826c80e17e6cf6c29861eb2933@10.10.1.99
     CSeq: 102 INVITE
     User-Agent: Asterisk PBX
     Max-Forwards: 70
     Date: Wed, 06 Dec 2009 14:12:45 GMT
     Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY
     Supported: replaces
     Content-Type: application/sdp
     Content-Length: 258
    
     v=0
     o=root 1821 1821 IN IP4 10.10.1.99
     s=session
     c=IN IP4 10.10.1.99
     t=0 0
     m=audio 11424 RTP/AVP 0 8 101
     a=rtpmap:0 PCMU/8000
     a=rtpmap:8 PCMA/8000
     a=rtpmap:101 telephone-event/8000
     a=fmtp:101 0-16
     a=silenceSupp:off - - - -
     a=ptime:20
     a=sendrecv
  10. blanchae Guru

    Preliminary SIP stress testing results.

    Got the SIPp server up and running. It is a dual Quad core Xeon 3.4 Ghz 2 GB RAM system. With an internal test UAC to UAS (loopback to loopback) I was getting 1200 calls per second (72000 calls per minute) and with an average of 40% loading across 4 CPUs. I have a screen capture I'll post later. These are sequential calls - one right after the other.

    Tested a P4 2.4 GHz (dual internal core) PiaF box with 512 MB RAM. RAM doesn't seem to make too much difference. Sending UAC sequential SIP invites as outlined above, the CPU started to max out at 18 cps (1080 calls per min). The CPU usage as reported in FreePBX stats would be around 65% and bounce between 15% and 95%.

    I had the asterisk CLI open with -rvvvvv and there was a lot of logging going on. It seemed to chew up a lot of CPU resources. I'll have to check on that and the logging to the hard-drive to see what performance can be gained by reducing the logging.

    I'll have to do the same test with tops to see what the true cpu usage is tomorrow and with an rtp stream rather than just a pause then hangup. Fun times! :D
  11. jroper Guru

    Hi

    I'd be interested to see what happens if you turn off FOP. My last test some years ago saw a massive increase in capacity.

    Joe
  12. blanchae Guru

    You're right on the money. With FOP on, the P4 2.4 GHz processor maxed out at about 20 calls per second (cps) with 90%+ cpu utilization. This was directly attributed to FOP - it was bouncing around 90%. Asterisk was around 17% cpu utilization.

    Disabling FOP, the number of calls was approximately equivalent to 1% cpu per cps. At 60 cps, asterisk was hitting around 65%. Surprisingly fail2ban was hitting between 20% and 25% utilization! That's another program that may need a second look.

    In summary, with FOP the max was about 20 cps (1200 calls per min) sequential. Without FOP, 60 cps (3600 calls per min) sequential.

    My recommendation is don't use FOP! I'll install FOP2 and test it next.
  13. blanchae Guru

    SIPp Internal Tests

    Screen capture of the initial SIPp server test running at over 1200 calls per sec (72000 calls per min) as discussed earlier. Sorry for the large image.

    [IMG]
  14. blanchae Guru

    Testing with FOP enabled

    Here's the results of testing with FOP enabled at 20 cps. op_server.pl is the FOP application sitting at 94.3% cpu utilization.

    [IMG]
  15. blanchae Guru

    Testing with FOP disabled

    Here's SIPp testing with FOP disabled and running 50 cps. Asterisk is at 52.9% and fail2ban at 19.5% is now a concern.

    [IMG]
  16. ukstevef Guru

    There are a few posts about fail2ban and the size of /var/log/secure that is read to find peeps to ban.
    Thinking aloud, can you move /var/log to ramdisk or lograte it if it gets too huge ?
  17. blanchae Guru

    SIPp testing with FOP2

    Screwed up my baseline when I went to test with FOP2. I was using FreePBX 2.6 and upgraded to 2.8. Base asterisk cpu utilization now sits at 20-25% without FOP2 running.

    Without FOP2 running, each call adds about 1%. So at 50 cps, the cpu utilization is about 75-95%. Fail2ban is definitely an issue.

    With FOP2 back at 20 cps, there are some very interesting results. FreePBX reports 99% cpu utilization and top reports anywhere from 100 to 195% cpu utilization for asterisk! fop2_server will pop up once in a while at high cpu utilization from 40 to 90%.

    [IMG]
    I don't know if this is an unfair evaluation of FOP2 as it polls in real-time but it sure seems to put a load on asterisk.

    [IMG]
  18. blanchae Guru

    Baseline without FOP2 and Fail2ban

    So here's testing without FOP2 and fail2ban.

    Without any calls, asterisk uses 0.3% cpu. With 20 cps, asterisk uses approximately 1% per call (bounces between 16 and 22%), with 50 cps, FreePBX reports 30-37% and top about 50%. So with some pretty basic hardware we can get some pretty high numbers. But remember this is no audio or transcoding and just sequential calls consisting of call and hang-up - no authentication either.

    With 100 cps (6000 calls per min!), FreePBX reports 75% utilization and top 115%.

    [IMG]
  19. blanchae Guru

    Asterisk log "full" is full, full, full, full!

    Checking the asterisk log file "/var/log/asterisk/full", it is over 300 MB in size! That may be why fail2ban is pulling high cpu utilization numbers! I'll have to see how to reduce the log file and/or rotate it quicker based on /etc/asterisk/logger.conf and logrotate's config file /etc/logrotate.d/asterisk
  20. blanchae Guru

    FreePBX Modules...

    When I was returning to a baseline, I started disabling the FreePBX 2.8 modules to see if any that were added in the upgrade from 2.6 were loading the system. The modules didn't make any difference.

    The ones that I disabled were: conference, queue, queue priority, parking lot, sys info, route congestion messages and fop2admin. I have a pretty basic system for testing.

Share This Page