TIPS Stress Testing Asterisk

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Yes its SIPp 3.2 for Linux, there is a SIPp 3.1 that works for WinXP - maybe Win7. The WinXP version would crash after about 80 cps... which is still pretty damn good for real world testing. In reality, you may get 80 calls per hour.

I'm going to try to put an ISO together or instructions on how to build your own server. I think that on a laptop would be a better place. This way you can test the hardware via the LAN and then across the Internet to test your ISP connection.
 

dswartz

Guru
Joined
Feb 17, 2009
Messages
1,056
Reaction score
0
thx. i'm curious to see how my virtualized (esxi) piaf on a 3.2ghz xeon will fare.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
There's still a lot of work to do to come up with a true baseline test that represents real world conditions. This stress testing has found some unexpected areas of performance gains.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,833
Reaction score
71
Hi

Thanks for the clarification on this, its been a while since I looked at this.

entries in the messages log would go to /var/log/asterisk/messages, not /var/log/messages which is used by a lot of services for logging.

Joe
 

gregc

Guru
Joined
Sep 8, 2008
Messages
433
Reaction score
3
Might look at a virtualbox install. Would be an appliance you could export that any other platform's virtualbox could import.

Just a thought.

-Greg
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Hi

Just to make it clear, turning the verbosity down to 0, without the full log being populated will prevent fail2ban from working.

Joe
Double-checked, the full log is still populated with error, warning and notices with the verbosity set to 0. Only the verbose messages are stopped. Setting verbosity to on, the verbose messages appear in the log. Fail2ban only checks the "notices" in the log so it still works.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
In Summary...

PiaF platform under test:

Low end system: P4 2.4 GHz dual core, 512 MB RAM, 80 GB hd, 2x Diguim TE410P 4 channels T1 cards, Xorcom astribank. System is idle with no active calls. Asterisk 1.6, CentOS 5.4 100 Mbps Ethernet card

Stress Test server:

DL380 G4 dual 3.4 Ghz quad core Xeon processors, 2 GB RAM, 36 GB RAID (small drives...), CentOS 6, SIPp3.2, Gigabit Ethernet card

Network:

Isolated for testing only, directly connected LAN with 100 Mbps Cisco 2900XL switch.

Testing:

Sequential SIP UAC test, no RTP payload. Call consisting of initiate call, call is answered on PiaF test server, then call is closed (hung up). The call rate was varied from 10 calls per second (cps) in increments of 10 cps until something broke (figuratively).

To put the cps rate in perspective, the testing was started at 10 cps which is the equivalent of 600 calls per min or 36,000 calls per hour which would be at an extremely large call center rate!

The SIPp WebFrontEnd application allows us to monitor the call rate, the number of successful calls and the number of failed calls. The CPU performance of the PiaF test server was monitored using FreePBX's System Status and the Linux command line tool "top". CPU % Utilization was the performance reference monitored.

Results:

How valid is this testing? It is not representative of real world loads as no RTP payload was sent. At this point it was used to stress test the hardware and software to see what would happen. It did bring up some unexpected services that loaded down the system and subsequent discussions resulted in system mods that dramatically increased performance.

Things that affected performance at these high call rates:

  • FOP - Flash Operator Panel version 1. It chews up a lot of resources. Not recommended. It was disabled for testing.
  • FOP2 - Flash Operator Panel version 2. Much better than FOP but still chews up resources. Before we blame FOP2, there is more to discuss as to how the Asterisk Manager Interface AMI works and its affect on performance. The AMI is the real culprit and not FOP2. This will be covered in another thread. FOP2 was disabled for testing.
  • Asterisk Logging - This is the major source of poor performance. Two specific areas: log files and the asterisk CLI.

Solution to improve performance.

The biggest problem was the verbose messages being logged in the /var/log/asterisk/full log file and displayed in the asterisk CLI. This puts an incredible load on the system. It is compounded because fail2ban parsed the full log file. If the log file is large, then fail2ban has to parse a large text file every few seconds. This can put an incredible load on the server.

The solution is in /etc/asterisk/asterisk.conf, set "verbose = 0". This will stop the verbose messages from appearing in the log file and from appearing on the asterisk CLI. It will not stop warnings, debug, error and notice messages from appearing. Notices are used specifically by fail2ban so fail2ban continues to work.

If verbose messages are required for troubleshooting, they can be enabled via the asterisk CLI by issuing the command "core set verbose 7". Then verbose messages are shown on the CLI and logged to the full log file.

The performance results:

  • Base system with FOP enabled: 20 cps
  • Base system with FOP disabled: 60 cps
  • Base system with FOP disabled and fail2ban disabled: 100 cps max
  • Base system with FOP disabled, verbose=0, fail2ban enabled: 150 cps max

The last test showed an amazing 150 cps max load (9000 cpm, 540,000 calls per hour)! Now it becomes apparent how simple logging can affect performance.

Recommendations:

Disable verbose messages from a production server unless needed for troubleshooting.

On a side note, several FreePBX modules were disabled to see if the modules would affect performance. None did.
 
Last edited by a moderator:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
15,325
Reaction score
2,705
Not to look a gift horse in the mouth, but...

It would be wonderful to compare results with a test using Asterisk 1.8 both with CentOS 5 and with the new 6.2 which is installed automatically with a PIAF2 install now. :biggrin5:
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,833
Reaction score
71
Hi

One point to note is that you are using the FreePBX dashboard to monitor concurrent calls.

That in itself generates a quite a number of AMI calls, and it must take a certain amount of processor to display all the different graphs.

It would be interesting to complete these tests to see if watching the dashboard does make a difference.

To count the calls in progress, simply do
Code:
rasterisk -x "core show calls"
from the command line at intervals.

Joe
 

dad311

Guru
Joined
Jan 13, 2008
Messages
604
Reaction score
2
Very interesting tool. Could this tool possibly be used for a DoS attack?

What would happen if I pounded someone else server with Sip requests? Would call processing be affected?
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
No, the setup for this testing is to create a SIP extension on the target PBX to connect to. From what I've seen, if there is no target SIP extension response then after about 2 or 3 failed calls the SIP test UAC times out waiting for a response and the program exits on an error. It is a UAC (user agent client) to a UAS (user agent server) communications. In addition, you would need your network firewall open on the SIP port plus network address translation pointing to your PBX. And finally allow anonymous SIP calls in your PBX.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,833
Reaction score
71
Could this tool possibly be used for a DoS attack?

The way that Blanchae has configured SIPP, it could not be used as a DOS attack as per his comments.

However, if you were to send many calls, unauthenticated, e.g to sip/[email protected], and those SIP messages were reaching your PBX, because of your firewall configuration, then yes, DOS will occur.

...And finally allow anonymous SIP calls in your PBX.
On Blanchae's server, if you were to set anonymous SIP calls to YES , and set the catchall destination to hangup.you would have to send about 100 CPS to make the PBX unusable, according to his tests.

If you disallow anonymous SIP calls, then FreePBX will play a message (sorry not in service), and to create a DOS, you would have to send a far lower number of calls, as Asterisk will have to deal with the overhead of playing this message, transcoding if necessary, and while the message is playing, the hacker can send hundreds more calls per second.

My advice is to set anonymous SIP calls to yes, if you are exposing SIP to the internet, and set the catchall destination to hangup. You will survive a DOS far longer this way.

Joe
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
When I get back to work in Jan, I'll check what happens when this is attempted and what appears in the log file. If a standard message appears than a fail2ban rule could be made.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,833
Reaction score
71
Hi

On DID not matched, you could put any string you like in at any of the log levels, see http://www.voip-info.org/wiki/view/Asterisk+cmd+Log.

So a 2 line piece of custom dial plan which was called when a DID was not matched, (i.e. ended up in the catchall DID route), the first line to write a message to log, so fail2ban could watch for it, and the second line to hangup would put a stop to DOS attacks, or "SPIT"

Joe
 

Members online

No members online now.

PIAF 5 - Powered by 3CX

Forum statistics

Threads
22,514
Messages
138,531
Members
14,644
Latest member
goseph