TIPS Stress Testing Asterisk

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
I guess at this point, the question should be asked "What is a reasonable load on a PiaF server?" The testing so far has been pretty extreme at the baseline 20 cps (1200 calls per min) and up to 100 cps (6000 calls per min) for sequential calls.

The next step for testing is to start concurrent call testing to see what the maximum number of concurrent calls for a piece of hardware. I feel that this would give a better real world picture. So the next question is "what would be an acceptable scenerio for concurrent call testing?".

The next tests would require authentication and an audio payload of 20 seconds or so. I can test different codecs too.

Later I'll try transcoding from one codec to another.
 

gregc

Guru
Joined
Sep 8, 2008
Messages
433
Reaction score
3
Just want to say I appreciate the testing you are doing. Very interesting stuff.

Are you slow this time of year or something? haha

-Greg
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
I'm a teacher at SAIT Polytechnic and the Fall semester finished for me on Monday. Fortunately, l'm all prepared for the Winter semester which starts Jan 9. I've got a couple of weeks to play in the labs and a sh*t load of equipment to play with. I get to be a mad scientist so to speak.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

Just to test the fail2ban issue, do "core set verbose 0" in the asterisk CLI, this will stop logging to the full log.

We can then decide which logging to do from the standard.

full => notice,warning,error,debug,verbose

Joe
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Log testing - performance increase!

This morning, I reconfigured the logs for asterisk and fail2ban and gained a substantial performance increase. On the baseline of 20 cps, the cpu load was 15% (top reporting) - one call roughly corresponded to 0.75% cpu load. This is the results without the asterisk CLI open. (This is with FOP2 stopped)

LogsCorrected-noFOP2.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
asterisk CLI load

When the asterisk CLI was open, using only "asterisk -r", the load increased by about 20% due to logging to the console. I use "top" as the reference and have the FreePBX status up just for comparison.

LogsCorrectedCLI-noFOP2.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Stressing the P4 2.4Ghz 512 MB server

Here's pushing 80 cps (4800 calls per min) through it with the improved logging configuration (I'll post the fixes soon). Great performance: 55% CPU! :crazy:

LogsOK80cps-noCLI-noFOP2.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Asterisk CLI affect...

Still running the same 80 cps. When the asterisk CLI was turned on with just "asterisk -r". The performance took a nose dive. 74% cpu which would spike up in the 90%. Just for interest sake, the CLI info was just whizzing by and not static like the screen capture!

Recommendation: don't leave your servers with the CLI open unless necessary!

LogsOK80cps-CLI-noFOP2.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Log fixes...

1. Modify /etc/asterisk/logger.conf

In /etc/asterisk/logger.conf, comment out the following line with a leading ";"

Code:
;full => notice,warning,error,debug,verbose

replace with the following which will create 4 log files in /var/log/asterisk that will be the first to show up in the directory listing and easily identify their purpose. Notice that the verbose log file is commented out as it is the main culprit for generating huge log files. If you need to do some serious troubleshooting, you could uncomment the line and reload asterisk from the CLI.

Code:
asterisk-debug.log => debug
asterisk-notice.log => notice
asterisk-warning.log =>warning
asterisk-error.log =>error
;asterisk-verbose.log=>verbose

2. Modify /etc/logrotate.d/asterisk

Modify /etc/logrotate.d/asterisk for the following. This will increase the number of log files to rotate to 10 and limit the file size to 100kB:

Code:
/var/log/asterisk/*log {
   missingok
   rotate 10
   daily
   size=100k
   create 0640 asterisk asterisk
   postrotate
       /usr/sbin/asterisk -rx 'logger reload' > /dev/null 2> /dev/null
   endscript
}

The new log files end with .log so they will fall under the same rules. It only checks once a day but the logs don't grow very fast anymore. :biggrin5:

3. Modify fail2ban

Point /etc/fail2ban/jail.conf to the new asterisk-notice.log file. fail2ban only exams the notices. You can verify this by examining /etc/fail2ban/filter.d/asterisk.conf

Code:
[asterisk-iptables]

enabled  = true
filter   = asterisk
action   = iptables-allports[name=ASTERISK, protocol=all]
           sendmail-whois[name=ASTERISK, dest=root@localhost, [email protected]]
logpath  = /var/log/asterisk/asterisk-notice.log  <= change
#logpath  = /var/log/messages
maxretry = 5
bantime = 1800

Restart and now you've gained a performance increase!
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Just realized that by changing the log file names from "full", it might of broke the log monitoring in FreePBX. I'll check on it tomorrow and if need be, change the asterisk-notice.log to "full" for backwards compatibility.
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

In earlier versions of Asterisk, (i've not checked later ones) edit logger.conf to uncomment the messages line

Code:
;messages => notice,warning,error
or just
Code:
messages => notice
This appears to log independently of the verbosity of Asterisk, so is better suited to fail2ban.

In asterisk.conf, uncomment this line and set to
Code:
;verbose = 0

If you don't restart Asterisk - do core set verbose 0

This turns off logging to the full log, and the verbosity in the console, and of course logging in "Asterisk Logs" in FreePBX, however my view is that if you want to do some trouble shooting, you will turn on the logging with core set verbose 99 at the asterisk CLI, and the full log will be populated.

This may be a more straight forward solution, and gives the flexibility of logging, or not, in FreePBX without leaving the GUI.

Unless I have missed it, you don't say whether you are using 64 bit or 32 bit version of CentOS, and it is not really clear what the hardware specs are for your PiaF server. I wonder if there will be a change with a different architecture.

These are pretty stunning results. I look forward to seeing what happens when you start introducing RTP to the testing.

Joe
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Just realized that by changing the log file names from "full", it might of broke the log monitoring in FreePBX. I'll check on it tomorrow and if need be, change the asterisk-notice.log to "full" for backwards compatibility.

Yep, it broke the FreePBX LogFile module as it looks for the "full" log file. Joe's solution seems more elegant. I'll give it a try pronto and report on how it works.
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Got so excited about the testing, forgot to take a look at the test PiaF server to see what versions its running. It is old as I set it up about 2 years ago. P4 2.4 Ghz dual core, 512 MB ram, 80 GB PATA drive 2x 4 channel T1 cards, Xorcom 16x FXS channel bank. No active connections.

Status.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Joe's suggestion works better than great! Set verbose = 0 in /etc/asterisk/asterisk.conf and it is amazing. At 20 cps, the CPU load was between 6% to 10%. The full log file was miniscule at 35 kB! Set the logging through the CLI using "core set verbose 7" and we were back to the original test results and the full log file was being populated as before.

Verbose=0.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Pushing the limit!

With verbose=0, I pushed the little server to its limit. 150 cps (9000 calls per min). :eek: The cpu utilization was occasionally peaking to 80+% and back to what you see here. Any faster and then errors would appear. Joe's suggestion is highly recommended at this time. Set no verbosity unless needed.

Verbose=0-150cps.JPG
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Lastly 150 cps plus "speak your extension"....

So I wondered what the audio would sound like if I used my Xlite and dialed *65 for "speak your extension" with 150 cps going on. Perfect audio - not a blip. The cpu utilization went up but not a hiccup in the audio. The cpu utlization is not fixed - it bounces between 50-85%. I just happened to capture it on the low swing.

SpeakYourExtension150cps.jpg
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
New thread dedicated to FOP2 performance.....

I had a good discussion with Asternic about FOP2 and its performance. I'll start a new thread to discuss fop2 stress testing, explaining the asterisk manager interface (AMI) limitations and how to improve fop2 performance. The good news is that combining Asternic's optimization with Joe's verbose fix, we can reach 40 cps (2400 calls per min) with FOP2 working with the current testing. Here's a teaser:

WithFOP2-40cps.JPG
 
Last edited by a moderator:

Speedy2k

Member
Joined
Oct 11, 2008
Messages
387
Reaction score
0
Wow! this is such a good thing to do!!! Good job!! Optimizing all used part of PIAF will be so great! thanx a lot!
 

jroper

Guru
Joined
Oct 20, 2007
Messages
3,832
Reaction score
71
Hi

It's quite amazing to see such capacity out of what is quite an ancient server.

Just to make it clear, turning the verbosity down to 0, without the full log being populated will prevent fail2ban from working.

You should enable the "messages" log as previously stated, then point fail2ban at that, with only notices enabled, as my tests some time ago suggested that the messages log file gets populated irrespective of the verbosity level.

Joe
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Just to make it clear, turning the verbosity down to 0, without the full log being populated will prevent fail2ban from working.

The full log is configured as follows:

full => notice,warning,error,debug,verbose

Setting verbose =0, just disables the verbose messages, the notices, warnings, error and debug info still come through. fail2ban checks the notices in the full log and it still works. That's one of the things that I checked during testing - including registering with the wrong password. It won't hurt to double-check tomorrow to verify.

BTW you can call a log file any name you want to, like "full", "messages" or "asterisk-notice.log" which I did in an earlier post in the thread. It just happens that the messages log file is used by a lot of other services.
 

Members online

Forum statistics

Threads
25,779
Messages
167,505
Members
19,199
Latest member
leocipriano
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top