TIPS Asterisk 100% CPU usage

[hbonath]

Hey guys, one of my PBXs machines is having an issue where Asterisk is consuming 100% of the CPU somewhat randomly.

The only way to get it to settle down is to restart asterisk, which I've been doing with a 'core restart gracefully'

This particular system has about 80 users and is relatively busy, however the CPU usage doesn't seem to warrant actual CPU needs, it seems like something is getting stuck and overtaxing the CPU.
When the CPU is pegged out, what's surprising is that it doesn't seem to affect call quality at all.

This particular system is running the "other guys" distro, so please go easy on me!
I just trust this forum for this question way more then theirs!

Asterisk version is 1.8.21.0, CentOS 6.2.
Phones are mostly Aastra 6737i, with a few Aastra 610D SIP DECT devices.

Right now I'm running VoipMonitor as well as log watching to try to figure out if I can track the CPU spike to a specific device/app/user but I haven't been able to find any correlation up until this point.

I've also run some of the pstack stuff from this link: http://www.moythreads.com/wordpress/2009/05/06/why-does-asterisk-consume-100-cpu/
and haven't been able to see anything interesting when I view the offending thread.

It's driving me crazy, do any of you guys out there have some better/different methods for tracking down rogue high CPU usage from the asterisk process?
Thanks!
-Henry

 

[hbonath]

I've confirmed with Top. Asterisk process hits 99%

 

[wardmundy]

Are HylaFax and AvantFax running on this machine?? We've seen iaxmodem cause this.

 

[hbonath]

Nope no FAX applications running

 

[wardmundy]

Probably time to try a newer release of Asterisk 1.8.

 

[reconwireless]

Which version of the FreePBX Distro are you using? Since you have CentOS 6.2 I'm going to assume 2.210.62, PBX in a Flash has a very different update/upgrade procedures than the FreePBX Distro. Utilizing the Distro you can run the update scripts provided in the wiki to get you to the latest version of Asterisk, all of the 6.2 releases have been EOL for updates since April, so you can follow the update scripts to update to the latest stable version. See: http://wiki.freepbx.org/display/FD/FreePBX-Distro-2.210.62 You can also utilize SysAdmin Pro to run updates directly from the GUI, if you have a license for that.

 

[hbonath]

Which version of the FreePBX Distro are you using? Since you have CentOS 6.2 I'm going to assume 2.210.62, PBX in a Flash has a very different update/upgrade procedures than the FreePBX Distro. Utilizing the Distro you can run the update scripts provided in the wiki to get you to the latest version of Asterisk, all of the 6.2 releases have been EOL for updates since April, so you can follow the update scripts to update to the latest stable version. See: http://wiki.freepbx.org/display/FD/FreePBX-Distro-2.210.62 You can also utilize SysAdmin Pro to run updates directly from the GUI, if you have a license for that.

Correct, 2.210.62-7 is where I'm at.
Will that upgrade keep me on asterisk 1.8?
Do you think that is the cause of the CPU spikes? Asterisk version?
Thanks for your help!
-Henry

 

[reconwireless]

If you follow the upgrade path you will eventually reach 1.8.22.0 seems I remember some locking thread issues in 1.8.21.0 but in my old age I don't remember exactly what was going on there, you can see the release notes here for the latest stable version http://wiki.freepbx.org/display/FD/3.211.63+Release+Notes

Using the FreePBX Distro you can also switch Asterisk Versions on the fly, so you could swap out to Asterisk 10 using version 2.210.62 or switch between 1.8, 10 or 11 utilizing the latest stable build, usually only takes a few minutes with a fast internet connection. See the details here: http://wiki.freepbx.org/display/FD/Changing+Major+Asterisk+Versions+on+the+Fly

 

[tm1000]

I must have missed the part where the freepbx distro is taboo here?? I thought we all get along!

If not I just want to clarify that, yes, we all do get along quite well. Ward talks to us privately on the phone all the time

I don't want people to think there is ill will between freepbx and pbx in a flash. There's none. I'm offended that we are referred to as "other guys". I started off here and learned everything from pbx in a flash (besides programming skills that was college) and now that I'm a developer for freepbx I come back here quite frequently

Anyways I'd hate for our users to feel out of place here and vice-versa...

 

[hbonath]

I must have missed the part where the freepbx distro is taboo here?? I thought we all get along!

If not I just want to clarify that, yes, we all do get along quite well. Ward talks to us privately on the phone all the time

I don't want people to think there is ill will between freepbx and pbx in a flash. There's none. I'm offended that we are referred to as "other guys". I started off here and learned everything from pbx in a flash (besides programming skills that was college) and now that I'm a developer for freepbx I come back here quite frequently

Anyways I'd hate for our users to feel out of place here and vice-versa...

Thanks for clarifying!! I hope I didn't offend anyone! I like both distros for their different strengths, and use them both in my business. However this forum is most definitely my fave!

 

[hbonath]

If you follow the upgrade path you will eventually reach 1.8.22.0 seems I remember some locking thread issues in 1.8.21.0 but in my old age I don't remember exactly what was going on there, you can see the release notes here for the latest stable version http://wiki.freepbx.org/display/FD/3.211.63+Release+Notes

Using the FreePBX Distro you can also switch Asterisk Versions on the fly, so you could swap out to Asterisk 10 using version 2.210.62 or switch between 1.8, 10 or 11 utilizing the latest stable build, usually only takes a few minutes with a fast internet connection. See the details here: http://wiki.freepbx.org/display/FD/Changing+Major+Asterisk+Versions+on+the+Fly

Very good, I did read the asterisk release notes on the 1.8.22.0 version on digium's site and it mentioned a fix for thread locking, which I am hoping will do the trick!
I have tried the switch versions in the past which has worked well for me.
I have not done the major release update before so I will definitely test this on a copy of this particular VM to see what the process looks like before I roll it out to the production server.
Thanks for your help, I'll report back if I continue to have issues!

 

[Stewart]

If you can install pstack, this post helps to narrow down which module is causing the issue.

 

[hbonath]

If you can install pstack, this post helps to narrow down which module is causing the issue.

Thanks for the tip, however if you note my original post references that thread.
I wasn't able to get anywhere with pstack, unless there's a better option.

 

[bucasia]

I've seen Asterisk do this when the astdb database is corrupt.

 

[Stewart]

Wow, not sure how i missed that! Can you unload any unused modules to see if it goes away? I've also seen that happen when an authorized user is actively trying to register into the system over and over again. Do you have 5060 forwarded?

 

[hbonath]

AstDB "seems" to look fine, this particular PBX has about 3900 entries. I noticed many of them stated "bad value" however I think that is OK as it was referencing things like language under extensions, which is blank in the FreePBX interface. I also compared this one's astdb against some of my other PBX boxes running the same version of the software without issue.

Stewart; This particular system is open publicly (yes, yes i know!) however I haven't seen any overloads of reg attempts with any of my monitoring systems. I can adjust the firewall if needed.
I see you mention an "authorized" user, are you thinking there could be a phone with a bad SIP Secret trying to reg over and over again?

I appreciate you guys help with trying to troubleshoot this, it's driving me nuts trying to figure it out!
I haven't had a chance to upgrade the distro version as of yet, but again, I probably have 10 other systems running the same distro version with same version of asterisk without issue...

 

[reconwireless]

Are all of your FreePBX modules up to date to the latest versions?

 

[hbonath]

Yes, all modules were up to date aside from pinsets (which I'm not even using)
I did just upgrade it a minute ago.

 

[tm1000]

Use asterisk-version-switch to test a different asterisk version

 

[Stewart]

You should be able to see in the Call Logs some random weird attempts to place the call to some very long phone numbers if someone is trying to hit you from unregistered accounts. And I'm sorry, I meant to say UNauthorized user.

EDIT: I've also frequently seen it in the chan_ooh323.so module. Try disabling that.