TIPS PIAF restarts automatically

[nikhilroshania]

Hello,

I am looking for expertise advice to troubleshoot the strange issues I am having with our asterisk server.
The issues is:
>> the asterisk restarts automatically on random occasions. The restart frequency is random, sometime it restarts twice in a day and sometime it restarts just ones in a month.
>> The restart event sometime happens in middle of the night, when there are no active calls.
>> Everytime it restarts a dump file is created in /tmp, this file is created. now I dont know how to read that file and how to take the troubleshooting actions.

Server Information:
>> Server is hosted in datacentre.
>> I am using PIAF Purple version now.
>> I also tried pbxinaflash with asterisk 1.8 and asterisk 11 (piaf-green).
>>We also changed the server hardware.
>>None of the above help.
I have no idea what triggers to restart asterisk.
if someone is providing commercial tech support then please advice.
thanks for going through this email.
--
Nikhil.

 

[nikhilroshania]

isnt there anyone having similar issue?
please share your thoughts.

 

[MGD4me]

Since you have tried replacing the server hardware, as well as the software, there's not much left. My trouble-shooting would have to start by looking elsewhere.

Where does the server get it's AC power? Check where it actually is plugged in, to see if the plug or outlet is loose, or corroded. Try a different outlet. Is the phyiscal location prone to power interruptions. Is any other hardware at this location experiencing 'strange problems'?

I would try installing a battery backup UPS to help 'condition' the AC power feed, and protect the server from 'dirty' AC or dropouts. Could there be other equipment in the building that might be drawing short term heavy AC loads?

 

[nikhilroshania]

Since you have tried replacing the server hardware, as well as the software, there's not much left. My trouble-shooting would have to start by looking elsewhere.

Where does the server get it's AC power? Check where it actually is plugged in, to see if the plug or outlet is loose, or corroded. Try a different outlet. Is the phyiscal location prone to power interruptions. Is any other hardware at this location experiencing 'strange problems'?

I would try installing a battery backup UPS to help 'condition' the AC power feed, and protect the server from 'dirty' AC or dropouts. Could there be other equipment in the building that might be drawing short term heavy AC loads?

=======================================
Thanks for the reply.

its not the OS that restarts but the asterisk.
the attached is the screenshot of the pbx dashboard. here system uptime is 28 weeks but asterisk uptime is only 2 days. this is because asterisk crashed and restarted automatically just before 2 days.

I think there is something with the asterisk that is triggering a restart.

after every crash and restart a file is created in /tmp.
I dont know how to use that file for troubleshooting.

 

[randy7376]

nikhilroshania

Some thoughts...

Have you looked at the Asterisk log file, yet, for any clues? Check /var/log/asterisk/full. You can use grep to narrow things down a bit.
For example:

cat /var/log/asterisk/full | grep -i fail
cat /var/log/asterisk/full | grep -i error
cat /var/log/asterisk/full | grep -i warn

Is the server exposed directly to the Internet or is it secure behind a hardware-based firewall? Is port 5060 accessible from Internet whether through direct connection or via port-forwarding at the firewall? It's possible someone is pounding port 5060 in an effort to access your server. Too many connections at once could cause Asterisk to fail.

From the Asterisk command-line, run sip show channels. How many SIP channels are shown? If it's in the hundreds or thousands, you may have just found your problem.

 

[nikhilroshania]

Guys,
thanks for the inputs.

The PBX is behind a software firewall but open to remote extensions as extensions are cross multiple locations. I have allowed only static IP address on firewall,
but I still notice a flood of incoming calls from random numbers.

randy7376 I also share the same thoughts that someone is trying to gain access of the system, which ends up with asterisk crash and automatic reload.
because such events happens on random interval I cannot take the channel info with 'sip show channels', unless there is any option to save history of such information.

there must be a better way of deploying asterisk. can anybody share their deployment structure?

Thanks again.

--
Nikhil

 

[randy7376]

Guys,
thanks for the inputs.

The PBX is behind a software firewall but open to remote extensions as extensions are cross multiple locations. I have allowed only static IP address on firewall,
but I still notice a flood of incoming calls from random numbers.

...

there must be a better way of deploying asterisk. can anybody share their deployment structure?

From your description, it appears you have your PBX wide-open to the Internet. Especially, if you're receiving a "flood of incoming calls from random numbers".
This is the most-likely reason Asterisk is restarting. You need to limit access to just your "multiple locations" and your VoIP provider(s).

In a nutshell: If each location has it's own static IP address, white-list it in your firewall along with your VoIP provider's IP addresses. Black-list everything else.

As an example, my firewall at home has the ability to make use of IP address lists in the firewall rules. I have an address list (my white-list is named SIP Allowed on my firewall) that contains any remote users and my VoIP provider. My firewall rule for UDP Port 5060 is to drop all traffic except for the entries in the SIP Allowed address list. My provider and the rmote users are allowed through while any other traffic is dropped. Easy!

At work, we have a VPN tunnel back to our provider which does not expose our PBX to the Internet. This was a requirement from our current ITSP. Zero exposure to the Internet. All remote extensions are connected via site-to-site VPN. Again, no exposure to the Internet.

If you have users using SIP phones or softphones from dynamic IPs, then you may want to look into PIAF's Travelin' Man. wardmundy may be able to provide a more-recent article/link regarding Travelin' Man.

As long as port 5060 is open to the Internet, your PBX will continued to get pounded. What I've written above should give you enough of an idea on what you should be doing.

 

[billsimon]

nikhilroshania review your /var/log/messages and see whether what you are experiencing is Asterisk crashes. In addition to the segfault shown in the messages file, you will see the restart in /var/log/asterisk/full.

You may be experiencing the bug with sqlite modules. Seems to be triggered with heavier call volume (or in your case, a lot of CDRs being generated from SIP scanners). See http://pbxinaflash.com/community/index.php?threads/sqlite2-in-piaf.14864/#post-95573

 

[billsimon]

From your description, it appears you have your PBX wide-open to the Internet. Especially, if you're receiving a "flood of incoming calls from random numbers".
This is the most-likely reason Asterisk is restarting. You need to limit access to just your "multiple locations" and your VoIP provider(s).

randy, I am concerned about this statement and think it has the potential to mislead people. This makes it sound like Asterisk is not suited for high call volume, and when it gets a flood of incoming calls, it will just roll over on its back and die. Have you actually seen this kind of behavior? If so, what was the root cause?

I think that Asterisk can handle high call volume just fine when properly configured.

 

[randy7376]

randy, I am concerned about this statement and think it has the potential to mislead people. This makes it sound like Asterisk is not suited for high call volume, and when it gets a flood of incoming calls, it will just roll over on its back and die. Have you actually seen this kind of behavior? If so, what was the root cause?

I think that Asterisk can handle high call volume just fine when properly configured.

billsimon

Yes, you make a good point and I can see how that could be misinterpreted. I made that statement based upon my own personal experience. Some background:

I have seen older versions (1.4/1.6) of Asterisk go off the deep end when flooded with a couple of thousand calls all happening at once. In my case, I had a new router/firewall that I was still learning to configure. I had set port-forwarding to port 5060 up from the firewall to the PBX , but had missed defining the interface on the firewall itself. This caused the five IPs we had at the time to port-forward to the PBX. I also hadn't set up any source/destination IPs either as I didn't yet have complete information on who/what would be connecting, but need the system to be up and running. In those days, I was also completely unaware (like many folks new to VoIP) of the level of SIP scanning that was going on. It's much worse today than it was in 2008.

Live and learn.

Anyway, how do we define "high call volume"? What is a normal high call volume versus an abnormal high call volume due to crack attempts from multilple IPs? I'm sure later versions of Asterisk handle this much better now than these earlier versions did. Since his PBX appears to be exposed to the world, what's the incoming call rate? Are we talking a few calls per second or tens of thousands per second? It's an unknown at this point since "flood of incoming calls" is not really defined here. Even Asterisk is going to have a threshold that's going to get exceeded if it's too much for one server/instance to handle.

My point is that if the call level are abnormally high, is it not a possibility that Asterisk is getting overwhelmed in this case causing a restart?

 

[wardmundy]

Take a look at this thread for a possible cause.

 

[JoeTalbot]

nikhilroshania review your /var/log/messages and see whether what you are experiencing is Asterisk crashes. In addition to the segfault shown in the messages file, you will see the restart in /var/log/asterisk/full.

You may be experiencing the bug with sqlite modules. Seems to be triggered with heavier call volume (or in your case, a lot of CDRs being generated from SIP scanners). See http://pbxinaflash.com/community/index.php?threads/sqlite2-in-piaf.14864/#post-95573

This is the bug that I've been seeing for the past couple of months. For our radio station (high volume) customers, we've rolled back to 1.7.5.5.4 (asterisk 1.4/zaptel) and the problems went away. Today I was told by a colleague (also working on this) that he moved to the Schmooze ISO and is no longer experiencing problems. It seems to be related to high call volume, possibly DAHDI, and leave behind a core dump that points you to the SQLite issue. Disabling SQLite will make the problem occur less frequently, but it will not start over again by itself when it does crash.

I am playing with Asterisk 11/Green now, and am building a traffic generator to bash the systems. You are not alone!

Joe

 

[JoeTalbot]

billsimon

Yes, you make a good point and I can see how that could be misinterpreted. I made that statement based upon my own personal experience. Some background:

I have seen older versions (1.4/1.6) of Asterisk go off the deep end when flooded with a couple of thousand calls all happening at once. In my case, I had a new router/firewall that I was still learning to configure. I had set port-forwarding to port 5060 up from the firewall to the PBX , but had missed defining the interface on the firewall itself. This caused the five IPs we had at the time to port-forward to the PBX. I also hadn't set up any source/destination IPs either as I didn't yet have complete information on who/what would be connecting, but need the system to be up and running. In those days, I was also completely unaware (like many folks new to VoIP) of the level of SIP scanning that was going on. It's much worse today than it was in 2008.

Live and learn.

Anyway, how do we define "high call volume"? What is a normal high call volume versus an abnormal high call volume due to crack attempts from multilple IPs? I'm sure later versions of Asterisk handle this much better now than these earlier versions did. Since his PBX appears to be exposed to the world, what's the incoming call rate? Are we talking a few calls per second or tens of thousands per second? It's an unknown at this point since "flood of incoming calls" is not really defined here. Even Asterisk is going to have a threshold that's going to get exceeded if it's too much for one server/instance to handle.

My point is that if the call level are abnormally high, is it not a possibility that Asterisk is getting overwhelmed in this case causing a restart?

The call volumes are not that high. We've seen systems restart with 6 calls. It seems to be related to how many calls happen at a time, or how many CDR records are created at a time... Not sure yet, but the number is lower that we'd expect. Also, with the older Zaptel/1.4 combinations, we'd run insane call volumes without difficulty, 5,000+ an hour daily at a system in the Dominican Republic.

 

[Capstone Works]

See here http://pbxinaflash.com/community/index.php?threads/sqlite2-in-piaf.14864/#post-95573
and here http://www.pbxinaflash.com/communit...-causing-asterisk-to-crash.14134/#post-101111

This seems to have fixed it for us.