TIPS PBX Server Freaked Out Today. Why?

[Alex Hackney]

I'm running a vultr instance of IncrediblePBX13.2.iso.
Asterisk 13.13.1
Incredible PBX 12.0.39
SL 6.7

I'm actually running a couple. One in particular today started having issues. Immediately after the office lost power the phones would not reconnect. The server has quit accepting any endpoint connections from whitelisted addresses or ips. The office has a static and was working fine before this happened.

I logged in and rebooted the server and now the PJSIP settings are gone. I tried converting all extensions to sip and that didnt fix it either.

I also cant use the sip show endpoints commands. I'm going to build another instance from a backup. Luckily this happened 30 mins before the close of business.

System is still accepting calls and doing voicemails just fine. Really weird.

Thoughts?

 

[krzykat]

I'm not a fan of pjsip, why change what works? I always make my extensions SIP and I go into the settings and tell it to only use SIP and to have SIP listen on 5060, apply, and reboot if necessary.
To track it down to pjsip versus whitelist, etc. HTTP or SSH to the server (if service enabled) from the endpoint IP, that will tell you that the server likes talking to you or not.

The fact that the system is still accepting calls and doing VM fine tells you that the server and asterisk is up ... I'd really change over to SIP.

 

[Alex Hackney]

I did try converting those extensions to sip and it still wasnt accepting the connections. I tried disabling ip2ban and it didn't solve the connection issue. What else could be blocking it?

 

[wardmundy]

Sounds like the MySQL databases may have been damaged. You can rebuild them. Instructions are around here somewhere. We're on the road so my searching prowess is no great shakes.

 

[krzykat]

Sounds like the MySQL databases may have been damaged. You can rebuild them. Instructions are around here somewhere. We're on the road so my searching prowess is no great shakes.

Perhaps this: https://gngtech.wordpress.com/2013/11/15/repairing-asterisk-mysql/

The other thing I'd try if you've gotten things towards SIP is now create a new SIP extension and try to register it.

 

[Alex Hackney]

So it turned out to be a couple of things.

1) The server crashed and the pjsip modules got disabled. I went in and reenabled them and that fixed pjsip.
2) One of the endpoints had a bad password and was getting blocked. I determined this when I was able to get another phone and my office to connect to the server and stay connected.
3) The ./add-ip for the site was blank. I know I added it correctly but it wasn't there.
4) The ./add-fqdn for the site was there but wasn't stopping the site from getting banned for too many incorrect attempts.

Adding the sites ip back in and restarting fail2ban fixed it after reenabling pjsip. That was a nightmare at 12 am to figure out and apparently sort of a perfect storm of issues. But I got it all back up and going without having to spin up another instance. I do nightly backups on my other servers and now this one. Probably over kill but better safe than sorry.

Thanks for the help though guys!

Alex

 

[wardmundy]

Now that nightly backups cost an additional 50 cents a month on Vultr's $2.50/mo. platforms, hopefully these kinds of nightmares will be a thing of the past.

 

[Alex Hackney]

Absolutely. I use the $5 plans for the 1gb of ram. But still really worth it to have it backed up for me on the fly.

Alex

 

[krzykat]

How often do you do auto-backups? I've done them weekly and never had an issue.

 

[Alex Hackney]

I was doing the asterisk backup nightly. With Vultr you basically just enable a nightly or weekly backup of the server. It takes a snapshot of it and stores it. For the plan I'm on its $1 a month to do this. Way worth it.

 

[wardmundy]

Just to be safe, you probably need to backup all of your databases separately in /root. Then, if it doesn't restore safely, you'll have the missing pieces to restore separately.

mysqldump -u root -ppassw0rd --all-databases > /root/mysql_full_backup.sql