SOLVED synagogue pbx I rebooted and came up with errors

[wa4zlw]

dont know whats going on but things just stopped this evening. when I rebooted it remotely I got this on the status gui page:

Critical Errors found
Please check for errors in the notification section


retrieve_conf failed, config not applied
Reload failed because retrieve_conf encountered an error: 1

root@pbx:~ $ status

  Incredible PBX 13.7.2 for Scientific Linux

  Asterisk: UP  Apache: UP  MySQL: UP
  SendMail: UP  IPtables: UP  SSH: UP
  LAN port: UP  Fail2Ban: UP  Webmin: UP
  GV OAUTH: UP  PortKnock: DN  NR VPN: UP
  FaxGetty: UP  IAX Modem: UP  HylaFax: UP

  RAM:3443MB Scientific Linux 6.7  Disk:47GB

  Asterisk 13.7.2  Incredible GUI 12.0.30

  Private IP: 10.196.4.10 10.196.100.10

  Public Info: 173.x.y.z

  System Time: Wed May 11 20:42:57 EDT 2016

  <  OK  >




WARNING: Always run Incredible PBX behind a secure hardware-based firewall.
root@pbx:~ $

thanks leon

 

[wa4zlw]

neither voip.ms or vitelity are registered. Vitelity is ok as they didn't pay their bill voip.ms still has funds and here's what I get:

Host                                    dnsmgr Username       Refresh State                Reg.Time                 
newyorkx.voip.ms:5060                   N      1#####_Shul        120 Unregistered                                 
inboundx.vitelity.net:5060              N      caaaaaa_shul          120 Request Sent                                 
2 SIP registrations.

                               
Name/username             Host                                    Dyn Forcerport Comedia    ACL Port     Status      Description                     
vitel-inbound/caaaaaal   64.a.b.c                                Yes        Yes            5060     UNREACHABLE                                 
vitel-outbound/caaaaaa  64.x.y.z                              Yes        Yes            5060     UNREACHABLE                                 
voipms/1#####_Shul        107.x.y.z                               Yes        Yes            5060     UNREACHABLE                                 
3 sip peers [Monitored: 0 online, 3 offline Unmonitored: 0 online, 0 offline]

ignore the vitelity stuff...leon

 

[wa4zlw]

PS: checked and both SIP and PJSIP are still set; sip on extensions and pjsip on trunks

 

[wa4zlw]

Bill yes I can get out. I've rebooted the Watchguard and the pbx. I can get in and out but can not mange the pbx from my management network so I have watchguard looking at that. I can get to it via a browser.

I'm also concerned about that error on the GUI status page. I've seen it before but can't remember how we fixed it. :-(

thanks leon

 

[wa4zlw]

well all of a sudden both voip.ms and vitelity are passing calls weird.
going to sleep up early

thanks leon

 

[wa4zlw]

a week goes by and we went down again.

this time I'm seeing the following weird errors:

[2016-05-15 19:09:09] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:09] NOTICE[3436] chan_iax2.c: Peer 'iax-fax2' is now REACHABLE! Time: 1
[2016-05-15 19:09:24] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:24] NOTICE[3422] chan_sip.c: Peer 'voipms' is now UNREACHABLE! Last qualify: 0
[2016-05-15 19:09:24] NOTICE[3422] chan_sip.c: Peer 'vitel-inbound' is now UNREACHABLE! Last qualify: 0
[2016-05-15 19:09:24] NOTICE[3422] chan_sip.c: Peer 'vitel-outbound' is now UNREACHABLE! Last qualify: 0
[2016-05-15 19:09:34] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:34] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:34] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:59] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:09:59] NOTICE[3422] chan_sip.c: -- Registration for '[email protected]' timed out, trying again (Attempt #2)
[2016-05-15 19:10:14] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:14] NOTICE[3422] chan_sip.c: -- Registration for '[email protected]' timed out, trying again (Attempt #2)
[2016-05-15 19:10:14] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:14] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:14] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:28] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:28] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2016-05-15 19:10:28] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data

I've rebooted the pbx and restarted asterisk as well as rebooted the Watchguard firewall. I did an upgrade to current this afternoon just as a precaution and still zip. my pjsip extensions are connected. just the SIP trunks down, again.

I tried to get them to move to the cloud but it's like pulling teeth. My personal DO has been up for months.

The synagogue is on a COMCAST BUsiness circuit. I'm wondering if they're playing around. I know someone in Connecticut and they have a known CISCO firmware bug. DOn't know if COmcrap has resolved it for him. I wouldn't put it past them. I don't know which modem we have but it is in bridge mode. I'm wondering if they're playing around with blocking ports?

THanks leon

 

[wa4zlw]

dunno about comcast it seems remote and local extensions are registering as well as my extension coming through my VPN tunnel. So this is all about trunks.

ldz

 

[atsak]

This is all about your watchguard firewall; almost certainly there. Comcast business doesn't block ports and I have no problem with them in several different cities (other than reliability at night sometimes).

 

[wa4zlw]

I beg to differ. This has happened twice in two weeks except this time we get this error:

[2016-05-15 19:10:14] ERROR[3422] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data

5060 PJSIP is working in and out of the firewall to endpoints. 5060 to the carriers is leaving the pbx.

Regarding comcast, please see this:

-------- Forwarded Message --------
Subject: [Uronode] Link status
Date: Sun, 08 May 2016 11:50:34 -0400
From: Brian <[email protected]>
Reply-To: [email protected]
Organization: Amateur Radio Services
To: [email protected]


It appears after doing numerous tests with both the manufacturer (Cisco)
and my VoIP provider (PhonePower) that there is definitely a critical
bug in these devices with regard to the firmware. In doing deep sniffs,
PhonePower confirmed that the CPE was blocking their outbound frames
from reaching me as the MAC address was different from that of my PBX,
and equaled that of the router device. Even taking down my amprnet DMZ
host and setting the asterisk server to it made no difference.

A very simple test to verify this firmware bug (which Cisco themselves
admitted to) is a ping reply from my public IP. The firewall on the DMZ
host is set to drop any incoming ICMP sourced from any non-44 IP. This
is proof that the firmware has at least one bug.

For those on Comcast be on the look out for:
Model: Cisco DPC3941T
Firmware: dpc3941-P20-18-v303r2042173-151003a-CMCST version 7 (current
as of this date)
These will deny you services and violate Open Internet/Net Neutrality
laws. While Cisco wished they could take action, these units are custom
manufactured for Comcast and have an exclusive contract. A patch
work-order must come to Cisco from Comcast.

For now it looks as if I've been able to engineer a couple hacks to
those I have link with only, SIP, and email. HTTPS, ssh, openvpn, and
other encrypted functions are still very impacted. Comcast engineering
is suppose to contact me by today (I'm not holding my breath). I hate
having to use a bandaid but its semi-working for now for those links I
need to maintain.

--
<rhetorical> Why is it linux users can install and operate *any* version of M$
Windoze but the same can't be said in reverse?</rhetorical>

73 de Brian - N1URO

 

[wa4zlw]

weird had to set 5061 as bind port for sip. didnt have to before. looks like we're registered again

thanks leon

 

[wa4zlw]

turns out making it 5061 caused one-way audio. now have it set to 5062. That was on inbound calls. did not try outbound calls prior to me fixing it.

 

[Asher]

Anything else changed? I went through couple of weeks of extensions randomly going unreachable. Turned out it was a disk IOPS issue. Moved the VM to a faster datastore and all settled.

 

[wa4zlw]

no but comcast could be part of the problem or not. this pbx is on bare metal. My pbx is on a Digital Ocean VM and just purrs.

Leon