ALERT Outgoing Issues (Local and External)

[krusta80]

I setup a PIAF Purple system for my office about a month ago (approximately 25 Cisco 7961's) and had originally instructed my employees to actively using it for outgoing calls only. I wanted to make sure that they had gotten the hang of the new system -- as well as ensuring that the call volume could be handled properly -- before permanently rerouting all incoming calls from our old PBX system.

As of Wednesday evening, however, all incoming calls have been rerouted to the new system, and for the first 24 to 36 hours, everything was working flawlessly! This morning, however, I noticed that several calls that I was placing from my own extension would ring and ring, but no one would answer. This would occur for both internal and external calls, which I found quite odd.

To make matters worse, I now have everyone on the system complaining of the same issue. I am rebooting the server to see if that resolves the issue for now, but clearly something has gone awry. The only change that I made this morning, which corresponds to the beginning of the problems, is that I added a new SIP trunk. The outgoing route associated with that, trunk, however, is only to be activated when dialing "9" as the prefix, so I don't think it's related. The timing does seem suspicious however.

I plan to start looking through the log files, but I was wondering whether anyone on here could help guide me. I really don't want to lose my boss's confidence in the system.

 

[Porch]

This is very strange. Put up the log of a failed call so we can take a look at it.

 

[krusta80]

Looks like some sort of congestion issue. FYI, I have 8 Google Voice trunks setup for all outgoing domestic calls. I assume that asterisk automatically slides down to the next trunk in line in case the current one is busy? Also, this wouldn't explain the problem with extension-to-extension calls that I experienced.

I will try to track down an example of an internal call gone bad. The log file is quite huge needless to say.

[EDIT]: Reposted the log file...was missing the last few lines

 

[rossiv]

You're using GV for a business's outbound calling? Honestly, you're out of your mind. Not trying to be rude, but your integration will end in May and unless you have a SIP provider or PSTN backup, you'll be SOL.
Asterisk *should* go to the next trunk.

 

[krusta80]

Ross,

I'm aware of everything you've stated...with the exception of my lunacy. As I mentioned in my OP, the system has been working flawlessly for all outgoing calls for over a month, and I saw no reason not to utilize GV as a primary (free domestic and dirt-cheap international is hard to ignore). That said, there are six POTS lines also available for outgoing calls (and currently last in line for our outgoing routes), and I am in the process of adding other SIP providers (Obivoice for starters) for when May arrives. Oh boy I better hurry!!

With your wonderful, technical opinion of "*should* go the next trunk", it's no wonder that you think you've earned the right to be rude to newcomers.

Weak indeed,
John

 

[rjaiswal]

Ross,

I'm aware of everything you've stated...with the exception of my lunacy. As I mentioned in my OP, the system has been working flawlessly for all outgoing calls for over a month, and I saw no reason not to utilize GV as a primary (free domestic and dirt-cheap international is hard to ignore). That said, there are six POTS lines also available for outgoing calls (and currently last in line for our outgoing routes), and I am in the process of adding other SIP providers (Obivoice for starters) for when May arrives. Oh boy I better hurry!!

With your wonderful, technical opinion of "*should* go the next trunk", it's no wonder that you think you've earned the right to be rude to newcomers.

Weak indeed,
John

I have to agree with Ross. Running a business on GV is lunacy.

Now on to your issue. Have you setup your GV trunks with only 1 channel?

GV might not be giving asterisks an all circuits busy signal, so it doesn't know to fail over to the next trunk.

 

[briankelly63]

New member John...... slow down a little. Ross I'm sure will have his own reply but he's one of the most respected and helpful members here. I was fine with your remarks until you got to the "should" comments.

The answer truly is "should" but it depends on the kind of failure that is encountered by the trunk. Its one thing for it to be truly "out" its quite another if some of the trunk communications but not all are happening the way that they should. GV adds other layers of issue.That's why it takes a lot of log and even firewall work.
Unexpected "ringing" can be in the VOIP set, Asterisk or even a provider. It could even be network related as Asterisk and VOIP tel sets don't handle network issues too well.
Multiple outgoing GV trunks should not be used as a primary business solution as Ross noted. There are many major, dirt cheap providers that are a much better idea. Take Anveo Direct for example that provides SIP traces on every call! Great for figuring out if its them or you.
Part of making all this work is a lot of research and taking member comments and researching those. If you research the history of Asterisk failover trunks you'll see why its not so simple.
Provide more information and I'm sure you'll get some assistance.

 

[rossiv]

Nice catch, rjaiswal .
What you are looking for in reference to rjaiswal's post is in FreePBX > Connectivity > Trunks > $YourGVTrunkNameHere

If you set Maximum Channels to 1 and have the GV trunks in sequence in the Outbound route, it should failover, as pictured above.

By should, I mean (and meant in my previous post) that it's a documented issue that in some cases depending on the provider's response, FreePBX/Asterisk didn't follow through to the next trunk. Hence the addition of the Continue if Busy checkbox in the trunk configuration. I think that was added in 2.11 if I recall correctly.

Please pardon my unintentional tone in my previous post. I've just never had GV be all too reliable myself. I'm all for trying to help you solve your issue.

Thanks for your kind words, briankelly63. Your reply is spot-on from what I've noticed about the failover.

 

[krusta80]

Ross,

I'm sorry for snapping as I did. I will be the first to admit that I have much to learn with regard to PIAF and all that it has to offer, but I am very mindful not to ever put all of my eggs in one basket. In the meantime, I'm still not sure how/why extension-to-extension calls would also be impacted if the issue were isolated to the GV trunks. Give me some time to look over what you all have said, and I promise to better behave myself.

 

[rossiv]

I agree that extension to extension calls being affected is quite strange. That says network issue to me, which brings me to some questions.
1. Is the PBX on-site on the same LAN as the phones, or is it hosted elsewhere?
2. What SIP firmware are you using on the 7961s? I've noticed issues with v9 on my 7961 and 7970.

 

[rossiv]

Looking at your log file, here's the crucial part. (Censored the dialed # with 15555555551.

[2014-02-28 10:50:13] VERBOSE[11452] app_dial.c:     -- Called local/15555555551@googlevoice-jag47hafcom
[2014-02-28 10:50:13] VERBOSE[11453] pbx.c:     -- Executing [15555555551@googlevoice-jag47hafcom:1] Dial("Local/15555555551@googlevoice-jag47hafcom-00000e46;2", "Gtalk/jag47hafcom/[email protected],tr") in new stack
[2014-02-28 10:50:13] VERBOSE[11453] app_dial.c:     -- Called Gtalk/jag47hafcom/[email protected]
[2014-02-28 10:50:13] WARNING[11453] app_dial.c: Invalid timeout specified: 'tr'. Setting timeout to infinite
[2014-02-28 10:50:13] DEBUG[7850] chan_gtalk.c: redirect [email protected]/srvenc-MjgC4QDu4CNXMp3t5v6Jrp4BGTA/Bma3
[2014-02-28 10:50:13] VERBOSE[11453] app_dial.c:     -- Gtalk/[email protected] is ringing
[2014-02-28 10:50:13] VERBOSE[11452] app_dial.c:     -- Local/15555555551@googlevoice-jag47hafcom-00000e46;1 is ringing
[2014-02-28 10:50:14] VERBOSE[11453] app_dial.c:   == Everyone is busy/congested at this time (1:0/0/1)
[2014-02-28 10:50:14] VERBOSE[11453] pbx.c:     -- Executing [15555555551@googlevoice-jag47hafcom:2] NoOp("Local/15555555551@googlevoice-jag47hafcom-00000e46;2", "GoogleVoice Call to 15555555551 failed") in new stack
[2014-02-28 10:50:14] VERBOSE[11453] pbx.c:     -- Auto fallthrough, channel 'Local/15555555551@googlevoice-jag47hafcom-00000e46;2' status is 'CHANUNAVAIL'
[2014-02-28 10:50:14] VERBOSE[11452] app_dial.c:     -- Local/15555555551@googlevoice-jag47hafcom-00000e46;1 is circuit-busy
[2014-02-28 10:50:14] VERBOSE[11452] app_dial.c:   == Everyone is busy/congested at this time (1:0/1/0)
[2014-02-28 10:50:14] VERBOSE[11452] pbx.c:     -- Executing [s@macro-dialout-trunk:31] NoOp("SIP/201-00000f92", "Dial failed for some reason with DIALSTATUS = CONGESTION and HANGUPCAUSE = 0") in new stack

Not sure how much you know about reading through these logs, but here's what I see. I see (above the snipped) Asterisk's usual call processing when an outbound call is started. Nothing odd there, so I didn't re-post it. Here we see Asterisk tried to dial out on the GV trunk jag47hafcom. Something's up with the timeout, but it appears to get a ringing signal from GV and then GV rejects the call. Asterisk did, indeed, failover to the next trunk, as seen in the next snipped.

[2014-02-28 10:50:14] VERBOSE[11452] app_dial.c:     -- Called local/15555555551@googlevoice-jfkalexhafcom
[2014-02-28 10:50:14] VERBOSE[11454] pbx.c:     -- Executing [15555555551@googlevoice-jfkalexhafcom:1] Dial("Local/15555555551@googlevoice-jfkalexhafcom-00000e47;2", "Gtalk/jfkalexhafcom/[email protected],tr") in new stack
[2014-02-28 10:50:14] VERBOSE[11454] app_dial.c:     -- Called Gtalk/jfkalexhafcom/[email protected]
[2014-02-28 10:50:14] WARNING[11454] app_dial.c: Invalid timeout specified: 'tr'. Setting timeout to infinite
[2014-02-28 10:50:14] NOTICE[7849] chan_gtalk.c: Remote peer reported an error, trying to establish the call anyway
[2014-02-28 10:50:14] NOTICE[7849] chan_gtalk.c: Ignoring duplicate call setup on SID 2d5519e41a2f4084
[2014-02-28 10:50:14] DEBUG[7850] chan_gtalk.c: The client is guest
[2014-02-28 10:50:14] VERBOSE[11451] app_dial.c:     -- Gtalk/[email protected] answered Local/18004321000@googlevoice-jag47hafcom-00000e45;2
[2014-02-28 10:50:14] VERBOSE[11449] app_dial.c:     -- Local/18004321000@googlevoice-jag47hafcom-00000e45;1 answered SIP/127-00000f91
[2014-02-28 10:50:14] VERBOSE[11451] pbx.c:     -- Executing [h@googlevoice-jag47hafcom:1] Hangup("Local/18004321000@googlevoice-jag47hafcom-00000e45;2", "") in new stack

Here I see in the same second (10:50:14) where it tried calling out on the jfkalexhafcom trunk, and Google sent back some kind of error. In that same second, it looks like someone hungup a call to Bank of America that was initiated earlier or something. Not sure there, the rest of it isn't in the log. I don't see in this snippet where Google rejected the call or where the call was disconnected.

 

[briankelly63]

Good exchange by all!

 

[krusta80]

OK, back to the problem at hand...

1. RJ, I did NOT have each GV trunk limited to one channel, but I have just made that change. Thank you for the that. Just out of curiosity, why would this problem not have presented itself sooner and why in such a sporadic manner? I always find inconsistent problems the most unnerving, as they are the hardest to reproduce and therefore troubleshoot.
2. Ross, I suppose my version of FreePBX is older; I do not see the "continue to next" checkbox. Thank you so much for your help...it's normally not like me to be a pain.
3. Brian, thanks for bringing me back to the planet. I'm honestly just a little stressed out with this mess, which by the way has temporarily been resolved by rebooting the server. I'm all for doing whatever needs to be done to get to the bottom of this, and I"m grateful for whomever is willing to help along the way. Please let me know what I can provide to dig a little deeper. One of my concerns is that the issue may be network traffic related, but the office's infrastructure was recently upgraded so who knows.

 

[rossiv]

I also saw this snippet, possibly related to your phone-to-phone issues.

[2014-02-28 10:50:39] WARNING[7857] chan_sip.c: Retransmission timeout reached on transmission [email protected] for seqno 102 (Critical Response) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response

If your phones are on the 192.168.17.x subnet, it looks like Asterisk is having trouble reaching them. I know that when I have used Cisco phones in the past, they were picky about NAT settings, and something else in FreePBX, but I don't remember what. Re-invite, maybe? Regardless, I'm thinking it may be helpful to see a screenshot of a config page in FreePBX of one of your extension's settings, with the secret sanitized.

 

[rossiv]

Well Drat, haha, the darned "I was typing something while you were posting and you beat me to it."
If you don't have the checkbox, then you are probably anywhere from 2.8 to 2.10. The best way for me to know is if you would SSH into your box and post the output of 'status' here. It should give you a blue screen with status of services and your PIAF version, FreePBX version, and Asterisk version. If your public IP shows, feel free to remove it.

 

[krusta80]

Would it help if I provided a link to the full log? I suppose I didn't grab every related line of text. For what it's worth, I first noticed the "endless ringing" symptom when calling 2123367320, followed by calling 5165574069, so I would suggest searching for those strings and scrolling down. Again, the problem was sporadic.

Hopefully this log doesn't have any passwords in it. FYI, it is about 20 MB in size.

http://pandora.dyndns.biz/full.log

 

[krusta80]

 

[krusta80]

I also saw this snippet, possibly related to your phone-to-phone issues.

[2014-02-28 10:50:39] WARNING[7857] chan_sip.c: Retransmission timeout reached on transmission [email protected] for seqno 102 (Critical Response) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 32001ms with no response

If your phones are on the 192.168.17.x subnet, it looks like Asterisk is having trouble reaching them. I know that when I have used Cisco phones in the past, they were picky about NAT settings, and something else in FreePBX, but I don't remember what. Re-invite, maybe? Regardless, I'm thinking it may be helpful to see a screenshot of a config page in FreePBX of one of your extension's settings, with the secret sanitized.

Yes, I didn't mention that the phone I was testing from is connected via Sonicwall VPN to the office from my apartment. The reason I didn't mention it yet is because the other users are all onsite and were complaining of the same symptoms. Regardless, a sample extension setup is on the way...

 

[krusta80]

I agree that extension to extension calls being affected is quite strange. That says network issue to me, which brings me to some questions.
1. Is the PBX on-site on the same LAN as the phones, or is it hosted elsewhere?
2. What SIP firmware are you using on the 7961s? I've noticed issues with v9 on my 7961 and 7970.

1. Yes it hosted on the same LAN
2. I'm using firmware 8-5-4TH1.6 (app load id)

 

[rjaiswal]

"1. RJ, I did NOT have each GV trunk limited to one channel, but I have just made that change. Thank you for the that. Just out of curiosity, why would this problem not have presented itself sooner and why in such a sporadic manner? I always find inconsistent problems the most unnerving, as they are the hardest to reproduce and therefore troubleshoot."


GV is a consumer product. It can have, at most, only 2 active channels. And using 2 channels can only happen in 2 scenarios. Call waiting, and 3 way calling.


Since you are only using it for outbound, you can only have one channel. Specifying limits in asterisks will make sure that asterisks won't try to use more channels than the trunk will allow, and might cause wierd dial plan issues.