ALERT GV: The Sky Has Fallen... Really

restamp

Member
Joined
Apr 24, 2016
Messages
97
Reaction score
53
Here is the likely reason for the "network unreachable" errors with stun from Josh Culp on the DSLR Forum. If someone with IPv6 connectivity from their server could test whether you're still seeing the unreachable errors, that would solve the last remaining issue.
As luck would have it, I do have IPv6 on my test server. I do not have any "network unreachable" errors in my logs and a quick "lsof" of asterisk shows two open IPv6 ports:
Code:
lsof -c asterisk | grep -i ipv6
asterisk 30586 asterisk   12u  IPv6         3188795947      0t0        TCP *:8089 (LISTEN)
asterisk 30586 asterisk   17u  IPv6         3188796008      0t0        UDP *:51967
Hope this helps
 
Last edited:

SMTC

Member
Joined
Jan 22, 2009
Messages
190
Reaction score
13
@SMTC: A lot has changed since that release. I'd start over just to be sure I had a reliable platform. Because of all the changes, I think cut-and-paste is your safest bet for moving things over.

Not to sound goofy, but in your experience how and what do you use to do "cut and paste". I'm envisioning having the management GUI for two live systems, one old and one new, open at same time and cutting and pasting. But that presumes you remember that I bought a spare Pi 3 - which is unlikely I think :). So then the question I have would be if I were to use WinSCP to move files from the existing config to my laptop what directories do I find my trunk, route, extensions configs? And can they be just "FTP'd" over? What about the CDR DB? What about the saved VM's? What about my IVR recordings? There are a lot of directories to rummage through on the installation... Thanks!
 

Eliad

Active Member
Joined
Aug 13, 2017
Messages
619
Reaction score
127
If I understand this correctly the STUN server issue applies only on the systems who has the GV-NAF applied, am I right?
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,206
Reaction score
5,229
Not to sound goofy, but in your experience how and what do you use to do "cut and paste". I'm envisioning having the management GUI for two live systems, one old and one new, open at same time and cutting and pasting. But that presumes you remember that I bought a spare Pi 3 - which is unlikely I think :). So then the question I have would be if I were to use WinSCP to move files from the existing config to my laptop what directories do I find my trunk, route, extensions configs? And can they be just "FTP'd" over? What about the CDR DB? What about the saved VM's? What about my IVR recordings? There are a lot of directories to rummage through on the installation... Thanks!

You can't use FTP to move trunks, routes, and extensions. They're buried deep in MySQL with cross-linked tables.
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
You can't use FTP to move trunks, routes, and extensions. They're buried deep in MySQL with cross-linked tables.
True. Good news @SMTC , the new release of FreePBX 15 (the app IPBX has been based on - after removing nonfree proprietary code etc), is coming with the long anticipated "portable backups" feature. This lets you save massive time migrating to new PBX version. Tap or click, to get all your data out of IPBX 15, and restore to any other IPBX 15, 16, 17, or newer version. You're no longer limited to restoring all yer PBX data to the exact same PBX version. All the db tables stuff, conf file stuff, voicemail recordings locations, IVR recordings locations, CDR, conf files, are no longer a headache of copy pasting or writing programming code to get that data out of PBX version 12 database into PBX version 13 database.

By the way @wardmundy what's your method for making these IPBX releases, do you have an automated script tool that you run to find and auto remove the non-free stuff from freepbx github releases, the core, modules, everything...? I hope so, it's too much work to do that manually...?!
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
Here is the likely reason for the "network unreachable" errors with stun from Josh Culp on the DSLR Forum. If someone with IPv6 connectivity from their server could test whether you're still seeing the unreachable errors, that would solve the last remaining issue.
Which would explain why I've never seen the error even with the testing done yesterday - one of the first things we do on every system is disable ipv6 unless needed. No ipv6 enabled would mean no ipv6 attempts and no error message.

It would also imply what I believed anyway which was the error message not meaningful.

I had always thought the res_monitor_stun.conf entry was irrelevant.
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
@wardmundy @jerrm @Eliad @restamp
Today I read the source code for Asterisk and PJSIP looking for how it uses ICE STUN, RTP, SRTP and DTLS, for max quality reliability and security/privacy. [1] [2] [3] [4] [5] [6] [7]

Looking for answers to this Big Question:
How do you get THE MOST reliable safe secure private call connections in all network conditions, and the least amount of user complaints and time and money wasted at the support help desk ??

Answer:
1. Enable Asterisk's RTP's (res_rtp) ICE STUN to get the audio/video media thru reliably. It uses the bundled PJSIP's ICE STUN libraries to find the best pathway for media under any network conditions.
2. Enable PJSIP ICE STUN, to let the SIP signaling find the most reliable network path to the endpoints, trunks etc
3. Enable IP6, and use it, with IP4. Mobile devices connect with IP6, some ONLY have an IP6 connection. The planet is upgrading to IP6. Asterisk PBX servers must accept connections on IP6 for maximum compatibility, speed, and lowest total cost of ownership. ICE STUN helps make whatever networking your PBX has, work smoothly and optimally. https://blog.apnic.net/2018/02/15/bad-ipv4-address-exhaustion/
4. For security and privacy of calls, ICE STUN and SRTP DTLS and Direct Media (calls must not be terminated aka decrypted by any PBX server in the middle) are mandatory. ICE STUN is absolutely essential to make sure Direct Media finds a pathway from endpoint directly to endpoint under all network conditions / network topology layout.

[1] http://www.pjsip.org/pjnath/docs/html/ice_demo_sample.htm
[2] https://trac.pjsip.org/repos/wiki/IceNegotiationFailure
[3] https://github.com/pjsip/pjproject
[4] https://git.pjsip.org/gitpub?p=pjproject.git;a=summary
[5] https://github.com/asterisk/asteris...61da398373b8d234/res/res_rtp_asterisk.c#L3395
[6] https://trac.pjsip.org/repos/wiki/Getting-Started/Autoconf
[7] https://stackoverflow.com/questions/47950966/how-to-use-ice-and-directmedia-together-in-asterisk
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
@wardmundy @jerrm @Eliad @restamp
Today I read the source code for Asterisk and PJSIP looking for how it uses ICE STUN, RTP, SRTP and DTLS, for max quality reliability and security/privacy. [1] [2] [3] [4] [5] [6] [7]

Looking for answers to this Big Question:
How do you get THE MOST reliable safe secure private call connections in all network conditions, and the least amount of user complaints and time and money wasted at the support help desk ??

Answer:virtually everywhere.
1. Enable Asterisk's RTP's (res_rtp) ICE STUN to get the audio/video media thru reliably. It uses the bundled PJSIP's ICE STUN libraries to find the best pathway for media under any network conditions.
2. Enable PJSIP ICE STUN, to let the SIP signaling find the most reliable network path to the endpoints, trunks etc
3. Enable IP6, and use it, with IP4. Mobile devices connect with IP6, some ONLY have an IP6 connection. The planet is upgrading to IP6. Asterisk PBX servers must accept connections on IP6 for maximum compatibility, speed, and lowest total cost of ownership. ICE STUN helps make whatever networking your PBX has, work smoothly and optimally. https://blog.apnic.net/2018/02/15/bad-ipv4-address-exhaustion/
4. For security and privacy of calls, ICE STUN and SRTP DTLS and Direct Media (calls must not be terminated aka decrypted by any PBX server in the middle) are mandatory. ICE STUN is absolutely essential to make sure Direct Media finds a pathway from endpoint directly to endpoint under all network conditions / network topology layout.

[1] http://www.pjsip.org/pjnath/docs/html/ice_demo_sample.htm
[2] https://trac.pjsip.org/repos/wiki/IceNegotiationFailure
[3] https://github.com/pjsip/pjproject
[4] https://git.pjsip.org/gitpub?p=pjproject.git;a=summary
[5] https://github.com/asterisk/asteris...61da398373b8d234/res/res_rtp_asterisk.c#L3395
[6] https://trac.pjsip.org/repos/wiki/Getting-Started/Autoconf
[7] https://stackoverflow.com/questions/47950966/how-to-use-ice-and-directmedia-together-in-asterisk
Trust me, time at the support desk is our #1 priority and we do almost none of the above, and often the polar opposite. No ICE, no STUN, (almost) no pjsip, no IPv6, no direct media to endpoints (calls are usually managed, recorded, etc.).

If we had issues we would change in a heartbeat, we don't.
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
Trust me, time at the support desk is our #1 priority and we do almost none of the above, and often the polar opposite. No ICE, no STUN, (almost) no pjsip, no IPv6, no direct media to endpoints (calls are usually managed, recorded, etc.).

If we had issues we would change in a heartbeat, we don't.
If it works for you, great. And granted, ultra super secure encrypted direct media calls are probably less than 10% of all calls around here, they're absolutely required when necessary, yet the 90% majority of calls are your normal managed recorded calls.

Now, what about this scenario, which only works when all advanced new tech is enabled (IP6, ICE STUN TURN).. You've got many users, and some need VOIP Dialer apps on their Android and Apple smart phones, for making and receiving their managed recorded business calls while away from their desk phones. They're on the move sometimes, they walk from one WiFi hotspot, to another, which causes their IP to change. They still need the call to stay up, and it will, with IP6, ICE STUN and TURN. They wander outside the WiFi coverage, the WiFi IP connection drops, yet they're on 4G LTE IP address, maybe it's IP6, maybe it's IP4, depending on the whims of the LTE data network. This user obviously expects and needs their call to stay up, and the VOIP app will keep the call up, as long as your Asterisk PBX is configured with IP6 and ICE STUN and TURN all enabled. PJSIP detects the user's VOIP app coming back in over the 4G LTE data connection and does what's called "IP HANDOVER", it hands off the call to the user's smart phone VOIP app thru its new 4G LTE IP6 address so the user can keep on listening and talking and doing their business conversation.
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
I move from WiFi <-> Mobile almost every day on plain old no ice/stun/turn chan_sip. Biggest issue is when the phone can't make up it's mind and bounces between connections in gray areas which would give any system grief.

Don't get me wrong, pjsip et al are the future, but a lot of the stuff you list works fine without it. We mix and match and use what works best.
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
I'm glad it works for your users to go from WiFi to Mobile data and calls stay up. What I'm saying is, it'd be pretty easy to come up with a common scenario where that'd break without ICE STUN TURN IP6 and PJSIP's superior traversal of NAT gateways to keep the audio and SIP signalling flowing in both directions.

For instance, you're on call, on your VOIP smartphone app, on office WiFii data. You walk outside, stil talking on call, on 4G LTE. You keep talking and walking, and arrive at the local starbucks or burger pub for lunch, your smart phone auto connects and logs into the free WiFi there, which you like because it's a rock solid 50 megabit connection. AND it's got a typical WiFI NAT gateway router. With a commonly available smart phone, VOIP app, it's easy to demonstrate the call drops when there's no ICE STUN TURN and IP6. These newer technologies are needed to get RTP UDP audio packets and SIP signalling thru many NAT gateways, especially when you went from not having a NAT gateway, then you walk to a place that has a NAT gateway, and the call is still ongoing. Why take the chance of letting those walking smart phone VOIP app users' calls drop, interrupt business, and lose money and time? Letting their calls continue by enabling ICE STUN TURN and IP6 is the cleanest safest cheapest and smartest way to deal with this walking around and getting a new IP on a WiFi behind a NAT gateway which you don't own or control yet still want to continue your calls and win business in all scenarios IMHO.
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
Letting their calls continue by enabling ICE STUN TURN and IP6 is the cleanest safest cheapest and smartest way to deal with this walking around and getting a new IP on a WiFi behind a NAT gateway which you don't own or control yet still want to continue your calls and win business in all scenarios IMHO.
Your roaming scenario scenario works probably 95%+ of the time now.

PJSIP is just now really becoming stable IMO. Disregarding PJSIP stability in Asterisk itself at the time, PJSIP support was barely beta quality with FreePBX 12. Its was mostly usable in FBPX13, we haven't tested 14 enough to be comfortable with cutting over and have no reason to - there is no financial incentive. Eventually, yes, but PJSIP with all the extras might squeeze out another 3%. There will always be some remote scenarios that are FUBARed. It's not worth it from a support scenario to change until we make it standard across the board.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,206
Reaction score
5,229
Here's what works for me. "Hey, I'm about to walk into Starbucks. Let me call you back." :ban:
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
Here's what works for me. "Hey, I'm about to walk into Starbucks. Let me call you back." :ban:
I guess my thinking is, why be a luddite and make users go thru that unnecessary interruption to their business conversation and thought flow, due to an unnecessary technical difficulties interruption. This PBX system is equipped with an advanced software stack specially designed to perform fast relatively seamless handover to a new IP address, in about a second, even when the user's endpoint suddenly auto logs on to a WiFi access point behind a NAT such as you find at all coffee shops retail stores public places of interest etc.
 
Last edited:

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
Your roaming scenario scenario works probably 95%+ of the time now.

PJSIP is just now really becoming stable IMO. Disregarding PJSIP stability in Asterisk itself at the time, PJSIP support was barely beta quality with FreePBX 12. Its was mostly usable in FBPX13, we haven't tested 14 enough to be comfortable with cutting over and have no reason to - there is no financial incentive. Eventually, yes, but PJSIP with all the extras might squeeze out another 3%. There will always be some remote scenarios that are FUBARed. It's not worth it from a support scenario to change until we make it standard across the board.

When you say PJSIP stability, I think what you're referring to was the fact that Asterisk developers neglected to learn PJSIP very well, they overlooked they had left PJSIP on a sort of "developer mode" setting. [1] By default, PJSIP builds with ASSERTIONS enabled, this is a developer tool to help find issues and fix them faster, yet assertions is a tech for devs and are to be used normally during the development phase only. So, consequently, on a live production PBX, many normal error-ish type conditions, such as incoming calls with corrupted caller ID or missing SIP headers, were causing PJSIP to ASSERT which totally halted Asterisk, crashed Asterisk, dropped all calls, etc. That PJSIP setting has since been put into normal mode, assertions disabled. PJSIP is actually a quite mature and awesome SIP library, it's the best one.
[1] https://issues.asterisk.org/jira/browse/ASTERISK-21696
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
Interruption. This PBX system is equipped with an advanced software stack specially designed to perform fast relatively seamless handover to a new IP address, in about a second, even when the user's endpoint suddenly auto logs on to a WiFi access point behind a NAT such as you find at all coffee shops retail stores public places of interest etc.
But you are ignoring the fact this works 95% of the time with the "traditional" chan sip and nat transversal methods. The incremental benefit is minor (or less) for our deployments.
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
When you say PJSIP stability, I think what you're referring to was the fact that Asterisk developers neglected to learn PJSIP very well, they overlooked they had left PJSIP on a sort of "developer mode" setting. [1] By default, PJSIP builds with ASSERTIONS enabled, this is a developer tool to help find issues and fix them faster, yet assertions is a tech for devs and are to be used normally during the development phase only. So, consequently, on a live production PBX, many normal error-ish type conditions, such as incoming calls with corrupted caller ID or missing SIP headers, were causing PJSIP to ASSERT which totally halted Asterisk, crashed Asterisk, dropped all calls, etc. That PJSIP setting has since been put into normal mode, assertions disabled. PJSIP is actually a quite mature and awesome SIP library, it's the best one.
[1] https://issues.asterisk.org/jira/browse/ASTERISK-21696
The only thing that matters is how it works for my implementation. Overall histories and other uses are irrelevant.

In the Asterisk/FreePBX context:
Does it do some things better than chan_sip - definitely.
Has it been as stable/reliable overall - no.
Has it been as easy to support - no.
Are we near a turning point - yes, but no need to rush.​
 

chris_c_

Active Member
Joined
Aug 19, 2010
Messages
509
Reaction score
67
The only thing that matters is how it works for my implementation. Overall histories and other uses are irrelevant.

In the Asterisk/FreePBX context:
Does it do some things better than chan_sip - definitely.
Has it been as stable/reliable overall - no.
Has it been as easy to support - no.
Are we near a turning point - yes, but no need to rush.​
Just for fun, if you would, reply with detailed step by steps demonstrating the situations you describe above:
1. a detailed step by step where you get PJSIP to be not "stable/reliable overall" ,
2. a detailed step by step where you get PJSIP to be not "easy to support".
 

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
Just for fun, if you would, reply with detailed step by steps demonstrating the situations you describe above:
1. a detailed step by step where you get PJSIP to be not "stable/reliable overall" ,
2. a detailed step by step where you get PJSIP to be not "easy to support".
Ring groups not working, blfs not working, general weird behavior - call parking/pickup issues if I recall, etc. Some things like the ring group issues definitely got better in FPBX13, we at least still had some weird blf issues IIRC. Had two smaller locations we tried starting out as pure PJSIP(it's the future after all) - that didn't last long. Not all of this (probably most of it) were technically PJSIP issues, more likely FPBX, (and maybe some of our own config), but that doesn't matter, there's no need to waste hours debugging when chan_sip just works. I'm perfectly willing to wait and let others get it worked out.
 

Members online

Forum statistics

Threads
25,825
Messages
167,856
Members
19,250
Latest member
mark-curtis
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top