ALERT GV: The Sky Has Fallen... Really

[wardmundy]

Asterisk uses a stun server if defined. Issue a "stun set debug on" and make some calls to see the activity.

To find out how much trouble a broken or down stun server can cause, set the stun address to a valid address that is not a stun server.

It may be firewall/router dependent, but I've never found a stun server needed for SIP/IAX endpoints. WebRTC is another matter. I don't set it anymore because of the potential bad server issues - got bit once and it took way too long to figure out what the issue really was.

The show status is dependent on the server being listed in res_stun_monitor.conf:

;
; Configuration file for the res_stun_monitor module
;
; The res_stun_monitor module sends STUN requests to a configured STUN server
; periodically.  If the monitor detects a change in the external IP address or port
; provided by the STUN server an event is sent out internally within Asterisk
; to alert all listeners to that event of the change.
; The current default listeners for the network change event include chan_sip
; and chan_iax.  Both of these channel drivers by default react to this event
; by renewing all outbound registrations.  This allows the endpoints Asterisk
; is registering with to become aware of the address change and know the new
; location.
;
[general]
;
; ---- STUN Server configuration ---
;  Setting the 'stunaddr' option to a valid address enables the STUN monitor.
;
;stunaddr = mystunserver.com    ; Address of the STUN server to query.
                                ; Valid form:
                                ;   [(hostname | IP-address) [':' port]]
                                ; The port defaults to the standard STUN port (3478).
                                ; Set to an empty value to disable STUN monitoring.
                                ;   Default is disabled.
;stunrefresh = 30               ; Number of seconds between STUN refreshes.
                                ;   Default is 30.

Problem is that this stun stuff is all over the place in the Asterisk config files. Here's what we're now doing, and it seems to work most (but not all) of the time using stun show status. No idea which config setting Asterisk actually relies upon.

sed -i 's|;stunaddr|stunaddr=stun.counterpath.net\n;|' /etc/asterisk/res_stun_monitor.conf
sed -i 's|;stunrefresh|stunrefresh|' /etc/asterisk/res_stun_monitor.conf
echo "stunaddr=stun.counterpath.net" > /etc/asterisk/rtp_custom.conf
echo "stunrefresh=30" >> /etc/asterisk/rtp_custom.conf
sed -i 's|stun.counterpath.net|stun.l.google.com:19302|' /etc/asterisk/res_stun_monitor.conf
sed -i 's|stun.counterpath.net|stun.l.google.com:19302|' /etc/asterisk/rtp_custom.conf
sed -i 's|stun.counterpath.net|stun.l.google.com:19302|' /etc/asterisk/sip_general_custom.conf
chown asterisk:asterisk /etc/asterisk/*.conf
amportal restart

 

[wardmundy]

Ward, I stopped when the installer reported this:

"This appears to be a fresh install of GVSIP for Incredible PBX.
Do you wish to install all of the required components?"

It shouldn't appear to be a fresh install; it is an existing, recent install that incorporates and that is running GVSIP.

Just a verbiage mis-match?

Just a nomenclature issue. We're using one installer to build and update the GVSIP platform as well as to install GVSIP trunks. When the installer doesn't find the default pjsip_custom.conf setup for GVSIP, it reinstalls Asterisk with the latest patches from the GVSIP-NAF tarball. When the installer doesn't find the correct version of OpenSSL, it first updates OpenSSL to the 1.1 version. If it finds the GVSIP default setup in pjsip_custom.conf, then it starts prompting whether you wish to add GVSIP trunks and skips everything else. If you want to update your GVSIP platform, you have to move pjsip_custom.conf out of the way to trigger a fresh install. Then you can put the file back to restore your previously installed GVSIP trunks. Hope that clarifies the upgrade procedure documented here.

 

[chris_c_]

Asterisk uses a stun server if defined. Issue a "stun set debug on" and make some calls to see the activity.

To find out how much trouble a broken or down stun server can cause, set the stun address to a valid address that is not a stun server.

It may be firewall/router dependent, but I've never found a stun server needed for SIP/IAX endpoints. WebRTC is another matter. I don't set it anymore because of the potential bad server issues - got bit once and it took way too long to figure out what the issue really was.

The show status is dependent on the server being listed in res_stun_monitor.conf:

;
; Configuration file for the res_stun_monitor module
;
; The res_stun_monitor module sends STUN requests to a configured STUN server
; periodically.  If the monitor detects a change in the external IP address or port
; provided by the STUN server an event is sent out internally within Asterisk
; to alert all listeners to that event of the change.
; The current default listeners for the network change event include chan_sip
; and chan_iax.  Both of these channel drivers by default react to this event
; by renewing all outbound registrations.  This allows the endpoints Asterisk
; is registering with to become aware of the address change and know the new
; location.
;
[general]
;
; ---- STUN Server configuration ---
;  Setting the 'stunaddr' option to a valid address enables the STUN monitor.
;
;stunaddr = mystunserver.com    ; Address of the STUN server to query.
                                ; Valid form:
                                ;   [(hostname | IP-address) [':' port]]
                                ; The port defaults to the standard STUN port (3478).
                                ; Set to an empty value to disable STUN monitoring.
                                ;   Default is disabled.
;stunrefresh = 30               ; Number of seconds between STUN refreshes.
                                ;   Default is 30.

[2018-08-04 13:18:58] ERROR[18926]: pjproject:0 <?>:    icess0xb0f554a ......Error sending STUN request: Network is unreachable

Thanks @jerrm !
The asterisk docs on ICE/STUN/TURN appear to omit mention of this essential config file "res_stun_monitor.conf" to get STUN to actually work, therefore, ipbx/freepbx isn't setting this config file setting, and STUN has been silently failing to run!
Have a look at the docs, there's no "res_stun_monitor.conf" : https://wiki.asterisk.org/wiki/display/AST/Interactive+Connectivity+Establishment+(ICE)+in+Asterisk
To get STUN working for SIP and PJSIP, you should set settings in these 4 config files!
1. res_stun_monitor.conf (IPBX/freebpx was omitting this one, causing STUN to fail!)
[general]
stunaddr = stun.counterpath.net
2. rtp_additional.conf
[general]
stunaddr=stun.counterpath.net
3. pjsip.conf
[global]
icesupport=yes
4. sip_general_custom.conf
icesupport=yes
stunaddr=stun.counterpath.net


"amportal restart"...
STUN is FINALLY working inside asterisk:
:~# asterisk -r
[2018-08-04 13:27:03] NOTICE[28059]: res_stun_monitor.c:151 stun_monitor_request: Old external address/port 0.0.0.0:0 now seen as xxx.yyy.zzz.212:38593.
pbx*CLI> stun show status
Hostname Port Period Retries Status ExternAddr ExternPort
stun.counterpath.net 3478 30 3 OK xxx.yyy.zzz.212 38593
pbx*CLI>

 

[tycho]

Just a nomenclature issue. We're using one installer to build and update the GVSIP platform as well as to install GVSIP trunks. When the installer doesn't find the default pjsip_custom.conf setup for GVSIP, it reinstalls Asterisk with the latest patches from the GVSIP-NAF tarball. When the installer doesn't find the correct version of OpenSSL, it first updates OpenSSL to the 1.1 version. If it finds the GVSIP default setup in pjsip_custom.conf, then it starts prompting whether you wish to add GVSIP trunks and skips everything else. If you want to update your GVSIP platform, you have to move pjsip_custom.conf out of the way to trigger a fresh install. Then you can put the file back to restore your previously installed GVSIP trunks. Hope that clarifies the upgrade procedure documented here.

Thanks; that's what I thought but it was easy to stop and wait for clarification rather than break things. And your linked upgrade procedure is what I am following.

 

[jerrm]

[2018-08-04 13:18:58] ERROR[18926]: pjproject:0 <?>:    icess0xb0f554a ......Error sending STUN request: Network is unreachable
ln -s /root/myPBX/backup/backup-crond /etc/cron.d/backup-crond
re

Thanks @jerrm !
The asterisk docs on ICE/STUN/TURN appear to omit mention of this essential config file "res_stun_monitor.conf" to get STUN to actually work, therefore, ipbx/freepbx isn't setting this config file setting, and STUN has been silently failing to run!
Have a look at the docs, there's no "res_stun_monitor.conf" : https://wiki.asterisk.org/wiki/display/AST/Interactive+Connectivity+Establishment+(ICE)+in+Asterisk
To get STUN working for SIP and PJSIP, you should set settings in these 4 config files!
1. res_stun_monitor.conf (IPBX/freebpx was omitting this one, causing STUN to fail!)
[general]
stunaddr = stun.counterpath.net
2. rtp_additional.conf
[general]
stunaddr=stun.counterpath.net
3. pjsip.conf
[global]
icesupport=yes
4. sip_general_custom.conf
icesupport=yes
stunaddr=stun.counterpath.net


"amportal restart"...
STUN is FINALLY working inside asterisk:
:~# asterisk -r
[2018-08-04 13:27:03] NOTICE[28059]: res_stun_monitor.c:151 stun_monitor_request: Old external address/port 0.0.0.0:0 now seen as xxx.yyy.zzz.212:38593.
pbx*CLI> stun show status
Hostname Port Period Retries Status ExternAddr ExternPort
stun.counterpath.net 3478 30 3 OK xxx.yyy.zzz.212 38593
pbx*CLI>

The res_stun_monitor.conf has NOTHING to do with stun failing. It is pretty much useless and irrelevant if stun is set in the sip/rtp config files because stun will be queried on each call.

Stun functionality works just fine if defined in the sip config files with no entry at all in res_stun_monitor.conf. As far as I know res_stun_monitor has no impact at all for pjsip.

A misbehaving stun server in sip/rtp config can still bring your pbx to a halt.

 

[chris_c_]

The res_stun_monitor.conf has NOTHING to do with stun failing. It is pretty much useless and irrelevant if stun is set in the sip/rtp config files because stun will be queried on each call.

Stun functionality works just fine if defined in the sip config files with no entry at all in res_stun_monitor.conf. As far as I know res_stun_monitor has no impact at all for pjsip.

A misbehaving stun server in sip/rtp config can still bring your pbx to a halt.

1. The "misbehaving stun server in sip/rtp config can still bring your pbx to a halt" - this is a bug in asterisk, the call shouldn't halt, asterisk should just save and re-use the most recently detected external IP.
2. "Stun functionality works just fine if defined in the sip config files with no entry at all in res_stun_monitor.conf. As far as I know res_stun_monitor has no impact at all for pjsip." This one PBX was giving warning and error messages about STUN failing, in exactly those conditions, STUN server defined in sip/pjsip conf files with no entry at all in "res_stun_monitor.conf".

I agree what you're saying about asterisk's usages of STUN. The STUN Monitor is intended to auto detect a sudden change of the DHCP public IP address by the ISP internet service provider, and when it happens, force all endpoint phones to re-register, so that calls will continue to work. That STUN Monitor's a different asterisk feature than asterisk's use of ICE, either the STUN protocol, or the TURN protocol, to get SIP and RTP packets flowing in both directions, thru a local NAT firewall, or remote NAT firewall, or double NAT, firewalls.. These two separate features may or may not be dependent on the other one being enabled, or having a valid STUN server setting, and working. They might still work independently of one another, and alone, as long as they're enabled and have a valid STUN server themselves..

 

[jerrm]

1. The "misbehaving stun server in sip/rtp config can still bring your pbx to a halt" - this is a bug in asterisk, the call shouldn't halt, asterisk should just save and re-use the most recently detected external IP.

I won't argue with that. It was mostly a matter of timeouts in our case. Inbound calls were being terminated by the provider because asterisk was taking too long to negotiate the connection.

When the stun server wasn't responding, Asterisk was retrying stun four times with something like an insane two or three second timeout each during call setup. By the time asterisk moved on, the provider had given up. Standard logging was not helpful.

Most stun servers are probably up 99.999, but for something with such a potential impact, I wish we could specify multiple servers for round robin, or better yet simultaneous, attempts.

Admittedly, this was two years ago. Maybe they have improved it and I missed it when skimming the changelog.

My fix is pointing to a hosts file entry with a cron job to test stun servers every so often and update the hosts file address if the current server is slow or down. Probably not needed, probably overkill, but gives peace of mind.

 

[wardmundy]

@chris_c_: We have a fresh build this morning that addresses the GVSIP patch issues (with @dziny fixes) as well as the stun problem. To upgrade existing servers without losing any existing Google Voice GVSIP trunks, here's the drill:

mv /etc/asterisk/pjsip_custom.conf /etc/asterisk/pjsip_custom.tmp
touch /etc/asterisk/pjsip_custom.conf
cd /root
rm gvsip-naf-gui.tar.gz
wget http://incrediblepbx.com/gvsip-naf-gui.tar.gz
tar zxvf gvsip-naf-gui.tar.gz
rm gvsip-naf-gui.tar.gz
cd gvsip-naf
./install-gvsip
mv /etc/asterisk/pjsip_custom.tmp /etc/asterisk/pjsip_custom.conf
amportal restart

@chris_c_ and @dziny: Not sure these patches are quite right yet with Asterisk 13. If you place a call from the Google Voice trunk and the other party answers and then hangs up, the Google Voice side never disconnects until you manually hang up the phone. Also seeing lots of "SSL 6 [SSL_ERROR_ZERO_RETURN] (Read)" messages as well as some "Error sending STUN request: Network is unreachable" messages even though stun show status displays a proper registration with google.

 

[tycho]

So, hold off on the patch that I am in the middle of executing?

 

[restamp]

We have a fresh build this morning that addresses the GVSIP patch issues (with @dziny fixes) as well as the stun problem. To upgrade existing servers without losing any existing Google Voice GVSIP trunks, here's the drill:

mv /etc/asterisk/pjsip_custom.conf /etc/asterisk/pjsip_custom.tmp
touch /etc/asterisk/pjsip_custom.conf
cd /root
rm gvsip-naf-gui.tar.gz
wget http://incrediblepbx.com/gvsip-naf-gui.tar.gz
tar zxvf gvsip-naf-gui.tar.gz
rm gvsip-naf-gui.tar.gz
cd gvsip-naf
./install-gvsip
mv /etc/asterisk/pjsip_custom.tmp /etc/asterisk/pjsip_custom.conf
amportal restart

I noticed when compiling the above that it generated two warnings:

  [CC] sdp_srtp.c -> sdp_srtp.o
sdp_srtp.c: In function ‘ast_sdp_crypto_process’:
sdp_srtp.c:263: warning: ‘crypto’ may be used uninitialized in this function
sdp_srtp.c:275: warning: ‘tag_from_sdp’ may be used uninitialized in this function

Here's a before and after diff of sdp_srtp.c:

diff asterisk-13.21.1/main/sdp_srtp.c asterisk-13.22.0/main
263c263
<    if (sscanf(tag, "%30d", &tag_from_sdp) != 1 || tag_from_sdp < 0 || tag_from_sdp > 999999999) {
---
>    if (sscanf(tag, "%30d", &crypto->tag) != 1 || crypto->tag < 0 || crypto->tag > 999999999) {

Although the new Asterisk is up and running, and apparently stable, the modified code looks to me like a SIGSEG waiting to happen if link encryption is attempted. The crypto pointer is indeed never initialized that I can tell. I also notice that this code is completely different in NAF's "git" clone version, which I don't understand.

My guess is that the patch.patch encryption patch is not being applied. When I tried to apply the patch.patch manually, it didn't work for me when done from the asterisk-13.22.0 directory; I had to run it from /usr/src. If this is not some cockpit error on my part, and you also see these errors, I'd tread carefully here (especially with link encryption) if stability is important to you.

 

[chris_c_]

@chris_c_ and @dziny: Not sure these patches are quite right yet with Asterisk 13. If you place a call from the Google Voice trunk and the other party answers and then hangs up, the Google Voice side never disconnects until you manually hang up the phone. Also seeing lots of "SSL 6 [SSL_ERROR_ZERO_RETURN] (Read)" messages as well as some "Error sending STUN request: Network is unreachable" messages even though stun show status displays a proper registration with google.

I suspect (not sure yet) that the STUN Monitor (for catching change of public dynamic IP address and signal all phones to re-register) is a different feature than the SIP/RTP ICE/STUN NAT firewall traversal intended for protecting two way UDP/TCP audio and two-way SIP TCP channel from behind one or both sides NAT gateway routers.

So, the STUN Monitor can show good status, while the PJSIP code can fail to interact with the other end of the SIP/RTP call and share its STUN public IP detection results with the GV SIP server, or whatever ICE/STUN RFC it's trying to perform.

What this all means is, I suspect Asterisk either has a bug with STUN, or we are still failing to set expected proper conf file settings for Asterisk's STUN feature - we're making Asterisk mess up by not putting the conf settings in the format it is expecting - whatever that is.

 

[billsimon]

In what situations are you guys finding that setting up STUN is required? I would suggest removing all the STUN settings you're playing with and let ICE do its job as usual. (Works for me)

 

[chris_c_]

In what situations are you guys finding that setting up STUN is required? I would suggest removing all the STUN settings you're playing with and let ICE do its job as usual. (Works for me)

STUN is the more basic, free (no account required), of the two methods of accomplishing ICE (Interactive Connection Establishment .... from device to device or server... thru NAT firewalls which normally would muck it up and prevent you from achieving it), the other method is more extreme it's called TURN.
ICE cannot "do its job" all by itself.
Asterisk does ICE by doing either STUN or TURN.
For Asterisk to do achieve STUN success, we need to tell Asterisk to enable STUN and have it tell PJSIP or SIP to use a STUN server of our choice, to establish reliable SIP and RTP connections, regardless of any NAT firewalls in the between, or not.
For some reason, Asterisk PJSIP is erroring while attempting to perform the STUN RFC task.
We need to investigate, find out why, and fix it.
Once STUN is working for PJSIP and SIP, Asterisk will be able to uphold reliable calls across any NAT gateway routers.
Working STUN will result in far less user complaints about one way audio and unreliable dropped calls.

 

[billsimon]

I feel like I just got man-splained.

 

[jerrm]

Working STUN will result in far less user complaints about one way audio and unreliable dropped calls.

Pretty sure @billsimon knows what stun is for, I think his point was that it is generally not needed even behind NAT on the asterisk side for SIP/PJSIP/IAX traffic. The standard NAT transversal generally works fine.

WebRTC is the only thing I've had issues without it, and that was apparently only to get things running, the traffic was all LAN<->LAN no NAT involved.

We don't use it anywhere unless WebRTC is involved and don't have any issues.

Maybe when the server is behind a dynamic IP, but the monitor option should be fine for that - but honestly is nothing we would ever install or support to begin with.​

 

[chris_c_]

Sorry you felt mansplained to Bill. Respect!

The fact remains, with all STUN settings removed, back to defaults, ICE don't do nothin'.

Example of how fast and straightforward a decent ICE STUN function can run.
This javascript ICE STUN code takes 0.2 - 0.4 seconds to completely "Gather candidates" (click the button):
https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/

"In what situations are you guys finding that setting up STUN is required?" -@billsimon
This situation: I've experienced consistent, impossible to fix, one way audio when running an endpoint and asterisk/ipbx connecting onward thru two NAT gateway routers to the destination trunk. ICE STUN is required. This is a common setup, NAT routers are everywhere. A valid STUN server was set in Asterisk SIP Settings. Yet, it failed to achieve two way connection, yet it should have, if it were running right.

It seems just about everyone online is sweeping this Asterisk STUN issue under the rug or saying just fold yer hand and migrate asterisk onto a public IP VPS. This is proof that the STUN bug is real, STUN is obviously broken in Asterisk, whether partly or totally. It's either a straight up bug in the implementation, or the docs omit sharing the necessary conf file settings to make the STUN code achieve a successful list of candidates for RTP and SIP setup thru any and all NATs along the way.

[2018-08-04 23:57:37] ERROR[28041]: pjproject:0 <?>:    icess0xb107b14 ......Error sending STUN request: Network is unreachable
pbx*CLI> stun show status
Hostname                  Port  Period  Retries  Status  ExternAddr       ExternPort
stun.counterpath.net      3478  30      3        OK      aaa.bbb.ccc.212  50794

My 2 cents. I could be wrong. I'm 90% sure this is 100% accurate.

 

[wardmundy]

I noticed when compiling the above that it generated two warnings:

  [CC] sdp_srtp.c -> sdp_srtp.o
sdp_srtp.c: In function ‘ast_sdp_crypto_process’:
sdp_srtp.c:263: warning: ‘crypto’ may be used uninitialized in this function
sdp_srtp.c:275: warning: ‘tag_from_sdp’ may be used uninitialized in this function

Here's a before and after diff of sdp_srtp.c:

diff asterisk-13.21.1/main/sdp_srtp.c asterisk-13.22.0/main
263c263
<    if (sscanf(tag, "%30d", &tag_from_sdp) != 1 || tag_from_sdp < 0 || tag_from_sdp > 999999999) {
---
>    if (sscanf(tag, "%30d", &crypto->tag) != 1 || crypto->tag < 0 || crypto->tag > 999999999) {

Although the new Asterisk is up and running, and apparently stable, the modified code looks to me like a SIGSEG waiting to happen if link encryption is attempted. The crypto pointer is indeed never initialized that I can tell. I also notice that this code is completely different in NAF's "git" clone version, which I don't understand.

My guess is that the patch.patch encryption patch is not being applied. When I tried to apply the patch.patch manually, it didn't work for me when done from the asterisk-13.22.0 directory; I had to run it from /usr/src. If this is not some cockpit error on my part, and you also see these errors, I'd tread carefully here (especially with link encryption) if stability is important to you.

You are correct. The srtp patches needed to be in the Asterisk source directory before executing the patches. That solved the patch errors and fixed the hangup failure issue as well, but still have the network unreachable errors with stun. Installers have all been updated.

 

[wardmundy]

Here is the likely reason for the "network unreachable" errors with stun from Josh Culp on the DSLR Forum. If someone with IPv6 connectivity from their server could test whether you're still seeing the unreachable errors, that would solve the last remaining issue.

The "network unreachable" message is documented on naf's wiki, as he tracked it down:

In order to work behind firewalls, SIP uses a protocol named ICE to try to detect your NAT settings automatically by connecting to stun.l.google.com using every local interface it can find (and asking what its external IP is so it can populate the ICE candidate list in the sdp it uses when making an outbound call). Most of the interfaces probably work, but for any one that fails it claims it to be an error. On my machine, its just the ipv6 address being unable to connect.

As the system likely has an IPv6 link local address and as Google has an IPv6 candidate, the ICE negotiation attempts to initiate. It doesn't work though and provides "network unreachable" as there is no global scope IPv6 address so for that attempt the network really is unreachable.

The res_stun_monitor module also has no impact on calling. It just raises an event internally when it detects that the public IP address has changed. Since TCP/TLS Is in use with a keep alive this dropped connection would get detected and a re-registration occur regardless.

 

[wardmundy]

So, hold of on the patch that I am in the middle of executing?

Just download the latest version and repeat the drill one more time.

 

[wardmundy]

Sunday morning edition from the OBi Forum: