SOLVED GV stopped connecting today

[yozh]

There is a bug report that matches this: https://issues.asterisk.org/jira/browse/ASTERISK-24712

There are four possible paths in this function that lead to the error message you are getting. http://doxygen.asterisk.org/trunk/dc/d6b/res__xmpp_8c.html#061c4cbf39cf2ba5a8d8175f2a6716ee

It seems like an OpenSSL compatibility problem.

What version of OpenSSL are you running?

I saw that bug, but there is no real solution is there ?

This is my OpenSSL version. I just updated this morning.

[root@pbx ~]# openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013
[root@pbx ~]#

 

[billsimon]

@yozh - one more thing

ldd `which asterisk`

 

[yozh]

@yozh - one more thing

ldd `which asterisk`

Sure

[root@pbx ~]# ldd `which asterisk`
linux-gate.so.1 => (0x00afc000)
libasteriskssl.so.1 => /usr/lib/libasteriskssl.so.1 (0x006d9000)
libc.so.6 => /lib/libc.so.6 (0x0016d000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00529000)
libz.so.1 => /lib/libz.so.1 (0x006c3000)
libm.so.6 => /lib/libm.so.6 (0x00305000)
libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x0032f000)
libssl.so.10 => /usr/lib/libssl.so.10 (0x006f8000)
libcrypto.so.10 => /usr/lib/libcrypto.so.10 (0x00760000)
libdl.so.2 => /lib/libdl.so.2 (0x00676000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00d25000)
libtinfo.so.5 => /lib/libtinfo.so.5 (0x003bf000)
libresolv.so.2 => /lib/libresolv.so.2 (0x00481000)
libedit.so.0 => /usr/lib/libedit.so.0 (0x003d8000)
libncurses.so.5 => /lib/libncurses.so.5 (0x00402000)
/lib/ld-linux.so.2 (0x00b0e000)
libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x00425000)
libkrb5.so.3 => /lib/libkrb5.so.3 (0x00928000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x00ad4000)
libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x0049b000)
libkrb5support.so.0 => /lib/libkrb5support.so.0 (0x00465000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00471000)
libselinux.so.1 => /lib/libselinux.so.1 (0x004c6000)
[root@pbx ~]#

 

[billsimon]

Hmm. My Asterisk installation doesn't build libasteriskssl. One more query...

ldd /usr/lib/libasteriskssl.so.1

just trying to make sure nothing is linking to old SSL libraries somewhere.

 

[billsimon]

If it seems like that SSL wrapper library is causing the problem, a workaround might be (from https://wiki.asterisk.org/wiki/display/AST/New+in+11) :

• The Asterisk build system will now build and install a shared library (libasteriskssl.so) used to wrap various initialization and shutdown functions from the libssl and libcryptolibraries provided by OpenSSL. This is done so that Asterisk can ensure that these functions do not get called by any modules that are loaded into Asterisk, since they should only be called once in any single process. If desired, this feature can be disabled by supplying the --disable-asteriskssl option to the configure script.

 

[yozh]

ldd /usr/lib/libasteriskssl.so.1
linux-gate.so.1 => (0x00694000)
libssl.so.10 => /usr/lib/libssl.so.10 (0x00eb4000)
libcrypto.so.10 => /usr/lib/libcrypto.so.10 (0x003b8000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00113000)
libc.so.6 => /lib/libc.so.6 (0x007c8000)
libgssapi_krb5.so.2 => /lib/libgssapi_krb5.so.2 (0x0012e000)
libkrb5.so.3 => /lib/libkrb5.so.3 (0x00d32000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x0016e000)
libk5crypto.so.3 => /lib/libk5crypto.so.3 (0x00173000)
libresolv.so.2 => /lib/libresolv.so.2 (0x00b92000)
libdl.so.2 => /lib/libdl.so.2 (0x006ab000)
libz.so.1 => /lib/libz.so.1 (0x006c3000)
/lib/ld-linux.so.2 (0x00961000)
libkrb5support.so.0 => /lib/libkrb5support.so.0 (0x00bcc000)
libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x0019e000)
libselinux.so.1 => /lib/libselinux.so.1 (0x001a2000)
]0;root@pbx:~[root@pbx ~]#



thank you for your help

 

[billsimon]

Does restarting (not just reloading) Asterisk fix the problem?

 

[billsimon]

Note from the quoted changelog above: "This is done so that Asterisk can ensure that these functions do not get called by any modules that are loaded into Asterisk, since they should only be called once in any single process."

The part of the code that is generating the error is the part that initializes SSL for the XMPP connection or reconnection. So it should work the first time, then fail thereafter if your Google Chat session is disconnected and you need to reconnect, due to the restrictions of this wrapper.

 

[yozh]

Even restarting the server doesnt help it. This just started, well I noticed it yesterday. This was working fine for years. No updates where applied until this morning.

Recompiling whole pbx might be a problem since this was a working PBIF solution, I might have to build a new VM with new asktrisks install and try that, but you dont think its an issue on Google end ? Seems strange that it stopped working for multiple people and then started again for some. I wonder if google banned my IP for xmpp/jabber connection.

 

[billsimon]

Oh. Then I think I see the problem.

This was working fine for years.

This implies you are using plain auth, not OAUTH2. I thought you had set up the OAUTH2 patch on your server.

Your log shows that Google isn't offering you the option to log in with plain username/password:

<stream:stream from="gmail.com" id="94E57E52572E4E57" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client"><stream:features><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"><required/></starttls><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>X-OAUTH2</mechanism><mechanism>X-GOOGLE-TOKEN</mechanism></mechanisms></stream:features>

...Only OAUTH2; otherwise you would get an option that shows

<mechanism>PLAIN</mechanism>

.

Do you have "Less secure apps" enabled at https://www.google.com/settings/security/lesssecureapps ?

 

[yozh]

Oh. Then I think I see the problem.

This implies you are using plain auth, not OAUTH2. I thought you had set up the OAUTH2 patch on your server.

Your log shows that Google isn't offering you the option to log in with plain username/password:

<stream:stream from="gmail.com" id="94E57E52572E4E57" version="1.0" xmlns:stream="http://etherx.jabber.org/streams" xmlns="jabber:client"><stream:features><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"><required/></starttls><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>X-OAUTH2</mechanism><mechanism>X-GOOGLE-TOKEN</mechanism></mechanisms></stream:features>

...Only OAUTH2; otherwise you would get an option that shows

<mechanism>PLAIN</mechanism>

.

Do you have "Less secure apps" enabled at https://www.google.com/settings/security/lesssecureapps ?

Yeah I do. I went thru each account about 2 months ago and made sure it was set for less secure apps and that they where allowing me to login from my IP. It was working pretty solid since then, until yesterday. Ill check again.

Whats the patch ? Can I apply it to PBIF with out much disruption ?

 

[billsimon]

Well, that's interesting. You have the "less secure apps" enabled and it's not letting you do the less secure login method. I don't know why, unless they are rolling out changes gradually. I have an Asterisk server doing plain login to Google Chat strictly for sending jabber messages (to notify me of phone calls) and it hasn't been disrupted.

The OAUTH2 patch is a code change, recompile, and reconfigure. It is certainly a disruption. I think this tutorial is still right. http://pbxinaflash.com/community/resources/incredible-pbx-google-voice-oauth.46/

 

[wardmundy]

Several of my Plain-text password trunks have started failing as well. Sounds like...

 

[yozh]

Well, that's interesting. You have the "less secure apps" enabled and it's not letting you do the less secure login method. I don't know why, unless they are rolling out changes gradually. I have an Asterisk server doing plain login to Google Chat strictly for sending jabber messages (to notify me of phone calls) and it hasn't been disrupted.

The OAUTH2 patch is a code change, recompile, and reconfigure. It is certainly a disruption. I think this tutorial is still right. http://pbxinaflash.com/community/resources/incredible-pbx-google-voice-oauth.46/

Yeah, just went to check again and turned it off/on and still get the same error and no "PLAIN" offered. The tutorial doesnt look too bad, maybe Ill try it.

 

[yozh]

Several of my Plain password trunks have started failing as well. Sounds like...

Yeah I guess it is well it was a good run, I have to try the workaround listed above, maybe with that it`ll work again.

 

[sub0]

If only it was just google voice that stopped working.... Starting at 3:16pm PST, all of our PIAF instances ran out of memory and we ended up with a full outage as it also affected our a2billing core switch.

Fixed it by adding 'noload => res_xmpp.so' to modules.conf

All systems show the same error:

[2016-06-14 16:12:50] WARNING[1831] res_xmpp.c: JABBER: socket read error
[2016-06-14 16:12:50] WARNING[1831] res_xmpp.c: JABBER: socket read error
[2016-06-14 16:13:00] WARNING[1831] res_xmpp.c: JABBER: socket read error
[2016-06-14 16:13:00] WARNING[1831] res_xmpp.c: JABBER: socket read error
[2016-06-14 16:13:05] ERROR[2009] /usr/src/asterisk/include/asterisk/utils.h: Memory Allocation Failure in function internal_ao2_alloc at line 564 of astobj2.c

 

[yozh]

Well I`m making a clone, and going to try to bring my trunks back ...

 

[Huckda]

So, it's not just me... was getting res_jabber.c Socket read error and something about TLS...

 

[yozh]

Well I`m making a clone, and going to try to bring my trunks back ...

So, followed instructions, id everything. Now getting this errors below.

[2016-06-14 21:45:52] WARNING[3916]: res_xmpp.c:3536 xmpp_client_receive: Parsing failure: Hook returned an error.
[2016-06-14 21:45:52] WARNING[3916]: res_xmpp.c:3595 xmpp_client_thread: JABBER: socket read error
[2016-06-14 21:45:52] NOTICE[3916]: res_xmpp.c:3420 xmpp_client_reconnect: Connecting to client token : XXXX6EXXXXXdXXXXikXXXXXXXX
[2016-06-14 21:45:52] NOTICE[3916]: res_xmpp.c:3630 fetch_access_token: Command CURL(https://www.googleapis.com/oauth2/v3/token,client_id=XXXXXn8j3d.apps.googleusercontent.com&client_secret=XXXXxxxXXXXo2&refresh_token=1XXXXXXXXXXXXXXXXXXyXXXxxxxE-8XXXXXXXXXXXdyk&grant_type=refresh_token)
[2016-06-14 21:45:52] NOTICE[3916]: res_xmpp.c:3634 fetch_access_token: Command status : {
"access_token": "XXXXxxxxxxxxxxxxxXXXew",
"token_type": "Bearer",
"expires_in": 3600
}
[2016-06-14 21:45:52] NOTICE[3916]: res_xmpp.c:3643 fetch_access_token: access Token : yXXXXCi8XXXXXX
[2016-06-14 21:45:52] ERROR[3916]: res_xmpp.c:2563 xmpp_client_requested_tls: TLS connection for client 'XXXXXXXX' cannot be established. OpenSSL initialization failed.

 

[wardmundy]

You'll probably need to follow the tutorial to implement Google Voice OAUTH 2 authentication on your server before attempting to register an account.