SOLVED Asterisk 13.17.2 Crashes and More

[wardmundy]

Just a heads up that there appear to be major problems with Asterisk 13 releases beginning with Asterisk 13.16. Something (still don't know what) is causing frequent core dumps typically after two dialplan reloads in FreePBX 12. In addition, something (still don't know what) broke Google voice recognition and Google TTS.

For the time being, the Incredible PBX 13 installer will load Asterisk 13.15.1 which appears to be rock solid with either CentOS 6.9 or the latest release of CentOS 7.

core.incrediblepbx-mia6.wardmundy.net-2017-10-13T11:51:22+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T11:53:39+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T12:29:09+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T12:30:49+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T12:31:29+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T12:45:14+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:02:23+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:07:32+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:13:37+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:19:18+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:21:05+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:22:04+0000
core.incrediblepbx-mia6.wardmundy.net-2017-10-13T13:24:21+0000

 

[rdoac]

Is there a way to go back? I have a live server 13.16.0 (built in June 2017) which is doing this. I tried the upgrade (which would have gone to 13.17.2) but have pjsip compilation errors. Can I hardcode the update script to "downgrade asterisk" or do I need to rebuild the box? I'd like to be able to do it one evening, out of hours.

The box is stable as long as I don't configure anything.

Its an IncrediblePBX on Ubuntu 14.04 running FreePBX btw.

 

[wardmundy]

# THIS IS FOR UBUNTU 14.04 64-BIT PLATFORM ONLY!!
amportal kill
mv /var/lib/asterisk/sounds/custom /tmp
cd /usr/src
wget http://downloads.asterisk.org/pub/telephony/asterisk/releases/asterisk-13.15.1.tar.gz
tar zxvf asterisk-13.15.1.tar.gz
cd asterisk-13.15.1
contrib/scripts/get_mp3_source.sh
# if you're using GV OAuth uncomment next 2 lines
# wget http://incrediblepbx.com/res_xmpp-13.tar.gz
# tar zxvf res_xmpp-13.tar.gz
# otherwise uncomment the next line for plaintext GV
# sed -i 's/SSLv3_method()/SSLv23_method()/g' res/res_xmpp.c
make distclean
autoconf
./bootstrap.sh
./configure
make menuselect.makeopts
menuselect/menuselect --enable-category  MENUSELECT_ADDONS menuselect.makeopts
menuselect/menuselect --enable-category  MENUSELECT_CODECS menuselect.makeopts
menuselect/menuselect --enable CORE-SOUNDS-EN-GSM --enable MOH-OPSOUND-WAV --enable EXTRA-SOUNDS-EN-GSM --enable cdr_mysql menuselect.makeopts
menuselect/menuselect --disable app_mysql --disable app_setcallerid --disable func_audiohookinherit --disable res_fax_spandsp menuselect.makeopts
sed -i 's|march=native|mtune=native|' Makefile.rules
make menuselect
# just tab to SAVE and press ENTER to save selections
make
make uninstall
make install
cd ..
ldconfig
cd Asterisk-Flite*
make clean
make
make install
mv /usr/lib/asterisk/modules/app_flite.so /usr/lib64/asterisk/modules
mv /tmp/custom /var/lib/asterisk/sounds
amportal start
# be sure to comment out UDP 10000-20000 line in /etc/iptables/rules.v4 and then
iptables-restart

 

[wardmundy]

# THIS IS FOR CENTOS 64-BIT PLATFORM ONLY!!
amportal kill
mv /var/lib/asterisk/sounds/custom /tmp
cd /usr/src
wget http://downloads.asterisk.org/pub/telephony/asterisk/releases/asterisk-13.15.1.tar.gz
tar zxvf asterisk-13.15.1.tar.gz
cd asterisk-13.15.1
contrib/scripts/install_prereq install
contrib/scripts/get_mp3_source.sh
# if you're using GV OAuth uncomment next 2 lines
# wget http://incrediblepbx.com/res_xmpp-13.tar.gz
# tar zxvf res_xmpp-13.tar.gz
# otherwise uncomment the next line for plaintext GV
# sed -i 's/SSLv3_method()/SSLv23_method()/g' res/res_xmpp.c
make distclean
# for centos 7 uncomment next line
# wget http://incrediblepbx.com/menuselect-incredible13.87.tar.gz
# otherwise for centos 6 uncomment next line
# wget http://incrediblepbx.com/menuselect-incredible13.8.tar.gz
tar zxvf menuselect-incredible*
rm -f menuselect-incredible*
# for centos 7 uncomment next line
# CFLAGS='-DENABLE_SRTP_AES_256 -DENABLE_SRTP_AES_GCM' ./configure --with-pjproject-bundled
# otherwise for centos 6 uncomment next line
# CFLAGS='-DENABLE_SRTP_AES_256 -DENABLE_SRTP_AES_GCM' ./configure --libdir=/usr/lib64 --with-pjproject-bundled
 make menuselect.makeopts
 menuselect/menuselect --enable-category MENUSELECT_ADDONS menuselect.makeopts
 menuselect/menuselect --enable-category MENUSELECT_CODECS menuselect.makeopts
 menuselect/menuselect --disable-category MENUSELECT_TESTS menuselect.makeopts
 menuselect/menuselect --enable CORE-SOUNDS-EN-GSM --enable MOH-OPSOUND-WAV --enable EXTRA-SOUNDS-EN-GSM --enable cdr_mysql menuselect.makeopts
 menuselect/menuselect --disable app_mysql --disable app_setcallerid --disable func_audiohookinherit --disable res_fax_spandsp menuselect.makeopts
 menuselect/menuselect --disable test_named_lock --disable test_res_pjsip_scheduler --disable test_file menuselect.makeopts
 sed -i 's|march=native|mtune=native|' Makefile.rules
make menuselect
# just tab to SAVE and press ENTER to save selections
make
make uninstall
make install
cd ..
ldconfig
cd Asterisk-Flite*
make clean
make
make install
mv /usr/lib/asterisk/modules/app_flite.so /usr/lib64/asterisk/modules
mv /tmp/custom /var/lib/asterisk/sounds
amportal start
# be sure to comment out UDP 10000-20000 line in /etc/sysconfig/iptables and then
iptables-restart

 

[rdoac]

Thanks Ward - this will be my weekend project - it's on vmware - so have snapshot support - I'll report back.

 

[wardmundy]

Thanks Ward - this will be my weekend project - it's on vmware - so have snapshot support - I'll report back.

Good luck. And you were right. As long as you made no changes in the GUI, Asterisk worked reliably except for the Google TTS and STT stuff. By all means, have a backup ready before you begin.

 

[dallas]

This also happens on Incredible for Raspberry Pi 3B.

 

[rdoac]

Thanks Ward, I got it working. I had to go back to the snapshot the first time as I still got the pjsip compilation errors. I ended up installing pjsip 2.6 from the upgrade script, which fixed that error. (Wondering if 2.3 has an issue?)
The only thing I noticed was that I lost my on hold music - simple enough to reupload. I'll leave the snapshot as is for the moment - and remove it Monday night after the first day of operation.

 

[wardmundy]

This also happens on Incredible for Raspberry Pi 3B.

Yep. It's on my list.

 

[restamp]

Thanks for this post, Ward. I had indeed noticed the problem on my CentOS box and presumed it was something I had done wrong. I'm now running on Asterisk 13.15.1 and it looks good.

One caveat: After running the above script, I had to "mv /usr/lib/asterisk/modules/app_flite.so /usr/lib64/asterisk/modules" and do the usual stuff (ldconfig; amportal restart) before I could coax the gentle voice of Flite to speak. (And after doing this, the Google TTS stuff worked for me. Indeed, it probably had been working all along had I only known when to speak to it.)

 

[dallas]

Yep. It's on my list.

Thanks Ward, I can supply freepbx and asterisk logs if you need them.

 

[billsimon]

FYI - no need to downgrade or go to a lot of trouble. The workaround was posted on DSL Reports (https://www.dslreports.com/forum/r31657410-PSA-Asterisk-13-7-1) : Upgrade to next version or touch /etc/asterisk/ari.conf

 

[restamp]

Can anyone still running 13.17.2 verify that the dslreports "touch /etc/asterisk/ari.conf" workaround works? I can't try it without rolling back (forward) to 13.17.2 again. However, I'm dubious because (1) the article refers to Asterisk 13.7.1, not 13.17 which we are on, and (2) even if 13.7 is a typo, it suggests upgrading to the next version, but 13.17.2 is still broken.

FWIW, here's the reference describing the problem (and workaround) in the Asterisk issues database:

https://issues.asterisk.org/jira/browse/ASTERISK-27026

One other data point: I'm also currently running FreePBX 13 with Asterisk 13.17.2 on a former Pogoplug (ARM architechture, think Raspberry Pi). There is currently no /etc/asterisk/ari.conf file on *this* box, yet it has been quite stable since upgrading to 13.17.2. (I know, different architecture, so the bug could be latent here.)

 

[billsimon]

Well, I can tell you I recently built FreePBX 14 and Asterisk 13.17.1 using the instructions here (https://wiki.freepbx.org/display/FOP/Installing+FreePBX+14+on+Debian+8.8) which include that touch command, and I have not experienced this crashing.

 

[jeff.h]

I had issues with gvoauth on 13.71, but when I upgraded to 13.7.2 that went away. I am running CentOS 6.9

 

[wardmundy]

FYI: This is not the "missing ari.conf bug." Digium "improved" things by breaking FreePBX 12 GPL modules beginning with Asterisk 13.16. And that's the primary reason we're moving to FreePBX 13 GPL modules. I don't have high hopes that Digium is going to fix this.

 

[billsimon]

Digium "improved" things by breaking FreePBX 12 GPL modules beginning with Asterisk 13.16.

Do you have the bug tracker link for this?

 

[wardmundy]

I haven't opened a ticket. Just sent some folks an email. FreePBX 12 is EOL so they're not going to spend a bunch of time on this, and I certainly appreciate that.

 

[dallas]

And that's the primary reason we're moving to FreePBX 13

Is there time line to update Incredible GUI to version 13?

 

[wardmundy]

Is there time line to update Incredible GUI to version 13?

Hope to have the Lean version available for pioneers before this weekend... with a little luck.

LEAN = no GUI mods at all, i.e. extensions, trunks, routes, or Incredible PBX apps.

FYI: Digium thinks they have fixed the 13.17.2 crashing bug in 13.18 RC which we have not yet tested.