SOLVED UCP Crashes Asterisk 13

[Joe Lones]

I've just installed a new box on ESXi, this time with ubuntu 14.04(64bit mini.iso, 1gb ram, 1 cpu.) and I'm now noticing random crashing of asterisk especially after a reboot.

I don't have much to go on. I see this is the syslog:

Mar  7 16:10:01 incrediblepbx CRON[2290]: (asterisk) CMD ([ -e /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php)
Mar  7 16:10:01 incrediblepbx CRON[2291]: (root) CMD (/root/ipchecker > /dev/null 2>&1)
Mar  7 16:10:06 incrediblepbx /usr/bin/crontab[2317]: (asterisk) LIST (asterisk)
Mar  7 16:10:06 incrediblepbx /usr/bin/crontab[2320]: (asterisk) REPLACE (asterisk)
Mar  7 16:10:06 incrediblepbx /usr/bin/crontab[2322]: (asterisk) LIST (asterisk)
Mar  7 16:10:06 incrediblepbx crontab[2324]: (asterisk) LIST (asterisk)
Mar  7 16:11:01 incrediblepbx cron[770]: (asterisk) RELOAD (crontabs/asterisk)
Mar  7 16:11:01 incrediblepbx CRON[2509]: (asterisk) CMD ([ -e /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php)
Mar  7 16:11:17 incrediblepbx kernel: [  177.683006] traps: asterisk[2523] general protection ip:54ab24 sp:7f4e7fc9eff0 error:0 in asterisk[400000+294000]
Mar  7 16:12:01 incrediblepbx CRON[2578]: (asterisk) CMD ([ -e /var/www/html/admin/modules/dashboard/scheduler.php ] && /var/www/html/admin/modules/dashboard/scheduler.php)

[2016-03-07 16:08:30] ERROR[1977] pbx.c: Function CURL not registered
[2016-03-07 16:08:30] ERROR[1977] res_xmpp.c: object is NULL
[2016-03-07 16:08:30] WARNING[1977] res_xmpp.c: Parsing failure: Hook returned an error.
[2016-03-07 16:08:30] WARNING[1977] res_xmpp.c: JABBER: socket read error
[2016-03-07 16:08:33] ERROR[1781] pbx.c: Function ODBC_TIMECLOCK already registered.
[2016-03-07 16:08:33] ERROR[1781] pbx.c: Function ODBC_ASTERIDEX already registered.

I recompiled to no avail. Should I look anywhere else?

I've installed the g729 codec using this, doubt this is problem:

cd /usr/lib/asterisk/modules
wget http://asterisk.hosting.lv/bin/codec_g729-ast130-gcc4-glibc-x86_64-pentium4.so
mv codec_g729-ast130-gcc4-glibc-x86_64-pentium4.so codec_g729.so
chmod 755 codec_g729.so

I'd really love it if any body would provide some help here. Please.

 

[wardmundy]

You need a license for G729 in the U.S. Not sure where you are. If you have the wrong processor choice for G.729, it would definitely cause crashes.

From their site:

• Asterisk crashing at startup
  If Pentium4 codec is not working, try -no-sse builds, try Pentium3. Try ICC builds. Doesn't work - try Pentium and Debug builds.
• Asterisk still crashing at startup
  Compile asterisk with CFLAGS=-g
  Run asterisk in gdb (with original and -debug version of the codec):
  $ gdb /usr/sbin/asterisk
  (gdb) r -cvvv
  when it crashes do:
  (gdb) set disassembly-flavor intel
  (32-bit gdb 6.x) disassemble $eip $eip+20
  (32-bit gdb 7.x) disassemble $eip, $eip+20
  (64-bit gdb 6.x) disassemble $rip $rip+20
  (64-bit gdb 7.x) disassemble $rip, $rip+20
  (gdb) info registers
  (gdb) info threads
  (gdb) bt
  (gdb) l
  (gdb) q

 

[Joe Lones]

@wardmundy. Not sure if the g729 codec is causing the crash. Really just a wild stab in the dark. I don't know why it's crashing. That's the only thing I did differently from the guide.

 

[Joe Lones]

@wardmundy.
I can reproduce the crashing now, every time i hit the UCP it crashes and all i see at the very top is on the UCP page is "Connecting...." and then it brings asterisk down
what could be causing this?

EDIT: I remember doing this to fix the icon:

cd /var/www/html
mv recordings recordings2
ln -s ucp recordings

EDIT2: error.log

[Mon Mar 07 17:17:49.559275 2016] [access_compat:error] [pid 1850] [client 192.168.1.90:59346] AH01797: client denied by server configuration: /var/www/html/ucp/index.html, referer: http://192.168.1.78/admin/config.php?display=index
[Mon Mar 07 17:17:49.559337 2016] [access_compat:error] [pid 1850] [client 192.168.1.90:59346] AH01797: client denied by server configuration: /var/www/html/ucp/index.cgi, referer: http://192.168.1.78/admin/config.php?display=index
[Mon Mar 07 17:17:49.559361 2016] [access_compat:error] [pid 1850] [client 192.168.1.90:59346] AH01797: client denied by server configuration: /var/www/html/ucp/index.pl, referer: http://192.168.1.78/admin/config.php?display=index
[Mon Mar 07 17:17:51.016462 2016] [:error] [pid 1854] [client 192.168.1.90:59349] PHP Fatal error:  Call to a member function MailboxCount() on a non-object in /var/www/html/admin/modules/voicemail/ucp/Voicemail.class.php on line 629, referer: http://192.168.1.78/ucp/

 

[Joe Lones]

Just so you know the error:

[Mon Mar 07 17:17:51.016462 2016] [:error] [pid 1854] [client 192.168.1.90:59349] PHP Fatal error:  Call to a member function MailboxCount() on a non-object in /var/www/html/admin/modules/voicemail/ucp/Voicemail.class.php on line 629, referer: http://192.168.1.78/ucp/

stems from when I add a virtual extension (for voicemails) to a new user's "Additional Assigned Extensions:" settings.

Unless I;m doing something wrong here, the rationale was to show the mailbox, (virtual extension , which has vm emabled) on the ucp.

Also now seeing this error which crashes asterisk:

Call to a member function PresenceState() on a non-object in /var/www/html/admin/modules/presencestate/ucp/Presencestate.class.php on line 143, referer: http://192.168.3.77/admin/config.php

All to do with the UCP. I'm also seeing:

Tue Mar 08 11:46:58.558383 2016] [access_compat:error] [pid 2152] [client 192.168.3.90:58839] AH01797: client denied by server configuration: /var/www/html/ucp/index.html, referer: http://192.168.3.77/admin/config.php
[Tue Mar 08 11:46:58.661276 2016] [access_compat:error] [pid 2152] [client 192.168.3.90:58839] AH01797: client denied by server configuration: /var/www/html/ucp/index.cgi, referer: http://192.168.3.77/admin/config.php
[Tue Mar 08 11:46:58.661339 2016] [access_compat:error] [pid 2152] [client 192.168.3.90:58839] AH01797: client denied by server configuration: /var/www/html/ucp/index.pl, referer: http://192.168.3.77/admin/config.php

@wardmundy. All this with a clean install and 4 extensions created.

 

[Joe Lones]

@wardmundy this member @chris_c_ seems to have the say type of errors warnings as me, could this be related to the crashing I'm experiencing http://pbxinaflash.com/community/threads/asterisk-bombs-ipbx-13-12-ubuntu-14.17534/page-2

Only thing I did different was to compile with wget http://incrediblepbx.com/res_xmpp-13.tar.gz for gv oauth.

I must have now reinstalled 4 times using the iso and ubuntu and I always see the errors.

 

[silox]

I'm experiencing the same problem unfortunately.

I'm running Ubuntu 16.04 LTS on a Dell 2950 server (8 CPUs, 16GB of RAM, RAID10 with 4x 1TB SATA drives)

The PBX is running Ubuntu 14.04 LTS within a Linux KVM VM that has 2 CPUs, 4GB of RAM and 50GB of hard drive space assigned to it.

I'm using Incredible PBX 13-12 with the Google OAUTH patch applied (not using Google Voice at all though, just did it for potential future use) and the system has 10 active extensions, 1 custom extension to dial a cell phone from an IVR and a virtual extension for a general VM box.

I wasn't sure what was causing the problem was until I was doing some work in the UCP to setup the general VM box's 'unavailable' message and asterisk crashed about 3 times in just a few minutes. The first time it crashed was while I was logging into the UCP (credentials were good and login process just hung until asterisk was restarted automatically and page refreshed).

I don't remember exactly what I was doing in the UCP when it happened the other 2 times other than I was looking for the area to upload the greeting which I found and did so. Luckily I'm using the script someone posted in this or another thread to automatically detect if asterisk has died and if so, restart it automatically then it sends an email to you about the occurrence.

I'm looking to get this resolved ASAP as this is a new deployment for a customer that just switched from Vonage and I'd like to avoid the 'this didn't happen before' remarks if at all possible.

Please let me know what I can do to help make progress on resolving this problem.

Thanks and I appreciate all of the great work made available through IncrediblePBX, the Nerd Vittles website and this forum!

 

[silox]

Wanted to provide additional details concerning this problem. It's definitely related to the UCP and I can replicate the issue when I successfully login to the UCP interface. Not sure what additional actions may trigger the problem within the UCP, too scared to try right now in the middle of the business day. I also receive the following errors in the log file when the asterisk crash occurs after logging into the UCP:

[Thu Aug 11 12:10:00.711055 2016] [:error] [pid 24280] [client 192.168.2.240:56222] PHP Strict Standards:  Non-static method DB::isManip() should not be called statically, assuming $this from incompatible context in /usr/share/php/DB/common.php on line 2200, referer: http://192.168.2.8/admin/config.php?display=index
[Thu Aug 11 12:10:00.712424 2016] [:error] [pid 24280] [client 192.168.2.240:56222] PHP Strict Standards:  Non-static method DB::isError() should not be called statically, assuming $this from incompatible context in /usr/share/php/DB/common.php on line 1217, referer: http://192.168.2.8/admin/config.php?display=index

I was able to resolve the 'AH01797: client denied by server configuration:' errors by modifying the UCP .htaccess file to include the file extensions .pl, .html and .cgi. I was hoping this would cause the crashes to cease but unfortunately that's a no go. I'm not sure if making the .htaccess file modifcations was the best thing to do in regards to security however this particular PBX is located on the customer's internal network so I'm not too worried about it.

 

[silox]

I saw a mention on the FreePBX forums about updating the UCP module. Is this something you think I should try? If so, how would I go about doing that?

 

[wardmundy]

It's available in their GitHub repo, and a tutorial for installing those packages is available here on the forum.

 

[wardmundy]

@silox: Some good news and bad news. Asterisk still crashes with the latest ucp package every time you log in to UCP on CentOS 7. But it worked fine on Ubuntu 14.04 and CentOS/SL 6.x using Incredible PBX ISO.

Have you tried XiVO? Their XiVO Clients have much of the same functionality... minus the core dumps.

 

[silox]

Woohooo! I have an update which is great news.

So, I updated the UCP module using the tutorial Ward provided and unfortunately I had the same problem. I did however start to do some digging and troubleshooting on my own. During that process I noticed that I received the following error below after logging into the UCP right before asterisk crashing/restarting

[Sat Aug 13 01:20:02.854208 2016] [:error] [pid 15834] [client 192.168.2.240:56020] PHP Fatal error:  Call to a member function PresenceState() on a non-object in /var/www/html/admin/modules/presencestate/ucp/Presencestate.class.php on line 106, referer: http://192.168.2.8/ucp/?display=dashboard

At first I tried commenting out that line in the file the error refers to which led to similar errors at other lines in the same file as well as in a different file. I know this is probably not a good idea in the long run but I wanted to see if that impacted the crashing. The errors eventually went away after commenting out all of the lines they referenced however the crashing problem did not. I removed the comments to begin additional troubleshooting which is when I noticed that the top of the UCP would always say 'Connecting...' after logging in and it would stay like that. I then tried changing the presence settings in the UCP and BAM, the problem went away! Just FYI, I set the 'upon login' and 'upon logout' presence settings to 'Available' in case you're wondering however I think you can set them to anything you want and it will likely resolve the issue. I feel that this variable isn't set like it should be without making this change first as I was able to set it back to 'Do nothing' like it originally was after setting it to 'Available' and still no crash (I had to logout and log back in with the 'Available' setting enabled first). The 'PresenceState' error I referenced above also went away.

Long story short, I changed all of the active users/extensions to 'Available' for both login/logout and left them that way, no more crashing. Soooo happy to have that squared away

I thought I would share this information as I'm sure this will be a sigh of relief for the handful of folks experiencing this issue and hopefully provide some insight on permanently resolving the problem in future releases.

Thanks Ward and everyone else for your input!