Incoming calls go straight to voicemail with high memory usage

[mattsl]

A couple things first:

Yes, I know this has been discussed before, but I have read every thread I can find on the topic and have yet to find a solution. If there is one, please ignore everything else in the post, and link to it. Yes, I know how Linux likes to cache files in RAM and that high memory usage isn't a problem unless something stops working; something has stopped working.


I have seen numerous users with this issue, and now I have having it. Things that I know:

My memory usage used to be lower, hovering around 50% or so when leaving the system running for months. Now this climbs so that within less than 24hrs it is in the 90s. At some point during this process, the incoming call routing ceases to functions and all calls go directly to voicemail.

I do use Google Voice, so disabling those functions is not a viable solution for me.

I recently performed module upgrades and installed SSL. Given that my system ran fine for months and this problem occurred shortly thereafter, I would suspect that there is something in one of those upgrades that is causing the problem.

Unfortunately, the problem did not present itself immediately after the upgrades, and I don't have a full backup to which I can rollback.


Answers that would be of great help to me in troubleshooting this so that we can hopefully all squash this bug permanently rather than have to decipher whether a new thread is due to a premature freakout at seeing a red bar or is actually a real problem:

Where can I find information that will provide myself and users here with detailed data regarding what, exactly, is eating up that RAM?

Is there an easy way to roll the modules back so that I can see if this is the issue and if so go through one by one?

Is there any other information I can provide that will help in troubleshooting this issue?

Also, just out of curiosity, I decided to play with Proxmox's ability to reallocate RAM in real time. When I decreased the memory size gradually, incoming calls continued to work at 448 and 384 which gave 68% and 72% usage respectively. However, when I dropped it to 256, which pushed usage to 84%, incoming calls died.


Thanks in advance for the help!

Matt

PS -It told me that my message size was too large. I will follow up immediately after this post with my specs, etc.

 

[mattsl]

System info

Hardware:

Proxmox OpenVZ container allocated 512MB memory and 512MB swap, 8GB HD, on an i3 processor.

Output from status:

Asterisk = ONLINE | Zap/Dahdi = UNKNOWN | MySQL = ONLINE │
│ SSH = ONLINE | Apache = ONLINE | Iptables = OFFLINE │
│ Fail2ban = ONLINE | Internet = ONLINE | Ip6Tables = OFFLINE │
│ BlueTooth = OFFLINE | Hidd = OFFLINE | NTPD = OFFLINE │
│ SendMail = ONLINE | Samba = OFFLINE | Webmin = ONLINE │
│ Ethernet0 = OFFLINE | Ethernet1 = OFFLINE | Wlan0 = N/A │
│ PBX in a Flash Version = 1.7.5.5 Running on *HARDWARE* │
│ FreePBX Version = 2.8.1.4 │
│ Running Asterisk Version = 1.8.6.0 │
│ Asterisk Source Version = 1.8.6.0 │
│ Zap/Dahdi Source Version = N/A │
│ Libpri Source Version = 1.4.11.4 │
│ IP Address = xxx.xx.xx.xxx on eth0 │
│ Operating System = CentOS release 5.7 (Final) │
│ Kernel Version = 2.6.32-4-pve - 64 Bit

Output from top:

top - 08:29:05 up 4:50, 1 user, load average: 0.02, 0.01, 0.00
Tasks: 36 total, 1 running, 35 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 0.0%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1048576k total, 661268k used, 387308k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1718 asterisk 20 0 302m 28m 4508 S 1.3 2.8 0:08.14 httpd
1257 asterisk 20 0 849m 148m 9060 S 0.7 14.5 1:33.38 asterisk
1345 asterisk 20 0 45252 12m 1896 S 0.3 1.3 0:24.88 op_server.pl
1 root 20 0 10364 680 568 S 0.0 0.1 0:00.12 init
29 root 20 0 100 16 4 S 0.0 0.0 0:00.16 init-logger
79 root 16 -4 12632 708 356 S 0.0 0.1 0:00.00 udevd
213 root 20 0 32728 484 364 S 0.0 0.0 0:00.00 brcm_iscsiuio
519 root 20 0 5924 616 492 S 0.0 0.1 0:00.02 syslogd
625 root 20 0 62652 1216 652 S 0.0 0.1 0:00.03 sshd
633 root 20 0 78356 2644 1740 S 0.0 0.3 0:00.00 cupsd
641 root 20 0 21656 904 696 S 0.0 0.1 0:00.00 xinetd
684 root 20 0 11972 1312 1088 S 0.0 0.1 0:00.00 mysqld_safe
734 mysql 20 0 169m 24m 3908 S 0.0 2.4 0:04.72 mysqld
766 root 20 0 66960 2328 788 S 0.0 0.2 0:00.34 sendmail
774 smmsp 20 0 57712 1752 596 S 0.0 0.2 0:00.00 sendmail
790 root 20 0 282m 12m 6408 S 0.0 1.2 0:00.29 httpd
806 root 20 0 20884 1176 580 S 0.0 0.1 0:00.02 crond
828 xfs 20 0 20260 1168 740 S 0.0 0.1 0:00.00 xfs
843 root 20 0 222m 5408 1664 S 0.0 0.5 0:12.34 fail2ban-server
845 root 20 0 13064 1248 980 S 0.0 0.1 0:01.48 gam_server
911 root 20 0 18744 448 296 S 0.0 0.0 0:00.00 atd
923 root 20 0 46756 524 328 S 0.0 0.0 0:00.00 saslauthd
924 root 20 0 46756 296 100 S 0.0 0.0 0:00.00 saslauthd
956 root 39 19 197m 15m 2008 S 0.0 1.5 0:00.01 yum-updatesd
1255 root 20 0 11972 612 384 S 0.0 0.1 0:00.00 safe_asterisk
1342 asterisk 20 0 11972 564 308 S 0.0 0.1 0:00.00 bash
1343 asterisk 20 0 9876 1136 952 S 0.0 0.1 0:00.00 sh
1357 root 20 0 18428 480 276 S 0.0 0.0 0:00.00 smartd
1362 root 20 0 79436 14m 1852 S 0.0 1.4 0:00.18 miniserv.pl
1720 asterisk 20 0 291m 18m 3460 S 0.0 1.8 0:00.13 httpd
1721 asterisk 20 0 287m 14m 3548 S 0.0 1.4 0:04.91 httpd
1722 asterisk 20 0 304m 30m 4524 S 0.0 3.0 0:04.21 httpd
1724 asterisk 20 0 291m 18m 3548 S 0.0 1.8 0:04.73 httpd
2379 root 20 0 90156 3408 2660 S 0.0 0.3 0:00.03 sshd
2381 root 20 0 12204 1660 1228 S 0.0 0.2 0:00.01 bash
5638 root 20 0 12624 1036 820 R 0.0 0.1 0:00.00 top

Output from free:
total used free shared buffers cached
Mem: 1048576 662716 385860 0 0 0
-/+ buffers/cache: 662716 385860
Swap: 0 0 0
0 0

 

[mattsl]

I don't know if this will be helpful, but I intentionally broke this, by lowering memory allocation in Proxmox, to see if I could track down what exactly was failing. I placed one incoming call before breaking it and one incoming call after breaking it. The difference occurred here:

Working:
-- Executing [s@macro-dial:3] AGI("SIP/myinboundroute-00000009", "dialparties.agi") in new stack
-- Launched AGI Script /var/lib/asterisk/agi-bin/dialparties.agi
dialparties.agi: Starting New Dialparties.agi

Broken:
-- Executing [s@macro-dial:3] AGI("SIP/myinboundroute-0000000b", "dialparties.agi") in new stack
-- Executing [s@macro-dial:4] NoOp("SIP/myinboundroute-0000000b", "Returned from dialparties with no extensions to call and DIALSTATUS: ") in new stack

 

[dad311]

I had this exact issue last week. I changed my Google password and forgot to update the Google password in the PBX. Memory went to 90%, calls started going to Voicemail.

I fixed the password issue, now im just waiting to see if the problem returns.

 

[mattsl]

Thanks dad! I have two Google Voice accounts registered in my server, and this is exactly what happened. I'm assuming this comes about for exactly the same reason that users have to disable those plug-ins if they don't use GV. Great catch! I will also leave this running for a few days to ensure that I don't have any further issues.

On a related note: When I tried to update the password for the GV account through FreePBX, it would not work. I'd submit changes and then it would give me the orange reload bar, but would show the old password again. I tired the reload anyway and no dice. I also tried changing it in the box again before hitting reload and still nothing. I had to delete and recreate the trunk to get it to let me change the password. Obviously not ideal.

 

[dad311]

Thanks dad! I have two Google Voice accounts registered in my server, and this is exactly what happened. I'm assuming this comes about for exactly the same reason that users have to disable those plug-ins if they don't use GV. Great catch! I will also leave this running for a few days to ensure that I don't have any further issues.

On a related note: When I tried to update the password for the GV account through FreePBX, it would not work. I'd submit changes and then it would give me the orange reload bar, but would show the old password again. I tired the reload anyway and no dice. I also tried changing it in the box again before hitting reload and still nothing. I had to delete and recreate the trunk to get it to let me change the password. Obviously not ideal.

Next time just edit /etc/asterisk/jabber.conf to update your google password.

 

[phinphan]

Thanks for the tip. I had this same problem. Fixed jabber.conf and rebooted.