TRY THIS Incredible PBX Ram Question/Issue

[M_O_P]

Setup:

Incredible PBX 13-13 (pre-gvsip) Enchilada
Installed gvsip
Cloud Based Setup, Centos 6.10 (6.9 originally), 1GB Ram. 0.5GB Swap. OpenVZ

Note: Ram usage below is actual % (no cache included)

Issue:

After my most recent install of gvsip + asterisk upgrade (13.22) my system ram has become unstable. After reboot, ram starts around 16% and then jumps to approx 22% (on first launch of gui). Eventually it settles around 35% and seems to stay there for quite some time. It seems to last for a day or so and then the ram usage gets wonky. It will start rising to around 75% and then heavy swap usage kicks in. Swap eventually maxes out, and then ram will climb to 95%+ at which time I reboot so as to avoid oom. I have never let it fully crashout.

What I have noticed is that when the memory increases it staircases up on the :29 of every hour. At the same time. I have these entries in my process list with the same time stamp:

php /var/lib/asterisk/bin/module_admin listonline
php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

Not sure why these tasks would be persistent and not released from memory when finished???

Any suggestions? Something definitely changed as it never maxed out like it is now.

Thanks,
M_O_P

-------------------------------------------------------------------------

My “ps aux --sort -rss” :

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 1320 0.3 5.8 1599292 61160 ? Sl Oct18 12:39 /usr/sbin/asterisk -f -vvvg -c

asterisk 3566 0.0 4.3 322716 45260 ? S 14:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 7048 0.0 4.3 322716 45260 ? S 13:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 25286 0.0 4.3 322716 45260 ? S 08:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 10933 0.0 4.3 322716 45256 ? S 12:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 14199 0.0 4.3 322716 45256 ? S 11:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 21405 0.0 4.3 322716 45256 ? S 09:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 28580 0.0 4.3 322716 45256 ? S 07:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 17424 0.0 3.7 322716 39424 ? S 10:29 0:00 php /var/lib/asterisk/bin/module_admin listonline

asterisk 7002 0.0 3.2 309604 33940 ? Ss 13:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 10885 0.0 3.2 309604 33940 ? Ss 12:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 14151 0.0 3.2 309604 33940 ? Ss 11:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 3517 0.0 3.2 309604 33936 ? Ss 14:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 17378 0.0 3.2 309604 33936 ? Ss 10:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 21356 0.0 3.2 309604 33936 ? Ss 09:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 28511 0.0 3.2 309604 33936 ? Ss 07:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 25226 0.0 3.2 309604 33932 ? Ss 08:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

asterisk 23298 0.0 2.5 457016 26832 ? S Oct18 0:01 /usr/sbin/httpd

asterisk 2348 0.0 1.7 439432 18804 ? S Oct18 0:02 /usr/sbin/httpd

asterisk 15565 0.0 1.6 434780 16784 ? S Oct19 0:00 /usr/sbin/httpd

asterisk 15543 0.0 1.4 434328 15324 ? S Oct19 0:00 /usr/sbin/httpd

mysql 1168 0.1 1.1 610908 12268 ? Sl Oct18 3:24 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid

asterisk 31738 0.0 0.9 471316 9944 ? S Oct19 0:01 /usr/sbin/httpd

asterisk 22999 0.0 0.7 322716 7600 ? S Oct19 0:01 php /var/lib/asterisk/bin/module_admin listonline

root 24479 0.0 0.6 697400 7020 ? Sl 00:15 0:17 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock

asterisk 22918 0.0 0.6 309604 6872 ? Ss Oct19 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

root 1265 0.0 0.4 325704 4664 ? Ss Oct18 0:02 /usr/sbin/httpd

root 1279 0.0 0.4 10364 4240 ? Ss Oct18 0:00 /usr/sbin/knockd -d -i venet0:0

root 28292 0.0 0.4 98012 4208 ? Ss 16:28 0:00 sshd: root@pts/4

root 692 0.0 0.3 183944 3824 ? Sl Oct18 0:06 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5

root 2608 0.0 0.3 188584 3296 ? Sl Oct18 0:05 /usr/bin/nrservice

root 809 0.0 0.2 92656 2104 ? Ss Oct18 0:00 NetworkManager --pid-file=/var/run/NetworkManager/NetworkManager.pid

root 2709 0.0 0.1 94636 2040 ? Ss Oct18 0:02 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf

root 28390 0.0 0.1 11608 1732 pts/4 Ss 16:28 0:00 -bash

root 3512 0.0 0.1 38808 1536 ? S 14:29 0:00 CROND

root 6994 0.0 0.1 38808 1536 ? S 13:29 0:00 CROND

root 10881 0.0 0.1 38808 1536 ? S 12:29 0:00 CROND

root 14144 0.0 0.1 38808 1536 ? S 11:29 0:00 CROND

root 17370 0.0 0.1 38808 1536 ? S 10:29 0:00 CROND

root 21351 0.0 0.1 38808 1536 ? S 09:29 0:00 CROND

root 25221 0.0 0.1 38808 1536 ? S 08:29 0:00 CROND

root 28505 0.0 0.1 38808 1536 ? S 07:29 0:00 CROND

root 28812 0.0 0.1 38808 1536 ? S 16:29 0:00 CROND

root 870 0.0 0.1 92292 1532 ? Ss Oct18 0:00 cupsd -C /etc/cups/cupsd.conf

root 1 0.0 0.1 19236 1168 ? Ss Oct18 0:00 init

asterisk 17422 0.0 0.1 106100 1156 ? S 10:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 21403 0.0 0.1 106100 1156 ? S 09:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 25283 0.0 0.1 106100 1156 ? S 08:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

496 28818 0.0 0.1 9236 1156 ? S 16:29 0:00 bash /etc/hetrixtools/hetrixtools_agent.sh

asterisk 3564 0.0 0.1 106100 1152 ? S 14:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 7046 0.0 0.1 106100 1152 ? S 13:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 10931 0.0 0.1 106100 1152 ? S 12:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 14196 0.0 0.1 106100 1152 ? S 11:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

asterisk 28578 0.0 0.1 106100 1152 ? S 07:29 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

root 1060 0.0 0.1 11344 1128 ? S Oct18 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr

root 1370 0.0 0.1 13808 1108 ? S Oct18 0:04 /usr/libexec/gam_server

root 941 0.0 0.1 386164 1096 ? Ssl Oct18 0:14 automount --pid-file /var/run/autofs.pid

root 1323 0.0 0.1 20064 1060 ? Ss Oct18 0:02 crond

496 28816 0.0 0.1 9236 1060 ? Ss 16:29 0:00 /bin/sh -c bash /etc/hetrixtools/hetrixtools_agent.sh >> /etc/hetrixtools/hetrixtools_cron.log 2>&1

root 29879 0.0 0.1 13508 1060 pts/4 R+ 16:29 0:00 ps aux --sort -rss

root 2606 0.0 0.0 11340 1040 ? S Oct18 0:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/nrservice

68 892 0.0 0.0 37852 1012 ? Ssl Oct18 0:00 hald

root 22913 0.0 0.0 38808 1012 ? S Oct19 0:00 CROND

asterisk 22996 0.0 0.0 106100 1012 ? S Oct19 0:00 sh -c /var/lib/asterisk/bin/module_admin listonline > /dev/null 2>&1

root 1231 0.0 0.0 86240 976 ? Ss Oct18 0:00 /usr/sbin/abrtd

root 1254 0.0 0.0 83868 968 ? Ss Oct18 0:01 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages

root 1209 0.0 0.0 86920 952 ? Ss Oct18 0:02 sendmail: accepting connections

root 814 0.0 0.0 58048 916 ? S Oct18 0:00 /usr/sbin/modem-manager

daemon 742 0.0 0.0 23536 840 ? Ss Oct18 0:03 /usr/sbin/slpd

uucp 1461 0.0 0.0 18188 808 ? S Oct18 0:00 /usr/sbin/iaxmodem

uucp 1462 0.0 0.0 18188 808 ? S Oct18 0:00 /usr/sbin/iaxmodem

uucp 1463 0.0 0.0 18188 808 ? S Oct18 0:00 /usr/sbin/iaxmodem

uucp 1464 0.0 0.0 18188 808 ? S Oct18 0:00 /usr/sbin/iaxmodem

root 893 0.0 0.0 20400 720 ? S Oct18 0:00 hald-runner

smmsp 1218 0.0 0.0 78372 684 ? Ss Oct18 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

496 29877 0.0 0.0 6260 648 ? S 16:29 0:00 vmstat 3 2

root 1012 0.0 0.0 21716 596 ? Ss Oct18 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid

uucp 2585 0.0 0.0 68652 584 ? Ss Oct18 0:00 /usr/sbin/faxgetty -D ttyIAX0

uucp 2591 0.0 0.0 68652 584 ? Ss Oct18 0:00 /usr/sbin/faxgetty -D ttyIAX2

uucp 2594 0.0 0.0 68652 584 ? Ss Oct18 0:00 /usr/sbin/faxgetty -D ttyIAX3

uucp 2588 0.0 0.0 68652 580 ? Ss Oct18 0:00 /usr/sbin/faxgetty -D ttyIAX1

uucp 1427 0.0 0.0 66152 572 ? Ss Oct18 0:00 hfaxd -i hylafax

root 2601 0.0 0.0 11608 552 ? S Oct18 0:00 /bin/bash /etc/rc3.d/S99nrservice.sh start

root 722 0.0 0.0 18252 540 ? Ss Oct18 0:01 irqbalance --pid=/var/run/irqbalance.pid

496 29876 0.0 0.0 9236 524 ? S 16:29 0:00 bash /etc/hetrixtools/hetrixtools_agent.sh

496 29878 0.0 0.0 4104 516 ? S 16:29 0:00 tail -1

root 1001 0.0 0.0 66292 508 ? Ss Oct18 0:00 /usr/sbin/sshd

uucp 1423 0.0 0.0 68592 504 ? Ss Oct18 0:00 faxq

root 2712 0.0 0.0 4064 468 tty1 Ss+ Oct18 0:00 /sbin/mingetty console

root 2713 0.0 0.0 4064 468 tty2 Ss+ Oct18 0:00 /sbin/mingetty tty2

root 1316 0.0 0.0 11340 464 ? S Oct18 0:00 /bin/sh /usr/sbin/safe_asterisk

dbus 796 0.0 0.0 21552 452 ? Ss Oct18 0:00 dbus-daemon --system

root 763 0.0 0.0 13396 408 ? Ss Oct18 0:03 lldpad -d

root 1402 0.0 0.0 21108 316 ? Ss Oct18 0:00 /usr/sbin/atd

root 1454 0.0 0.0 16052 300 ? Ss Oct18 0:00 /usr/sbin/iaxmodem

root 1187 0.0 0.0 66560 284 ? Ss Oct18 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2

root 871 0.0 0.0 45004 280 ? Ss Oct18 0:00 /usr/sbin/wpa_supplicant -c /etc/wpa_supplicant/wpa_supplicant.conf -B -u -f /var/log/wpa_supplicant.log -P /var/run/wpa_supplicant.pid

root 152 0.0 0.0 10668 264 ? S<s Oct18 0:00 /sbin/udevd -d

root 1189 0.0 0.0 66560 32 ? S Oct18 0:00 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam -n 2

root 2 0.0 0.0 0 0 ? S Oct18 0:00 [kthreadd/7071]

root 3 0.0 0.0 0 0 ? S Oct18 0:00 [khelper/7071]

EOL

 

[atsak]

Is fail2ban reading a log that's largish to ban or something like that? I always provision with 2GB. Find 1GB has "weird stuff" like this.

 

[M_O_P]

My fail2ban logs don't show anything unsual, vps is pretty locked down and jails are empty.

 

[M_O_P]

Ram at critical levels again.

total used free shared buffers cached
Mem: 1024 1022 1 3 0 20
-/+ buffers/cache: 1002 21
Swap: 512 512 0
WARNING: Always run Incredible PBX behind a secure hardware-based firewall.

 

[M_O_P]

It's these items using the bulk...

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6788 0.6 4.2 1589528 44336 ? Sl 09:43 0:01 /usr/sbin/asterisk -f -vvvg -c
asterisk 2764 0.0 3.6 322716 38292 ? S Oct22 0:02 php /var/lib/asterisk/bin/module_admin listonline
asterisk 12215 0.0 3.6 322716 38292 ? S 08:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 15204 0.0 3.6 322716 38292 ? S 07:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 18361 0.0 3.6 322716 38292 ? S 06:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 21822 0.0 3.6 322716 38292 ? S 05:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 6159 0.0 3.6 322716 38288 ? S Oct22 0:02 php /var/lib/asterisk/bin/module_admin listonline
asterisk 9342 0.0 3.6 322716 38288 ? S Oct21 0:02 php /var/lib/asterisk/bin/module_admin listonline
asterisk 28471 0.0 3.6 322716 38288 ? S 03:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 31529 0.0 3.6 322716 38288 ? S Oct22 0:02 php /var/lib/asterisk/bin/module_admin listonline
asterisk 8545 0.0 3.6 322460 38024 ? S 09:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 16752 0.0 3.5 322716 37264 ? S Oct21 0:03 php /var/lib/asterisk/bin/module_admin listonline
asterisk 2716 0.0 2.6 309604 27460 ? Ss Oct22 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 8496 0.0 2.6 309604 27460 ? Ss 09:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 12147 0.0 2.6 309604 27460 ? Ss 08:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 6106 0.0 2.6 309604 27456 ? Ss Oct22 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 9293 0.0 2.6 309604 27456 ? Ss Oct21 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 15156 0.0 2.6 309604 27456 ? Ss 07:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 18312 0.0 2.6 309604 27456 ? Ss 06:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 21774 0.0 2.6 309604 27456 ? Ss 05:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 31478 0.0 2.6 309604 27456 ? Ss Oct22 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 28418 0.0 2.6 309604 27452 ? Ss 03:29 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 25211 0.0 2.1 322716 23044 ? S 04:29 0:00 php /var/lib/asterisk/bin/module_admin listonline
asterisk 13095 0.0 1.8 309604 19056 ? Ss Oct21 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php
asterisk 16702 0.0 1.7 309604 18744 ? Ss Oct21 0:00 php /var/lib/asterisk/bin/freepbx-cron-scheduler.php

And my swap is this:
(Note: These are all the same types of processes just push out of main memory.)
grep VmSwap /proc/*/status | sort -n -r --key=2.1 | head -10
/proc/27618/status:VmSwap: 37200 kB
/proc/31282/status:VmSwap: 36928 kB
/proc/6866/status:VmSwap: 36504 kB
/proc/20191/status:VmSwap: 36236 kB
/proc/24552/status:VmSwap: 36140 kB
/proc/3003/status:VmSwap: 36068 kB
/proc/29810/status:VmSwap: 35892 kB
/proc/13141/status:VmSwap: 30204 kB
/proc/6762/status:VmSwap: 26340 kB
/proc/31241/status:VmSwap: 26280 kB

 

[wardmundy]

Try this. In Settings -> Advanced Settings, set Update Notifications to NO.

 

[M_O_P]

Here is an update.

total used free shared buffers cached
Mem: 1024 454 569 3 0 286
-/+ buffers/cache: 168 855
Swap: 512 28 483


It has been stable for over a week now with Ward's suggested fix!