Debugging Memory Problems

[dbaum]

I am running Purple release on the Acer Revo Box recommended in Nerd Vittles. I contnue to see creep in the used App Memory. We have a total of three users on the system and peering with threee service providers. The only thing that seems to help is rebooting the host system. Is anyone else seeing this? Anyone have a fix?

 

[wardmundy]

Linux manages memory and will use all it can get its hands on for caching if not needed for other purposes. So... this is not necessarily a bug. If your server crashes as a result of the memory creep, then that's a problem. You'll recognize the difference.

 

[dswartz]

the easiest way to tell if you have (or rather, don't have) a problem: get a shell and do 'free -m'. on mine it looks like:

root@pbx:~ $ free -m
total used free shared buffers cached
Mem: 2026 1383 643 0 808 299
-/+ buffers/cache: 275 1750
Swap: 760 0 760

If the increasing usage is in the 'cache' section, it is not an issue.

 

[dbaum]

Rebooted - here's data

total used free shared buffers cached

Mem: 883 468 414 0 25 194
-/+ buffers/cache: 249 633
Swap: 760 0 760

PIAF shows app mem at 29%

Will check later today for values.

PS Ward, I know Linux managed the memory, but this was NOT experienced with last version running under Centos 5.5 32 bit. Only happened since purple release was loaded

 

[wardmundy]

There could very well be memory leaks in Asterisk 1.8.0. Wouldn't surprise me in a .0 product release. Just need some solid evidence. That's all.

 

[dbaum]

Since App memory is now reaching 91% we also see Swap memory climbing and it hits 89%. Having to reboot is annoying.

 

[darmock]

I have had my purple system in production since before we released it to the public and it does not behave as yours is doing. Dual core dell something with 4gb ram / 200gb hd etc. One I had laying around. Also has some digium cards in it that were donated by digium (thanks) . This is running as my office pbx. I also have a couple of other boxes running purple on long term testing in the lab none of which shows a similar problem.
Sorry you are having this problem but I couldn't begin to tell you where to look. Does the box fail perform when your app memory and swap memory is high? Or are you rebooting just because free -m shows a lot of use? What does top show during one of these periods you are talking about? Perhaps if you were to attach a copy of the full version of statustofile that might provide some clues

Tom

 

[bmore]

Since App memory is now reaching 91% we also see Swap memory climbing and it hits 89%. Having to reboot is annoying.

Not a fix but a potential workaround for you...

To save power on my pbx I run a cron job each night that shuts down the computer.

shutdown -h now

The computer powers up back automatically each morning using the Bios RTC setting to power on each day at the same time. If you have a service that calls go to voice mail or forward to another number, the alternate handling kicks in... In my case you get busy... You can either wait until morning or if its an emergency you probably have my cell number.

In your case you could shutdown for say 5 minutes and then power back on. Run the cron job at say 11: 59 pm, set the bios to power on at 12:04 am.

Not necessary for you but I also use Smart Strip LCG3 Energy Saving Surge Protector to power off all the ancillary devices. Note that this may not work with a real low power computer, it requires a minimum difference between power drawn when on versus when off.

 

[dbaum]

Tom,
The box hit 89% app mem and 81% SWAP (from FreePBX stsus screen) last night and call processing was not functional. All calls (internal and external) went to VM defined as last termination location in dialplan.
I rebooted 11 hours ago (unfortunately forgetting to run statustofile first). Call processing was restored.
11 hours later with no calls processed/attempted APP memory is at 60% and SWAP 0%. Current statustofile.txt is attached.


I will run it when higher levels are reached

 

[dbaum]

Thanks, that's food for thought

 

[blanchae]

Have you tried "top" and checked to see what is hogging your memory? Add swap size with the commands f then p.

 

[dbaum]

Eugene, yes I have and it is the process associated with command "asterisk"

 

[jroper]

Hi

You may get more idea of what is happening by turning up the verbosity, and watching the full log.

rasterisk -vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
tail -f /var/log/asterisk/full

This may show something not quite right.

Joe

 

[dbaum]

Joe,
Thank for your suggestion
The following series of messages repeat constantly

[2010-12-03 15:14:37] ERROR[3111] res_jabber.c: JABBER: encryption failure. possible bad password.
[2010-12-03 15:14:37] ERROR[3111] res_jabber.c: aji_act_hook was called with out a packet
[2010-12-03 15:14:37] WARNING[3111] res_jabber.c: Parsing failure: Hook returned an error.
[2010-12-03 15:14:37] WARNING[3111] res_jabber.c: JABBER: Got hook event.
[2010-12-03 15:14:37] WARNING[3111] res_jabber.c: JABBER: socket read error

I didn't know we used JABBER in asterisk or Free PBX or PIAF.

When I go to WEBMIN and look at active processes i get the data shown in attached file.

 

[darmock]

in /etc/asterisk/modules.conf

ensure that under autoload=yes

that you have

noload = res_jabber.so
noload = chan_gtalk.so

By default purple installs jabber and gtalk. It is disabled by default however if you ran incredible pbx it turns it on and you have to configure the incredible with your username and password for your account. I suspect you have not done this. If you are not using google voice trunks then this will turn it off.

Tom

 

[dbaum]

I have not configured these trunks, but they are disabled in Free PBX. I assume from your note that disabling them there is not sufficient and that I must modify the configuration files as you suggest. May I inquire where this is documented - I must have missed it.

 

[darmock]

Where is it documented? Good question. However if you modify the file the problem will go away how does that sound. I am the guy that wrote the install program for purple so you *CAN* take my word for it.

Just because they are disabled in freepbx really means nothing as freepbx is a *shell* around asterisk.

enjoy

Tom

 

[robf]

Is it possible that something else is listening on the jabber port?
The socket read error made me think of this.

FWIW, My purple build has been running fine since I built it a couple of weeks ago.

Obviously your asterisk process is using significant RAM.

If you're not using GV.. (outbound is broken at this writing) you could unselect with make menuselect jabber, chan_gtalk and and re make.
Not sure thats an option.

did you do as darmock suggested and choose not to load jabber and chan_gtalk? If that works its obviously easier.

This may be WAY off base but my gigabit ethernet eth0 was a bit flaky (skge) so I've been running on my eth1 which is an nforce 100mbit. Likely an irrelevant detail.

I'm hoping you get it resolved soon.

A top from my system sorted by resident memory consumption.

 

[dbaum]

Tom, I wasn't complaining, I just wanted to learn from my mistakes. I am visually impaired so sometimes I miss things in documentation. I have made the change and the messages have gone away. I will monitor the APP MEM and SWAP percentages.

 

[darmock]

No worries I understand. Documentation always lags behind product I am afraid. When I get paid to do windows projects I actually hire a scribe to build the docs while the programming staff is coding. Unfortunately since this is a non paying project that gets put by the wayside at times. There has been some movement to help out with docs and some have contributed! <Thanks> I still am working on getting all of it in shape prior to posting it.

Unfortunately the error you have was never codified in the docs.


Tom