1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. If you had a PIAF Forum account in the vBulletin days, log in with your old credentials. Otherwise, sign up again and we'll get you back in business as soon as we can.
  3. Guest: We think the problem with locked threads from long message subjects has been resolved. Post a link here if you still see a problem.

Debugging Memory Problems

Discussion in 'Bug Reporting and Fixes' started by dbaum, Nov 19, 2010.

  1. dbaum Guru

    I am running Purple release on the Acer Revo Box recommended in Nerd Vittles. I contnue to see creep in the used App Memory. We have a total of three users on the system and peering with threee service providers. The only thing that seems to help is rebooting the host system. Is anyone else seeing this? Anyone have a fix?
  2. wardmundy Nerd Uno

    Linux manages memory and will use all it can get its hands on for caching if not needed for other purposes. So... this is not necessarily a bug. If your server crashes as a result of the memory creep, then that's a problem. You'll recognize the difference. :wink5:
  3. dswartz Guru

    the easiest way to tell if you have (or rather, don't have) a problem: get a shell and do 'free -m'. on mine it looks like:

    root@pbx:~ $ free -m
    total used free shared buffers cached
    Mem: 2026 1383 643 0 808 299
    -/+ buffers/cache: 275 1750
    Swap: 760 0 760

    If the increasing usage is in the 'cache' section, it is not an issue.
  4. dbaum Guru

    Rebooted - here's data

    total used free shared buffers cached

    Mem: 883 468 414 0 25 194
    -/+ buffers/cache: 249 633
    Swap: 760 0 760

    PIAF shows app mem at 29%

    Will check later today for values.

    PS Ward, I know Linux managed the memory, but this was NOT experienced with last version running under Centos 5.5 32 bit. Only happened since purple release was loaded
  5. wardmundy Nerd Uno

    There could very well be memory leaks in Asterisk 1.8.0. Wouldn't surprise me in a .0 product release. Just need some solid evidence. That's all. ;)
  6. dbaum Guru

    Since App memory is now reaching 91% we also see Swap memory climbing and it hits 89%. Having to reboot is annoying.
  7. darmock PIAF Developer

    I have had my purple system in production since before we released it to the public and it does not behave as yours is doing. Dual core dell something with 4gb ram / 200gb hd etc. One I had laying around. Also has some digium cards in it that were donated by digium (thanks) . This is running as my office pbx. I also have a couple of other boxes running purple on long term testing in the lab none of which shows a similar problem.
    Sorry you are having this problem but I couldn't begin to tell you where to look. Does the box fail perform when your app memory and swap memory is high? Or are you rebooting just because free -m shows a lot of use? What does top show during one of these periods you are talking about? Perhaps if you were to attach a copy of the full version of statustofile that might provide some clues

    Tom
  8. bmore Guru

    Not a fix but a potential workaround for you...

    To save power on my pbx I run a cron job each night that shuts down the computer.
    Code:
    shutdown -h now
    The computer powers up back automatically each morning using the Bios RTC setting to power on each day at the same time. If you have a service that calls go to voice mail or forward to another number, the alternate handling kicks in... In my case you get busy... You can either wait until morning or if its an emergency you probably have my cell number.

    In your case you could shutdown for say 5 minutes and then power back on. Run the cron job at say 11: 59 pm, set the bios to power on at 12:04 am.

    Not necessary for you but I also use Smart Strip LCG3 Energy Saving Surge Protector to power off all the ancillary devices. Note that this may not work with a real low power computer, it requires a minimum difference between power drawn when on versus when off.
  9. dbaum Guru

    Tom,
    The box hit 89% app mem and 81% SWAP (from FreePBX stsus screen) last night and call processing was not functional. All calls (internal and external) went to VM defined as last termination location in dialplan.
    I rebooted 11 hours ago (unfortunately forgetting to run statustofile first). Call processing was restored.
    11 hours later with no calls processed/attempted APP memory is at 60% and SWAP 0%. Current statustofile.txt is attached.


    I will run it when higher levels are reached
  10. dbaum Guru

    Thanks, that's food for thought
  11. blanchae Guru

    Have you tried "top" and checked to see what is hogging your memory? Add swap size with the commands f then p.
  12. dbaum Guru

    Eugene, yes I have and it is the process associated with command "asterisk"
  13. jroper Guru

    Hi

    You may get more idea of what is happening by turning up the verbosity, and watching the full log.

    Code:
    rasterisk -vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
    tail -f /var/log/asterisk/full
    This may show something not quite right.

    Joe
  14. dbaum Guru

    Joe,
    Thank for your suggestion
    The following series of messages repeat constantly

    [2010-12-03 15:14:37] ERROR[3111] res_jabber.c: JABBER: encryption failure. possible bad password.
    [2010-12-03 15:14:37] ERROR[3111] res_jabber.c: aji_act_hook was called with out a packet
    [2010-12-03 15:14:37] WARNING[3111] res_jabber.c: Parsing failure: Hook returned an error.
    [2010-12-03 15:14:37] WARNING[3111] res_jabber.c: JABBER: Got hook event.
    [2010-12-03 15:14:37] WARNING[3111] res_jabber.c: JABBER: socket read error

    I didn't know we used JABBER in asterisk or Free PBX or PIAF.

    When I go to WEBMIN and look at active processes i get the data shown in attached file.

    Attached Files:

  15. darmock PIAF Developer

    in /etc/asterisk/modules.conf

    ensure that under autoload=yes

    that you have

    noload = res_jabber.so
    noload = chan_gtalk.so

    By default purple installs jabber and gtalk. It is disabled by default however if you ran incredible pbx it turns it on and you have to configure the incredible with your username and password for your account. I suspect you have not done this. If you are not using google voice trunks then this will turn it off.

    Tom
  16. dbaum Guru

    I have not configured these trunks, but they are disabled in Free PBX. I assume from your note that disabling them there is not sufficient and that I must modify the configuration files as you suggest. May I inquire where this is documented - I must have missed it.
  17. darmock PIAF Developer

    Where is it documented? Good question. However if you modify the file the problem will go away how does that sound. I am the guy that wrote the install program for purple so you *CAN* take my word for it.

    Just because they are disabled in freepbx really means nothing as freepbx is a *shell* around asterisk.

    enjoy

    Tom
  18. robf Member

    Is it possible that something else is listening on the jabber port?
    The socket read error made me think of this.

    FWIW, My purple build has been running fine since I built it a couple of weeks ago.

    Obviously your asterisk process is using significant RAM.

    If you're not using GV.. (outbound is broken at this writing) you could unselect with make menuselect jabber, chan_gtalk and and re make.
    Not sure thats an option.

    did you do as darmock suggested and choose not to load jabber and chan_gtalk? If that works its obviously easier.

    This may be WAY off base but my gigabit ethernet eth0 was a bit flaky (skge) so I've been running on my eth1 which is an nforce 100mbit. Likely an irrelevant detail.

    I'm hoping you get it resolved soon.

    A top from my system sorted by resident memory consumption.
    [IMG]
  19. dbaum Guru

    Tom, I wasn't complaining, I just wanted to learn from my mistakes. I am visually impaired so sometimes I miss things in documentation. I have made the change and the messages have gone away. I will monitor the APP MEM and SWAP percentages.
  20. darmock PIAF Developer

    No worries I understand. Documentation always lags behind product I am afraid. When I get paid to do windows projects I actually hire a scribe to build the docs while the programming staff is coding. Unfortunately since this is a non paying project that gets put by the wayside at times. There has been some movement to help out with docs and some have contributed! <Thanks> I still am working on getting all of it in shape prior to posting it.

    Unfortunately the error you have was never codified in the docs.


    Tom

Share This Page