1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.
  2. If you had a PIAF Forum account in the vBulletin days, log in with your old credentials. Otherwise, sign up again and we'll get you back in business as soon as we can.
  3. Guest: We think the problem with locked threads from long message subjects has been resolved. Post a link here if you still see a problem.
Cookies Welcome Back Long Message Subjects SOLVED

Debugging Memory Problems

Discussion in 'Bug Reporting and Fixes' started by dbaum, Nov 19, 2010.

Page 2 of 2

  1. dbaum Guru

    Tom,
    Thanks for your note.
    I am a telephony systems architect and one of the inventors of distributed paging, cellular and satellite services. I would love to talk with you about some ideas for PIAF - can't seem to get Ward on the phone he is motion too much :). Pls email me with contact info at david.baum@nofrills.net.
    dbaum, Dec 4, 2010
    #21

  2. troycarpenter Guru

    I see the same memory problem on 1.8.0 (purple install). Every couple of days I have to restart asterisk...however, the standard amportal script doesn't seem to kill asterisk in this state, and I have to kill asterisk manually. Once I do, the memory usage shoots down to a normal level.

    I first notice it when I call in and my ring groups stop working. Eventually everything just fails to voice mail.
    troycarpenter, Dec 17, 2010
    #22

  3. RizSher Guru

    Had the same issue, App memory kept creeping up over time, and at some point, all routing would go haywires... incoming calls would't follow inbound routes/time conditions, go straignt to IVRs when they were supposed to goto ring groupts etc.

    That was on a HP Laptop Core Duo with 2 GB ram.. same one I've used with various versions of PiaF over the last 1.5 year. LAst night I downloaded the latest iso and went back to Asterisk 1.6...

    Riz
    RizSher, Jan 2, 2011
    #23

  4. troycarpenter Guru

    My temp workaround was to create cronjobs that will stop amportal, kill any remaining asterisk processes, then restart. I have it running nightly.

    I disabled the chan_iax2.so module as suggested elsewhere. We'll see if that changes my memory usage.
    troycarpenter, Jan 5, 2011
    #24

  5. unsichtbarre Member

    So is it safe to say that App Memory in the 60-90% range is acceptable (possibly to be expected in Linux) so long as you are not also consuming some/all of the swap?

    -J
    unsichtbarre, Jan 22, 2011
    #25

  6. RizSher Guru

    I think the way Linux manages memory etc and the fact that 90% usage is all fine has been discussed many many times in the past, and most folk understand its different to windows and ... the high number shouldn't cause any concern.

    In this particular case, some of us, myself included, saw the usage creep up, followed by complete freakiness of the * server....
    RizSher, Jan 22, 2011
    #26

  7. dbaum Guru

    Thanks to all of you who have weighed in on my original post. The bottom line is that the behavior (APP memory and SWAP memory utilization) and the consequences (call processing not working) is Asterisk version dependent.

    It is only observed with the 1.8.x versions of Asterisk. Specifically that used on the first PIAF Purple build was most troublesome. Afte rreaching APP memory use of 50% or greater, with or without any measured SWAP use, calls (internal and external) would route directly to voicemail without regard to contents of the dial plan and without every ringing on any phone. I have recently migrated to Asterisk source version 1.8.2 and observed APP memory utilization way above 50% with no SWAP utilization, and no call processing anomalies. I am continuing to let the machine run without reboot to see if the anomalies are ever observed.
    dbaum, Jan 22, 2011
    #27

  8. wardmundy Nerd Uno

    Last post was in January. Now it's April. Has the issue been resolved? :confused5:
    wardmundy, Apr 5, 2011
    #28

  9. dbaum Guru

    No it has not, we continue to see this problem with PURPLE build/.
    dbaum, Apr 6, 2011
    #29

  10. blanchae Guru

    Purple build
    Asterisk 1.8.3
    FreePBX 2.8.10

    System uptime 26 days, 3 hours
    512 MB RAM
    Memory usage 55%
    Swap 0%

    I suggest taking a look at the FreePBX modules:

    Asterisk Info and System Info

    They give more detailed information on what's happening.
    blanchae, Apr 6, 2011
    #30

  11. darmock PIAF Developer

    cat /proc/meminfo will give you more results.

    Since this thread happened I have been checking our long term purple test machines (real and virtual) plus a couple of purple installs that my clients *insisted* on. So far none of them show the memory leak as you are describing. These are all "bare" machines without any addons or extras. Essentially what was included with the payload files and some options from the pbx-menu.

    Our office system is the latest purple on an Intel D510 platform with 2gbs and the most memory use it has ever shown was about 58% during a few calls plus transcoding. it normally runs about 49% - 51% used with no swap used at all. While this is high it just does not show the creep you are experiencing.


    I keep thinking you are having hardware problems. Have you tested you hardware? Have you tested your RAM and motherboard?

    I have run across times when certain hardware configurations just don't work or I have had a few lemon computers. Most of our lab computers are Dells with AMD processors and white box atoms with Intel motherboards only.


    Tom
    darmock, Apr 6, 2011
    #31

  12. blanchae Guru

    One of things that we should be posting is the version of Asterisk that is running with the memory problems. Otherwise we are just guessing as to the cause. 1.8.3 seems to be fine.
    blanchae, Apr 6, 2011
    #32

  13. darmock PIAF Developer

    I agree but people just don't understand. We have any number of tools to use from status to statustofile built in. They just have to use them and post them. I can't think of a way to force this issue other than not answering questions.

    I think the current version of asterisk 1.8 tree does not suffer from the problems that are outlined in this thread or we would have hundreds of responses demanding we fix this...... (like we write the asterisk source....)


    Tom
    darmock, Apr 6, 2011
    #33

  14. pjaric New Member

    I have an Intel D510 as well with the same issue. The noload seems to have worked for me. The App Memory is stable for now.

    noload = res_jabber.so
    noload = chan_gtalk.so
    pjaric, Apr 27, 2011
    #34

  15. dbaum Guru

    Update on experience as of 2011-05-05

    Anomalous call processing behavior (all calls to voicemail w/o regard to dialplan) noted with app memory reaching 79%

    here is cat /proc/meminfo output:
    MemTotal: 904620 kB
    MemFree: 12140 kB
    Buffers: 62848 kB
    Cached: 111864 kB
    SwapCached: 0 kB
    Active: 779352 kB
    Inactive: 59736 kB
    HighTotal: 0 kB
    HighFree: 0 kB
    LowTotal: 904620 kB
    LowFree: 12140 kB
    SwapTotal: 779144 kB
    SwapFree: 779040 kB
    Dirty: 264 kB
    Writeback: 0 kB
    AnonPages: 664460 kB
    Mapped: 26624 kB
    Slab: 40028 kB
    PageTables: 2984 kB
    NFS_Unstable: 0 kB
    Bounce: 0 kB
    CommitLimit: 1231452 kB
    Committed_AS: 986948 kB
    VmallocTotal: 114680 kB
    VmallocUsed: 6820 kB
    VmallocChunk: 100384 kB
    HugePages_Total: 0
    HugePages_Free: 0
    HugePages_Rsvd: 0
    Hugepagesize: 4096 kB



    Data from status command:
    │ PBX in a Flash Version = 1.7.5.5 │
    │ FreePBX Version = 2.8.1.4 │
    │ Running Asterisk Version = Asterisk 1.8.0 │
    │ Asterisk Source Version = 1.8.2 │
    │ Dahdi Source Version = 2.4.0+2.4.0 │
    │ Libpri Source Version = 1.4.11.5 │
    │ IP Address = 70.167.244.152 on eth0 │
    │ Operating System = CentOS release 5.5 (Final)


    Box is white revo from acer ... considering rebuilding, again, or possibly putting in acer blackbox revo. Only Purple installs on white boxes have shown this behavior. Purple on others have not, and bronze on white boxes have not. To quote Yul Brenner - "a puzzlement"!
    dbaum, May 5, 2011
    #35

  16. darmock PIAF Developer

    I see you have a runnning vs source mismatch on asterisk 1.8.0 This indicates that when you attempted to upgrade using update-source it was unsuccessful. You could try the EXPERIMENTAL update-source program again and see if asterisk 1.8.3.3 has the same behavior. This is really a Digium problem as opposed to a PIAF problem. (one the entire development staff has been unable to duplicate yet) Digium will tell you to upgrade to the latest source 1.8.3.3. If it still occurs open up a bug report at the asterisk web site and let them know as it is the only way it will get fixed. If you want to permanently donate us one of the boxes that are giving you the problems to us we can try and see what it is.

    The fact that it does not happen to all boxes running current 1.8 asterisk to me indicates that it may be related to hardware. I have any number of boxes in the lab and a LOT of commercial installs now running purple with 1.8.3.3 none exhibit the same S&S as what you are describing.

    If you have a different box to install it on perhaps to see if the problems persist. etc...


    Tom
    darmock, May 5, 2011
    #36

  17. randy7376 Guru

    This might be related...

    There was an update awhile back for a resource exhaustion issue with AMI about 6-7 weeks ago that affected Asterisk 1.6.x and Asterisk 1.8.

    We were experiencing this on version 1.6.2.17. Upgrading to 1.6.2.17.2 resolved our issue. It manifested itself primarily when people would attempt paging.

    All the phones would go off-hook randomly, the overhead audio would be grossly out-of-sync, and eventually the phones just stayed off-hook with no response. Rebooting would temporarily resolve the issue.

    randy7376, May 5, 2011
    #37

  18. AnyTime New Member

    Memory Creep



    We are showing this exact issue. This morning the customers machine would send all calls to voicemail no matter what time group/conditions were supposed to be set to. The memory was @ 79% and the memory line was YELLOW. I restarted amportal and the memory went back to 13%.

    This machine was an elastix machine before, but we decided to go with PBXiaF so to get rid of all the extra non-needed crap. This issue is so random though I think there is a problem.


    The Server is an INTEL White Box with 2 gigs of ram.

    PBX in a Flash Version = 1.7.5.6
    FreePBX Version = 2.8.1.4
    Running Asterisk Version = 1.8.4
    Asterisk Source Version = 1.8.4
    Dahdi Source Version = 2.4.1.2+2.4.1
    Libpri Source Version = 1.4.11.5
    IP Address = 192.168.0.5 on eth0
    Operating System = CentOS release 5.6 (Final) Kernel Version = 2.6.18-238.9.1.el5 - 32 Bit
    AnyTime, May 23, 2011
    #38

  19. malcolmd Guru

    Asterisk can optionally be compiled with a memory allocation debugger. To build this option in, from the menuselect, browser to "Compiler Flags" and then enable "MALLOC_DEBUG." Recompile and install Asterisk.

    Then, from the CLI, you can do:

    memory show summary

    to see memory utilization by file. So, you can run it, wait a bit till your system says memory's been leaked, and then run it again to compare and provide a pointer as to the culprit.

    Once you've tracked down the file that's at fault, you can then do:

    memory show allocations offending_file.c

    and it'll provide the allocations by source line number. If something's gone bonkers, there should be an allocation (or allocations) with really crazy numbers.

    Armed with some leading information, you can open an issue on the issue tracker at:

    https://issues.asterisk.org

    Reporting guidelines can be found here:

    http://www.asterisk.org/developers/bug-guidelines

    Cheers
    malcolmd, May 24, 2011
    #39

  20. wardmundy Nerd Uno

    Very helpful. Thanks, Malcolm.

    As for the actual steps in the Land of PIAF:


    wardmundy, May 25, 2011
    #40
Page 2 of 2

Share This Page