R.I.P. Asterisk using a lot of memory

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
Hello,

My PIAF server been running for 2 years without any issue. But lately it has problem. The first time happened last month, phones cannot connect to the server. I try to ssh to the server and fail. The server's screen doesn't display anything but black screen when hit the keyboard and move the mouse. I have to hard restart the server.

Then similar thing happened last week, phone cannot connect to the server. I got this error message from a sip monitor system:
RESPONSE-> Duration:32085;Error Code:408;Reason:Request Timeout (Couldn't find the user in time).

I was able to ssh to the server and restart the asterisk service. Then things became normal.

I tried to find out what the cause is. I see the asterisk using a lot of memory. It's like 70%. I don't think it was using this much before.

I think it's because I changed something 2 month ago. I added several different trunks to test different ISPs for price and quality. During the set up of callcentric trunk, I think I change something in sip_general_custom.conf. But I changed back after I tested their services. And I also deleted the other trunks I added recently. The asterisk is still using a lot of memory.

Memory usage lows down to 2% after I restart the asterisk. But it spikes back to 40% after a day. And then it will keep increasing.

I checked online and I guess I should disable some modules to find out the broken modules?

Any suggestions will be appreciated.


Regards,
William
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
+-------------------SYSTEM INFORMATION *VERIFIED*---------------------+
| Asterisk = ONLINE | Dahdi = ONLINE | MySQL = ONLINE |
| SSH = ONLINE | Apache = ONLINE | Iptables = ONLINE |
| Fail2ban = ONLINE | Internet = ONLINE | Ip6Tables = ONLINE |
| Disk Free = ADEQUATE| Mem Free = ADEQUATE| NTPD = ONLINE |
| SendMail = ONLINE | Samba = OFFLINE | Webmin = ONLINE |
| Ethernet0 = ONLINE | Ethernet1 = N/A | Wlan0 = N/A |
| |
| PIAF Installed Version = 2.0.6.2 under *KVM* |
| FreePBX Version = 2.8.1.5 |
| Running Asterisk Version = 1.8.13.0 |
| Asterisk Source Version = 1.8.13.0 |
| Dahdi Source Version = 2.6.1+2.6.1 |
| Libpri Source Version = 1.4.12 |
| IP Address = 192.168.10.131 on eth0 |
| Operating System = CentOS release 6.2 (Final) |
| Kernel Version = 2.6.32-220.17.1.el6.i686 - 32 Bit |
| |
+---------------------------------------------------------------------+
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Run top from the Linux CLI and see what's at the top of the list.
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
The speed of memory usage is faster than before. You can see the CPU usage is 78.6 now. Thanks.



Code:
top - 00:30:58 up 35 days, 12:27, 1 user, load average: 0.23, 0.22, 0.14
Tasks: 94 total, 1 running, 93 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 0.3%sy, 0.0%ni, 90.4%id, 8.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 2071368k total, 2012664k used, 58704k free, 31468k buffers
Swap: 786424k total, 173956k used, 612468k free, 138600k cached
 
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30813 root 20 0 2560m 1.6g 7828 S 1.7 78.6 71:06.91 asterisk
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Looks like a problem with your version or configuration of Asterisk. You're running very old Asterisk and FreePBX code with a very old operating system. None of them are supported any longer. It's probably time to bite the bullet and move up to Asterisk 11 and FreePBX 2.11. That would be PIAF-Green or Incredible PBX 11 in our universe.
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
Thanks Ward. I was thinking about that. But it was always running beautifully and I didn't bother to do it. Is there something like a update script to help me to do it or should I better install a new system and manually input my settings to the new system?

I think I will install a new system and configure everything in the new one. This is easier for me and the system is 'cleaner'. Thanks.
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
Some updates:
I later installed a new PIAF system and manually migrate the settings from the old to the new. Same thing happens. Memory goes up quickly. It takes a little more than a day to reach 90% usage. And then Asterisk will reload automatically. When the memory usage is high as 80%, ring groups feature doesn't work as expected. Ring group will only ring the first extension even I set it as ring all. After some experiments, I found that it's the google voice trunk. I have about 20 google voice account connected to my PIAF server. After I delete all of them, the memory problem solved. Thanks!
 

mainenotarynet

Not really a Guru - Just a long time user
Joined
May 29, 2010
Messages
754
Reaction score
155
Sorry wzheng, you didn't FIX the problem, you are avoiding it -- by deleteing your, what did you say 20, google voic entries, you may not see the spike in memory usage but it doesn't say WHY it spikes.

Also you mention it is a memory issue (and sorry Mr wardmundy) the output of status give PiaF, FreePBX and Asterisk versions, but NOT how much memory your system has.

I had a system do that a while back (RPBX) as the base system only had 256MB, I had to up the memory and the problem went away.

If you have a New PiaF-Green, use the GoogleVoice/MOTIF module (there are 2 in FreePBX (the other is just Google Voice. Put in your 20 accounts again and see what happens.

If you have low memory to begin with it will happen again. By deleting the problem you also deprive your users of getting calls (and making them through the GV accounts)

For this many GV accounts I would suggest no less than 1 or 2GB for memory the issue soounds like your box has only 256 or 512MB in it.

Good luck
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
Hello Kevin,

Thanks for the reply. You are right, I just know it's about GV but don't know why it happens. I always use more than 2GB memory but less than 3 GB memory. I use both virtual and physical machine to test.
Here is the status:
-------------------SYSTEM INFORMATION *VERIFIED*---------------------+
| Asterisk = ONLINE | Dahdi = ONLINE | MySQL = ONLINE |
| SSH = ONLINE | Apache = ONLINE | Iptables = ONLINE |
| Fail2ban = ONLINE | Internet = ONLINE | Ip6Tables = ONLINE |
| Disk Free = ADEQUATE| Mem Free = ADEQUATE| NTPD = ONLINE |
| SendMail = ONLINE | Samba = OFFLINE | Webmin = ONLINE |
| Ethernet0 = ONLINE | Ethernet1 = N/A | Wlan0 = N/A |
| |
| PIAF Installed Version = 2.0.6.5 under *KVM* |
| FreePBX Version = 2.10.1.19 |
| Running Asterisk Version = 1.8.28.0 |
| Asterisk Source Version = 1.8.28.0 |
| Dahdi Source Version = 2.9.0 |
| Libpri Source Version = 1.4.14 |
| IP Address = 72.x.x.x on eth0 |
| Operating System = CentOS release 6.5 (Final) |
| Kernel Version = 2.6.32-431.1.2.0.1.el6.i686 - 32 Bit |
| |
+---------------------------------------------------------------------

I am using Purple-PiaF. I will try the PiaF-Green with GV/MOTIF module later. Thanks.


William
 

mainenotarynet

Not really a Guru - Just a long time user
Joined
May 29, 2010
Messages
754
Reaction score
155
If using purple -- that may explain it as the 'old way' is not supposed to work anymore (search the Google Voice posts on the forum) the XMPP was supposed to be dropped (nut I believe that is what the MOTIF uses (not sure - I have 1 GV account -- actually had 4 but only added 1, deleted the numbers on 2 of them and never added the last one) ) .

When I used the MOTIF the first time it said disconnected on the page for that number and I deleted it and started over - now it works. Be careful to follow the install procedures carefully or GV won't work at all (also search the Nerd Vittles articles that Mr Ward Mundy (Nerd Uno) does. I believe it says something like after installing motif module you need to restart asterisk or it won't ever work.

The other issue could be that the sign in is being blocked by Google as an insecure app (there are references to how to correct these as well.

If you are still on purple -- great that it has lasted you for so long, but consider coming up to Green (and also installing incredible PBX 11 and Incredible Fax 11) for the most up to date stuff. You may find the FreePBX layout a little differnet but its all there still.

I have the Tar files for the pwhite Text to speech which I find better that what FreePBX hasbundled in new versions if interested -- one issue I have though is filenames with underscores or hyphens fail in Google Chrome but will take just fine in Firefox -- wierd, shouldn't matter .. but it does.

Good luck in your migrations
 

wzheng

New Member
Joined
Oct 20, 2013
Messages
19
Reaction score
1
Thank you very much, Kevin.

Your information is very helpful. I will need some time to digest and catch up. I will definitely upgrade to Green very soon. Thanks again!
 

Members online

Forum statistics

Threads
25,778
Messages
167,504
Members
19,198
Latest member
serhii
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top