Stress Testing Part 2: FOP2

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
This is a continuation from the initial stress testing performed in this thread. This thread discusses the load placed on a PiaF server with the FOP2 (flash operator panel version 2) installed. Specifically it discusses how FOP2 interacts with the Asterisk Manager Interface AMI and the issues surrounding the AMI interface.

The developer of FOP2, Asternic responded to the initial performance query:

FOP(1) uses a lot more cpu than FOP2. You can quickly verify that by
running both and checking the cpu time used by each one. The main reason
for that is that FOP(1) supports REGEX based channel matching, and Regex
are expensive to match on every AMI event you get.

FOP2 is much more nice to cpu because of the above. However, as you have
seen in your stress tests, it analyzes every AMI event that is
broadcasted. AMI is *much* more verbose than the CLI or probably event
the full log.

So, if you run hundreds of calls per minute, and you pass them via the
humongous FreePBX dialplan, AMI will generate hundreds of events to be
processed by external applications, like fop/fop2. Take into account
that each priority in a dialplan generates an event, and depending the
application you run in that priority, it might generate *another* event.
You can run fop2_server in debug level 1 to see the events being
broadcasted by asterisk and received by fop2. "fop2_server --debuglevel
1" or use option "-X 1"

If you plan to run 50 calls per seconds on an asterisk box, then you are
probably not using it as a regular PBX, so fop2 is not to be installed
there. Even if you plan to do that, you should consider on installing
all monitoring/reporting tools on a separate box.

There are ways to optimize the manager to reduce the load on the
machine, they are described in a blog post here:

http://www.fop2.com/blog/make-fop2-snappier-using-ami-eventfilter
The AMI is MORE verbose than the logging that we saw in the initial stress testing! The AMI sends out many "event" notices based on a single request from FOP2. Asternic has described a method of reducing the number of AMI event notices by using the AMI eventfilter.
 
Last edited by a moderator:

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Event Filtering

In /etc/asterisk/manager.conf, create a new user with the following event filtering:

Code:
[fop2]
secret = [I]<appropriate password>[/I]
deny=0.0.0.0/0.0.0.0
permit=127.0.0.1/255.255.255.0
read = system,call,agent,user,command,originate
write = system,call,agent,user,command,originate
writetimeout = 1000
eventfilter=!Event: RTCPSent
eventfilter=!Event: RTCPReceived
eventfilter=!Event: VarSet
eventfilter=!Event: Cdr
eventfilter=!Event: ExtensionStatus
eventfilter=!Event: ChannelUpdate

In /usr/local/fop2/fop2.cfg, change the global settings to match the new user:

Code:
[general]
; AMI definitions
manager_host=localhost
manager_port=5038
manager_user=fop2
manager_secret=[I]<appropriate password>[/I]

From the Linux command prompt, restart FOP2:

Code:
service fop2 stop
service fop2 start
 

blanchae

Guru
Joined
Mar 12, 2008
Messages
1,910
Reaction score
9
Location
Calgary, Alberta, Canada
Results

Combining Asternic's optimization with Joe's verbose fix, we can reach 40 cps with FOP2 using the current sequential stress testing (including fail2ban running): stress test server initiates call, PiaF server responds, call hangs up - no RTP payload (audio transmitted). So FOP2 looks like it's doing pretty good at 40 cps (2400 calls per min/144,000 calls per hour!)! :biggrin5:

The CPU utilization was jumping between 50% and 90% when these screenshots were taken. There's several services (such as httpd) that come and go every couple of seconds not related directly to fail2ban, asterisk and fop2. I just happened to capture the action on the low end. Any cps higher would reach over 90% CPU loading. Here's the screenshots:

 

sigmaz

Member
Joined
Dec 17, 2010
Messages
100
Reaction score
1
Be sure to restart asterisk after creating the fop2 user.. I couldn't connect until after I had rebooted..
 

Members online

PIAF 5 - Powered by 3CX

Forum statistics

Threads
22,514
Messages
138,531
Members
14,644
Latest member
goseph