SUGGESTIONS Email Alert On Trunk Failure

[swilkey]

Updated and enhanced script

This is a new version of the previous script. The following modifications have been made:
- the script now automatically tries to re-establish the trunks if it detects a failure. It then sleeps for 20 seconds before checking the trunks again. If you find that this is too short a time to reestablish all your trunks lengthen the sleep time.
- moved email addressing details to the top of the script so that they are easier to customise
- original script assumed both iax2 and sip trunks and would fail if you didn’t have both – now the one you don’t want can be easily commented out at the top
- improved the documentation in the script about what files to create and what to include in those files in order to customise best to individual environments
- simplified the code to use the same code for both SIP and IAX2 and thus reduce the length of the script. PLEASE NOTE: the iax trunk alert file has now changed to having iax2 in the name. If you are upgrading from an older version you will need to rename the original iax trunk alert file.
- it is now clear that if the sip host at the other end of your trunks is not unique you can list the usernames for each of your trunks with that sip host and the script will check each of the trunks - previously it would simply check one trunk multiple times if they all connected to the one host.

I hope this is helpful to others.
--
#!/usr/bin/perl -w
use strict;
use warnings;
#no warnings; #Remark out to suppress ALL warnings, also add remark to above line.
#### email address must have \@ as perl needs to escape the @ symbol. Obviously you’ll need to edit these email addresses
my $alertemail = "YOURNAME\@YOURDOMAIN.COM" ;
my $pbxdomain = "YOURDOMAIN.COM" ;
#### comment out whichever of the following two lines are not relevant for your server
checktrunk("sip");
checktrunk("iax2");
##################################################
##################################################
####
#### Trunk Alerts script written by Jim Hribnak Oct 7th 2007
#### if there is any questions please feel free to drop me an email at jimh at domain nucleus.com
####
##################################################
##################################################
####
#### Script modified by Charles Pippin Feb 21st 2009 - Added counters for loop retry.
#### if there is any questions please feel free to drop me an email at cwpippin at domain gmail.com
####
##################################################
##################################################
####
#### Script modified by Stephen Wilkey Mar 18th 2010
#### - moved email addressing details to the top of the script so that they are easier to customise
#### - original script assumed both iax2 and sip trunks and would fail if you didn’t have both – now the one you don’t want can be easily commented out at the top
#### - improved the documentation below about what files to create and what to include in those files in order to customise best to individual environments
#### - simplified the code to use the same code for both SIP and IAX2. PLEASE NOTE: the iax trunk alert file has now changed to having iax2 in the name. If you are upgrading from an older version you will need to rename the original iax trunk alert file.
##################################################
##################################################
####
#### Create the following files in /etc/asterisk (if you want to only check sip trunks don’t create the other file etc)
#### - /etc/asterisk/trunkalerts_sip.conf
#### - /etc/asterisk/trunkalerts_iax2.conf
####
#### in the files below add the any unique entry from asterisk -rx "sip show registry" and
#### from asterisk -rx "iax2 show registry". The original specification was to use
#### the hostname, however if you have multiple trunks from the same supplier it is more
#### likely that the username might be unique. You don't need to be consistent either
#### - you can use hostname for some and username for others - as long as each
#### line of the show registry result is specified uniquely once by something
#### unique in that line for this script to grep on.
####
##################################################
##################################################
####
sub checktrunk {
# the next line gets the trunktype parameter passed to the subroutine
my $trunktype=shift;
####
print "=============================================\n";
print "$trunktype Trunk information\n";
print "=============================================\n";
open(TRUNKLIST,"</etc/asterisk/trunkalerts_$trunktype.conf");
while (<TRUNKLIST>) {
chomp;
my $recheckcounter = 1;
my $upcounter = 5;
my $dncounter = 5;
until ($upcounter == 0) {
my $trunks = `/usr/sbin/asterisk -rx "$trunktype show registry" | grep \"$_\" | awk '{
print \$4
}
'`;
#comment next line if you don't want to see the output
#print "$trunktype trunks = $trunks\n"; #Remove remark to see output from "sip show registry" command
if ($trunks =~ "Registered") {
$upcounter = $upcounter - 1;
#print $upcounter,"\n"; #Remove remark to see counter decrement
print "$_ is up\n" ; #Remove remark to see notification of trunk being "UP"
}
else {
$dncounter = $dncounter - 1;
#print $dncounter,"\n"; #Remove remark to see counter decrement once trunk is found to be down
print "$_ is down\n" ; #Remove remark to see notification of trunk being "DOWN"
if ($dncounter == 0) {
if ($recheckcounter == 1) {
# we now reload the trunks and force the check to occur again
`asterisk -rx "module reload"`;
print "Reloading trunks (will wait for reconnection before checking again\n";
# adjust the length of time on the next line (seconds) if you find it takes longer to re-establish
# trunk connections after a reload
`sleep 20`;
$recheckcounter = 0;
$dncounter = 5;
}
else
{
mailalert();
$upcounter = 0;
#print $upcounter,"\n"; #Remove remark to see counter decrement once trunk is found to be down
print "houston we have a problem\n";
print "$_ trunk is not registering\n";
$recheckcounter = 1;
}
}
}
}
}
#end of while loop (read trunk file)
}
##################################################
##################################################
####
#### Email Subroutines
#### Change anywhere below where there is an email address an email address
#### must have \@ as perl needs to escape the @ symbol
####
##################################################
##################################################
sub mailalert {
my $sendmail = "/usr/sbin/sendmail -t";
my $from= "FROM: <pbx\@$pbxdomain>\n";
my $reply_to = "Reply-to: <pbx\@$pbxdomain\n";
my $subject = "Subject: $_ is DOWN!!!!\n";
my $content = "PBX TRUNK $_ is DOWN!!!!\n";
my $send_to = "To:<$alertemail>\n";
open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";
print SENDMAIL $from;
print SENDMAIL $reply_to;
print SENDMAIL $subject;
print SENDMAIL $send_to;
print SENDMAIL $content;
close(SENDMAIL);
print "An email has been sent!\n\n";
}

 

[ghurty]

I have a outbound only trunk, so there is no registration, Any suggestions on how I can monitor it?


Thanks

 

[jroper]

Hi

You only know that it is not going to work when you actually make a call - so presumably, this will throw up a line in the log file that you can use Fail2Ban to monitor and notify you.

Joe

 

[Hat]

I am trying to update the original script to the latest version posted by swilkey with mixed results. On the SIP side everything works great. I am running into problems with IAX2 monitoring. I have only one IAX2 trunk (Fonicatec) and it is registered, however, the script seems to think it is not. I have renamed my original trunkalerts_iax.conf file to trunkalerts_iax2.conf as required. Inside the file I only have the Host as shown when I issue the command "iax2 show registry" (64.27.1.153:4569).

My guess (feeble as it may be) is the "grep" command in the script doesn't like the contents of the IAX2 file. Any assistance with this would be greatly appreciated.

This is from the original script.

asterisk -rx "sip show registry" |grep \"$_\" | awk '{print $4}'`;
asterisk -rx "iax2 show registry" |grep \"$_\" | awk '{print $5}'`;

 

[dswartz]

even easier, use the trunkdown setting for that outbound trunk and have it run the notification script.

 

[geoff]

I modified the script to monitor IAX2 trunks that do not have a registration string, it breaks checking sip trunks, but if anyone wants the code please let me know.

Geoff

 

[geoff]

I am trying to update the original script to the latest version posted by swilkey with mixed results. On the SIP side everything works great. I am running into problems with IAX2 monitoring. I have only one IAX2 trunk (Fonicatec) and it is registered, however, the script seems to think it is not. I have renamed my original trunkalerts_iax.conf file to trunkalerts_iax2.conf as required. Inside the file I only have the Host as shown when I issue the command "iax2 show registry" (64.27.1.153:4569).

My guess (feeble as it may be) is the "grep" command in the script doesn't like the contents of the IAX2 file. Any assistance with this would be greatly appreciated.

This is from the original script.

asterisk -rx "sip show registry" |grep \"$_\" | awk '{print $4}'`;
asterisk -rx "iax2 show registry" |grep \"$_\" | awk '{print $5}'`;

try changing the code on the line above to:

asterisk -rx "iax2 show registry" |grep \"$_\" `;

That should fix your problem

Geoff

 

[swilkey]

Script updates

Sorry for not responding before now to those who asked questions - I've been away and without access to a PC.

My understanding of outbound only trunks is that the PABX identifies that they cannot be contacted only when you try to use them. Maybe you could ping the sip server associated with them on a regular basis and assume that if it responds to a ping it is up, but that isn't necessarily a valid assumption. If the outbound only trunk is down, there doesn't seem to me to be anything you can do about it as the fault is likely to be at the other end, so is there value in knowing it is down? Wiser minds than me will need to give thoughts on this.

As far as IAX goes, sorry I can't advise as I have no way of testing these. I attempted to preserve IAX coding in the new version of the script I provided without change from the original, but with no way of testing it I can't be 100% certain. It would appear that a solution has now been provided for that though, so thanks!

 

[swilkey]

Geoff, if you can give me a copy of your modified code that works for IAX then I can test it on SIP and try to work out why it doesn't work. Between us we should be able to get it working for both. Unfortunately I don't have any IAX trunks and therefore cannot test that part of the code - I just tried to preserve it from the original version found in the forum.

 

[centoasa]

Often I've some trunk in "request sent" state, but the script of the friend swilkey don't work, don't send no email. My pbx is not on internet, is only for lan, so my hostname is pbx.local
I've put this two lines:
my $alertemail = "email\@pbx.local" ;
my $pbxdomain = "pbx.local" ;

but it don't work

 

[completetech]

so this should work to monitor both type of trunks and email out via... whatever method we are using for mail? i.e. gmail for notifications?

 

[centoasa]

I use internal email pbxinaflash email server

 

[tbrummell]

If anyone tries to use this with 1.8.5 (could be the whole 1.8 stream, not sure), I had to change this line from:

print \$4

to:

print \$5

to make it work. Just thought I'd throw that out there.

 

[shane]

I ran into a problem with this script running on * 1.8. I found that this

my $trunks = `/usr/sbin/asterisk -rx "$trunktype show registry" | grep \"$_\" | awk '{
            print \$[B][COLOR=Red]4[/COLOR][/B]
        }
        '`;

needs to be changed to this:

my $trunks = `/usr/sbin/asterisk -rx "$trunktype show registry" | grep \"$_\" | awk '{
            print \$[COLOR=Red][B]5[/B][/COLOR]
        }
        '`;

Hope that helps someone else.

 

[tbrummell]

Wow. Look ^^^ 1 post. LoL

 

[kdaffef02]

Big Thank

I ran into a problem with this script running on * 1.8. I found that this

my $trunks = `/usr/sbin/asterisk -rx "$trunktype show registry" | grep \"$_\" | awk '{
            print \$[B][COLOR=Red]4[/COLOR][/B]
        }
        '`;

needs to be changed to this:

my $trunks = `/usr/sbin/asterisk -rx "$trunktype show registry" | grep \"$_\" | awk '{
            print \$[COLOR=Red][B]5[/B][/COLOR]
        }
        '`;

Hope that helps someone else.

Hi shane,
I'm running * 1.6, and I was turning around and around... after your correction, everything is ok ...

Thank you

 

[krzykat]

Disable Failed Trunks

Has anyone made any mods so that when a trunk is determined to match your failed criteria (such as 5 failures in a row), that it would disable the trunk?

Then if you have an outbound route that tries:
Trunk 1
Trunk 2
Trunk 3

And let us say that Trunk 1 doesn't reply and just times out waiting for ringback, then it would be nice to have that trunk changed to disable, and then that route would just go immediately to Trunk 2.

I know this is related, but perhaps I should start this as a new thread ??

 

[LesD]

I wanted to add a script to my trunks to report a trunk failure on outgoing calls so I tried the script listed by thunderheart back in 2008 as I could not find anything more recent.

It sort of works but needs a fair bit more work to make it useful.

The script at the moment stands as follows:

#!/bin/bash
 
declare -a array
while read -e ARG && [ "$ARG" ] ; do
array=(` echo $ARG | sed -e 's/://'`)
export ${array[0]}=${array[1]}
done
 
[email protected]
 
DATE=`date "+%m-%d-%Y %H:%M"`
 
HOST=`PIAF-DA`
 
CID=`echo $agi_callerid`
CIDNAME=`echo $agi_calleridname`
EXTENSION=`echo $agi_extension`
CHANNEL=`echo $agi_channel`
 
REPORT=$EXTENSION
 
if [ "$EXTENSION" = "s-CHANUNAVAIL" ]
then
REPORT="Channel Unavailable"
fi
 
# Message Formating
 
MSG="\n\n A call from channel [$CHANNEL] \n"
MSG="$MSG With CID : [$CIDNAME] - [$CID] \n\n"
MSG="$MSG has failed on host [$HOST] at $DATE \n\n"
MSG="$MSG with error : [$REPORT] \n\n"
 
# log
 
echo >> /var/log/asterisk/trunkfailure.log
echo -e "$MSG" >> /var/log/asterisk/trunkfailure.log
 
# Send an email
 
echo -e "$MSG" | mail -s " - Failed call on host [$HOST] -" "$EMAIL"

I installed that in </var/lib/asterisk/agi-bin> as <trunk_monitor.agi> and I set 'Monitor Trunk Failures' in the trunk to just the name of the file.

I also had to do:

chmod +x /var/log/asterisk/trunkfailure.log
chown asterisk:asterisk /var/log/asterisk/trunkfailure.log

before it would work. This is all as per the earlier post.

I failed the trunk by changing the secret and tried a call and immediately got an email

Subject: - Failed call on host [] -
 
A call from channel [SIP/210-00001750]
With CID : [unknown] - [0208802yyyy]
 
has failed on host [] at 09-14-2014 02:39
 
with error : [continue]

So it is sort of working but the information coming through is not very useful nor is it completely correct. I would like to address the following shortcomings.

1. The HOST ID is not coming through. I would anyway have preferred the script to determine and report the machine's hostname or something similar to automatically identify the system from which it originates.

2. The CID being reported is that of my outgoing trunk which does not uniquely identify which of several trunks it is - they all have the same CID. I really need the name of the trunk.

3. It would also be useful to see the number being called.

4. The reported error message is wrong.

5. I need to be able to differentiate between the trunk failing and a call failing due to an invalid number. At the moment both are resulting in an email.

The log shows the following when the trunk fails to authenticate

[2014-09-14 13:23:50] VERBOSE[6486] app_dial.c: -- Called SIP/BestVoip/0044208802xxxx
[2014-09-14 13:23:51] WARNING[6458] chan_sip.c: Received response: "Forbidden" from '"0208802xxxx" <sip:[email protected]>;tag=as718056c8'
[2014-09-14 13:23:51] VERBOSE[6486] app_dial.c: == Everyone is busy/congested at this time (1:0/0/1)
[2014-09-14 13:23:51] VERBOSE[6486] pbx.c: -- Executing [s@macro-dialout-trunk:23] NoOp("SIP/210-0000178a", "Dial failed for some reason with DIALSTATUS = CHANUNAVAIL and HANGUPCAUSE = 21") in new stack
[2014-09-14 13:23:51] VERBOSE[6486] pbx.c: -- Executing [s@macro-dialout-trunk:24] GotoIf("SIP/210-0000178a", "1?continue,1:s-CHANUNAVAIL,1") in new stack
[2014-09-14 13:23:51] VERBOSE[6486] pbx.c: -- Goto (macro-dialout-trunk,continue,1)
[2014-09-14 13:23:51] VERBOSE[6486] pbx.c: -- Executing [continue@macro-dialout-trunk:1] GotoIf("SIP/210-0000178a", "0?noreport") in new stack
[2014-09-14 13:23:51] VERBOSE[6486] pbx.c: -- Executing [continue@macro-dialout-trunk:2] AGI("SIP/210-0000178a", "trunk_monitor.agi") in new stack
[2014-09-14 13:23:51] VERBOSE[6486] res_agi.c: -- Launched AGI Script /var/lib/asterisk/agi-bin/trunk_monitor.agi

For an invalid number, the log shows

[2014-09-14 13:08:20] VERBOSE[5380] app_dial.c:    -- Called SIP/BestVoip/00442000000000
[2014-09-14 13:08:21] VERBOSE[5380] app_dial.c:    -- SIP/BestVoip-00001778 is making progress passing it to SIP/210-00001777
[2014-09-14 13:08:32] VERBOSE[6458] chan_sip.c:    -- Got SIP response 480 "Temporarily not available" back from 77.72.174.129:5060
[2014-09-14 13:08:32] VERBOSE[5380] app_dial.c:    -- SIP/BestVoip-00001778 is circuit-busy
[2014-09-14 13:08:32] VERBOSE[5380] app_dial.c:  == Everyone is busy/congested at this time (1:0/1/0)
[2014-09-14 13:08:32] VERBOSE[5380] pbx.c:    -- Executing [s@macro-dialout-trunk:23] NoOp("SIP/210-00001777", "Dial failed for some reason with DIALSTATUS = CONGESTION and HANGUPCAUSE = 19") in new stack
[2014-09-14 13:08:32] VERBOSE[5380] pbx.c:    -- Executing [s@macro-dialout-trunk:24] GotoIf("SIP/210-00001777", "1?continue,1:s-CONGESTION,1") in new stack
[2014-09-14 13:08:32] VERBOSE[5380] pbx.c:    -- Goto (macro-dialout-trunk,continue,1)
[2014-09-14 13:08:32] VERBOSE[5380] pbx.c:    -- Executing [continue@macro-dialout-trunk:1] GotoIf("SIP/210-00001777", "0?noreport") in new stack
[2014-09-14 13:08:32] VERBOSE[5380] pbx.c:    -- Executing [continue@macro-dialout-trunk:2] AGI("SIP/210-00001777", "trunk_monitor.agi") in new stack
[2014-09-14 13:08:32] VERBOSE[5380] res_agi.c:    -- Launched AGI Script /var/lib/asterisk/agi-bin/trunk_monitor.agi

Can anyone suggest changes to the code to fix/improve it?

 

[krzykat]

#!/bin/bash
 
declare -a array
while read -e ARG && [ "$ARG" ] ; do
array=(` echo $ARG | sed -e 's/://'`)
export ${array[0]}=${array[1]}
done
 
[email protected]
 
DATE=`date "+%m-%d-%Y %H:%M"`
 
HOST=`hostname`
 
CID=`echo $agi_callerid`
CIDNAME=`echo $agi_calleridname`
EXTENSION=`echo $agi_extension`
DNID=`echo $agi_dnid`
STATUS=`echo $agi_dialstatus`
CHANNEL=`echo $agi_channel`
CAUSE=`echo $agi_causecode`
CODEC=`echo $agi_causecode`
HANGUPREASON=`echo $agi_hangupcause`
REASON=${REASON}
 
REPORT=$EXTENSION
 
if [ "$EXTENSION" = "s-CHANUNAVAIL" ]
then
REPORT="Channel Unavailable"
fi
 
# Message Formating
 
MSG="\n\n A call from channel [$CHANNEL] \n"
MSG="$MSG With CID : [$CIDNAME] - [$CID] \n\n"
MSG="$MSG has failed on host [$HOST] at $DATE \n\n"
MSG="$MSG Number that was dialed : [$DNID] on trunk [VITELITY] \n\n"
MSG="$MSG with error : [$STATUS] and hangup cause is : [$REASON] \n\n"
# log
 
 
echo >> /var/log/asterisk/trunkfailure.log
echo -e "$MSG" >> /var/log/asterisk/trunkfailure.log
 
# Send an email
 
echo -e "$MSG" | mail -s " VITELITY - Failed call on host [$HOST] - dialing [$DNID] - " "$EMAIL"

Here's my script that works. It does show me what number is called, but couldn't figure out the host, so I have a different script for each trunk, so that it tells me the trunk via fixed code:

 

[LesD]

See thread I started at <http://pbxinaflash.com/community/in...tion-delays-drop-through-to-2nd-trunk.15620/>

It relates to an apparent delay of some 50 seconds or so while the email is sent resulting in a long delay till the fall-through trunk can process the call - if there is such a trunk defined.