TIPS FCC RoboCall BlackList

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
CSaXc20WUAAD7Kb.jpg


UPDATED: Oct. 29 at 4:30 pm EDT (special thanks to adamgoldberg for elegant code addition)

Found something interesting on the FreePBX Forum this morning. It was a link to a new FCC Robocall Blacklist which apparently will be updated weekly. With help, we've turned this into a few little scripts that will import the list into Asterisk automatically with the following caveat: ALL existing BlackList entries are ALWAYS erased before the import begins. If you want to preserve your personally-added BlackList entries from FreePBX or Incredible GUI, be sure you enter them with a Description value of 1. That's the FreePBX default value. Using a value that sorts higher than FCC will mean they won't get exported in Step #1 below. Using this 1 methodology, your personal BlackList entries can be parsed and preserved while the FCC entries will be replaced weekly and have a unique Description of FCC.

HOW IT WORKS. First, we export your personal BlackList entries from the Asterisk DB. Next, we erase ALL BlackList entries from the Asterisk DB. Then, we import the latest FCC BlackList into the Asterisk DB (10-digit numbers only). Finally, we restore your personal BlackList entries to the Asterisk DB.

You can run these 3 scripts weekly or whenever you like to keep the FCC BlackList current in Asterisk. Here is the correct procedure. The scripts MUST be run in the correct order. It only takes a minute or two to complete the 3-step procedure. If you don't have any personal BlackList entries, just skip steps 1 and 3. How can you review your personal BlackList entries?
Code:
sqlite3 /var/lib/asterisk/astdb.sqlite3 'SELECT key FROM astdb where KEY LIKE "/blacklist%" and value="1"'

TO GET STARTED: Download import-fcc-blacklist.tar.gz and untar it into /root: tar zxvf import-fcc-blacklist.tar.gz

1. Save your existing personal BlackList entries by running: ./export-my-blacklist

2. Import the new FCC BlackList after deleting ALL existing Asterisk blacklist entries: ./import-fcc-blacklist

3. Restore your existing personal BlackList entries by running: ./import-my-blacklist

The script also blocks ANONYMOUS calls so, if you don't want that, comment out line 38 in the import-fcc-blacklist script:

Code:
#asterisk -rx "database put blacklist blocked FCC"

WARNING: This makes a BIG BLACKLIST... currently 6889 non-duplicate entries. No idea how well Asterisk is going to handle this although Asterisk 11 and 13 astdb now use a modern, indexed, lightning-fast SQLITE 3 database so you should be fine. Doesn't seem to blow up the GUI either.

DO NOT USE WITH OLDER ASTERISK VERSIONS, before Asterisk 11 or DISASTER AWAITS!


OTHER HELPFUL COMMANDS

Add Personal Entry to BlackList: asterisk -rx "database put blacklist 4045551212 1"

Add the Anonymous Call Blocker to BlackList: asterisk -rx "database put blacklist blocked 1" (NOTE: This is done automatically in Step #2 above)

List and Count the entries in your current BlackList: asterisk -rx "database show blacklist"
 

geopeterwc

Guru
Joined
Aug 17, 2010
Messages
385
Reaction score
131
I've been using Vitelity DIDs for my installations ... there's a "blocklist" feature in the Vitelity account (Settings | Blocklist) into which I've been adding numbers that show up in my CDR and that I verify to be scam numbers periodically. The process is anything but "automatic" and can be time consuming, but blocking the numbers before they arrive at my PIAF, together with the IVR on incoming calls (http://pbxinaflash.com/community/in...lemarketer-block-pat-fleet.16015/#post-103529), I've been able to eliminate 99.999% of the scam calls ringing through to any extension. And my household is no longer running to the phone to answer a scam call!

I saw this on the Freepbx forums yesterday as well and wondered how it could be implemented. Thanks, Ward ... for staying ahead of the curve on things like this. I like this approach too, where calls that might slip through could be greeted by Lenny ... but nearly all of the telemarketers that hit my PIAF are recorded announcements - that don't have the ability to "press 8" as my incoming IVR requests of callers.

None of our friends or businesses that have encountered "Press 8" have complained ... mostly, they want the ability for their own home phones!

/Pete./
 

Trimline2

Guru
Joined
May 23, 2013
Messages
524
Reaction score
96
To get all of my blacklist entries, I used
Code:
 sqlite3 /var/lib/asterisk/astdb.sqlite3 'SELECT key FROM astdb where KEY LIKE "/blacklist%" and value>"0"'
. This will list all of the manual entries you may have made with names in the entry.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
The trick is you don't want the FCC entries. Unfortunately, the logic you used grabs all of them as well since FCC sorts higher than zero. You could use: value<"F"
 
Joined
Mar 26, 2010
Messages
17
Reaction score
3
I wonder why the scripts to extract a good, valid, unique list of numbers are more complicated than this:
Code:
 cat Telemarketing_RoboCall_Weekly_Data.csv | awk -F"," '{print $3 "\n" $4}' | sort -u | grep "[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]" | sed "s|-||g" > blacklist
You need both columns $3 and $4, because they're both telephone number fields, and they're sometimes different, sometimes the same, and sometimes one or the other is missing.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
I wonder why the scripts to extract a good, valid, unique list of numbers are more complicated than this:
Code:
 cat Telemarketing_RoboCall_Weekly_Data.csv | awk -F"," '{print $3 "\n" $4}' | sort -u | grep "[0-9][0-9][0-9]-[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]" | sed "s|-||g" > blacklist
You need both columns $3 and $4, because they're both telephone number fields, and they're sometimes different, sometimes the same, and sometimes one or the other is missing.

Thanks, adamgoldberg. Looks & works great. I'm not sure I'd call your approach "not complicated," but it certainly is elegant. Great job! We'll use it in the next update if you don't mind.

You are correct. We need both columns of numbers, they need to be unique numbers, and our script gets the same results... just not in one line. :aureola:

Tortoise-and-Hare.jpg
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
briankelly63 suggested providing code to allow you to export a Blacklist from an earlier release of Asterisk that could subsequently be imported into newer Asterisk 11 or 13 platforms.

Here you go:
Code:
asterisk -rx "database show blacklist" | cut -f 1 -d ":" | cut -f 1 -d " " > myblacklist
sed -i 's|/blacklist/||' myblacklist
sed -i '/blocked/d' myblacklist
 

AndyInNYC

Active Member
Joined
May 23, 2013
Messages
772
Reaction score
124
I must be missing something.

After downloading the tar file and unpacking it I have run: ./import-fcc-blacklist. I get the following output:

Code:
Processing FCC blacklist...
--2015-10-29 16:21:24--  https://consumercomplaints.fcc.gov/hc/theme_assets/513073/200051444/Telemarketing_RoboCall_Weekly_Data.csv
Resolving consumercomplaints.fcc.gov... 192.161.148.10
Connecting to consumercomplaints.fcc.gov|192.161.148.10|:443... connected.
ERROR: cannot verify consumercomplaints.fcc.gov’s certificate, issued by “/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M”:
  Self-signed certificate encountered.
To connect to consumercomplaints.fcc.gov insecurely, use ‘--no-check-certificate’.
cat: Telemarketing_RoboCall_Weekly_Data.csv: No such file or directory
Importing blacklist into Asterisk...
1 database entries removed.
Updated database successfully
Done
WARNING: Always run Incredible PBX behind a secure hardware-based firewall.

My system specs are:

Code:
Asterisk  = ONLINE  | Dahdi    = ONLINE  | MySQL    = ONLINE    x
                  x  SSH        = ONLINE  | Apache    = ONLINE  | Iptables  = ONLINE    x
                  x  Fail2ban  = ONLINE  | Internet  = ONLINE  | Ip6Tables = ONLINE    x
                  x  Disk Free  = ADEQUATE| Mem Free  = ADEQUATE| NTPD      = ONLINE    x
                  x  SendMail  = ONLINE  | Samba    = OFFLINE | Webmin    = ONLINE    x
                  x  Ethernet0  = ONLINE  | Ethernet1 = N/A    | Wlan0    = N/A      x
                  x                                                                    x
                  x  PIAF Installed Version  = 2.0.6.5 under *HARDWARE*                x
                  x  FreePBX Version          = 2.11.0.38                              x
                  x  Running Asterisk Version = 11.10.0                                x
                  x  Asterisk Source Version  = 11.10.0                                x
                  x  Dahdi Source Version    = 2.9.0                                  x
                  x  Libpri Source Version    = 1.4.14                                  x
                  x  IP Address              = 192.168.40.29 on eth0                  x
                  x  Operating System        = CentOS release 6.5 (Final)              x
                  x  Kernel Version          = 2.6.32-431.1.2.0.1.el6.x86_64 - 64 Bit  x
                  x  Incredible Version      = 11.10


The one file added is Number: "blocked", Description: "FCC" and that's it.


What am I missing?


Andrew
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
Hi guys,

To break the job into steps, as is done in the the three scripts, but with some IMHO useful improvements

A) get the unique phone numbers into a file

curl -ks https://consumercomplaints.fcc.gov/...051444/Telemarketing_RoboCall_Weekly_Data.csv |grep -Eo "[0-9\-]{12}"|sort -u |awk -F '-' '{print "/blacklist/"$1$2$3"|FCC"}'> blacklist

B) purge the old entries

rasterisk -x ' database query "delete from astdb WHERE key LIKE \"/blacklist%\" AND value = \"FCC\" "'


C) insert the new entries

sqlite3 /var/lib/asterisk/astdb.sqlite3 ".import blacklist astdb"

This method uses the asterisk record locking described in the source code ./main/data.c

If you are unsure of the safety of .import, ( I believe it is) then iterate as done now through

curl -ks https://consumercomplaints.fcc.gov/...051444/Telemarketing_RoboCall_Weekly_Data.csv |grep -Eo "[0-9\-]{12}"|sort -u |sed 's/-//g' > blacklist

Thoughts?
 

randy7376

Defnyddiwr Gweithredol
Joined
Sep 29, 2010
Messages
864
Reaction score
144
dicko

FYI... your awk statement is shorting the numbers by 4-digits.

They're showing up like this:
Code:
/blacklist/1844432|FCC
For what it's worth, here's my fix:
Code:
curl -ks https://consumercomplaints.fcc.gov/...051444/Telemarketing_RoboCall_Weekly_Data.csv |grep -Eo "[0-9\-]{12}"|sort -u |awk -F '-' '{print "/blacklist/"$1$2$3$4"|FCC"}'> blacklist

Cheers!
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
Well that will also work because $4 will always be a null , my grep line should return lots of lines like "^321-608-4860$", awk separates that string on "-" (the $n is 1 based) I can't see how i screwed that up , show me your output
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
So by reductio ad absurdum the cron job and even the the most conservative understanding of asterisk's database locking, you could call early on sunday morning

/usr/sbin/rasterisk -x ' database query "delete from astdb WHERE key LIKE \"/blacklist%\" AND value = \"FCC\" "'&&for i in $(/usr/bin/curl -fks https://consumercomplaints.fcc.gov/hc/theme_assets/513073/200051444/Telemarketing_RoboCall_Weekly_Data.csv|/bin/grep -Eo "[0-9\-]{12}"|/usr/bin/sort -u |/bin/sed 's/-//g');do /usr/sbin/rasterisk -x "database put blacklist ${i} FCC" >/dev/null;done

Not quite technically a oneliner but cron compliant. If it blows up then if your cron is set up correctly you will get an email soon after running it (stderr is not redirected)
 
Last edited:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
We've tried to sift through all the suggestions and end up with a single script that can be run at any time without shutting down Asterisk.

Nerd Vittles article will be out tomorrow, but here's the code:
Code:
cd /root
wget http://incrediblepbx.com/fcc-blacklist.tar.gz
tar zxvf fcc-blacklist.tar.gz
rm -f fcc-blacklist.tar.gz

And here's a one-liner to add a weekly cron job to /etc/crontab but I prefer @dicko tip below:
Code:
echo "10 1 * * 6 root /root/import-fcc-blacklist > /dev/null 2>&1" >> /etc/crontab

You will note we've also added a second script to recover from a catastrophe in the event the FCC makes "improvements" and blows everything up. :shuriken:

Special thanks to @adamgoldberg and @dicko for their terrific suggestions/improvements. Open source development at its finest!
 
Last edited:

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
Ideally crontab entries should email any useful output , in this case if the wget or the other calls fail that would be noted to stderr. perhaps leave out the 2>&1 ?

I would suggest

echo "$(($RANDOM%60)) $(($RANDOM%24)) * * $(($RANDOM%2 +6 )) /root/import-fcc-blacklist > /dev/null" >> /etc/crontab

to "spread the load" on the site incase this code gets some traction.
 
Last edited:

Arne Bolen

New Member
Joined
May 1, 2015
Messages
5
Reaction score
1
The script adds 10 digit phone numbers to the blacklist. Some VoIP providers (like Anveo and Callcentric) use 11 digit (with or without the + prefix) phone numbers in the CID field causing the script to be useless.

Any solution to this problem?
 

Members online

No members online now.

Forum statistics

Threads
25,778
Messages
167,504
Members
19,198
Latest member
serhii
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top