I HAVE A DREAM FCC Robocall Consumer Complaints List

Discussion in 'Wish List' started by geopeterwc, May 16, 2016.

  1. geopeterwc

    geopeterwc Guru

    Joined:
    Aug 17, 2010
    Messages:
    228
    Likes Received:
    51
    A few months ago, a procedure for implementing a call-blocking "feature" for the iPBX by populating the Blacklist using the FCC Unwanted Calls List (https://consumercomplaints.fcc.gov/hc/en-us/articles/205239443-Data-on-Unwanted-Calls). The scheme was presented in the Forums (http://pbxinaflash.com/community/threads/fcc-robocall-blacklist.17853/) for iPBX host systems.

    Having tested it out the time on an Atom-based iPBX, it worked admirably! It was working with callers being routed to Lenny, too!

    However, because most of those unwanted "scam" callers are robocalls - recordings, really - it was a rare that a caller would have a conversation with Lenny. But, by intercepting the calls, there were fewer scam calls to interrupt the day (and evening)!

    I'm considering running this up on a RasPi (2 or 3) - but curious if anyone has tried this on a RasPi, and what their experience has been. ... or is the RasPi just to limited on bandwidth or available memory (the current list from the FCC runs to tens-of-thousands of "suspect" numbers that would be added to the iRasPi's blacklist.

    Thoughts, anyone?

    /Pete./
     
  2. ostridge

    ostridge Guru

    Joined:
    Jan 22, 2015
    Messages:
    372
    Likes Received:
    66
    As I am in UK the FCC list seems a bit pointless here, however my blacklist is currently 56 entries. Any
    UNKNOWN, anonymous, private, restricted, unavailable, unknown CLIs are after SIT tones and "Calls may be recorded" are routed to Lenny, and other calls are either answered or go to voice-mail. If any nuisance call is encountered I add the CLI to blacklist, check on Whocalled....com or whatever. I'm on the do not call list (TPS), and frequently complain to ico.org.uk (the uk regulator).
    Fake CLIs also get blacklisted, whether somebody's real number or perhaps truncated numbers - a shame for any genuine numbers that are used or given out by scammers.

    Conversations with Lenny mostly are brief, and the frequency of blacklisted callers is really minimised.

    My local doctors surgery called twice while witholding their number - a couple of short conversations with lenny whilst asking for Mrs xxxx, can you say that again ... again.. again.... Then followed with another call with the surgery number released but with a 00 prefix, yes the provider was adding an extra '0'.

    I would like the Information Commissioners Office or Ofcom to provide an equivalent to the FCC Robocallers list, but they say they have no such plans. I find that attitude to be short sighted, as the public are providing large numbers of complaints, and only the regulator can validate and maintain such lists. Some of the robocalls are for call blocking hardware, so unregulated lists abound and I guess anarchy rules.

    As regards Pi2 or Pi3 coping with the FCC entries, I can only assume it should cope OK.
     
    wardmundy likes this.
  3. lrosenman

    lrosenman Guru

    Joined:
    Oct 17, 2014
    Messages:
    222
    Likes Received:
    30
    Part of my problem with this is Amazon's CS number and Dish Network's CS number are on the list, so I discontinued it.
     
  4. dicko

    dicko Still learning but earning

    Joined:
    Oct 30, 2015
    Messages:
    590
    Likes Received:
    211
    What you were running stopped being effective May 23

    I added my whitelist concept, try out

    http://community.freepbx.org/t/finding-balance-with-the-fcc-robocall-list/35391/14

    edit: apparently that post was censored by the"suits" at Sangoma because it "wasn't" licensed

    I will post a copy here:-


    #!/bin/bash
    # you are licensed by by me "dicko" who wrote it to do anything you want with this code without restriction, there is nothing but bash primitives in here as any sane but sentient linux scripter would recognize
    LOGFILE=/var/log/asterisk/fccblacklist
    CIDPREFIX='' # What you need to prepend to NXXNXXXXXX to match your carriers CID presentation "+1" or "1" are likely options
    SPAMLEVEL=2 # Ignore numbers with less than this number being reported, DO NOT use 0!
    START=$(date +%s)
    echo -n "$(date +'%Y/%m/%d %H:%M:%S') BL was $(/usr/sbin/rasterisk -x 'database show'|grep FCC|wc -l), " >> ${LOGFILE}
    wget -q https://opendata.fcc.gov/api/views/t3tq-usu8/rows.csv -O -|sort -n|uniq -c|grep -v " [$(seq -s '' $SPAMLEVEL)] "|sed "s:^\s*\([0-9]*\)\s*\(.*\)-\(.*\)-\(.*\):/usr/sbin/rasterisk -x \"database put blacklist ${CIDPREFIX}\2\3\4 FCC$
    echo -n "$(/usr/sbin/rasterisk -x 'database show'|grep FCC|wc -l) FCC level ${SPAMLEVEL} in $(( $(date +%s) - ${START} )) secs, " >> ${LOGFILE}
    /usr/sbin/rasterisk -x "database show whitelist"|grep -v results|cut -d "/" -f3|cut -d " " -f1|awk '{print "/usr/sbin/rasterisk -x \"database del blacklist "$1"\""}'|/bin/bash >/dev/null




    but previously run

    rasterisk -x "database put whitelist 8006665678 DishNetworks-thatapparently-pisses-people-off-(no-shit)-and-calls-them-from-this-number-for-unsolicited-sales"
    rasterisk -x "database put whitelist 8006661234 Amazon-serice-support-that-apparently-equally-pisses-people-off-and-calls-them-for-no-reason"

    TYhe other post I had removed said

    From

    http://pbxinaflash.com/community/threads/fcc-robocall-consumer-complaints-list.19363/1

    It would make sense to automatically whitelist anyone you have already called. hmmm, I will think about that, but it would probably include granny if you were a nice person [​IMG]

    Probably by looking at all calls since the last fcc update because otherwise it would be horrible [​IMG]

    edit2 :-

    Oh well, I was almost totally purged from that Sangoma thread so it might not make a lot of sense ,

    The concept remains however, FCC script has changed, I wrote a filter there (t3tq-usu8) that just downloads the numbers, the rest is just bash and asterisk's blacklist modified to remove your self entered whitelist which you can set with

    rasterisk -x "database put whitelist 3235550219 grannygotscammed"
     
    #4 dicko, Jun 13, 2016
    Last edited: Jun 14, 2016
    wardmundy likes this.
  5. wardmundy

    wardmundy Nerd Uno

    Joined:
    Oct 12, 2007
    Messages:
    14,701
    Likes Received:
    2,512
    We always welcome contributors here without the need to sign your life (and home) away. :dots:
     
  6. wardmundy

    wardmundy Nerd Uno

    Joined:
    Oct 12, 2007
    Messages:
    14,701
    Likes Received:
    2,512
    Adding a CallerID WhiteList to XiVO

    One problem noted in the FCC RoboCall BlackList project has been the existence of a few numbers that really shouldn't be blacklisted, such as UPS, FedEx, and a few major banks. Turns out the suggestion by @dicko for a WhiteList that overrides a matching BlackList entry (e.g. from FCC Robocall Blacklist) is extremely easy to implement on the XiVO platform. Here's how to manually update your server...

    Edit IPX Services:Configuration Files:cid-superfecta.conf and replace the top [xivo-subrgbl-did] context with the following:
    Code:
    [xivo-subrgbl-did]
    exten = s,1,NoOp(### DID Incoming Call Subroutine ###)
    same = n,NoOp(Calling FROM: ${XIVO_SRCNUM})
    same = n,NoOp(Blacklist: ${DB(blacklist/${XIVO_SRCNUM})})
    same = n,NoOp(Whitelist: ${DB(whitelist/${XIVO_SRCNUM})})
    same = n,GotoIf($[${LEN(${DB(whitelist/${XIVO_SRCNUM})}))} > 1]?keepon)
    same = n,GotoIf($[${LEN(${DB(blacklist/${XIVO_SRCNUM})}))} > 1]?byebye)
    same = n(keepon),GoSub(cid-superfecta,s,1)
    same = n(return),Return()
    same = n(byebye),Answer
    same = n,Congestion
    same = n(return),Return()
    
    To add numbers to the WhiteList using the Asterisk CLI: database put whitelist 8431234567 1

    To delete numbers from the WhiteList: database deltree whitelist/8431234567

    To list numbers on the Whitelist: database show whitelist

    All new Incredible PBX for XiVO builds after noon EDT, 6/29/2016 already have the update.

    To update existing Incredible PBX for XiVO servers the lazy way,
    issue the following commands.
    Code:
    cd /
    rm -f cid-superfecta.tar.gz
    wget http://incrediblepbx.com/cid-superfecta.tar.gz
    tar zxvf cid-superfecta.tar.gz
    rm -f cid-superfecta.tar.gz
    /etc/init.d/asterisk reload
    
     
    #6 wardmundy, Jun 29, 2016
    Last edited: Jun 29, 2016
  7. dicko

    dicko Still learning but earning

    Joined:
    Oct 30, 2015
    Messages:
    590
    Likes Received:
    211
    I will offer my latest efforts, first the script ("var/lib/asterisk/bin/fcc.sh"):-

    #!/bin/bash
    # by dicko
    # Licensed under the MIT (http://creativecommons.org/licenses/MIT/) license.
    LOGDIR="/var/log/asterisk"
    LOGFILE="${LOGDIR}/fcc.log"
    #LOGFILE="/dev/stdout"
    PIDFILE="/var/run/asterisk/fcc.pid"
    if [ -f ${PIDFILE} ];then echo "$0 already running, pid $(cat ${PIDFILE}), run rm ${PIDFILE} if you are sure it is not";exit;fi

    echo -n "$(date +'%Y/%m/%d %H:%M:%S')" >> $LOGFILE

    echo $$ > ${PIDFILE}
    SP=3 # spam level, 1 should be fine also
    if [ $1 ]; then SP=$1;fi
    WGET="wget -q https://opendata.fcc.gov/api/views/t3tq-usu8/rows.csv -O -"
    PRE= # Prefix to add to NXXNXXXXXX to suit your carriers CID
    SQL="sqlite3 /var/lib/asterisk/astdb.sqlite3"
    MYSQL="mysql -u root -pyourpassword"
    ASTPID=$(pidof asterisk)
    START=$(date +%s)
    CURRENT=$($SQL "select value from astdb where key like '/fcc/%'")
    ASTDBFCC=$($SQL "select * from astdb where key like '/fcc/%'")
    $SQL " select substr(substr(key,12),-10) from astdb where key like '/whitelist/%'"|sort >${LOGDIR}/fcc.whitelist

    $MYSL -ss -D asteriskcdrdb -e "select distinct substr(dst,-9) from cdr where length(dst) >= 10 order by calldate desc limit 100000" > ${LOGDIR}/fcc.calls


    ${WGET}|sed -e '1d' -e 's/-//g'|sort -n|tee ${LOGDIR}/fcc.numbers|uniq -c|tee ${LOGDIR}/fcc.unique |awk '{print "/blacklist/"$2"|FCC-"$1}'|sed -e "/-[$(seq -s '' 0 ${SP})]$/d" -e "s|list/|list/${PRE}|" > ${LOGDIR}/fcc.sp${SP}
    #grep -vwF -f ${LOGDIR}/fcc.whitelist ${LOGDIR}/fcc.sp${SP} > ${LOGDIR}/fcc.import
    grep -vF -f ${LOGDIR}/fcc.whitelist ${LOGDIR}/fcc.sp${SP}|grep -vF ${LOGDIR}/fcc.calls - > ${LOGDIR}/fcc.import
    FCC=$(cat ${LOGDIR}/fcc.numbers|wc -l)
    SPCNT=$(cat ${LOGDIR}/fcc.sp${SP}|wc -l)
    WLCNT=$(cat ${LOGDIR}/fcc.import|wc -l)
    echo -n " (wget in $(($(date +%s) - ${START})) secs) " >> $LOGFILE
    if [ "$1" == "" ] && [ "$CURRENT" == "$FCC" ] ;
    then
    echo "No new records, FCC=${FCC} astdb=${ASTDBFCC}" >>${LOGFILE}
    else
    if [ ${ASTPID} ] ; then /usr/sbin/rasterisk -x 'core stop when convenient' >/dev/null 2>&1;fi
    $SQL "DELETE from astdb WHERE key LIKE '/blacklist/%' AND value like 'FCC%'"
    $SQL "DELETE from astdb WHERE key LIKE '/fcc/%' "
    $SQL ".import ${LOGDIR}/fcc.import astdb"
    $SQL "insert into astdb values( strftime('/fcc/%Y-%m-%d--%H-%M','now','localtime') , ${FCC})"
    echo "FCC/${SP}=${SPCNT} of ${FCC} BL-WL=${WLCNT} Time ($(($(date +%s) - ${START})) secs) " >> $LOGFILE
    if [ ${ASTPID} ];then /usr/sbin/service asterisk start > /dev/null;fi
    fi
    rm -rf ${PIDFILE}
    exit 0


    Hopefully this is self documented and a little less obtuse.

    Then the context you can send calls to:-

    [from-nanp-trunks-susceptible-to-being-on-the-fcc-list]
    exten => _X.,1,Set(CALLERID(number)=${CALLERID(number):-10})
    same => n,Set(VALUE=${DB(blacklist/${CALLERID(num)})})
    same => n,Set(LEVEL=${CUT(VALUE,-,2)})
    same => n,gotoif($["${LEVEL}" = ""]?from-pstn,${EXTEN},1)
    same => n,Playback(custom/fccsorry)
    same => n,SayDigits(${CALLERID(num)})
    same => n,Playback(custom/fccisblacklisted)
    same => n,Set(PASSCODE=${RAND(100,999)})
    same => n,SayNumber(${level})
    same => n,Playback(custom/fccruhuman)
    same => n,SayDigits(${PASSCODE}
    same => n,PlayBack(custom/fccafterbeep)
    same => n,read(HUMAN,beep,3,,,5)
    same => n,gotoif($["${HUMAN}" = "${PASSCODE}" ]?voicemail:hangup)
    same => n(voicemail),PlayBack(custom/fccconfirmedcode)
    same => n,Voicemail(3101,s) ; WHERE 3101 is the the voicemail box of your sentient adult in charge
    same => n(hangup),hangup


    You get to record the custom/fcc*prompts, maybe something like

    fccsorry: . . . . . .We are sorry, but your callerID . . .
    fccisblacklisted: is on the fcc blacklist of robocallers and spammers for a count of
    fccruhuman: To prove you are a human please dial
    fccafterbeep: After the beep.
    fccconfirmcode: Thank you. Please leave a message after the beep, include your phone number,
    who you are and what your call is about.


    It covers the ongoing conversations at the other site that got fahrenheit 451'ed , first off it t confuses the machines, then it covers the FedEX guys to actually let you know they are regular, and you can whitelist them as soon as you get the email. You don't need Superfecta and it should cover the whitelist/blacklist contretemps as the traditionally the Asterisk blacklist just looks at the "value" this script filters on the existance of FCC in value,

    Also note that the context

    from-nanp-trunks-susceptible-to-being-on-the-fcc-list

    falls through to from-pstn "very quickly" if not needed. Those quick enough will notice that DB(blacklist*) could easily be replace with DB(fccblacklist*) making this whole snippet completely transparent to FreePBX yet attentive at level 1 to the FCC list (Please watch the logs, the list is NOT realtime, as some have stated, it updates at about every 25 to 28 hours, but might be real-time sometime real-soon (maybe) so please adjust down your cron job appropriately perhaps is sufficient (yes this is a little grumpy for root to do it:) )

    1 4,20 * * * su asterisk -c /"var/lib/asterisk/bin/fcc.sh"

    )

    Any peer comments would be much appreciated, especially if you have actual tried it :)
     
    #7 dicko, Jun 29, 2016
    Last edited: Jun 29, 2016
    wardmundy likes this.
  8. wardmundy

    wardmundy Nerd Uno

    Joined:
    Oct 12, 2007
    Messages:
    14,701
    Likes Received:
    2,512
  9. lrosenman

    lrosenman Guru

    Joined:
    Oct 17, 2014
    Messages:
    222
    Likes Received:
    30
    looks like you have a bad year on the originally published date:
    Originally published: Tuesday, September 13, 2015
     
  10. tbrummell

    tbrummell Guru

    Joined:
    Jan 8, 2011
    Messages:
    547
    Likes Received:
    43
    So, after spam calls started ramping up again on my "new" Ipbx 13-12.2 on SL with Incredible GUI 12.0.39, I decided to implement the FCC blacklist again. I followed Wards instructions from the Twitter post above, and also installed the Lenny Blacklist Mod. Imported the FCC entries successfully, and now when I attempt to go to Admin, Blacklist, I get returned with a 500 error "is currently unable to handle this request." Blacklist page did work before importing the FCC entries.

    Suggestions?
     
  11. tbrummell

    tbrummell Guru

    Joined:
    Jan 8, 2011
    Messages:
    547
    Likes Received:
    43
    And the relevent error in /http/error_log:

    [Tue Oct 18 08:57:25 2016] [error] [client xxx.xxx.187.227] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 40819223 bytes) in /var/www/html/admin/config.php on line 605
     
  12. tbrummell

    tbrummell Guru

    Joined:
    Jan 8, 2011
    Messages:
    547
    Likes Received:
    43
    So yeah, I should have used my Google Fu before posting. I increased the memory limit in php.ini, and the Blacklist loads, but it is slow.....but it loads.