PIONEERS Google Assistant & IncrediblePBX

Discussion in 'Developers' Corner' started by Kezzism, Oct 16, 2018.

  1. Kezzism

    Kezzism System scavenger

    Joined:
    Oct 12, 2018
    Messages:
    6
    Likes Received:
    2
    As a person who is new to PBX and generally engaged from a hobbyists perspective I've been searching for a way to make SIP phones relevant to me contextually, as that context has shifted from using the phone a lot to now mainly using other IP technologies and products.

    Since I'm just using this at home I have some freedom to experiment - while I'm excited to try out the Facebook Messenger webhooks people have been talking about here along with other WebRTC and XMPP APIs for web apps that I use (more frequently than I call people tbh), I think one of the biggest things I'd want from a phone system like this would be the ability to use Google Assistant from a SIP phone. I could finally return my ex-roommate's Google Home Mini, and have it stop eavesdropping on me all the time, listening in for an inopportunely placed BurgerKing Commercial.

    Thankfully, all I had to do was search Google to see if anyone had implemented something close to what I was imagining; thankfully it looks like they have:


    This is a great example, and it makes me want to go further setting up and experimenting with this functionality on my own instance. I have a lot in common with the setup the guy in that video describes, mainly because I'm running the ESXi image of IncrediblePBX 13-13 (So I can use Google Voice with ease primarily) CentOS is a given, unfortunately what isn't a given is... any level of instruction from this source! Like, he's an end solution contractor, he's not gonna give away the magic Google genie in the LAMP stack.

    But I have a suspicion that he got all his secret information from an open source, that's always how the story goes. Russell Grokett, the apparent progenitor of this, has done us all the favor of putting a pretty comprehensive guide to setting this up over on his GitHub, and it looks fantastic - even including detailed PDFs of a step-by-step process. The only thing is he's targeting Raspberry Pi as a platform, yes I know Linux is Linux for the most part; and I'm probably going to go through his setup guide when I have the time over the next few days. The reason I'm posting this here for now is because I probably don't have a ton of time on my hands considering, I searched the PIAF forms for threads referencing Google Assistant and I haven't found any - hence why I'm starting one now, to save folks time in the future~

    Take a look over the install instruction PDF for this if you have a sec, see if there's any red flags when it comes to doing things with CentOS / IncrediblePBX that are described; when I get around to trying it myself I'll report back here with what I've found.

    Thanks
     

    Attached Files:

    krzykat and wardmundy like this.
  2. Kezzism

    Kezzism System scavenger

    Joined:
    Oct 12, 2018
    Messages:
    6
    Likes Received:
    2
    So I've just put in a bunch of time into figuring this out, currently there are a few main limitations which are preventing the process from going smoothly, or presently, at all.

    The first seems to be CentOS / Yum relying on Python 2.6, a bunch of dependencies for this project will not compile for it and pip warns constantly about its immanent depreciation. I followed @wardmundy 's guide from the SSL tutorial on setting up yum -y install python27 and scl enable python27 bash which let me install things successfully to that shell, but sudo doesn't execute there and its unlikely in deployment that the perl .agi files that this project relies upon copying to /usr/share/asterisk/agi-bin (a directory that doesn't seem to exist in the Incredible distro) will run from there or with 2.7.

    The things that went well: I was able to run the test script, have google answer a recorded wav file prompt and generate an output of appropriate size. Generating OAUTH 2 tokens and getting the device registered to use assistant has been entirely straightforward, again all of this being run from the scl 2.7 enabled bash shell. Yum has different syntaxes for installing the rest of the dependencies cited in the installation pdf, but they're all within the rpm that Incredible 13-13 includes so once I found the names they all worked.

    Some other conflicts of interest exist within the way asterisk is configured here and there though; in the raspbian build that the tutorial centers around, the install script directly edits /etc/asterisk/extensions.conf and /etc/asterisk/sip.conf, which presumably we don't want to do since all the changes made by incredible to add wolfram-alpha and other stuff are made to extensions_custom.conf. Also the tutorial doesn't assume any trunks, GVSIP or otherwise like I've set up for my one extension, and I don't want it to overwrite my working setup / add a nonexistant extension.

    I might be hitting an upper limit to my capabilities and knowledge here with how to proceed further, the functionality being touted here is really exciting in my opinion, and would be a killer app / add on for both Incredible and 3CX users - so long as it isn't being used as an IVR itself or on an inward dialing number, the assistant api remains free to use, its great off the bat in a single use lab scenario like my own, but it could also be compartmentalized with different Oauth tokens for different extensions, giving people within an organization personal access to their own google assistants if they grant the app access. Being able to turn an old sip phone into a Google home that only listens when you want it to just seems, well, you know, incredible! And it would be super awesome to see a tool that sets up and configures this functionality in the same way one exists for adding GVSIP trunks!

    I'm just not sure at the moment if I'm capable of making that kind of thing, or making this work on my own. If I had to say I'm 'stuck' at a phase right now it would be copying the agi scripts into asterisk, and getting them to show up at an extension within incredible's webUI; I'm not sure where exactly they should go in my 13-13 centOS install, and I'm not in a rush to totally break what I have working, but I do have a backup so I could go further.

    Let me know what your thoughts are about this, I'd appreciate the input~ Thanks for reading!
     
  3. Kezzism

    Kezzism System scavenger

    Joined:
    Oct 12, 2018
    Messages:
    6
    Likes Received:
    2
    I think in the meantime, when I get home from work tonight I'll try this again with a 13-13 install from the Ubuntu Virtualbox image, I'd anticipate better luck because Bionic Beaver is already running Python 2.7, uses Apt as it's package manager so the installation will be much more similar to the ones for Raspian.
     
  4. krzykat

    krzykat Guru

    Joined:
    Aug 2, 2008
    Messages:
    1,331
    Likes Received:
    320
    Do you know that @ABSGINC (the person responsible for that integration youtube video) is a member and contributor here on the website? If you get stuck, perhaps if you ask him here, he may offer some assistance.
     
    ABSGINC and Kezzism like this.
  5. Kezzism

    Kezzism System scavenger

    Joined:
    Oct 12, 2018
    Messages:
    6
    Likes Received:
    2
    That's good to know! It seems like it's been a while since he's been online though,
    In the meantime I've had a pretty good time setting up Ubuntu and installing Incredible 13-13 on it ☑
    running most of the dependencies for the Assistant api and registering the device ☑
    running the test script ☑

    the real difference here that's getting to me now is the discrepancies between Incredible / FreePBX and vanilla asterisk. Like I said before, the example code relies heavily on modifying the extensions.conf (and I mirrored those changes to extensions_custom.conf because I know they get overwritten) but... none of that stuff shows up inside of the IncrediblePBX UI... ever...

    Apparently changes made to extensions_custom.conf need to be manually added to the webUI, okay cool - so I went to custom destinations and added a google assistant one google_api,s,1 to reference google_api in extensions_custom which looks like this:

    [google_api]
    exten => 6666,1,Answer()
    ; Play prompts
    exten => 6666,n,Playback(./custom/google_hello)
    exten => 6666,n,Playback(./custom/google_example)
    ; Google Assistant SDK API integration
    exten => 6666,n(record),agi(google.agi,en-us)
    ; Loop
    exten => 6666,n,Playback(./custom/google_another)
    exten => 6666,n,goto(record)
    ; These are not used currently
    exten => 6666,n(goodbye),Playback(vm-goodbye)
    exten => 6666,n,Hangup()

    Another major difference between this installer and my installation of Incredible is that this wants to put the agi scripts in /usr/share/asterisk/agi-bin/google.agi whereas I know on my machine those should be at /var/lib/asterisk/agi-bin .
    I've copied the content to both places just in case, but neither of them seems to work. With the custom destination set, and then this set up in the misc application pane I can get a little further:
    [​IMG]
    But when I dial this I get:
    [​IMG]

    So how can I add an agi script like this to an extension in IncrediblePBX such that it emulates how you'd normally do it in asterisk / this example? I'm sure for some of the verteran users here, this is a walk in the park. I'm not getting any super useful information from the logs yet (or anything that I understand for that matter), but right now it just seems like the agi script isn't executing at all. If I could go a step further to where I was actually contacting Google, it would make a world of difference.
     
  6. Kezzism

    Kezzism System scavenger

    Joined:
    Oct 12, 2018
    Messages:
    6
    Likes Received:
    2
    Actually there were some relevant things I saw go flying by in the logs, I'm just not sure what to make of them just yet,
    the only lines vaguely alluding to the script were:

    [2018-11-08 19:49:15] WARNING[2484][C-00000001]: pbx.c:4418 __ast_pbx_run: Channel 'SIP/701-00000001' sent to invalid extension but no invalid handler: context,exten,priority=google_api,s,1

    2018-11-08 19:49:15] DEBUG[1167]: cel_odbc.c:790 odbc_log: Executing SQL statement: [INSERT INTO cel (eventtype,eventtime,cid_name,cid_num,cid_ani,cid_rdnis,cid_dnid,exten,context,channame,appname,appdata,amaflags,accountcode,uniqueid,linkedid,peer,userdeftype,extra) VALUES ('HANGUP',{ts '2018-11-08 19:49:15.636145'},'701','701','701','','6666','s','google_api','SIP/701-00000001','','',3,'','1541706555.1','1541706555.1','','','{"hangupcause":0,"hangupsource":"","dialstatus":""}')]

    I'm not seeing any of the debug lines from the AGI script though so I don't think it's getting to the execution stage, it's just referencing the method I set aside in the custom destinations dialogue of Incredible's UI
     
  7. ABSGINC

    ABSGINC You can call me Scott.

    Joined:
    Oct 1, 2014
    Messages:
    57
    Likes Received:
    25

    there are no secrets, just a dialplan logic that plays typing keys while processing the curl request...all similar to the open source stuff out there..

    you can use a number of already existing examples using the Amazon Alexa or Google Assistant SDK with relative ease. My next efforts was going to create a web registration process front end that could walk a user dialing to the application via webhooks to "link" their amazon or google account to an assistant speed dial (per the PBX remote party ID / caller ID) haven't got that far yet..but surely you dont want an individual google account accessible to anyone dialing on a pbx.. because this is what that is.. my home pbx i guess. I was waiting for someone else to give a shit about this kind of project.. i work full time as a telephony engineer, but this is still a hobby of mine, and i certainly don't mind making enhancements or extended contributions or open source efforts.

    [​IMG]

    [​IMG]


    its been almost a year since i patched this concept together from googles documentation on there SDK, and yes the raspberry pi examples, --mostly need to have the right dependencies for the OS / setup you are targeting, I'm on Centos 6, and i cant remember the dependencies I had to have to first get the test scripts to communicate with google and their whole authentication process. once you tackle that monster it gets easier. Might as well do both google and amazon at once like I did last year. I'm sure there will prove to be benefits from having these webhooks mastered going down the road as a developer.

    . I planned on bringing it back to life on some commercial implementations i have on my work plate, but I'm always for providing hints to the open source community on how to hack some cool stuff together. i wont be able to give away any commercial stuff i was paid to design, but any of the open source concepts I'm proud to guide people to also achieve the same results.
     
  8. ABSGINC

    ABSGINC You can call me Scott.

    Joined:
    Oct 1, 2014
    Messages:
    57
    Likes Received:
    25
    [​IMG]

    [​IMG]
    [​IMG]

    [​IMG]

    This looks like the agi file i hacked together for freepbx to work with the dialplan adjustments i also shared in this thread. its not meant to handle more then one request, but can easily include adjustments for registering users, etc.. Have you seen the facebooks webhooks implementations on here? -- anyway here is what i use for google.agi


    Code:
    #!/usr/bin/perl
    # -*- coding: utf-8 -*-
    # vim: set et sw = 4 hay = utf-8:
    
    #
    # This program is free software, distributed under the terms of
    # the GNU General Public License Version 2.
    #
    #
    # Version 1.0 - 2018-05-13
    #
    #
    
    use warnings;
    use strict;
    
    $| = 1;
    
    # ----------------------------- #
    #   User defined parameters:    #
    # ----------------------------- #
    # Default language (future)     #
    my $language = "en-us";
    
    # Default max silence timeout   #
    my $timeout = 2;
    
    # Absolute Recording timeout    #
    my $abs_timeout = -1;
    
    # Default interrupt key         #
    my $intkey = "#";
    
    # Verbose debugging messages    #
    my $debug = 1;
    
    # ----------------------------- #
    
    my %AGI;
    my $ do;
    my $fh;
    my @result;
    my $name;
    my $audio;
    my $uarequest;
    my $uaresponse;
    my %response;
    my $endian;
    my $silence;
    my $filetype;
    my $json;
    my $results    = 1;
    my $beep       = "googleastchime";
    my $comp_level = -8;
    my $ua_timeout = 10;
    my $tmpdir     = "/tmp";
    my $tmpname    = "$tmpdir/google_audio";
    my $sox        = "/usr/bin/sox";
    my $format     = "wav";
    my $command    = "/root/googleast/bin/googlesamples-assistant-pushtotalk";
    my $oauth_dir  = "/home/asterisk/.config/google-oauthlib-tool";
    my $api_json_file = "$oauth_dir/credentials.json";
    my $thank    = "/var/lib/asterisk/sounds/custom/keystyping";
    
    # Store AGI input #
    ($ AGI {arg_1}, $ AGI {arg_2}, $ AGI {arg_3}, $ AGI {arg_4}) = @ARGV;
    while (<STDIN>) {
        chomp;
        last if (!length);
        $AGI{$1} = $2 if (/^agi_(\w+)\:\s+(.*)$/);
    }
    
    $name = " -- $AGI{request}:";
    console_log ("Starting...") if ($debug);
    
    # Setting language, timeout, interrupt keys and BEEP indication #
    if (length($AGI{arg_1})) {
        $language = $AGI{arg_1} if ($AGI{arg_1} =~ /^[a-z]{2}(-[a-zA-Z]{2,6})?$/);
    }
    
    if (length($AGI{arg_2})) {
        if ($ AGI {arg_2} == -1) {
            $silence = "";
        } elsif ($ AGI {arg_2} = ~ / ^ \ d + $ /) {
            $silence = "s=$AGI{arg_2}";
        } else {
            $silence = "s=$timeout";
        }
    } else {
        $silence = "s=$timeout";
    }
    
    if (length($AGI{arg_3})) {
        $intkey = "0123456789#*" if ($AGI{arg_3} eq "any");
        $ intkey = $ AGI {arg_3} if ($ AGI {arg_3} = ~ / ^ [0-9 * #] + $ /);
    }
    
    if (length($AGI{arg_4})) {
        $beep = "" if ($AGI{arg_4} eq "NOBEEP");
    }
    
    # Answer channel if not already answered #
    console_log ("Checking channel status.") if ($debug);
    print "CHANNEL STATUS\n";
    @result = checkresponse();
    if ($result[0] == 4) {
        console_log ("Answering channel.") if ($debug);
        print "ANSWER\n";
        @result = checkresponse();
        if ($result[0] != 0) {
            die "$name Failed to answer channel.\n";
        }
    }
    
    # Handle interrupts
    $ SIG {'INT'} = \ & int_handler;
    $ SIG {'HUP'} = \ & int_handler;
    
    
    ####
    # START OF REQ/RESP AUDIO HANDLING
    ####
    
    # RECORD REQUEST AUDIOFILE
    console_log ("RECORD FILE $tmpname $format $intkey $abs_timeout $beep $silence") if ($debug);
    print "RECORD FILE $tmpname $format \"$intkey\" \"$abs_timeout\" 0 \"$silence\"\n";
    @result = checkresponse();
    die "$name Failed to record file, aborting...\n" if ($result[0] == -1);
    
    
    # RE-ENCODE REQUEST WAV AUDIO FILE FROM 8000 TO 16000
    console_log ("Converting $sox $tmpname.$format.") if ($debug);
    my $cmd = "$sox $tmpname.$format -r 16000 ${tmpname}_in.wav";
    console_log ("CMD: $cmd") if ($debug);
    my $status = qx/$cmd/;
    
    
    # CLEAN
    if (!$debug)
    {
        unlink glob "$tmpname*";
    }
    
    
    
    exit;
    
    
    #------------------------------------------
    
    
    
    sub checkresponse {
        my $input = <STDIN>;
        my @values;
    
        chomp $input;
        if ($input =~ /^200 result=(-?\d+)\s?(.*)$/) {
            warn ("Command returned: $input\n") if ($debug);
            @values = ("$1", "$2");
        } else {
            $input .= <STDIN> if ($input =~ /^520-Invalid/);
            warn ("Unexpected result: $input\n");
            @values = (-1, -1);
        }
        return @values;
    }
    
    
    sub int_handler {
        die "$name Interrupt signal received, terminating...\n";
    }
    
    sub playback {
            my ($file, $keys) = @_;
            my @response;
    
            print "STREAM FILE $file \"$keys\"\n";
            @response = checkresponse();
            if ($response[0] >= 32 && chr($response[0]) =~ /[\w*#]/) {
                    console_log("Got digit chr($response[0])") if ($debug);
                    print "SET EXTENSION ", chr($response[0]), "\n";
                    checkresponse();
                    print "SET PRIORITY 1\n";
                    checkresponse();
            } elsif ($response[0] == -1) {
                    console_log("Failed to play $file.");
            }
            return $response[0];
    }
    
    sub console_log {
            foreach my $message (@_) {
                    warn "$name $message\n";
                    print "NOOP \"$name $message\"\n";
                    checkresponse();
            }
    }
    
    
    END {
        if ($tmpname) {
            console_log ("Cleaning temp files.") if ($debug);
            #unlink glob "$tmpname.*";
        }
    }
    
    
     
  9. kdthomas

    kdthomas Member

    Joined:
    May 13, 2016
    Messages:
    45
    Likes Received:
    8
    I'd be very interested in adding this to my Home PBX. Following...
     
    ABSGINC likes this.
  10. ABSGINC

    ABSGINC You can call me Scott.

    Joined:
    Oct 1, 2014
    Messages:
    57
    Likes Received:
    25
    :lurk5::lurk5:
    :iagree:

    I like the way you think @Kezzism -- this is exactly where i left off with it early this year.. seems you found my video on the absg blog midst of miles of distractions and job offers. It was eventually my intention to release a walk through and motivate @wardmundy like I did last year with the Facebook webhooks ;-) I always love marrying the open source AI and communication examples in our open source PBX's..

    :beta1:
     
    krzykat likes this.

Share This Page