PIONEERS Google Assistant & IncrediblePBX

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
As a person who is new to PBX and generally engaged from a hobbyists perspective I've been searching for a way to make SIP phones relevant to me contextually, as that context has shifted from using the phone a lot to now mainly using other IP technologies and products.

Since I'm just using this at home I have some freedom to experiment - while I'm excited to try out the Facebook Messenger webhooks people have been talking about here along with other WebRTC and XMPP APIs for web apps that I use (more frequently than I call people tbh), I think one of the biggest things I'd want from a phone system like this would be the ability to use Google Assistant from a SIP phone. I could finally return my ex-roommate's Google Home Mini, and have it stop eavesdropping on me all the time, listening in for an inopportunely placed BurgerKing Commercial.

Thankfully, all I had to do was search Google to see if anyone had implemented something close to what I was imagining; thankfully it looks like they have:

This is a great example, and it makes me want to go further setting up and experimenting with this functionality on my own instance. I have a lot in common with the setup the guy in that video describes, mainly because I'm running the ESXi image of IncrediblePBX 13-13 (So I can use Google Voice with ease primarily) CentOS is a given, unfortunately what isn't a given is... any level of instruction from this source! Like, he's an end solution contractor, he's not gonna give away the magic Google genie in the LAMP stack.

But I have a suspicion that he got all his secret information from an open source, that's always how the story goes. Russell Grokett, the apparent progenitor of this, has done us all the favor of putting a pretty comprehensive guide to setting this up over on his GitHub, and it looks fantastic - even including detailed PDFs of a step-by-step process. The only thing is he's targeting Raspberry Pi as a platform, yes I know Linux is Linux for the most part; and I'm probably going to go through his setup guide when I have the time over the next few days. The reason I'm posting this here for now is because I probably don't have a ton of time on my hands considering, I searched the PIAF forms for threads referencing Google Assistant and I haven't found any - hence why I'm starting one now, to save folks time in the future~

Take a look over the install instruction PDF for this if you have a sec, see if there's any red flags when it comes to doing things with CentOS / IncrediblePBX that are described; when I get around to trying it myself I'll report back here with what I've found.

Thanks
 

Attachments

  • RaspiAsteriskGoogleInstallation.pdf
    908.8 KB · Views: 30

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
So I've just put in a bunch of time into figuring this out, currently there are a few main limitations which are preventing the process from going smoothly, or presently, at all.

The first seems to be CentOS / Yum relying on Python 2.6, a bunch of dependencies for this project will not compile for it and pip warns constantly about its immanent depreciation. I followed @wardmundy 's guide from the SSL tutorial on setting up yum -y install python27 and scl enable python27 bash which let me install things successfully to that shell, but sudo doesn't execute there and its unlikely in deployment that the perl .agi files that this project relies upon copying to /usr/share/asterisk/agi-bin (a directory that doesn't seem to exist in the Incredible distro) will run from there or with 2.7.

The things that went well: I was able to run the test script, have google answer a recorded wav file prompt and generate an output of appropriate size. Generating OAUTH 2 tokens and getting the device registered to use assistant has been entirely straightforward, again all of this being run from the scl 2.7 enabled bash shell. Yum has different syntaxes for installing the rest of the dependencies cited in the installation pdf, but they're all within the rpm that Incredible 13-13 includes so once I found the names they all worked.

Some other conflicts of interest exist within the way asterisk is configured here and there though; in the raspbian build that the tutorial centers around, the install script directly edits /etc/asterisk/extensions.conf and /etc/asterisk/sip.conf, which presumably we don't want to do since all the changes made by incredible to add wolfram-alpha and other stuff are made to extensions_custom.conf. Also the tutorial doesn't assume any trunks, GVSIP or otherwise like I've set up for my one extension, and I don't want it to overwrite my working setup / add a nonexistant extension.

I might be hitting an upper limit to my capabilities and knowledge here with how to proceed further, the functionality being touted here is really exciting in my opinion, and would be a killer app / add on for both Incredible and 3CX users - so long as it isn't being used as an IVR itself or on an inward dialing number, the assistant api remains free to use, its great off the bat in a single use lab scenario like my own, but it could also be compartmentalized with different Oauth tokens for different extensions, giving people within an organization personal access to their own google assistants if they grant the app access. Being able to turn an old sip phone into a Google home that only listens when you want it to just seems, well, you know, incredible! And it would be super awesome to see a tool that sets up and configures this functionality in the same way one exists for adding GVSIP trunks!

I'm just not sure at the moment if I'm capable of making that kind of thing, or making this work on my own. If I had to say I'm 'stuck' at a phase right now it would be copying the agi scripts into asterisk, and getting them to show up at an extension within incredible's webUI; I'm not sure where exactly they should go in my 13-13 centOS install, and I'm not in a rush to totally break what I have working, but I do have a backup so I could go further.

Let me know what your thoughts are about this, I'd appreciate the input~ Thanks for reading!
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
I think in the meantime, when I get home from work tonight I'll try this again with a 13-13 install from the Ubuntu Virtualbox image, I'd anticipate better luck because Bionic Beaver is already running Python 2.7, uses Apt as it's package manager so the installation will be much more similar to the ones for Raspian.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
Do you know that @ABSGINC (the person responsible for that integration youtube video) is a member and contributor here on the website? If you get stuck, perhaps if you ask him here, he may offer some assistance.
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
Do you know that @ABSGINC (the person responsible for that integration youtube video) is a member and contributor here on the website? If you get stuck, perhaps if you ask him here, he may offer some assistance.
That's good to know! It seems like it's been a while since he's been online though,
In the meantime I've had a pretty good time setting up Ubuntu and installing Incredible 13-13 on it ☑
running most of the dependencies for the Assistant api and registering the device ☑
running the test script ☑

the real difference here that's getting to me now is the discrepancies between Incredible / FreePBX and vanilla asterisk. Like I said before, the example code relies heavily on modifying the extensions.conf (and I mirrored those changes to extensions_custom.conf because I know they get overwritten) but... none of that stuff shows up inside of the IncrediblePBX UI... ever...

Apparently changes made to extensions_custom.conf need to be manually added to the webUI, okay cool - so I went to custom destinations and added a google assistant one google_api,s,1 to reference google_api in extensions_custom which looks like this:

[google_api]
exten => 6666,1,Answer()
; Play prompts
exten => 6666,n,Playback(./custom/google_hello)
exten => 6666,n,Playback(./custom/google_example)
; Google Assistant SDK API integration
exten => 6666,n(record),agi(google.agi,en-us)
; Loop
exten => 6666,n,Playback(./custom/google_another)
exten => 6666,n,goto(record)
; These are not used currently
exten => 6666,n(goodbye),Playback(vm-goodbye)
exten => 6666,n,Hangup()

Another major difference between this installer and my installation of Incredible is that this wants to put the agi scripts in /usr/share/asterisk/agi-bin/google.agi whereas I know on my machine those should be at /var/lib/asterisk/agi-bin .
I've copied the content to both places just in case, but neither of them seems to work. With the custom destination set, and then this set up in the misc application pane I can get a little further:
NxJsOVm.png

But when I dial this I get:
WWTiXr5.jpg


So how can I add an agi script like this to an extension in IncrediblePBX such that it emulates how you'd normally do it in asterisk / this example? I'm sure for some of the verteran users here, this is a walk in the park. I'm not getting any super useful information from the logs yet (or anything that I understand for that matter), but right now it just seems like the agi script isn't executing at all. If I could go a step further to where I was actually contacting Google, it would make a world of difference.
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
Actually there were some relevant things I saw go flying by in the logs, I'm just not sure what to make of them just yet,
the only lines vaguely alluding to the script were:

[2018-11-08 19:49:15] WARNING[2484][C-00000001]: pbx.c:4418 __ast_pbx_run: Channel 'SIP/701-00000001' sent to invalid extension but no invalid handler: context,exten,priority=google_api,s,1

2018-11-08 19:49:15] DEBUG[1167]: cel_odbc.c:790 odbc_log: Executing SQL statement: [INSERT INTO cel (eventtype,eventtime,cid_name,cid_num,cid_ani,cid_rdnis,cid_dnid,exten,context,channame,appname,appdata,amaflags,accountcode,uniqueid,linkedid,peer,userdeftype,extra) VALUES ('HANGUP',{ts '2018-11-08 19:49:15.636145'},'701','701','701','','6666','s','google_api','SIP/701-00000001','','',3,'','1541706555.1','1541706555.1','','','{"hangupcause":0,"hangupsource":"","dialstatus":""}')]

I'm not seeing any of the debug lines from the AGI script though so I don't think it's getting to the execution stage, it's just referencing the method I set aside in the custom destinations dialogue of Incredible's UI
 

ABSGINC

You can call me Scott.
Joined
Oct 1, 2014
Messages
59
Reaction score
31
...he's not gonna give away the magic Google genie in the LAMP stack.


there are no secrets, just a dialplan logic that plays typing keys while processing the curl request...all similar to the open source stuff out there..

you can use a number of already existing examples using the Amazon Alexa or Google Assistant SDK with relative ease. My next efforts was going to create a web registration process front end that could walk a user dialing to the application via webhooks to "link" their amazon or google account to an assistant speed dial (per the PBX remote party ID / caller ID) haven't got that far yet..but surely you dont want an individual google account accessible to anyone dialing on a pbx.. because this is what that is.. my home pbx i guess. I was waiting for someone else to give a shit about this kind of project.. i work full time as a telephony engineer, but this is still a hobby of mine, and i certainly don't mind making enhancements or extended contributions or open source efforts.

Screen-Shot-2018-11-11-at-5.19.23-PM.png


Screen-Shot-2018-11-11-at-5.22.17-PM.png



its been almost a year since i patched this concept together from googles documentation on there SDK, and yes the raspberry pi examples, --mostly need to have the right dependencies for the OS / setup you are targeting, I'm on Centos 6, and i cant remember the dependencies I had to have to first get the test scripts to communicate with google and their whole authentication process. once you tackle that monster it gets easier. Might as well do both google and amazon at once like I did last year. I'm sure there will prove to be benefits from having these webhooks mastered going down the road as a developer.

. I planned on bringing it back to life on some commercial implementations i have on my work plate, but I'm always for providing hints to the open source community on how to hack some cool stuff together. i wont be able to give away any commercial stuff i was paid to design, but any of the open source concepts I'm proud to guide people to also achieve the same results.
 

ABSGINC

You can call me Scott.
Joined
Oct 1, 2014
Messages
59
Reaction score
31
That's good to know! It seems like it's been a while since he's been online though,


So how can I add an agi script like this to an extension in IncrediblePBX such that it emulates how you'd normally do it in asterisk / this example? I'm sure for some of the verteran users here, this is a walk in the park. I'm not getting any super useful information from the logs yet (or anything that I understand for that matter), but right now it just seems like the agi script isn't executing at all. If I could go a step further to where I was actually contacting Google, it would make a world of difference.

Screen-Shot-2018-11-11-at-7.09.21-PM.png


Screen-Shot-2018-11-11-at-7.09.43-PM.png

Screen-Shot-2018-11-11-at-7.10.17-PM.png


Screen-Shot-2018-11-11-at-7.10.34-PM.png


This looks like the agi file i hacked together for freepbx to work with the dialplan adjustments i also shared in this thread. its not meant to handle more then one request, but can easily include adjustments for registering users, etc.. Have you seen the facebooks webhooks implementations on here? -- anyway here is what i use for google.agi


Code:
#!/usr/bin/perl
# -*- coding: utf-8 -*-
# vim: set et sw = 4 hay = utf-8:

#
# This program is free software, distributed under the terms of
# the GNU General Public License Version 2.
#
#
# Version 1.0 - 2018-05-13
#
#

use warnings;
use strict;

$| = 1;

# ----------------------------- #
#   User defined parameters:    #
# ----------------------------- #
# Default language (future)     #
my $language = "en-us";

# Default max silence timeout   #
my $timeout = 2;

# Absolute Recording timeout    #
my $abs_timeout = -1;

# Default interrupt key         #
my $intkey = "#";

# Verbose debugging messages    #
my $debug = 1;

# ----------------------------- #

my %AGI;
my $ do;
my $fh;
my @result;
my $name;
my $audio;
my $uarequest;
my $uaresponse;
my %response;
my $endian;
my $silence;
my $filetype;
my $json;
my $results    = 1;
my $beep       = "googleastchime";
my $comp_level = -8;
my $ua_timeout = 10;
my $tmpdir     = "/tmp";
my $tmpname    = "$tmpdir/google_audio";
my $sox        = "/usr/bin/sox";
my $format     = "wav";
my $command    = "/root/googleast/bin/googlesamples-assistant-pushtotalk";
my $oauth_dir  = "/home/asterisk/.config/google-oauthlib-tool";
my $api_json_file = "$oauth_dir/credentials.json";
my $thank    = "/var/lib/asterisk/sounds/custom/keystyping";

# Store AGI input #
($ AGI {arg_1}, $ AGI {arg_2}, $ AGI {arg_3}, $ AGI {arg_4}) = @ARGV;
while (<STDIN>) {
    chomp;
    last if (!length);
    $AGI{$1} = $2 if (/^agi_(\w+)\:\s+(.*)$/);
}

$name = " -- $AGI{request}:";
console_log ("Starting...") if ($debug);

# Setting language, timeout, interrupt keys and BEEP indication #
if (length($AGI{arg_1})) {
    $language = $AGI{arg_1} if ($AGI{arg_1} =~ /^[a-z]{2}(-[a-zA-Z]{2,6})?$/);
}

if (length($AGI{arg_2})) {
    if ($ AGI {arg_2} == -1) {
        $silence = "";
    } elsif ($ AGI {arg_2} = ~ / ^ \ d + $ /) {
        $silence = "s=$AGI{arg_2}";
    } else {
        $silence = "s=$timeout";
    }
} else {
    $silence = "s=$timeout";
}

if (length($AGI{arg_3})) {
    $intkey = "0123456789#*" if ($AGI{arg_3} eq "any");
    $ intkey = $ AGI {arg_3} if ($ AGI {arg_3} = ~ / ^ [0-9 * #] + $ /);
}

if (length($AGI{arg_4})) {
    $beep = "" if ($AGI{arg_4} eq "NOBEEP");
}

# Answer channel if not already answered #
console_log ("Checking channel status.") if ($debug);
print "CHANNEL STATUS\n";
@result = checkresponse();
if ($result[0] == 4) {
    console_log ("Answering channel.") if ($debug);
    print "ANSWER\n";
    @result = checkresponse();
    if ($result[0] != 0) {
        die "$name Failed to answer channel.\n";
    }
}

# Handle interrupts
$ SIG {'INT'} = \ & int_handler;
$ SIG {'HUP'} = \ & int_handler;


####
# START OF REQ/RESP AUDIO HANDLING
####

# RECORD REQUEST AUDIOFILE
console_log ("RECORD FILE $tmpname $format $intkey $abs_timeout $beep $silence") if ($debug);
print "RECORD FILE $tmpname $format \"$intkey\" \"$abs_timeout\" 0 \"$silence\"\n";
@result = checkresponse();
die "$name Failed to record file, aborting...\n" if ($result[0] == -1);


# RE-ENCODE REQUEST WAV AUDIO FILE FROM 8000 TO 16000
console_log ("Converting $sox $tmpname.$format.") if ($debug);
my $cmd = "$sox $tmpname.$format -r 16000 ${tmpname}_in.wav";
console_log ("CMD: $cmd") if ($debug);
my $status = qx/$cmd/;


# CLEAN
if (!$debug)
{
    unlink glob "$tmpname*";
}



exit;


#------------------------------------------



sub checkresponse {
    my $input = <STDIN>;
    my @values;

    chomp $input;
    if ($input =~ /^200 result=(-?\d+)\s?(.*)$/) {
        warn ("Command returned: $input\n") if ($debug);
        @values = ("$1", "$2");
    } else {
        $input .= <STDIN> if ($input =~ /^520-Invalid/);
        warn ("Unexpected result: $input\n");
        @values = (-1, -1);
    }
    return @values;
}


sub int_handler {
    die "$name Interrupt signal received, terminating...\n";
}

sub playback {
        my ($file, $keys) = @_;
        my @response;

        print "STREAM FILE $file \"$keys\"\n";
        @response = checkresponse();
        if ($response[0] >= 32 && chr($response[0]) =~ /[\w*#]/) {
                console_log("Got digit chr($response[0])") if ($debug);
                print "SET EXTENSION ", chr($response[0]), "\n";
                checkresponse();
                print "SET PRIORITY 1\n";
                checkresponse();
        } elsif ($response[0] == -1) {
                console_log("Failed to play $file.");
        }
        return $response[0];
}

sub console_log {
        foreach my $message (@_) {
                warn "$name $message\n";
                print "NOOP \"$name $message\"\n";
                checkresponse();
        }
}


END {
    if ($tmpname) {
        console_log ("Cleaning temp files.") if ($debug);
        #unlink glob "$tmpname.*";
    }
}
 

ABSGINC

You can call me Scott.
Joined
Oct 1, 2014
Messages
59
Reaction score
31
:lurk5::lurk5:
......, its great off the bat in a single use lab scenario like my own, but it could also be compartmentalized with different Oauth tokens for different extensions, giving people within an organization personal access to their own google assistants if they grant the app access. ...

:iagree:

I like the way you think @Kezzism -- this is exactly where i left off with it early this year.. seems you found my video on the absg blog midst of miles of distractions and job offers. It was eventually my intention to release a walk through and motivate @wardmundy like I did last year with the Facebook webhooks ;-) I always love marrying the open source AI and communication examples in our open source PBX's..

:beta1:
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
This looks like the agi file i hacked together for freepbx to work with the dialplan adjustments i also shared in this thread. its not meant to handle more then one request, but can easily include adjustments for registering users, etc.. Have you seen the facebooks webhooks implementations on here? -- anyway here is what i use for google.agi

Thank you! This is exactly what I was looking for and will probably be the only thing between me and getting this working~ :tt1:
Sorry if I came off a little rough at first, but I've been reading the Facebook Webhooks thread and elsewhere and the contributions you've made here are nothing short of, incredible!

I really do think all this could scale well, you're absolutely correct about the next step being making a setup / UI such that people in an organization could use to log into their own accounts. I'm very new to all this and I'm not sure how I'd be able to help pull that off, but its been a ton of fun thus far just messing around with these VMs and test servers - I think we're making great strides towards having this be a thing we could provide instruction for in the PIAF community at least. All of this is thousands of times better than those Alexa equipped antique phones that started selling for $1500 today :icon bs:

I should have the free time over the next few days to try this out, best of luck in your endeavors in the meantime!!
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
So I'm at a bit of a loss when it comes to where to start with all this? I have can get through to the Google API and I've registered my pbx server as a device, that all works well - but I can't see if the file you've made your main edits to is extensions.conf or extensions_custom.conf, or even something else entirely.

I think a step by step process is what we've set out to make here, I just need to see things a bit more explicitly to get further.
Thanks
 

ABSGINC

You can call me Scott.
Joined
Oct 1, 2014
Messages
59
Reaction score
31
It's been a busy year, but I want to revisit this and make this awesome for 2019. I'll most likely be starting from near scratch again and I'll try to document the recipe better, but hope to find others to experiment with me on these procedures.

@Kezzism , You need something in extensions_custom.conf to catch and execute that agi, (record your voice, send to google, save response) then play back.. it's essentially a dialplan application.

UNFORTUNATELY. I have been asleep at the wheel in my dev lab where I had over 7 VPSs running in my lab on Hiformance, this example was one of them, and those boxes are toast. Slowly BUT SURELY I am rebuilding my dev lab in a new cloud environment where I plan to build a media server running asterisk to handle inbound Google Assistant and Alexa calls.

As more and more AI take over our daily lives I still think this is one of the coolest addons to our PBX, not as an inbound dial service, but as end user service. There are also lots of possibility in learning Googles Actions and Intents system and build a voice application you could invoke from your PBX to turn your lights on, open your garage door, etc..

LOL, the Amazon enabled antique phones, ... trying to not think about building these .....
 

Alberto M Gil C

Alberto Gil
Joined
Dec 29, 2018
Messages
1
Reaction score
1
Hello, I am interested in buying this development, or support with economic resources.
I have tried to connect with ABSGINC by multiple means, but I have not had luck. My wish is to connect dialogflow - google assistant - Isabell PBX.
I would be grateful if you would ask me for any help you need to make this a reality and inform me of any progress; I'm to order.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
I, too, have dropped the ball on this. But I'm paying attention now. CentOS 6 is definitely the most stable platform if we can get it to work. I'll wrestle with Ubuntu release of Incredible PBX as well. @ABSGINC lost almost as many HiFormance servers as we did. But we've found a new platform as long as you have good off-site backups. :) We were back in business in one afternoon. And the performance is even better than HiFormance... so far.

TIP to those posting photos/images in the forum: Post your image to Twitter first. Then link to the Twitter image here.
 
Last edited:

ABSGINC

You can call me Scott.
Joined
Oct 1, 2014
Messages
59
Reaction score
31
@wardmundy I agree, Centos 6 sounds most stable, dare I say to start with a vanilla install of Incredible13-13 .. I'm back online in various Cloud environments testing perforamnce this week and will make better documentation of the dependencies required above our standard 13-13 install and try to go from there.

First steps in these ventures I usually start with getting an SSL certificate on the BOX, might not even be necessary with these APIs for the minimal functionality I demonstrated in the video from last year, but if we want to allow for a webpage to do the hookbacks to google and amazon token stuff.. might as well get the SSL out the way.

Honestly the biggest challenge will be to automate a registration process that perhaps visiting a URL (behind our PBX's apache) , this URL to then "Authenticate" a google and amazon account. This is how we tie the assistant voice to a particular account data. There is a refresh token, a key, you know, the whole oauth token fun stuff.. In my past server examples and setups I have manually generated this token and access credentials via the linux shell, works fine for playing around, but anything that we want to be plug and play ready and friendly like voice assistants are supposed to be will benefit to some backend to front end web programming to make this part hook up..
 

aigyu

New Member
Joined
Aug 22, 2019
Messages
2
Reaction score
0
@ABSGINC, by any luck, did you get some time to make a tutorial for it so everyone can implement it?
 

Kezzism

System scavenger
Joined
Oct 12, 2018
Messages
9
Reaction score
4
Recently switched to VitalPBX since it runs well on a Pi and a large part of my inspiration was to see if I could finally use this to get Google Assistant up and running, the good news is it all works flawlessly! There are some minor differences in the overall design of the CGI script (you can see everything in greater detail on the github https://github.com/VitalPBX/VitalPBX_Google_Assistant) but the instructions were easy to follow and worked!

I think what I was missing in my previous attempts last year was changing the ownership of these files to belong to Asterisk, well, that and maybe putting certain ones in the right place. If you do follow these directions step by step you'll get a working Google Assistant instance on *789 but by default if you grab the sln files it starts with an annoying plug and a spoken example. Remove google_welcome and google_example to shorten the time until you get an initial prompt. Also you could encode your own prompts or grab some of rgrokett's original ones, as long as they start with google_ it'll find them in /var/lib/asterisk/sounds/en

Only negative observation is it's slow as hell to respond with an answer... But I entirely blame that on the fact it's using an older Pi 2,I'll have to see if upgrading to a 3B+ or 4 helps.
 

Members online

No members online now.

Forum statistics

Threads
25,781
Messages
167,507
Members
19,201
Latest member
troutpocket
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top