TUTORIAL Google Contacts to Asterisk Phonebook

Rrrr

Tink
Joined
May 28, 2009
Messages
343
Reaction score
25
Attached is my version with some additional key features, further explained in the code:
# 1. Can generate dial codes
# 2. Select Google contact group or all contacts
# 3. Unify international prefixes to + or 00 or 011
# 4. Cleanup names that contain characters with accents (most SIP devices do not support utf8)
# 5. Allow CLI manual input
# 6. Replace or create new contacts - Avoids Google from blocking you if you have many contacts. You can use this feature to download all your Google Contacts once in a while (manually or monthly cron) and have a daily cron for a group, eg PBX Phonebook, with dial codes
# 7. Tracing: you can set your level of print output to trace whats happening.

Warning: I am not an experienced developer and 1st time python. The code tends to provide me with much learning, so I explain a lot in the comments, mainly to myself.
I will be happy to receive input on improving the code, the use of API v3 and pythonizing.

Instructions:
Unpack and save the file to /root/Google/gdata-2.0.18/
chmod +x googlecontactsASTERIDEX0.91.py
cd /root/Google/gd*
Use an editor to set the global variables, such as your email name, password, maxresults, etc.
./googlecontactsASTERIDEX0.91.py


---
new version in #31
 

Rrrr

Tink
Joined
May 28, 2009
Messages
343
Reaction score
25
...
You can automate the import process to keep your Google contacts current in Asterisk by adding the script to /etc/crontab and running it each day:

Code:
9 0 * * * root /root/Google/gdata-2.0.18/googlecontacts.py >/dev/null 2>&1


As an alternative, I looked this up and I put googlecontacts.py into /etc/cron.daily
But that does not work, despite having this line in /etc/crontab:
Code:
25 6    * * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

Just curious for an explanation?
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Put the command in a bash script and run the script as a cron job.
 

centoasa

Member
Joined
May 31, 2009
Messages
118
Reaction score
1
Answering my own question, you have to insert 4-space indents just as shown in the OP of this thread. Simply copying the text from nerdvittles, as I did, leaves it unformatted.
Sorry, I don't understand: I've always the same error. I've put my country code (for Italy 39), but always error.
 

Rrrr

Tink
Joined
May 28, 2009
Messages
343
Reaction score
25
What exactly is your error?
If you use the first script and the error is something like in post #13, then you need to edit the line it complains about.
Best is to remove all empty space before the first word and then add back spaces, until it is aligned properly with the rest of the code.
 

bobkoure

Member
Joined
May 22, 2013
Messages
173
Reaction score
20
I approached this with a bit of trepidation as installing mysql for python on both Windows and Mac has been a minor nightmare for me in the past.
However, on Raspbian (and other Debians, I guess) it's as simple as apt-get install python-mysqldb. If you're not logged in as root, you'd need to prefix with sudo of course.
 

Rrrr

Tink
Joined
May 28, 2009
Messages
343
Reaction score
25
Let me know if you want my updated, yet more complicated/sophisticated app for syncing. It can be run twice to
1. Load all your contacts without dial code eg for incoming calls
2. Load a shorter list of contacts and create a dial code that can be used for outbound TTS dialing

I have tinkered with it a bit and removed a bug with printing certain characters in the debug log on the raspbmc.
 

bobkoure

Member
Joined
May 22, 2013
Messages
173
Reaction score
20
I'd be curious to see it.
However, all I really need is a simple "copy all phone numbers plus associated names from G contacts to Asteridex" that can be scheduled to run nightly. But then I don't do anything with dial codes or TTS dialing - so I'm probably not your target audience.
 

Rrrr

Tink
Joined
May 28, 2009
Messages
343
Reaction score
25
You can easily take the script of post #1 and try it out.
It did not work for me, I solved some issues that had to do with international characters in names, the Google API v3 and other stuff. Then I added functionality and used asteridex instead of asterisk phonebook. Works with BeagleBone Black and Raspbx.

Recently, the asteridex db got an extra field (email), my new script (attached) works with it, but does fill it with "not set". Further notes are in posts #19 and #22

Here it is, it should run (set it to execute) after inserting your account details.
It wont generate dial codes, unless you change it.
Also note to set g_mymaxresults and g_intn_prefix accordingly.
It will delete data in the existing asteridex/user1 table and add all your Google contacts (use phpMyAdmin to make a copy of user1 if you want to try first).

Just know that if you have many contacts, like 1.000 and a daily cron, Google will not respond to queries over 10.000 a month.

Therefore, for those who want to update just a selection (group) of all your Google contacts on a regular basis, use the below steps.

step 1. Run the script as is to get all your Google contacts in asteridex

step 2. Run the script again after you have made the following changes:
g_replacecontacts = 1 (means do not delete table, just update the entries for contacts
g_phonebookgroup = your selected contact group that needs to be updated
g_set_dial_code = 1 (optional, make dial codes for the selected contact group)

step 3. make a (daily) cron, leaving settings as in step 2 (assumes the selected group < 300 entries).
 

Attachments

  • googlepbxcontacts0.95.py.zip
    6.3 KB · Views: 58

Jordan

New Member
Joined
Feb 6, 2014
Messages
3
Reaction score
0
Hi All,

I have been trying to use the script from Post #15... but I just can't get my head around what is going on. I have not written python before so I'm a little lost..

Here is the error I am getting:

Code:
Traceback (most recent call last):
  File "./googlecontacts.py", line 70, in <module>
    main()
  File "./googlecontacts.py", line 49, in main
    feed = gd_client.GetContacts(query=qry)
  File "/usr/local/lib/python2.7/dist-packages/gdata/contacts/client.py", line 201, in get_contacts
    desired_class=desired_class, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/client.py", line 640, in get_feed
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gdata/client.py", line 278, in request
    version=get_xml_version(self.api_version))
  File "/usr/local/lib/python2.7/dist-packages/atom/core.py", line 520, in parse
    tree = ElementTree.fromstring(xml_string)
  File "<string>", line 124, in XML
cElementTree.ParseError: unbound prefix: line 100, column 369

I am running Python 2.7.3, on a Debian Wheezy machine. It seems to me there is an issue with the API, but I just can't really make sense of it... could anyone shed some light on this for me?

This would be a brilliant addition to my PBX.

Thanks in advance!
 

Chad Howell

New Member
Joined
Apr 12, 2014
Messages
4
Reaction score
1
If you are having a problem with the following error
HREF="https://www.google.com/m8/feeds/contacts/default/full?max-results=1000">here</A>.\n</BODY>\n</HTML>\n', 'reason': 'Moved Permanently'}
ERROR: Error getting contacts

I found this solution: https://code.google.com/p/gdata-python-client/issues/detail?id=693
I have discovered a work around for the issue. If you edit line 1088 of

gdata-2.0.18/src/gdata/services.py

from

elif server_response.status == 302:

to

elif server_response.status == 302 or server_response.status == 301:

and then reinstall with

python setup.py install

It will allow the redirect to be followed and for your script to complete.

This solution worked for me.

-Chad
 
Joined
Apr 17, 2009
Messages
829
Reaction score
9
I hate to dig up this old thread and ask a possible stupid question. But I followed the guid on nerdvittle.com and I have done this a few other times and it worked perfect, however I am getting the following error when trying to finalize the last step of syncing the contacts.

Code:
root@localhost:~/Google/gdata-2.0.18 $ ./googlecontacts.py
Traceback (most recent call last):
  File "./googlecontacts.py", line 64, in <module>
    main()
  File "./googlecontacts.py", line 43, in main
    feed = gd_client.GetContactsFeed(query.ToUri())
  File "/usr/lib/python2.6/site-packages/gdata/contacts/service.py", line 104, in GetContactsFeed
    return self.Get(uri, converter=gdata.contacts.ContactsFeedFromString)
  File "/usr/lib/python2.6/site-packages/gdata/service.py", line 1108, in Get
    'reason': server_response.reason, 'body': result_body}
gdata.service.RequestError: {'status': 301, 'body': '<HTML>\n<HEAD>\n<TITLE>Moved Permanently</TITLE>\n</HEAD>\n<BODY BGCOLOR="#FFFFFF" TEXT="#000000">\n<H1>Moved Permanently</H1>\nThe document has moved <A HREF="https://www.google.com/m8/feeds/contacts/default/full?max-results=1000">here</A>.\n</BODY>\n</HTML>\n', 'reason': 'Moved Permanently'}

I followed this guide to build the system http://nerdvittles.com/?p=9214
Then to get the google contacts I followed http://nerdvittles.com/?p=6103 just as I have always done since it became available..

If someone could help guide me in the right direction of what to look for I would greatly appreciate it.
 

Chad Howell

New Member
Joined
Apr 12, 2014
Messages
4
Reaction score
1
Follow the instructions in the post directly above yours. It works like a charm.
 

Maxime Baillargeon

New Member
Joined
Feb 3, 2014
Messages
2
Reaction score
0
Hi, I was suceffully using accountingnerd's version of the script for the last couple mounths until this sunday night (I have the cron job set on sunday night). Now the script halts in the middle of the job resulting to lost contact in asteridex....

When I run the job from terminal, I get this :


Code:
Traceback (most recent call last):
  File "./googlecontacts.py", line 99, in <module>
    main()
  File "./googlecontacts.py", line 88, in main
    phonetype = phone.rel.split("#")[1]
AttributeError: 'NoneType' object has no attribute 'split'

here's my version of the script

Code:
#!/usr/bin/python
# googlecontacts.py v0.1 By: John Baab Email: [email protected]
# Modified by: Aaron J. Clark, CPA.CITP Email: [email protected]
# Purpose: syncs contacts from google to asteridex
# Requirements: python, gdata python client, asterisk, MySQLdb
#
# License:
#
# This Package is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or (at
# your option) any later version.
#
# This package is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this package; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
# USA
#
# On Debian & Ubuntu systems, a complete copy of the GPL can be found
# under /usr/share/common-licenses/GPL-3, or (at your option) any later
# version
import atom,re,sys,os
import gdata.contacts
import gdata.contacts.service
import MySQLdb
import gdata.data
import gdata.auth
import gdata.contacts.client
import gdata.contacts.data
import unicodedata
 
email = '[email protected]'
password = 'removed'
domain = 'proitek.ca'
 
def strip_accents(s):
  # remove accents from characters in a name
  return ''.join(c for c in unicodedata.normalize('NFD', s)
                  if unicodedata.category(c) != 'Mn')
 
def main():
    # Change this if you aren't in the US.  If you have more than one country code in your contacts,
    # then use an empty string and make sure that each number has a country code.
    country_code = "1"
 
    gd_client = gdata.contacts.client.ContactsClient(domain=domain)
    gd_client.ClientLogin(email, password, 'google2asterisk')
    qry = gdata.contacts.client.ContactsQuery(max_results=2000)
    feed = gd_client.GetContacts(query=qry)
    # Create a connection to MySQL
    con = MySQLdb.connect(host='localhost', user='root', passwd='passw0rd', db='asteridex')
    cur = con.cursor()
    # delete all of our contacts before we refetch them, this will propogate deletions
    cur.execute("DELETE FROM user1")
    cur.close()
 
    # re-instantiate the cursor
    cur = con.cursor()
 
    # for each phone number in the contacts
    for i, entry in enumerate(feed.entry):
        for phone in entry.phone_number:
 
            # I don't want to bring in phone numbers of type "Other"
            if phone.rel != gdata.contacts.REL_OTHER:
 
                # Strip out any non numeric characters
                phone.text = re.sub('\D', '', phone.text)
             
                # Remove leading digit if it exists, we
                # will add this again later for all
                # numbers Only if a country code is
                # defined.
                if country_code != "":
                    phone.text = re.sub('^\+?%s' % country_code, '', phone.text)
             
                # Insert the number into the cidname
                # database, reinsert the country code if
                # defined.
             
                name = strip_accents(unicode(entry.title.text))
                if str(name).strip() != "None":
                    phonetype = phone.rel.split("#")[1]
                    name = name + " (" + phonetype + ")"
                    num = phone.text
                    if str(name).strip() != "None":
                        cur.execute("INSERT INTO user1(`name`, `out`) VALUES (%s,%s)",(name,num))
                        print str(name) + " [" + num + "] was inserted into the database"
             
    cur.close()
    con.close()
 
if __name__ == "__main__":
    main()
 

BinaryTB

New Member
Joined
Jul 22, 2014
Messages
1
Reaction score
0
Hi guys, hope you don't mind, but I adapted the script from this thread to do some things that I wanted for my setup. Figured I'd share. Don't worry, most of the code is comments so people can hopefully understand it better.

What the script does differently:

-Reads Google Contacts and writes to a MySQL database
-Supports multiple Google accounts
-Uses mysql.connector module (I *believe* it's pure python and doesn't require MySQL client libraries installed)
-Very fast, at least on my setup (2 accounts, ~1000 contacts total, ~2 seconds)
-Leverages Regex SQL query for better matching (International dial codes and country codes are much less of a problem)

I have it set up to run nightly and then set up the MySQL database as a caller id lookup source in FreePBX and it's been running smoothly for the past week.

I haven't tried it myself, but you could possibly set it up as a SFDatabase source in Superfecta, it may require a modification of the SQL query, haven't tried it myself.

Code:
#!/usr/bin/env python
 
# Author: Tashfeen Bhimdi
# Version: 2014-07-18a
#
# Modification of code by John Baab (googlecontacts.py v0.1)
# http://pbxinaflash.com/community/index.php?threads/google-contacts-to-asterisk-phonebook.10943/
#
# Purpose: syncs contacts from google to a MySQL database
# Requirements: python, MySQL, gdata python module, mysql.connector python module
# Notes: This is READ-ONLY from Google script:
#        -Contacts are pulled from Google
#        -Local database is purged
#        -Pulled contacts are stored in local database
#
# Debian:
# apt-get install python-gdata
# apt-get install python-mysql.connector
#
# License:
#
# This Package is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public
# License as published by the Free Software Foundation; either
# version 3 of the License, or (at your option) any later version.
#
# This package is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public
# License along with this package; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
#
# On Debian & Ubuntu systems, a complete copy of the GPL can be found under
# /usr/share/common-licenses/GPL-3, or (at your option) any later version
 
 
import re
 
import mysql.connector
import gdata.contacts.client
 
 
def Drive():
 
    # Change if not in the US
    # Prefixes are removed from contact numbers before inserted into MySQL database
    # Use "" if you don't want a prefix removed
    countryCode = "1"
    internationalDialingCode = "011"
 
    # countryCode and internationalDialingCode are removed because
    #  the MySQL lookup query checks the latter portion of the phone number in the database
    # e.g. Calling Numbers 12345678999 and 2345678999 will both match 2345678999
    #  in the MySQL database using this query in FreePBX/MySQL CNAM Lookup:
    #
    # SELECT name FROM phonebook WHERE '[NUMBER]' REGEXP CONCAT(number,'$') ORDER BY CHAR_LENGTH(number) LIMIT 1
    #
    # REGEXP means a regular expression search
    # CONCAT adds "$" to the end of the number in the db, signifying end of string
    #  So a number in the database must match all or just the end of the incoming number
    # ORDER BY CHAR_LENGTH tells the db to sort the record with the shortest db number first
    #  This should take care of an incoming number 6789 matching both 16789 and 3456789
    #  in the db (16789 is the "best match" in this case since it has the largest overlap)
    # LIMIT 1 tells MySQL to only return 1 result
 
    # Google login info
    # PLEASE use app specific passwords
    # If only using one account, make sure variable is stored in this format:
    # googleUserAccounts = (("[email protected]", "mypassword"))
    googleUserAccounts = (("[email protected]", "mypassword"),
                          ("[email protected]", "mypassword"))
 
    # MySQL login info
    mysqlUser = "username"
    mysqlPw = "password"
    mysqlHost = "servername"
    mysqlDatabase = "databasename"
 
 
    ################################
 
 
    rx = re.compile('\W+')
    allGoogleContacts = []
    for email, password in googleUserAccounts:
 
        # Login to Google
        gd_client = gdata.contacts.client.ContactsClient(source='googlecontacts_mysql')
        gd_client.ClientLogin(email, password, gd_client.source)
 
        # Retrieve contacts
        query = gdata.contacts.client.ContactsQuery()
        query.max_results = 10000
        feed = gd_client.GetContacts(q = query)
 
        for entry in feed.entry:
 
            # If contact has a name
            # If contact has a phone number
            # If contact is part of a group ("Other Contacts" autogroup is skipped this way)
            if entry.name is not None and len(entry.phone_number) > 0 and len(entry.group_membership_info) > 0:
 
                # Clean up characters in contact name; replace all non-alphanumerics with spaces
                fullName = entry.name.full_name.text
                fullName = rx.sub(' ', fullName).strip()
 
                for rawPhoneNumber in entry.phone_number:
                 
                    # Remove non-numeric characters from the phone number
                    phoneNumber = re.sub("[^0-9]", "", rawPhoneNumber.text)
 
                    # Remove country code from the phone number
                    if countryCode != "" and phoneNumber.startswith(countryCode):
                        phoneNumber = phoneNumber[len(countryCode):]
 
                    # Remove international dialing code from the phone number
                    if internationalDialingCode != "" and phoneNumber.startswith(internationalDialingCode):
                        phoneNumber = phoneNumber[len(internationalDialingCode):]
 
                    # Save contact for later insert
                    allGoogleContacts.append((fullName, phoneNumber))
 
    # Remove NAME+NUMBER duplicates (if an account has duplicates or multiple accounts have the same contact)
    allGoogleContacts = tuple(set(allGoogleContacts))
 
    # Connect to MySQL
    cnx = mysql.connector.connect(user=mysqlUser,
                                  password=mysqlPw,
                                  host=mysqlHost,
                                  database=mysqlDatabase)
    cursor = cnx.cursor()
 
    # Remove the contacts currently in the database
    cursor.execute("DROP TABLE IF EXISTS phonebook") # autocommit
    ddl = """
            CREATE TABLE phonebook (
                contact_id int(6) NOT NULL AUTO_INCREMENT,
                name varchar(32) NOT NULL,
                number varchar(32) NOT NULL,
                PRIMARY KEY (contact_id)
            )
          """
    cursor.execute(ddl) # autocommit
 
    # Insert all contacts into the database
    insertSql = """
                INSERT INTO phonebook
                    (name, number)
                VALUES
                    (%s, %s)
                """
    cursor.executemany(insertSql, allGoogleContacts)
 
    # Commit insert and close MySQL connection
    cnx.commit()
    cursor.close()
    cnx.close()
 
 
if __name__ == "__main__":
    Drive()
 
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top