Converting Google Directions API output to a Spatial Data Format

I’ve ran into a major road block of sorts in the past month on my project. My goal is to untangle the ugly output that is returned by the Google Directions API and write it to a file thats easy to plot and read in QGIS.

I found a great script that converts the encoded polyline into a series of latitude and longitude points, which I could then again convert into a geojson or shape file format.

However I am currently trying to pull the geocoded overview line out of the Google Directions output in Python. I am getting close by using the split function, but getting hung up on all the quotation marks and backslashes that muddle up the output and confuse python.

For example, I am trying to just isolate the text in the quotes after the “”overview_polyline” : {\n “points” :” text in file below.

"start_location" : {\n                        "lat" : 53.5779983,\n                        "lng" : -113.5965706\n                     },\n                     "travel_mode" : "DRIVING"\n                  }\n               ],\n               "via_waypoint" : []\n            }\n         ],\n 
        "overview_polyline" : {\n            "points" : "m{xeIdeusTyJxADtG{@|DaJ?kRBoRFsDhJiCvIs@vCyBnKwEhKuVfu@ma@zoAcPtc@yFvAyDlBwNHsSg@uBeF{AyPcDeCqD?

\\\\p^Jxu@kBtmBa@taCpCfi@hl@l}DlH`q@l@hUuJjbCqArdBn@p]~KxiAjFlq@hAld@^bpIo@|}HkFr{EVf{BvGrqEeDjvBy@pv@lGp}BlBbpBkCnvA]jxAHpqIgCn{BwHtbB
\\\\`}@tH|`E|G`pDLz~@aGbvAYlrFEjmPb@~z[s@l~HaOrzBuGhb@}|@loEmAtOl@fyJoBhx^UnyBgd@`nDoCzXcBzs@q@rkArDbWfPtd@`Olh@fExj@_A~yKm@fdB_NxyBgHx~@oCvu@mKlu@mDr^rCzyCoq@`cQoNnqB{IbmAgKf`CmMjxCqCtz@|@zXfFt]`M~e@zFv^`C|_@`@nc@{A~g@sCth@r@`y@fErdAqN~_DqPdxCiXxbFeM~nB}
\\\\beCoAlm@tBbc@|OrdAlFnj@[h\\\\mN|kBeQh}DrBpd@pQjkA|At_@i@|SgEv[_HpV{D|PkBrTaAdmCxDvj@fHnXvPr]~HvTbHr\\\\pUbdBlDrg@dAr_H_UjfJaD~RgZngAeE~`@bU~|Q~A`gArDj_@rb@tsDlF`x@h@`bBArwFpB~o@`OfiA~yAdaKjRtmAzL~_@`Mb^|Ifh@|LfgA`Db_A_E`tFpC|vFwDjzA{SzzBiEho@Dz\\\\dR~xFl@`dAkBboARxpAjG`mAxMnkAbDlh@cAfdAaKjx@ee@toCoFtp@Yds@xDzqAaBl^gJfz@kCrh@bBt_@xHlt@c@hMmOrf@a@tVnY`{A`nAzpGjAvx@IzoIFxgGbRlcArA~Uw@jXiThpAGbTdNpgBp@v]kAzVmLdi@wQbh@eFna@sAjd@n\\\\|wEbSllC|Mb|@ty@~uElPbuA~G`i@rHnXlj@t_BdIf^nFls@`KndFkBjg@}PliAZtpBnBpjD{Fbl@yNf{@}D~n@jA|gAjOzvC@pe@kCrx@qC`eAhBtY|m@v`DdBp
\\\\a@zUgd@tyCcBnWQjz@@p`A~DfcA`FzuAoBt_@aEtTyWvq@w_@z_AuG~ViRl_Au[xgCPzn@zI`h@bQpt@nC~Z~FbwB`Cpq@`EdW~NzZhy@~jAh{AvrA~L~Rd[vw@~o@~cBbEhSxH~eAvM`bAjNjx@dTj~@dWzj@nLx[xg@rwB`h@hlBne@t{BdyA~aEtGbLx]j]vJfWhZ~}AdStl@l~AfyDtUnk@~p@dvBbXbu@rz@nmBlElUr@|VoAnS"\n         },\n         "summary" : "Yellowhead Hwy W/AB-16",\n
         "warnings" : [],\n         "waypoint_order" : []\n      }\n   ],\n   "status" : "OK"\n}\n'

I think the great thing about scripting is that most people have already done a lot of the hard work and posted it online. With basic knowledge of the language and a clear understanding of process I know I will eventually get to my goal, step by step.

I will keep puttering about, but this is my script so far which doesn’t quite get there (it is splitting things in the wrong place).

import os
import sys
import fileinput


fileToSearch  = 'newfile1.geojson'

tempFile = open( fileToSearch, 'r+' )

for line in fileinput.input( fileToSearch ):
    mystring = line
    clipped = mystring[mystring.find("overview_polyline"):mystring.find('"\n')]
    print(clipped)
Advertisements

Putting Things Together – Saving the Google Directions API output to a File

Today I made significant progress in my script. I was able to open up a csv file with the list of To and From directions, then loop through each pair and send a request to the Google Directions API and get a GeoJSON object in return, and finally I was able to save each GeoJSON object as a unique file on my computer. I set up the script to add a counter to each file, so that each time it encountered a new TO and FROM pair in the spreadsheet it saved the output with a unique number.

Let’s dive into how we did it!

First, I import all my libraries, and open up the .csv file containing my TO and FROM locations. Then I create a new variable called counter that starts at 0.


import urllib.request
import csv

f = open('testtable.csv', 'r')
csvf = csv.reader(f)
counter = 0

Next I start looping through each row of my csv file row by row. I print each value within the row just so that I can see what TO and FROM areas it is reading. Also at this point within the loop I update the value of my counter to add 1 to what the previous value was.


for row in csvf:
        print(row[0])
        print(row[1])
        counter += 1

Now I create the new file name that the results of the first row will go into. Notice I add the counter number to the end of the file name so that each row creates a uniquely numbered file.


        filename = "newfile{}.xml".format(counter)
        print(filename)
        text_file = open(filename, 'w')

Its time to contact google and get the directions! I format the portion of the http request to input the to and from directions first. After that I tack it onto the rest of the url. I then contact the API and save the response from the API as a variable called googlejsonoutput.


        direction1="origin={}&destination={}".format(row[0], row[1])
        url="https://maps.googleapis.com/maps/api/directions/json?%s" % direction1
        print(url)
        response = urllib.request.urlopen(url)
        googlejsonoutput = response.read()

Finally, all I do is write the google output from the variable into my new file. I was having trouble writing it directly to the file, so the %s direction below simply tells Python to write my output as a string. I found that this does work. After the output has been written I close the text file so that there are no problems.


text_file.write("%s" % googlejsonoutput)

And so that things don't get messy, at the very bottom of my script outside of my loop, I close the csv file that we were reading from.


f.close()

I can double check my success, as my spreadsheet was 2 rows long, I have two new output files in my working folder called newfile1.geojson and newfile2.geojson. Each of the files have geojson objects populated in them.

However, there is a big problem! QGIS won't actually open the data. I think I will have to go through each file and do a bit of formatting to get it in GIS format. Thats the topic of the next blog post.

Thanks for reading! The entire script is pasted below if you don't wish to view it all cut up like above. I add lots of print() commands because I like to track the progress of my script in the Python shell while they execute. These commands are not necessary.


import urllib.request
import csv


f = open('testtable.csv', 'r')
csvf = csv.reader(f)
counter = 0

for row in csvf:
        print(row[0])
        print(row[1])
        print("end of row")
        
        counter += 1
        print(counter)
        
        filename = "newfile{}.geojson".format(counter)
        print(filename)
        
        text_file = open(filename, 'w')
        direction1="origin={}&destination={}".format(row[0], row[1])
        url="https://maps.googleapis.com/maps/api/directions/json?%s" % direction1
        print(url)
        
        response = urllib.request.urlopen(url)
        googlejsonoutput = response.read()

        text_file.write("%s" % googlejsonoutput)
        text_file.close()

##close files
text_file.close()
f.close()
 
        

Reading CSV files in Python

I know the easiest way to get my large table of From and To results passed to Python is to create a comma separated value (.csv) table. Its very easy to edit this, as I can simply open a text editor and start my data entry. Below is an example of how a very simple TO and FROM table would look like in csv:

From,To
Edmonton,Hinton
Vancouver,Burnaby
Edmonton,Calgary

To load and parse through these values in Python I need open the file and then return all the rows. The script do do this is below. Please note that it is very important to store the .csv file in your current working directory instead of some random folder on your hard drive!

import csv
f = open('testtable.csv')
csvf = csv.reader(f)
for row in csvf:
    print(row)

And from running that with the csv above we get the following output, with each row as a separate list of strings:

['From', 'To']
['Edmonton', 'Hinton']
['Vancouver', 'Burnaby']
['Edmonton', 'Calgary']

The next step is to pass each value in this list of strings to the Google Directions API and store the returns in a database!

Utilizing Python to send API requests to Google Directions

In the last post we talked about how to manually enter API requests into any browser and get a geoJSON file back that is readable in QGIS. In order to complete my project efficiently, I needed to find out how to pass these requests to the Google Direction API through Python so I could eventually pass in a large spreadsheet or database of origins and destinations and get back a series of files.

The first step in the process was to learn how to send a url request through python. I found directions on the google maps API website to preform a simple geocoding request, but when I passed it into python I got all sorts of errors! The code suggested by Google looks like this:

Screen Shot 2015-05-18 at 1.53.57 PM

After some quick googling I found that the urllib2 library isn’t in use for Python 3.0 and up. So I found that the replacement was urllib.request. Easy enough! I added that to the code, and included the print(jsongeocode) to the end and ta-da! it spits out the geoJSON response from the server that is now stored as a variable in Python for me. I modified the script so that it was sending a request to the directions API instead of the geocoding API simply by using the url example shown in my blog post below. Obviously I will want to refine this a bit more so that I can pass in the origin and destination as quick variables and have it generate the entire url from that. We will work on that in the next few blog posts. The basic code I made is located below for returning a route file from Edmonton to Calgary.


direction1="origin=Edmonton&destination=Calgary"
url="https://maps.googleapis.com/maps/api/directions/json?%s" % direction1

response = urllib.request.urlopen(url)
jsongeocode = response.read()
print(jsongeocode)

 

Using an API to get route files from Google and Open Street Map

The simple user interfaces I explained in my previous blog post work fine for typing in origins and destinations for a trip and exporting a GPX or KML file with the polyline information. Obviously if I want to create a database with hundreds or thousands of routes, I need to find out a way to do this more efficiently. To my satisfaction I found two API’s that will allow me to do this.

How Google Directions APIs work

The OSM YOURS and Google Directions APIs are actually quite simple in concept. Instead of typing in your route origin and destination into a website interface, you include your routing request in the text of a url, hit enter on your browser, and the API will return you some text or a file containing the routing directions. You can customize the response you get from the server by adding in flags to the url itself that are explained in the API documentation. For the purpose of brevity I’ll be focusing on the Google Directions API in this blog, but the OSM YOURS API offers similar capabilities with a gpx file output option instead of a geoJSON file.

Google Directions API and GeoJSON

The Google Directions API has options to return your routing results in XML or geoJSON format. For those unfamiliar with these file types, they are simply files that contain the route information coordinates in different formats. For this project, it is important that I can easily open the data in QGIS (an open source mapping software) to visualize how things look on a map and possibly edit my data. I chose geoJSON because it is easy to open in QGIS.

So how do you send an request to the Google Directions API and get back a geoJSON file? Its actually quite simple. Note that copying and pasting the link below into your browser gives you a file back with the route from Vancouver to Seattle. Note the origin=Vancouver and destination=Seattle text. Simple!

https://maps.googleapis.com/maps/api/directions/json?origin=Vancouver&destination=Seattle

Now for a look at the geoJSON text file that we get in return. Thanks to Chris Whong’s NYC Taxi blog for explaining this in detail. All the information below is a brief summary of his excellent blog post.

A quick look at the geoJSON file we get back looks like this in our browser:

Screen Shot 2015-05-18 at 1.17.10 PM

The important part of the file is the overview polyline section. Which is near the bottom.
Screen Shot 2015-05-18 at 1.19.31 PM

See that complicated block of characters after “points”? That is actually an encoded string of points that makes up the entire route. I won’t be needing to decode this since I can load the geoJSON file directly into my database and mapping software, but Google does offer an interface to decode this. Simply copy and paste the code into the encoded poly lines window at this page, and you will get a preview of the route along with a list of the lat long coordinates that are strung together to create the line.
Great! Now I’ve cracked the basics of how to send an API request to Google and get my precious route linework in return. Next in the process I will have to figure out how to send these requests from Python on my desktop and get a response. After that I will need to automate the url generation so I can make large list of the url routing codes based on a spreadsheet of all my trip destinations. I think my next step after that will be to create some sort of database to store all my data for web hosting, or for utilizing in QGIS to make some nice looking static maps!

Routing Services – Google (My Maps) and Open Street Map (YOURS)

My first idea with this project was to simply type in routes I have travelled in Google maps and save all of this information within their web interface. I soon learned that the instantly available google maps does not offer an export option, and only allows for 10 stops along each route. Google’s My maps does however offer an export option for each route. Its fairly simple; after signing in to google.com/mymaps you can create a new map and input directions into that map.  You can record up to 10 stops on the route and then choose the Export To KML feature to get a KML file containing the line information in the route.

Screen Shot 2015-05-17 at 6.29.52 PM

It is also possible to export route information from Open Street Maps (OSM) data, though not directly from the open street map website. There are a number of OSM routing services listed here. The service I found to be most useful was http://yournavigation.org since it had an easy to work with GUI, and the ability to export to .gpx format easily.

Screen Shot 2015-05-17 at 6.35.13 PM

While these simple user faces are easy enough to operate and generate a polyline file output, I hope to develop a solution where I can compile a large spreadsheet or database of trips, each with a FROM and TO column that I can pass to a routing service automatically and get a series of files in return. This will save huge amounts of time typing in locations and exporting each file manually.

Starting the Journey

I’ve always loved to revisit routes and paths in my memory. Recreating in my mind the roads I’ve travelled and trails I’ve walked helps fill out my personal narrative and provide me with a healthy dose of nostalgia. I think its impressive that the human body is so capable of transporting itself through space often at very high speeds. We are fairly unaffected by this rapid movement, and it is hard to read on someone’s face all of the places they have been. Wouldn’t it be fascinating if individuals recorded detailed personal cartographic logs as a form of spatial biography?

With these thoughts in mind, I thought I would attempt to create a visual representation of the places I have been. I will have to find the proper methods and technologies to properly achieve this and then mentally sort through my past to record my spatial movements since birth. My end goal is to publish an interactive web and a static print map of this data. Hopefully I can accomplish this using open source software and document it in this blog. Each phase along the process will a learning experience that I will share here. I am starting very fresh in most of the technologies I will be using in this process, so I expect to encounter many challenges and exciting learning oportunities along the way.