Geo+Django: Geo beyond the Django

I'm Joe Jasinski and I work for Imaginary Landscape and I've been doing Django development for about four years. And I wanted to talk to you today about a topic I find really interesting, and that's uh Geojo and GIS applications. So a little bit of background about how I got into this. A few years back, Chicago had a mobile app building competition And uh they made available a data portal for the city for use with this competition and other reasons. And um I built a simple little app that kind of showed uh various um cool interesting things to do around uh CTA train stops. And um though I didn't win any awards for this, it kind of inspired me to learn more about this technology over the next few years. And I continue to build on this project and

play with it. And some of the examples from this talk are going to be straight out of that project. So, first, I'd like to cover a few general definitions that I found useful. One of them is a coordinate reference system And this is a system to locate geospatial entities on a map. And there are two basic types. There's uh geographic coordinate systems, which represent the Earth as a sphere or spheroid, and there's uh projected coordinate systems which project Project maps into a flat surface. And you've probably heard all the problems with projected coordinate systems and that um You get map distortions if you flatten a map from a sphere. So um they're particularly good at representing local data and not so much like zoomed

out data. So coordinate systems are important because data could be encoded in any number of coordinate systems, and you need a way to like normalize that data so you can perform math and display them on the same on the same map. So a spatial coordinate or coordinate system registry is another thing. The primary registry is called the EPSG. And this is an authority that catalogs and maintains different coordinate systems And it uniquely identifies coordinate systems by a spatial reference ID. And so that's how spatial reference systems are kind of named, is based off that ID. There's a great resource called spatialeference. org, which is a nice index for coordinate systems, and it uh

it has a lot of information about each system and the and Defines the definition of the coordinate systems in a various variety of different formats. One very common coordinate system that you'll know you'll encounter a lot if you use this is WSG WSG84. And it's a geographic coordinate system, meaning it represents the Earth as a sphere, spheroid. And it's used by the GPS system. So you'll find a lot of data. encoded in this um system. It also goes by the name uh EPSG430 or 4326, which is the um spatial reference ID for it. Um so you'll probably encounter this a bunch. I'd like to talk about uh Postgres and PostGIS. And me, I personally, when I use Geo Django, my preferred back

end of choice is um uh post GI or is uh Post GIS and and Postgres. First of all, I you know I love Postgres and um GeoDjango typically has better support or more functions available for um Postgres. Um so it's a good choice for working with uh geodata. And Post GIS went through a major update not too long ago to a version a version 2. 0 update. And it's become a lot easier to install for an Ubuntu 1404 system. You just need to install a bunch of apt dependencies and um switch to the database you want to use and run a create extension command. It's now an extension. So it's um really easy to get started working with it on a new system.

I've also included a link on here, you'll be able to see in the slides for just some quick installation steps for Ubuntu 1404 and CentOS 7. So I'm going to refer to this diagram a few times throughout the talk. It's basically the architecture stack that I'm going to build up And let's assume that we've gone through the Django tutorial and uh installed Geo Django and hooked it up with our post -GIS database. And this is what we have so far. This is what it will look like. A few other notes about Post GIS, when you run that extension, it will create a bunch of different objects in your database. It will create functions and stored procedures to do geometric operations, but it will also create a new table.

called a spatial references table. And this is basically a tabalized form of the data from spatialeference. org. It has the spatial reference IDs and different um different definitions for the coordinate systems that that uh Geojango and or PostGIS can use to do conversions between coordinate systems. Also there's two new views that will be created in your database, one called the geometry columns and one called geography columns. views and these um have these identify which columns in your database are spatially enabled. Previous to PostGIS 2. 0, these were tables and not views. So You had to manually maintain these these tables. But uh uh and

well if you use Geojo, Geo Django would take care of that for you. But uh Uh as of 2. 0, you don't no longer have to do that since they're views and they're dynamically calculated. So one thing when you're working with uh after we've got the database set up and Django set up, um the next question is where do I get data? And um I found my data source on Chicago's um data portal, which was great. There was uh like hundreds of different um data files that I can use. But there's data sources all over the place. I don't know. I was really surprised. Every major city has their own data portal, it it seems like. Just do a Google and you'll find a lot of uh example data you can play with. Um so uh um

and that's just the open data you can get. Um Another place to get data is the OpenStreetMap project. And this is kind of like a crowdsourced Wikipedia of maps. thing and data is available um under an open license so you can get a lot of data out of out of this um website. It's a pretty cool um uh pretty cool site and tool So now we've identified some potential data sources, and uh these sources provide data in all sorts of formats, and now we have to import those uh formats, import that data into the database so we can start using it. So there's a few serialization formats I'd like to talk about. And the first one is called uh well-known text. And this is just

um a tech or a string-based format for identifying geometry. So here's some examples of point objects and string objects. They're just um you know basic text representations of the geometry that you can load into their in your database. Well-known text is used in a lot of places and including one of the columns in the spatial refsys table defines Each coordinate system in well-known text format. So it's not only for geographic data, it's also for defining reference systems. So Django has uh uh built-in support for loading well-known text data with using this OGR geometry class, which takes uh constructor argument for your geography or geometry data

and it will create a Python object that you can then assign to a model or uh or uh do whatever you want with. Another serialization format is called the Keyhole Markup Language. You might have heard of, it's KML. And it's an XML-based format developed by Keyhole Inc. which was then bought up by Google and is now used in Google Earth. So um it's obviously a good format for sending data to Google Maps and to Google Earth, but it's a serialization format so you can use it however you want. There's some Python utilities. called simple KML and fast KML that you can use to uh load and manipulate data and in KML. GeoJSON is another obvious choice for serialization format.

And um It's developed and maintained by a independent group on the internet, so um it's there's no specific authority, it's just like a mailing list that of people that uh Define how this format should work. And it's a JSON representation of your data. And it's a nice option to send data to a browser if you're um You know, is because JavaScript is nice to work with in the browser. There's a good Python utility for working with this on the Python level called uh GeoJSON. All of these formats are serializable by Geo Django, so you can print out various geometry objects in these respective formats. And finally, I'd like to mention um shapefile format, which is an extremely common format for persisting uh

geospatial data to files specifically. And it was a format developed by the ESRI, which is a uh a company that makes uh a very um popular open a very popular product called arc gis um for geo geospatial data. It's um an expensive program but it's uh used used in the industry a bunch And shapefiles are kind of misnamed because they're not really a single file, they're a collection of files. So you'll find at least three files in a shapefile archive. It's usually like a zip or something. But each file has a different purpose. And you can have up to like 10 or so different files representing the same geometry. Uh Django has some support for loading this type of um object using uh uh

this data source class. You can import um shapefile data. And also there's a Python library called PyShape, which um You can use to help get data into your application. So you can create a Python-based importer using the tools I just mentioned, or you can take advantage of some command line tools to load this data. One of them is uh shape to PGSQL or OGR to OGR. I'm not going to get into detail on those, but they're they're very powerful and they have a lot of different options you can pass to them to import different types of geodes. data. So I'd like to talk a little bit about um server-based geo utilities. And the first one is um the Project Reproj 4 library, which is uh a dependency for Geojango and it's uh responsible for uh converting between different projections.

And Also, the spatial ref sys table in PostGIS has a column called Proj4Text, which which also stores a definition for a projection in Proj 4 format. So there's there's an additional That's what Proj4 can use to do map projection conversions. Also, I've used Proj4, or more specifically, a Python wrapper for called PyProj. uh that um to translate points on a map. And in my particular use case, I filled in a shape uh with evenly spaced um points by uh by transforming a single point into multiple directions. And I use this for like generating a source, a group of source points for creating a heat map.

Another great utility for working with data on the server side, geodata on the server side is called Mapnik. And it's basically a tool that you use if you want to render maps to images. And it's great for it's there's a lot of Python support support for it, it's got some Python bindings. Um so you can write You can use this tool to uh write just flat JPEGs or images that you can use in reports. We can also um you can also be it also can be used by a map tiling engine, which I'll talk about in a little bit. So here's our stack architecture diagram again. You'll notice I added MapNick to it, and you see it's making a direct connection to PostGIS.

And uh Mapnik supports a variety of different data sources, um, shapefiles, CSV, GeoJSON, and in the last slide you saw Post GIS. Um, and each data source is added as a layer. So this kind of takes a layered approach and the topmost layers show up on top. But one of the nice features about Mapnik is the fact that you can style your maps. Just geo-dain geodata by itself isn't very useful unless you can visually represent it. So it's got a pretty um sophisticated way to uh style data. You can either use this Python API to like build up styles, or you can create an XML style definition that you can load and use

to style your different data layers. So here's a very simple map mapnik example, and I'm using the XML style sheet in this example. So I'm loading the XML file. Creating a new map object. I'm defining a subquery to pull in data from my map, and then I'm creating a connecting to the PostGIS database. I'm adding that data source as a layer to Mapnik, and then I'm rendering it to a JPEG. And this image at the top right here is uh uh partially constructed by this query I created here, so that's kind of uh what you can do with it. So Mapnik XML is not very pretty. It's um

it's XML and I never want to work with it. So that's where a program called TileMill comes into play. And TileMill is a Node. js, a Node. js-based desktop application. It's open source, but it's fine maintained by a company called Mapbox, which does um hosted map mapping. And it supports a bunch of standard formats for importing data, including post GIFs and shape shape files. And it's built on top of Mapnik. So It's making calls to Mathnik to render uh render the images that display in its interface. Um and as a result, uh It also can be used to generate that MapNIC XML. That's the most important thing that I use it for.

So the reason why I like tile mill is because it has something called map style sheets. And map style sheets is a is a language or a markup language similar to CSS or or most similar to lessjit. js And you can change the behavior of how your maps will look in a very intuitive fashion at different Zoom levels. And it's a really cool application to add lots of style to your app. or to your maps. On the back end, TileMill is using another Node. js library called Cardo. And Cardo will convert map style sheets to Mapnik XML. And uh so you can call this command on the command line if you've installed it

to just do this translation yourself so you don't have to even use Tile mill, if you don't want to, you can do this all from the command line. Another great tool for working with uh geodata is called Quantum GIS. It's another desktop application. It's got it's written in C in Python, and so you can make calls to it from your Python scripts. And I use this to uh visualize and and prototype my applications. And this I use it differently than tile mill because tile mill is I think better at um styling your maps whereas QGIS is more like a data editor. So they both they they both kind of can work in conjunction and um But if you really just want to get some data on your map,

QGIS is a to visualize and make sure it looks good. QGIS is a great application to do that. So now that we have like some tools to work with the data, let's pretend we've created some geoqueries using the Django documentation to do some fun queries with Geojango and now we need to take that data and deliver it to the browser. So when I say this I'm referring to two types of data that I typically send to the browser. And one is geography data, vector data like uh uh KML or um you know defined shapes or k KML or GeoJSON. And another one is uh rendered map tiles, um using Mapnik as kind of the source for that. So to create um geography data, you can use any type of serializer you want, uh, but one that I found that was really useful is one called

Rust Framework GIS. And it's basically a a plugin or an application that rides on top of Django Rust framework that defines the serializer so you can just easily um Uh turn query sets into GeoJSON output. And so this is particularly useful for again reading stuff from JavaScript. It's pretty easy to use and install. And then map tiling, you may be familiar with. Google Maps is a good example of it as you're zooming into a mapping application. It's loading in image tiles from a server and as you zoom in the tiles kind of get more and more detailed. But there's open source solutions to do map tiling, and one popular solution, Python-based solution, is called tile

stash. And um it it renders basic it renders map tiles in real time, it caches them and then it serves them up as you access them. So uh you can also use uh Mapnik to uh as the backend rendering engine for this, so um and and set a map Mapnik style XML file to uh uh to dynamically add your styles and you can set data sources um just like you can with Mapnik and and the other applications It takes a very simple URL scheme. If you visit this URL, you run the server, you visit this URL in your browser, you'll load a single map tile. You're basically giving it a layer name, which is something you define in a configuration, a zoom level. a column and row

which corresponds to a tile location um and a geo coordinate. So um Here's basically what the stack looks like now that I've added Tile stash on top of Mapnik and Django Rust Framework on top of and Rust Framework GIS on top of uh Django Now we have the back end set. We need something to read it from the front end, the JavaScript level. And there's a bunch of JavaScript client libraries that I used to actually render a slippy slippy map application so you can zoom and interact with it in the browser. And one of those is um obviously Google Maps. It's a a great tool for um uh just getting up and running and it's used everywhere it's it's got a great API

but it has some downsides in that it's um you're kind of beholden to their API so if it changes you've got to update your code. Um it's kind of got the same look and feel across sites. There's not a lot of level for or a lot of F ways you can customize it to look a lot different. There's I'm sure there's some, but and you also lose control of the mapping stack. Here's a quick example of a code using Google Maps. And in this example, I'm defining a map object. I'm assigning the map to a div and I'm adding a click event hand handler so I can click on a point on the map and it will load in, in this case, a KML file. Google Maps automatically takes care of the map tiling for me, so it's that's nice that I don't have to think about that.

It just pulls in uh map tile data as it needs it and it superimposes your vector data on top of it Open layers is another good option for map uh a JavaScript client library. It's one of the oldest uh open source options for map tiling and it's The only downside is it's very large. It's like 700 kilobytes. They just released a new version, so maybe they helped fix some of that size problem. But in the meantime, a project called Leaflet has come into play, which I personally like. It's uh very small, lightweight, um, it's got a nice plug-in system and it's easy to use. Here's a quick example of a leaflet code base. I'm defining a map, I'm setting a TAL server URL, I'm loading some data from uh a

JSON. Endpoint and then I'm adding a click handler to pull in uh some data as I click on a map. And you'll notice the big difference between this example and the Google Maps example is that I am setting a map map tiling server URL. And in the comments above there I have a few different map tile servers that I found that just have different um different types of tiles. You can include your own if you run your own map tiling server. But uh It gives you a little bit more customization in terms of what you see on the map. So after that we've kind of completed our stack. This diagram I'm using Leaflet as the JavaScript front end. It's pulling in map tile data and GeoJSON data,

connecting with the back end. And That's kind of that's how I've architected some of this stuff. Um this is uh um a pretty sophisticated Sophisticated topic. There's a lot to it, so there's only so much I can cover in about 25 minutes, but there's some other technologies that I just want to point out that um That you should Google because uh if you're interested in the stuff. One of them is uh PG routing, which is uh um another Postgres Postgres extension for uh defining graphs of data and doing routing with different uh routing calculations. Uh there 's another front-end um A JavaScript application called Modest Maps. It looks similar to Leaflet

to me. If you're looking for a JavaScript JavaScript client, that might be something to take a look at Uh Grash GIS is a um a tool similar to um uh QGIS. It's very uh it's been around for a long time and open source as well. Um Another thing to take a look at. Marble is a cool KDE project that is like a Google Earth replacement. Something to kind of look at as well. That's what I have for the talk so far. I have some links to more references in the top link there if you're interested. Some just links I found useful and definitions. Also, if you're interested in seeing like a working project, I have a GitHub project with

some of this experimentation that I've done. It's uh um it probably could use a little bit more polish, but it gets the idea across. Anyway, I appreciate uh your your attention and uh thank you for