The Power of GeoDjango by Anna Kiefer

Yeah, I think that's a good thing. Okay, sorry for those technical difficulties. But thanks for being here. My name is Anna Kiefer and I work at Kavala Analytics. I'm a software engineer. And we're an energy analytics startup that focuses on mapping and data analytics, data analytics, and mapping the US distribution grid. And so this is data related to production, capacity, generation, and investment, as well as soft energy infrastructure, things like substations, transformers, feeder lines. Things you can't just find uh say on Google or or searching in Google Maps.

Uh and so I deal a lot with geospatial data, uh, hence this talk, which is intended to be a gentle introduction to GIS and GeoJengo. So all of you probably recognize this. This is a human readable address, and it identifies where we are right now. It's useful for humans, but it's not so useful for computers. Humans misspell things, there's formatting differences, the difference between drive and DR, for example. And so To get around these problems, we use coordinate systems, which model a location on the Earth's surface. And they allow us to map features more easily. There are two common types of coordinate systems: geographic coordinate systems

and projected coordinate systems. And geographic coordinate systems define locations on the Earth as a 3D sphere, while projected use a 2D surface. And they're better for sort of distance queries, length, area. So let's see if we can do a little bit better than this address. So this oops. So this is our location in degrees, minutes, seconds, and this uses a geographic coordinate system. And it also uses what's known as World Geodetic System, 1984. And so that's a little bit better, but that's not the only geographic coordinate system we could use. This is decimal degrees, and it also uses WGS84.

And it's even simpler. Uh but again, so this isn't uh the only. There's I mentioned geographic coordinate systems and projected coordinate systems. This is a projected coordinate system, some of you may recognize. It's uh goes by the UTM. It uses um the universal Mercuter time, I believe. Um And it also uses map datum, uh, WGS84. First number is the grid zone designation, uh, followed by the coordinates easting and northing uh distance measurements. Um but so the WGS eighty four um was uh uh invented by the um Department of Defense. But they're not, they invented other ones as well,

them and other organizations. And so this is a military grid reference system, and it's used by NATO militaries. Uh but that's not the only one. There's a state plane coordinate system, which divides the US into 124 geographic zones. But that's not the only one as well. Because we're in the US we should probably use feet, not the metric system. So that's in feet , but there's also something known as the international foot, which does not equal the US survey foot So this is a representation in terms of the US survey foot. But there's additional ones. This is the Maidenhead reference. Oop. That's the Maidenhead reference system. It's used by amateur radio operators around the world.

And finally, uh, or no, not finally, sorry, um, this is a different uh global area reference system developed Finally, this is what I found actually yesterday. It's called What Three Words, and it tries to take all of our points on Earth and give them three words. And this is where we are right now. Does feels unity, which I take to mean we're all tired. And so all of these are valid ways of representing our location. But it you as you can see, they're all different formats. And so we need to get this into a standard format. And I basically mentioned these to show you all of the different coordinate reference systems there are

and the importance of standardizing our data and how Geo Django and PostGIS uses these and turns them into standardized while still being flexible So the next time one of your friends asks you where you live, use one of these other reference systems. Okay. So for those of you unfamiliar, GeoDjango is Django's robust framework for handling geospatial data. It gives developers the ability to store geospatial data, query, aggregate, and filter it. And it provides support for four spatial database backends: Postgres , SQL Lite, MySQL, and Oracle. And Postgres and SQLite are in bold because it offers most support for those two. And even amongst those two, Postgres

and Post GIS. have the most support. Don't fight me on this, but I think Spatial Light and SQLite has some limited support for some functions, some geospatial functions that are common. Um so I'll be using uh Postgres. Uh this is Postgres plus the Earth equals post GIS. Okay, so now how to discuss how PostGIS deals with all of those coordinate systems that we saw in our last slide. Okay, so for those of you who have used uh geospatial data, uh this is might look familiar to you. This is um known as the SRID. Uh Um what's um a spatial reference identifier. Um and uh the SRID defines what coordinate system you would like to use.

Now there are many SRIDs, hundreds, um And they're used for different geographies around the world. But to perform geospatial functions on the same data, it has to be of the same SRID. Uh so this uh defining your own SRID could be good if all of your data is, say, in one state or in one small geography Uh but oftentimes uh it's not. Uh so PostGIS offers the ability. It sort of throws the it has the ability to uh to put in unique SRIDs. Uh but um it sort of recommends that you use SRID4326, and that maps to uh the WGS84, which I mentioned earlier. And then when you need to use uh you need to measure something like area, distance, or length, you use uh a function to create that column, uh, and then in your projected coordinate system and measure it that way

Okay, so uh now I want to just discuss a little bit of sort of Geojjango data models, which look surprisingly similar to the regular data models, uh, except there's one spatial field type. And that's the point, line string, polygon, or multi-point. Not circles though, circles, it turns out, are much more difficult and complicated than you thought in elementary school. So you don't define circles, you define them as polygons. And GeoJengo also has support for raster uh data, which I'll get to in a little bit. And then as I mentioned, you can uh put in an SRID or use the default, which is 4326. A lot of mapping frameworks

and user facing use uh coordinates in lat -long order, uh but that's actually incorrect. Uh well according to PostGIS, it needs to be in XY order, which would be longitudinal So the very first application I developed. I did not know this, and all of my points turned up, I think, in Asia somewhere. So um okay. And then these can be uh initialized using uh WKT, WKB, hexadecimal, or geoJSON. And so this all looks well and good. Uh but as I was thinking when I first started, uh, wait, what the heck is WKT? I didn't know anything about what WKT was. Um So this is just a representation

of WKT. WKT stands for Well-Known Texts, and it's a text markup language defined by the Open Geospatial Consortium. Um of which the US Department of Defense is a part of, and many other 500 other government institutions. And it shows beneath it its binary equivalent, and that's how the data is actually stored in the database. All right, so now that we have a better understanding of what these data formats are , we can tackle what I had mentioned in the last slide, the differences between vector and raster data, uh, as it relates to GIS. Okay. Um so on the left we have uh raster data, and this is represented by um

Largely satellite imagery. A raster consists of a matrix of cells or pixels organized into rows or X and Y columns, where each cell contains a value of information. And this is good for representing continuous data, real-world phenomena like temperature, elevation, or other spectral data, continuous data. Um and vector data, on the other hand, is good for representing geometries. Um and all vector data consists of a list of coordinates that define vertices. So again, it's points, lines, and polygons, and that is usually small storage requirements, whereas raster data has large storage requirements.

And so that is um a polygon representing the county of San Diego. Okay. And so here are the steps. They look remarkably like setting up a regular Django model. You get or create your geospatial data somehow. You create your Django model as you would, defining your SRID and your geospatial field, point, line, polygon, etc. You find some way to import your data. And then you can start querying. And there's a ton of different querying methods that are really fun. So, um you can perform data discovery, min, max, uh you've got a bounding box area and length.

Similar and you can query and filter similar to the way you would as usual, querying using Django's ORM. It comes with contains within intersects, borders, overlaps, touches, tons, and you'll need to look up the documentation. as I did probably many times to figure out the differences between some of these. And then you can aggregate data based on filters and create new geometries using union unarary or unionary union or cascade. union which can merge lines, polygons, and multi-polygons, and um lines, polygons, and points. Okay, and now just to go over some other popular libraries that I use a lot. I use Shapely, which is good for vector geometries.

uh geopandas as well which is good for vector geomet geometries and rostereo which I don't have much experience with um but I have heard it's really popular uh to manipulate raster data. Okay. So so far we've covered on all of these things, uh, GeoDjango supported geography field types or geospatial field types. different coordinate systems projected versus geographic, raster vector data, distance and spatial querying methods, and data formats and data translations. So are we all done? No. Um we haven't discussed one important aspect um that I've spent a lot of time on, and that's debugging and troubleshooting.

Um Okay. So um some of these geospatial libraries can be frustrating, incredibly frustrating, uh especially when you're doing really complex geospatial analyses. I've often received uh geometry and topology exceptions that are difficult to debug, uh, GDAL being one of them notorious uh for having different versioning issues. Um I hope I don't know if the invent uh creator of GDAL is here, but I love GDAL, but it can have some issues. Um and so uh for example here's one that I just want to touch on. Um so you can see above, um I have GeoJSON with a multi-polygon, um, and an it's an empty the cool There's empty list as court as its coordinates. Um and I'm able to uh initialize my GOS uh geometry

and uh when I print its well-known text it comes out empty, but uh the valid method comes out true, and that's because Geojango is not validating that there are actual uh polygons within my multi-polygon. Um and this is different to the below block, uh which shows I'm trying to um instantiate a uh polygon and that is an empty geometry, an empty list, uh, and then I get a GDAL error that says, you know, basically invalid geometry pointer, you can't instantiate a polygon with an empty list. And so this showed up. I imported for hours a lot of multi-polygons. It turns out they were empty. And I thought they were valid, because I was printing is valid or in this. So I was printing uh valid and it was turning out okay.

So that's just one issue that I can't I've come across. So I've uh compiled uh a hopefully helpful list of uh tips to debug. Okay, so despite the example I just gave, I do think that uh the the valid method and Shapley's is valid method uh are really useful for figuring out whether or not uh a polygon is valid. If it's a line and there's a break in it, I believe is valid will return false. It'll validate whether or not polygons are entirely closed. And so uh and then when in doubt, plot it using QGIS. Um I mentioned that app before that I was working on. I was trying to plot the US landfills. Uh um in the US um and uh they weren't appearing on my map at all um and I realized that a lot of them were in the Pacific Ocean, which

Might be valid, but was not where I wanted them to be. So it helps to plot them using free software, QGIS being one of them. Also plotting them helps discover uh slivers in between features, maybe holes in polygons, or breaks in lines. Oftentimes these will import smoothly, but you'll get errors down the line when querying them. And so I found that very helpful. Other things are memory and performance issues that I've come across. Um I came across came upon this the other day. Uh one of my geospatial queries was taking uh hours, and I realized that the M uh was actually miles and not meters. Some somewhere along the line. I must have transformed it into miles. And um

and that was causing a lot of the other uh geometries around it to be contained in the filter. Uh and so make sure you have your units correct. And then there's also a helpful simplify method that you can use. And this you can add a tolerance decimal, that's between one and zero, and that simplifies the coordinates. So it'll cut out some of the coordinates, and depending on the precision needed in your app , that can be very helpful and speed up queries. Finally, the prepared also does something similar. It prepares it for more efficient geospatial analysis. Uh and then um some of you might be familiar with this, there's something called the right-hand rule, and that basically says that all of the interior coordinates need to go in a certain direction, they need to go clockwise, and the exterior need to go counterclockwise.

And this has resulted in a lot of GDAL errors that I've spent a lot of time trying to debug. Okay, so I'm going to give a short demo. This is the app, one of the apps that I've worked on, and I just think it shows how we've used uh PostGIS and uh Geojo. Uh let's see if I can sorry, I should have had this up before. And it's our app called Grid Assessor, and it maps uh US distribution data. Okay, so here's the US. Uh and actually I'm going to Just go ahead and search.

I think I've got that correct. This is our our location where we are currently. At the Marriott. There we go. There's some nice uh reverse geocoding going on. All right, and so the green or sorry, the the orange are feeder systems around us, uh the yellow are transmission lines, and then land parcels will will start to come in, and those are polygons. Or the polygons will start to come in and those are land parcels. And so you can see we've used polygons, multi-line strings, uh, and points to represent um where we are in the uh uh energy around us. Um Alright, so now I'm not gonna click around too much. Just wanted to

show that. Oh no, I lost the presentation. All the way left. Nope. There we go. Oh I see. I need to go back. Well, I think the last Okay, well I think the last slide is just thank you. Oh it'll go back? Oh there we go. Thank you. Sorry. All right, there we go. My last slide. Thank you all so much for staying till 5 p. m. This has been really fun. Um and yeah, thank you to the DjangoCon community and all of you.