Building Dynamic Dashboards With Django and D3 by Clinton Dreisbach

Speaker 1: Come on, yo.

Speaker 2: Um this is my first Django Con , which is super exciting for me. I am a little nervous. I'm a little nervous, not because of the talk but because I uh I brought my four-year-old son and he's in daycare and it's the first time I brought him to a conference so if my phone starts buzzing uncontrollably. We'll see what to do about that. And if you see a little hobbit running around tonight bowling, that would be mine. Say hello to him. He looks just like Bilbo Baggins. It's uncanny. Cool. So before we get started today, I just want to make one note about content. The dashboard that I'm going to show, this is all a case study based off a dashboard that I just built and just released as open source last week. It's all based around 911 data, which is, of course, police data.

Speaker 2: And the police are a very touchy subject right now. I just want to state that up front in case anyone has any issues with that, that is okay. I understand this is not a pro police talk. I 100% fully support the Black Lives Matter movement. This is not an anti-police talk either. It's just about 911 data. But with that said, I worked on this in coordination with my local police department where I also built an application to help them detect racial profiling and their traffic stops and prevent that. Great, so the problem that I had to solve. I started at my job and they came to me and said, hey, we want to build this dashboard that handles millions of 911 records. We have a prototype. Here it is. The prototype is built. It's a static site. It uses a CSV file for the back end.

Speaker 2: It serves everything to the front end. And it's cool, but it really doesn't scale. You can see it here. It looks nice, but it really doesn't scale. And so they said, hey, we would like for you to build something better that we can really use millions and millions of records for. This is what I ended up building, and we'll look at it in detail later. But with this, I built it with the back end of Django, where I do all my data processing, and then uh front end using D3. D3 , if you're not familiar, is a data visualization library in JavaScript. It's fairly intimidating. Who in here has ever seen or been intimidated by, particularly D3 or and or JavaScript? That's awesome. You're not gonna have to see too much of it. So

Speaker 2: part of my talk actually recommends hey maybe you don't want to use D3. So the architecture of it So here's the tools that I used. This is my stack. I use Django, of course. It's right now running on Django 1. 8, but I want to upgrade it. I don't mention the database I use here. I use Postgres SQL, which I see as kind of a necessity for this, and you'll see why later. I use the Django Rest framework, which has been great for me. Absolutely love it. Obscure library called Django URL filter. And then on the JavaScript side, I use something called RActive. js. This gives us the ability to do reactive programming. I'll kind of explain what reactive programming is as we get further into the dashboard.

Speaker 2: And then uh D3, of course, is my visualization library. NVD3 is a higher level library on top of that. And then Leaflet. Leaflet is a mapping library that allows to have dynamic maps. Before we get started, I think just so we have context, it'd be nice to see what we're looking at. So I'm going to pull that up. Right here. So the dashboard that I'm talking about This is it. This is using New Orleans data. I don't live in New Orleans. I live in Durham, North Carolina. The Durham one is the one that I created for my local police department. This is using public data, and that's a really good point. This is open source, so you can use it with public data. But here I can see the call volume over time.

Speaker 2: You see it dips at the end, that's just because we don't have today's data yet. I can click through and see if I want to see all general assistance calls only. Click that. Now I've just got journal assistance calls. I can see how heavy it is all over town. I can look at the response time and see That the uh median response time for general assistance calls is uh five minutes and twenty-six seconds for the last two weeks. That's not too bad overall. It could be worse. Um and then I can even look and see all the calls by location At that moment of like, wow, this isn't gonna work. That was awesome. Uh so yeah, this this clusters them, so I can

Speaker 2: go in, look at individual calls and see, like this call right here was uh complaint, other. Um and I can see high call volume locations, uh which this is being used uh actually to detect mental health problems of community in some places using that. Okay, now that we just had the like briefest overview, how do you build something like this with Django? Cool, so this is the architecture and I'm gonna come back to this again at the end. So you're gonna get to see this again, but if you look here, you can see I start with a request that comes in and I load a page. And then I sort of push everything off to the front end where I'm handling everything with D3. I'm using Django really to generate summary stats, which are the summarizations that you see on the front end.

Speaker 2: It takes the filters that come in and generates it. Okay, Django. How do I use Django? Well, so for Jang for for the Django side of this, I have an endpoint for every page that you saw when I flipped through uh the dashboard. Um so this is really simple just this three little endpoints here um that all they have an API prefix because they all serve up JSON responses They all serve very uh they I try to keep them compact and I they're all summarized ahead of time. Most of that's done in the database to make this as fast as possible. The way I approached that was I came up with this idea of a summary model where I put all the analysis that I would need to do to display an entire dashboard page.

Speaker 2: in here. This is just a very small subset of it. But you can see here in this I don't have a laser pointer, so it's gonna be hard to point, but that's okay. You can see here in this I have sort of a base one called call overview. And then I use in my subclasses this idea of annotations Each page is really about a specific subject. So we've got a page that's about the amount of calls. For especially for departments and planning, they want to see the number of calls. For the public, we really want to see the response time I I care a lot about what the response time is in my community. And so I can have a different annotation for that here. But every one of the particular aggregations I want to do, whether it's you know by day of week, by district, by type of call, by unit, all those different things I can do, and I just put the different annotations for how I want to aggregate that inside of each one here.

Speaker 2: You might notice in here down the bottom on that last annotation for the mean, the average seconds of the officer response time. Seconds is not a standard Django function for the ORL. So I'll talk a little bit about why that happened and how useful that's been. So I mentioned I use Postgres SQL and I find that to be a necessity for this sort of work because it really does have some great things inside of it. in terms of being able to do different types of queries that you can't necessarily do because it goes outside the SQL standard. But here's an example of where I was able to build a custom function around the Django RM and why Django was so useful. We had our option, in fact I was pressured at the beginning a little bit to use a different technology just because we had several people internally who were very

Speaker 2: good at JavaScript. The discussion came up, you know, maybe we should use Node for this. And one of the big arguments that I made in favor of using Django was how great it is at letting us customize the ORL and build things around it to make it fit our needs exactly. So here, I have a little helper function called precision where depending on the amount of data that I am looking at, whether I'm looking at a week, you know, I'm looking at a month, I'm looking at a whole year of data. It's going to use a different precision when we're looking at the data. And you know, maybe month, day, or hour. And then down here at the bottom on volume by date. You know, I have um I I truncate that date by the precision. So it's grouping it together.

Speaker 2: Like if I'm looking at a year of data, it's grouping it together by month. If I'm looking at less than that, it's grouping it together by day. If I'm looking at less than seven days, it's grouping it together by hour. And that is going to happen dynamically in our dashboard, just to take a quick look here. Alright, so last seven days, when I moved to last seven days here, we're suddenly looking at it by hour, and we can really see what's happening. If I go back here and um Say year to date, and then I'll change that to a more custom range. Okay, so that's by day. You can see by day, and then if I go in here and say, oh yeah, not January 2016, but um January 2015. It might be quicker to type that.

Speaker 2: This is uh this is live. Um love it. Love it. All right, here we are. Let me click apply. Wow. Okay.

Speaker 3: Maybe I'll use a future date.

Speaker 4: Yeah, that's best.

Speaker 2: Oh, was I looking at future date? I was, because I um Because I'm a nut. I thought that this was August. I don't know. Um take a second there, but yeah, now we're looking at it by month. It's a much flatter line. So this was a really cool feature that I was asked for that Django just made easy to do. Django plus Postgres in this case. What you see here with this date trunk, again, that's not a standard uh function. Um I had this whole thing in my blog and I go into that function a little bit further there. I don't really have time in 25 minutes to dissect the date trunk function. But it was something I was able to add easily. In this case, I was also able to add an aggregation easily. I needed percentiles. RTI International, I work specifically on our Center for Data Science.

Speaker 2: Um percentiles are not a particularly heavy uh uh data uh piece of data science, but in this case we really wanted not just to see the average response time But you know 75% of calls. What's their response time? Because that's gonna matter a lot more. The average response time can get artificially lowered, but we really want to be able to see what's the real response time on the ground. So with here I was able to go in and say, give me the quartiles, that is, you know, 25% of calls. How quickly are they responding to? 50%, 75% of the total number of calls. Lastly with these summary models, I just call a bunch of functions and return a bunch of of JSON. In this case, you're seeing a data structure here that gets turned into JSON by Django RestFramework. And this was this is for the volume page.

Speaker 2: I've got volume by date and source and all these different types of ways that we want to look at the volume. And then we look at the We also have the heat map, which shows by uh day of the week and hour what the amount of calls were. The key to this and the key to doing one of the big things that was demanded. was Django URL filter. So one of the big demands, one of the things that I had to have in this project, was the ability for every view to be bookmarked. If they click through on three different charts and they've drilled down to say, hey, in this district on Mondays for general assistance calls, this is what I'm looking at We they want to be able to send that to someone else, not just to bookmark it so they can look it up again, but you know, for people who aren't necessarily that technical or they just want to do something quick.

Speaker 2: You know, you've got you've got uh Let's say uh you've got uh a lawyer who's looking at this public public data and wants to send it to their client, or you've got someone within the police station who wants to send it to the chief. They would just be able to send that URL. So I use Django URL filter. Now A lot of people may be familiar with Django Filter, just because that's highly shown on the Django REST Framework website. It's a separate library, but it's often used with Django. uh REST framework. Jingle URL filter is its little brother that isn't nearly as good, but it's super hackable. And it's very small. I shouldn't even say it's not as good. It's just quirky. Right? And if you look at how many people use it, not many, but it's very, very hackable. And that was important because I had a couple of things I needed to do here. The big one was I needed to be able to call query set

Speaker 2: methods. I had certain ways of looking at my data that I couldn't just put in a like, you know. call. objects. filter statement. I need to be able to say, hey, I need everything during day shift, which is you know 7 a. m. to 7 p. m. And so I might make methods on my query set. You certainly don't have to understand this code up here, but note that what it does is it looks here. to see if there's an attribute with that filter name on the query set and if so it just goes ahead and calls that instead of sending it over to filter. And that became very, very useful. So you can see here how it translates it, right? If I have get parameters of district seven, nature 10, those will go straight to filter, but shift, because shift is a is a method on my query set will get called here if that's one of the parameters.

Speaker 2: The other big hack was I need to build my filter from a data structure instead of building it from classes and objects. The reason why is because I have the filter and I have the way ways to select it on my front end. I didn't want to have to recreate that, right? One of the easiest ways to make mistakes in programming is by repeating yourself because you will never get it exactly the same. And so I certainly didn't want to do that here. I wanted to be able to have this data structure right here, which is a pretty arcane data structure like most things that you put together while you're coding it. You know, this is sort of putting the airplane together while I while it was in it Um yeah, it it it it got a little funky, but it's you know relatively self-explanatory what's happening here. I've got filters based off time received and based off shift

Speaker 2: and district and nature group and whether or not the call was canceled. Um And with this, it can just translate it straight into JSON. And then my front end can consume that to build all the filters that I have along the top of the page. We'll see that again when we go back to look at it. My API endpoints were simple. There's not a lot of detail to go into on them. But you can see here it calls, it takes the those get parameters from the request, which are going to be up at the top of the page or be in the URL bar, right? So it's markable I can send to other people. And then it's just going to take my overview model or my summary model, call it to DIT donut, and send that message. Nothing major there. The front end is where this got really interesting and how it works with Django.

Speaker 2: So my front end, I mentioned that it's reactive. So what do I mean by that? Reactive really just means that there is a flow of data, there's events that happen, there's things that monitor those events and react to them In this case, the things that you might see someone do while using this page is they change filters, which they can do in one of two places, right? They can click a drop-down. Let's pull that up. So they can click one of these drop-downs, like I just want to look at Monday. You can see the URL changes up here immediately and it reloads. I might also say, I just want to look at this district. Right, so this updates in two different places. Um and the thing I just selected here, district three, I could select district seven here. Or I could go in here and clear it. I have two different ways that I can update these filters, but every time I do, the URL at the top of the page changes.

Speaker 2: The application watches for those changes in the URL, right? If I went and changed the URL just by manually changing it, if I went here and said I don't care about this nature group, I wouldn't really expect someone to do this. But they might have a bookmark. In that case, this would be necessary. It looks for those changes in the URL and then sends requests to the backend for new data. It gets that request back. We update the data. And when the data is updated, the page is updated. If anyone's ever written like a fairly complex JavaScript application using pretty standard tools, let's say you just use jQuery. uh which is an awesome library but whenever you have something new you need to update you know there's a lot of link linkages you have two different ways that you can update something now you've got to link both of them With this sort of way of looking at it, it made it very easy to add new controls, to add new charts, and not have to have sort of an exponentially growing set of linkages.

Speaker 2: Like I said, reactive programming, the big words that you might hear about are unidirectional, right? Everything flowed one way. Would this change happen, then this would happen, then this would happen. I don't have data syncing two different ways. And then it was uh data flow, right? Data going on, and event driven. So let's this is a uh sort of a summary of the different events And the reactions that I saw in my application. And all of these look kind of synchronous, right? When a user clips on a chart, the filter changes. With the filter changes, the URL hash is updated. But note that this is asynchronous, and that was part of this that works really well because it means I can come in at different points. There's no direct chain of things I have to do. If any of these things change, then the things that should happen happen. So like I showed, when I just go and change the URL hash manually,

Speaker 2: the Ajax request is sent for new call data. And then each one of those charts actually monitors a specific subset of the data. So if only that subset changes, then only that chart updates. It's a pretty slick way of doing things. This is a similar thing to what you see um if anyone's ever used React. js. It's the same model But it uh Ractive was what we used because it was uh a little simpler. Um it works really well with Django and it was written by people I respect over at The Guardian. Here's a simple component that you might see. So I've got some JavaScript here that shows I've got a template, and then I've got this data hidden true thing, right? So I have a little chart header I want to hide and open.

Speaker 2: In fact, you can see it right here. Pretty simple stuff. Not normally complicated. But here I have the template where you can see You know, it says like unless hidden, show this. Right? This isn't uh rendered once. It monitors the data, which is the whole point of the way uh this dashboard works, is it monitors the data to update. So as soon as the data updates, this is re-rendered. Okay. I haven't talked about D3 at all yet. We have five minutes. Awesome. Visualizations. So, but what I was going to say about D3 is there are D3 is really, it can be easily thought of as a toolkit for building visualizations. It is not particularly

Speaker 2: good to think of it as uh a chart library because it's not. A chart library would give you some charts. Uh D3 gives you a lot of tools. Uh I uh I sort of describe it as you know you could get you know It's like a bag of lotus parts. You can build your lotus, it's gonna be awesome, but you gotta build it. And so getting, you know, going and just picking out the Ford Taurus from the lot is sometimes a smarter move. And there's a lot of higher level libraries on top of D3 that I recommend. I used ME D3 because I liked its styling. It fit in really well for what I was doing, but there's a lot of other ones to look at. One I've been using recently is Plotly JS The team behind Plotly open sourced their JavaScript library and it's awesome. It's also like two megabytes, it's giant, that's the only downside. But here you can see a

Speaker 2: simple chart object that I built, this higher level object on top of D3. You know, it says, oh filter things by day of wheat received when I click on it. Um and has some formatting stuff in there. And then I have this monitor chart. And this is how I hook up the reactive nature of this. So whenever data in the volume by day and week subset of that data tree changes, then call update on this chart. All my chart objects have a create and update method. And that's really that's the entire API behind my entire system. Here is When the page loads, call create. When the data updates, call update. Monto chart is simple. It just takes a what's called a key path. Again, that's sort of a path into the tree of data and says, hey, call this function when that changes.

Speaker 2: unless the page is currently loading. That was a little protection I put in there because a lot of this stuff is asynchronously loading. The heat map was where I actually used D3 for real. The heat map was very, very cool. To show how to do this would require an entire class on D3, which I am not gonna give. But and the great part is I built it directly off of one of Mike Mostock's examples on his excellent site blocks. org. If you're interested in doing cool visualizations , He has amazing examples there. It was cool until I was teaching some people data science and then one of them made uh their sample project that had the exact same heat map in it. And went, oh, okay. I guess more than one person has looked at that example. Um but this heat map here is showing, you know, by day of week and by hour uh how many calls there are.

Speaker 2: And again, I just have a create and update function. So I was able to plug this into the way everything else works very easily by having this create and update function. So every one of my visualizations has the same API because of the way the data flow works. So, my lessons learned. First one was for building something that's so data intensive like a dashboard, reactive programming really simplifies those interactions. It made it much more simple to work with. I learned that I should always use higher level libraries on top of D3. I started by not doing that and I I I bled on my computer for it. It was not fun. I really, really that was like two weeks that I'm ever getting back. And then I didn't talk about this in here, but uh this is an open source application, you can look at it.

Speaker 2: When you have serious front-end work happening, when I have like simple front-end work, I like to just use the standard sort of uh Django asset pipeline tools. Um I like Django Compressor myself. But for serious burn -in work where you have a lot of stuff going on, using Webpack and Jangle Webpack Loader is really great and really simple. I was able to use it with this and even make a plugin system for it. It works really, really well. We're running, we're getting near the end, so I have this again. Um and it might make a little more sense to you now that I've talked through reactive programming how it works, how all these things are connected If you want to see the code behind this, there's a link at the bottom to get IO slash CFS. You can look at the code there. And if you go to cfsdemo.

Speaker 2: rticds. org, that's a mouthful, you can actually play with the application. with the live uh New Orleans data. It updates nightly, so it should continue to have live data. I know I don't have a lot of time for questions, but I'd like to take any if there's if there's a minute or so.

Speaker 3: You have a minute. So is this application like communicating with any of the officers like out and like make or doing controls right now or if you square it in-house and then they analyze the data and then they were like, okay, we send these officers in these locations today.

Speaker 2: So for my local police station, it's on the internal internet. It's mainly used at their like cop stat meetings, which I got to go to. I felt like I was extra on the wire. It was badass. Again, I am I am not crazy pro-police, but there's something about feeling like an extra on the wire that's pretty cool. So it's used in house on their internet. But uh yeah, it and it's been really useful for them to be able to find real problems in the city.

Speaker 5: My question is actually related. Um I was wondering, yeah, I I really like how simple it is when you go from the the the URL, you know, all the way down the stack. But how do you how do you control situations where You know, so someone can't put like, you know, dunder user or dunder email or something. How does that work?

Speaker 2: So yeah, uh I mean the things that they're able to access are solely filtering on that query set.

Speaker 5: Yeah, I mean I know there's solutions, but what's your solution?

Speaker 2: Oh I mean Django URL filter filters out the things that can't be used on the query set.

Speaker 5: Okay.

Speaker 2: And anything that comes in that can't be used is just cards. So I think that's probably all the time I have for questions, but I'm gonna go right outside the door here if anyone has any further ones. I know this is a big topic for 25 minutes, so I'd love to answer any.