Optimizing your webapp by using django-debug-toolbar

So um just an introduction uh as uh as was already done. Um I'm a software engineer at Venmo, my name is Chris Adams. Um My Twitter GitHub is AdamC64. Interestingly, I'm not the Chris Adams who's also a developer, Python developer, Django developer, and Django contributor. who um works at the Library of Congress, uh AC D H A. So that's not me. Um and neither of us are the gentleman Chris Adams, a 90s-era professional wrestler who's here for disambiguation's sake. There he is. That's neither of us, just in case you're wondering. Um yeah, it's kind of interesting having a very common name and seeing the collisions that can possibly happen. So we're here to talk about Django.

Django is great. We all love using Django, I'm assuming. But Django is really a set of tools. And um they those two work together um in in interesting ways and helpful ways. But at the end of the day, what we're using when we're using Django, when we're writing Django code is um Is uh is a set of tools that work a certain way. So tools are great, um and but tools can be used in good or bad ways. Okay. Um this um in one sense is very obvious. I mean but but when you're actually going into things uh it might not be so obvious to you when you're when you're tr when when you're using them, especially for the first time. Um so the Django ORM is one of the tools in uh

that Django provides to us. It's an ob uh object relational model, um, easy wrapper over um SQL queries, which used to have to use for Used to be much more tedious to be writing. So one thing to remember is that whenever we're using a a tool, especially new tool, is we should manage our own expectations for tools. So many many people approach a new tool with a broad set of expectations as to what they think the tool will do for them. However, this may have little correlation with what the project actually has implemented. So this is just kind of a long way of saying that tools can be misused , especially if you're new to them. So you know, as amazing as it would be if

they did, unicorns don't exist. Tools are tools are not really perfect. They never can be perfect in this imperfect world. So we have to really understand them and not deceive ourselves, especially when encountering new tools. That this tool is the one panacea that's gonna really get everything. It's gonna understand me, it's gonna know me, it's really gonna, you know, I'm gonna be understood by this tool that I'm using. Actually, that's not true. It's your responsibility to understand the tool and to use it correctly. So the Django ORM, uh, among its other um among other things you could say about it, is an abstraction layer. So not only is it set of tool uh is a tool, but it's an abstraction layer.

Um abstraction layers are great because they take us away from um messy details. But also risky for the same reason. They take us away from messy details. Same reason. Okay. So we don't have to do many things that we used to have to do before because we have the ORM. But we might forget or not understand that the ORM is doing things that we don't expect it to do. So don't forget you're far from the ground when using the ORM. You have um You have uh you have the ORM API itself, you have Django's implementation, you have um uh Python, you know, maybe uh you you misunderstand some some way about how di uh you know dictionaries or keyword keyword arguments work. You you have you have lots of layers when you're using the RM, but then you have the s uh SQL itself and you have your s

your specific database backend which might have its own quirks. um might you know manage indexes in a particular way, might um might do things slightly differently, load things from c uh uh return results from cache that you're not expecting things you know and and that depends on your on your configuration. So you you you have a lot of layers here. Um So one thing that I really like about Django is the query set um the the the the the uh programming interface that query sets implicitly give us. And uh uh explicitly give us. I mean we can use we can use the query, uh we can define query sets, we can do many things with them, we can tell them we want certain things to happen. um when we when we at when we define query sets.

And um so query sets have have two features that's worth keeping in mind when you're thinking about them. One is that query sets are lazy. And the other is queries, query sets are immutable. So this is a review for people who might know this already. So what do these terms mean? Lazy means that a query set doesn't evaluate until it needs to. So uh many people you know don't get the you know really don't realize this. And when you do realize it it's a great aha moment because um you're you're realizing what's going on behind the good uh behind the hood. When you define a query set, you know, you know, model. objects at all, what you're doing is you're you're defining uh an instruction, which is deferred. It doesn't get executed until it needs to. And there's certain cases when it does uh execute

And um many cases where it doesn't execute. Uh query sets are also immutable, which means that if you chain them together, you say, you know, I have a query set and then I say dot filter, what's returned from that operation is a brand new query set. However, that new query set um in as much as possible inherits all the features and the um the properties that you wanted of your old query set. So you can chain filters together, you return a query set Um apply conditional, have another filter that only happens in the case of that conditional. You can do all sorts of things. And uh every time the query sets are returning and you're getting a new one. Um so you don't have to worry about um uh the the the the references to the old query set you can you can store store one uh

So c query sets are always new. And so, you know, the the the the the Quick examples here, you know, e each of these operations is uh uh returns a new query set and does not actually hit the database So um worth keeping in mind that you that um what's not happening in it it's the same level as what is happening These, however, are situations where query sets are evaluated. There are a few of them. They're in the Django documentation. These are just a sample of three. There are other ones also. Query sets are evaluated when you uh make a list out of them, when you um use an index or range operator, um, and when you uh iterate over them. So but the interesting thing to

think about here is that you can define a query set um very very early in your view function. It's only at the for loop, for example, in in the third in the third case here, that the query says query set is actually evaluated, the SQL is sent to the database server. So um so anyway, the things to keep in mind when dealing with with with uh query sets. Okay. So um I want to highlight um some uh easily uh some some patterns which are problematic. Um and oftentimes people don't even know that the pattern is problematic because they're really doing things that look very straightforward. I'm writing my my view function. Um everything looks great. I mean, you know

But actually um when I understand what's going on, um Um cer certain things uh um it's it's easy to miss. So let me let me kind of point these out here. Um So we have we have a sample app which is uh a blog app, kind of boilerplate. Um and I've I've taken away like uh various other attributes of these models. These are just the relations between the models. And um One of the features of Django model objects is that um when I when I act well uh look uh I'll I'll just go through the example so we can say. So so we have um we have basically blog post and a comment on a post. Submitter, which is a foreign key to user model, post

which has um uh foreign key back to the blog, so you're posting on a blog. And um, you know, uh uh likers, many of the people who like that post, okay, um, and comments. on the post which are done by a uh a particular user and um and you know the foreign key back to the post that they're the comment on. Pretty basic model, pretty straightforward So here's a view, which can be used to just generate a list of the blogs that are on the site. Maybe this is a management view or something like that. Looks really simple. Here's the template that renders that that view.

Looks really simple. We uh have a you know uh a reference to the blog detail page, we put the blog name there, and we say submitted by the person who submitted it, right? Very simple. This seems like just Just home run, like really easy thing to write, the the push it to production, get this thing working. Um and it generates something like this, okay? As would be expected. Alright, so um and now I just want to talk about the detail view for a little bit. Um the detail view um uh is a way of seeing a particular blog and I I you know get the re Get the blog or 404 if the blog um uh doesn't exist. I um I return I I I request the posts that are uh that are for that blog.

And uh and then I render I render that um the blog detail uh HTML This is a little bit more complicated because there's a lot of different features that maybe my users want to see. They want to see who likes the blog, they want to see the comments that are on the blog. And so this is a little bit more complex. Um so and it generates something like this. Okay, you know, I I just use some sample uh Lorem Ripsum generator. So, you know, we have comments, we have people who liked it. Um okay. So you this looks a little bit more complex in terms of how how this is generated. So now as Django developers we're left um thinking, okay, I've done my job, right? Um I've I've gotten something out.

Um I've completed the work, right? It works, right? But really, at the end of the day, as I said, the Jenguram is a tool, and the tool is functioning a certain way One of the main interfaces that we have to um towards our data is through the SQL quer uh the uh SQL query language. And uh Django is generating, the Orm is generating SQL. So how do we find out what SQL is being generated to to to to to uh in a particular view? Um and the problem is if you can't measure it, um you never know if there are problems, right? You know, um how how do you know you there's there's i information lacking and um we can theorize based on what we think it's doing, right

This magical, I know what the ORM is doing. Like, you know, I it's just super easy, you know, I I'm using dot all, so it's doing my my query there, right? No, it's it's not. There there's lots of uh false assumptions we can make So really um in general, and this goes not just for Django, it goes for most tools and most technology, um you need to measure and uh you need to get an actual um printout or a display um and transparency into how the tool is functioning and what it's doing. And as precise as possible. The more information you can get here, the more the more power you have as a developer. And I think Uh I think everyone would everyone would agree with me here. So how do we get this information? You know, we can't be naive with our tools. How do we get this information?

Well, one really, really cool project is the Django debug toolbar. If you're not using it, I highly recommend you use it or have some way of getting information that's equivalent to the kind of things that this provides out of the box. So I'm not going to go over how to install it. They have good documentation. It's relatively straightforward. So let's use the Django Debug toolbar to take a look at our um our pages. So first we'll take a look at the blog list page. So um here's our blog list. And notice that one of the cool things that the Django Debug toolbar is it gives us a uh panel, which is um it basically injects HTML JavaScript into the page. and um

also the data about the execution of the page. And it gives us these tabs that we can click on. So we can get information about, you know, um the the timing that it took to generate the HTML Details about the settings, the HTTP headers, the request object. Okay. These are uh I I could speak for a long time about each of them. Uh I'm going to focus for this for this talk on the SQL. And notice that there's 52 queries being generated and being executed to generate this uh to actually generate this page, which um seems like a lot for just a list, right? So like we we did we did the thing, right? We used the ORM, it looked really straightforward, we we used it, and um

what what happened here is um we we actually we didn't realize it. We we generated quite a lot of queries. And and it this is a bad thing because there's a a a network um round trip that has to happen. The the server has to send the query, um the the the um And also the uh receives response and uh the database server has to process each individual query. So there's a lot of extra overhead for having all these things happening. Um and uh oh sorry, if we click on if we click on the SQL button, uh the page actually actually um this Django Devoke tool guard the tubar actually gives us a list of all the queries that happen, which is kind of cool when you think about it. Here it is. Right here. Visualization gives us a timeline. I cut off a little bit more of the things just for the sake of

uh displaying but you you should definitely check it out. Um so you know now now our next question is great we have we have the queries here. Where are they being generated? And actually you can click on the plus and you can get that information. The plus to the to the left side. um will give us the full query and also the actual line of code where the query is um is is is generated. But I should say here, this is um not a query set that's generating it. That that that's already happened. In this particular case, these individual select statements are being executed through the foreign key, blog. submitter So Django, the query sets are lazy and also the model objects are lazy.

The foreign key relations aren't going to be accessed, aren't going to be loaded. until they need to be or if they need to be. So that's why it saves in terms of an optim optimization. Maybe you never reference the submitter object. Okay? And so you don't need to have information about that the off-user um uh um model object. So here it is, uh blog. submitter is um being referenced, that's generating the these many, many, many individual queries. And the more objects you have, the more queries you have. So this is actually a very bad situation because there should be a way to do this much easier. So this is where Select Related comes in. Select related works by creating a SQL join and including the fields of the related object in the individual select statement.

for the um the the the for the model itself. So so the the auth user is going to be joined with um uh with uh the the the blog objects in this case. So the SQL is going to return back the blog um c mo um uh multiple times uh uh as as many auth users as there are And you think, well, is this bad? Well, it's actually much more efficient because you you're batching it, you're doing it in one shot, and so there's only one network uh back and forth to get that information. So it's actually um much better, much fewer queries. So it gets a related object in the same database query. And um however, to avoid much larger result set that result from many relationships, so like a many to many field as opposed to a foreign key.

or maybe a reverse foreign key when you're where you're where you're getting the the opposite side of the relation. Select related won't handle those cases. It'll just handle i think of it in terms of foreign keys and one-to-one fields. It's it's it's um helpful and when you're going to be iterating over them in a loop. So here we we use select related. We tell it we want to uh select uh the submitter and um uh of the blogs and you can have commas and have multiple uh multiple fields which are which it will join together and get all that all that information. And um So um so now we do it and uh notice two queries and there they are. Okay, and notice the inner join. um on the uh on the query. So uh we

we've just completely um uh r reduced the number of queries and even if we have much much more users and much much more blogs this isn't going to increase um So this is really, really great optimization, one a great tool, especially when you're using loops. Um So all right, the blog detail page now is a little bit more difficult. So now notice we start with uh 44 queries and um if we take a look at the queries in this case it's a it's a lot less straightforward, right? You don't just have a a list And what I usually do is I start from the top, uh open my code, and I actually go down and see see where these things are being invoked and I just try to be intelligent, try to identify patterns. Patterns are good. The same thing is happening over and over again. Might might be a group of things happening over and over again. And in this case, I'm just going to

reveal it for you here. The um patterns that you identify if you take a look at these things are that uh there is a submit or foreign key being accessed on the comment Okay, so now we're talking about comments that are being um generated and who submitted that comment. And there's also a query on the likers attribute, which is if you remember is a many-to-many relation. as people who like the post. So we have uh comment dot submitter and post likers. So prefix related is an optimization that's used for this case. And instead of using an inner join What Django is going to do is going to pre-fetch those uh those relations, those many to many relations. And it's it's all it then is going to do the uh the operation we expect of of our model and it's gonna join them in Python.

And that's actually more efficient than having database do it for you. So there's a reason why select related doesn't do this by default. Prefetch related is going to prefetch those many, many relations because it could just result in a massive you know, you know, result set. So it's much, much better to use it this way. So you you remember prefetch related is for many uh a a uh relation that has many uh many possi uh m uh many related members. It does a separate lookup for each relationship, does the joining in Python, and it allows it to uh uh this allows it to pre-fetch many to many and many to one objects in addition to the Farnkey and one-to-one that is in the Select related that Select related uh helps you use helps to helps you manage um and it also supports uh generic relation gener generic foreign key I don't know what version of Django that is

um that it does support that I would check that but it it you you can use it for those kinds of things So we put it in here. You can use double underscore syntax, and in this case, this is the most efficient um our set of arguments that um um there uh that you know at uh i that that will help us in our particular example. Um Uh there might be more that I that I overlooked. I I didn't spend that much time trying to find the optimal, optimal. But these these uh this simple optimization, this this one line of code is gonna is gonna save us um So if you see it's going to prefetch through the comments table and get the submitter, and it's also going to have that the comments information, which is going to be helpful for us.

It's going to implicitly do that. And it's going to select the likers. And one line and we're done. Well, you know, we only have seven queries now. Yeah, I'm sure you can get this uh even smaller. But uh Gino one line just uh amazing amount of work that th that you can do If you notice, it prefetches certain things before others. So, you know, it's it's just really, really helpful pattern to think about. and to bring into your uh your own tool toolkit um as you as you're working to um to uh make your cor make your views better. Um So in summary, the query set API method, select-related and prefetch related, automate some uh some of these best practices uh to avoid extra queries in views and in templates.

Okay. And uh use select related for one-to-many or one-to-one relations, and prefetch related for many to many or many-to-one relations, including the reverse foreign key relations. So um yeah that's basically it. So thank thanks. This is pretty straightforward. I hope this will um uh allow you to start using those uh these tools more effectively. Um I have I have a code on GitHub, so please uh please clone it. Um and yeah, thank you very much.