Why large Django projects need a data (prefetching) layer with Flávio Juvenal

It's very nice to be with you today. Thank you very much for having me. I'm going to talk about some problems we saw while working with many different Django projects. I'm Flavio Juvenal. Here's my Twitter handle. I'm from Vinta Software. And at Vinta we love Django REST framework. We've built many Django REST framework APIs over the last nine years. uh with different customers from Brazil, from the US, and we we did lots of stuff and we've seen many problems with Django Rest framework. We love it, but we've seen many problems with it. And we do uh custom software development uh for all sorts of clients, all sorts of types of companies.

And we either are the whole development team or we sell squads, teams that can work from the front-end design and back-end to our customers And we are very integrated with the software engineering of those companies. So we are very transparent about the problems we face on the code base. And the solutions we we find, we share with the CTOs of those companies and we validate them, those solutions with them. And I'm going to discuss with you some of those solutions today about problems related to pre-fetching on Django REST framework, but in Django in general as well. Okay, so uh

general rest framework loves dry, loves the don't repeat yourself principle. Um and it work it works very well uh in general Uh for HPA API development. There are lots of places in Django Res framework for you to plug in your own classes or configure classes uh for dealing with common concerns such as permissions, filters, uh validation, etc. So it 's great. Uh and we love that so much that we've built uh this documentation tool called Classy Django Res Framework. Which is based on classic class based views, another tool that's more focused in Django class based views.

And we built this one focusing on Django REST framework. And there you can find the like Methods and attributes for uh generic views, serializers, view sets, mixings, even pagination classes. You can find it out there and you can like find the exact place or on those generic classes that you need to override. And that avoids you to write like lots of custom code and that's great and it works overall. But what's the price we are paying for all this don't repeat yourself uh thing? Uh are there any trade-offs in respecting don't repeat yourself in Django Rest framework? For example, reusing serializers.

It's a very common thing to do in Django REST framework, right? For example, you have a movie list view Which lists like movies like a uh you can have on IMDb like the movie database. You could you could have that there uh the movie list view uh using the movie serializer Okay, and that same serializer you could have maybe in a movie detail view. And now you are reusing serializers across views. General REST framework makes that really easy to do. It kind of like guides you towards that, right? Okay, so what's the price for that? If you are not careful, an increase in this dryness thing uh in the don't repeat yourself philosophy

can lead to a decrease in performance Right, so imagine you want to see that on your API. You want to see uh the movie The Matrix, uh list of movies, uh, one of them is The Matrix. Inside each movie you want to see a list of the directors of the movie because the movie can have multiple directors as the matrix has. Okay, that's fine. To implement that in Django Res framework is quite easy, right? I just need a movie serializer with directors and the person serializer inside of that. Uh and the person serializer is another serializer I declare on the same or module or not, whatever. Because kind of the person serializer is asking to be reused.

uh like it it's asking to be nested inside the movie serializer and that works well for Django Rest framework but now unless you do a prefetch on the view that uses the movie serializer on all views that use movie serializer you have an N plus one issue right because for every movie uh instance model instance I'll have to get the directors from the database and that will be an N plus one queries issue. Right, so you need a prefetch there. Okay, so now you have to go back to all views that use that serializer and you have to add prefetch later. Okay, so we've seen that many times probably on blog posts, articles, other talks. That that seems fine, right?

But it it seems to be coupled, right? Those things. Every time I change the the movie serializer and I do something with pre-fetching or with annotations, I have to go back and change all the views So what's happening here because don't repeat yourself is making us repeat changes, right? Otherwise we have to pay the price of n plus ones. And it's a bit strange. It's not only a performance issue. Okay, it's more than that. Because an increase in this dryness. will lead to an increase in change amplification. And change amplification overall is a bad thing because change amplification is the number of places in your code base that

need to be changed during an anatomic change to the software. What I mean by an atomic change to the software is like you have a customer that asks you, or you have a boss that asks you, okay, uh I need you to add the Directors to all movies returned by the API. That's an atomic change from the point of view of business value, right? But now you have to change multiple places of your code base from your code base to make that change. So the atomic change leads to multiple places that need to be changed. And it's better to have a low uh change amplification for the majority of changes you are going to make in your software. And the problem, as I said, is not only about uh prefetching, uh, it's more than that.

So imagine you now have to add the nomination count uh of awards for each director uh inside each movie. So this nomination will be something like all the nominations from all the kinds of awards that a director has. Okay, so I need that field. Uh it's really easy to add that to Django REST framework. I just need to add an integer field to person serializer. That's fine. But where that comes from? There's nothing here in the serializer telling me where that comes from. So nomination count is asking for an annotate Okay, I I need to annotate that uh in my

query set in order for it to appear in the serializer. Um for the serializer to be able to get that out of model instances But that's implicit. Okay, that's not explicit on the serializer. And as the Zen of Python says, uh Explicit is better than implicit. So I have something here that's kind of conflicting with other Python philosophies, right And to solve that I have to go back to my views, all of them that use the movie serializer, and I have to do something like this. Okay, so now things are starting to get complicated, right? Because I have to Inside movie, prefetch directors, then I have the query set with of persons with an annotation

with the count of nominations, and I have to remember to put distinct. Okay, because it's a left order joint as we saw earlier today. Okay, so things are starting to get complicated. Things are implicit. Uh there are lots of places I have to change. And don't repeat yourself in general. Isn't helpful if you need to be careful. I guess that's the idea here. I have to be careful to make that those changes in many parts of my code base. So the don't repeat yourself has a price here. I can reuse the serializers, but I now have to be careful on how I use them and how do I use query sets for them And I think what matters most is not only the don't repeat yourself, it's how much change amplification do I have in my code base.

Uh how much code that changes together is close together. I guess that's uh even more important for maintenance than simply don't repeat yourself. I have to optimize my code for changes, not for the first time that I write it. Okay, and vanilla Django suffers from the same issues uh because template code can also use pre-fetches and annotations set by the view And I just forgot something. The annotation uh will simply not be available if I forget it. So it's not only a performance issue, as I said, it 's also a logic issue. Okay, the feature will simply Do not work if I'm not careful enough to prefetch and annotate everything my serializer

needs or my template code needs. And template code is even worse because it's uh mixed with uh HTML and everything. Okay, so how can I solve that? If if those things are coupled, uh query sets and serializers, should I just like accept that and couple that? Uh and the first solution I usually see in many Django REST framework code bases is to put some kind of method inside the serializer that gets the query set that serializer needs. So you can do that. You can like uh add a class method, get optimized query set inside the R serializer, uh and you can receive an initial query set from the view with filters and pagination and everything.

And you can do all the pre-fetches and annotations you need to render that serializer. That's fine, and lots of people do that. I've seen that in many code bases. That seems good enough, but There are different contexts where you can reuse and nest uh serializers. So serializers are not reused only with nesting. Sometimes they have their own views, right? So I have movie serializer with personal serializers inside of it, but I can have that same person serializer. Uh and use it in a person list view. Again, because Django REST framework makes that very easy, so people will eventually do that in your code base. And now that the things are contextual, right?

I'm not sure if the prefetches I need to make when person serializer is inside the movie serializer are the same ones that I have to make when a person serializes by itself in another view. I'm not even sure that I should use the same serializer for those different uh cases so things start to get complex because it's all contextual as well. Can you handle maybe all the cases that you need with just this method? So maybe we should parameteri parameterize a gap optimized query set. Maybe if I add a nested serializer, like a new one , I need also to nest the get optimized query set calls. So there will be nested get optimized query set

calls happening. Uh what if I remove unnecessary? Now I have to remember to remove uh the get optimized reset call on the other uh So it's quite hard to maintain this. There are lots of changes that can happen and you have to be careful once again. And even if you think you can handle all of that, there's always the serialized method field in Django REST framework code bases. And this one is quite hard because you can have any kind of logic inside of it. For example, imagine I have uh a serialized method field called has won any award and someone implemented that logic there uh checking if the person has any awards

Okay, so that's the logic there, but I need awards. I need that prefetch. And there's nothing here except for the code that tells me that I need uh that prefetch. So it's not that explicit Okay, someone has to read the code and to make sure everything that's uh this code is doing is also uh made on the uh prefetches of the query set. Okay, so that's hard. We know so what are the solutions? There are many incomplete solutions for that problem. One of them is maybe we shouldn't nest serializers at all. We should have a flat RESTful API, respect all the RESTful price principles and everything. Or even if we are if we need some nesting, maybe we should have a dedicated serializer for nesting.

Okay, we can try that. We've tried that in fact, but it's a fight against the framework. Eventually someone will do it and you won't be able to catch it on the code review. And it also solves the prefetch issue in most cases, but perhaps not the annotation or custom method issues. Another incomplete solution is let's test the number of queries in now views. Let's do that and ensure it's doing it's not it all views don't have any plus ones. That's fine, but that solves the past, not the future. Nothing is preventing people from adding new views. That will not have the right prefetches. Okay, so it solves the past, not the future.

Iliab has discussed a solution to that that can work but requires some automation, maybe You can add coverage logic that checks if your performance tests are covering all the views. Yeah, if you can build something as smart as that, that can work, but the naive solution won't work. And there are libraries for auto-prefetching as well and many people use that. That will solve most of the easy. uh cases for prefetching but will not solve the filtered prefetches issue and will not solve the annotations issue like the nomination count that we just saw you have to annotate that in order to have that value on the serializers. So it's not

also an incomplete solution. Maybe those incomplete solutions work for you, for your code base, for the moment you are on your code base, on your product. But as the code base grows, perhaps there won't be enough. So I I think that our complete solution needs uh Either to use non-DRF serializers that are very explicit about pre-fetching, or you can keep DRF, you can keep Django REST framework, but You must warn about missing prefetches for each serializer. You must run all the necessary prefetches automatically. You must prevent unnecessary ones, you must keep the serialized nesting support because people will want to do that, and you have to support the serialized method field.

So it it's hard. You have to do all of that in order to have a uh code base with Django REST framework that's really maintainable and fast, uh has performance. And we tried to build this with a tool called Django Virtual Models. It's something that we built at Finta for a specific customer and now we launched it open source. And I want to share with you today how do do we think that that solves this problem. And it's already available online, it's well tested, it's being used in production, but it's an like an early launch, so the API might change. Imagine you need to render all of this, okay? All this nesting. You have uh the direct the movie name, you have directors, you have the nomination count, and you have the awards.

This uh director one. Okay, so there is all this nesting here. That's not hard to implement with Django REST framework. All you need is something like this, three serializers nesting on them, uh movie serializer with person serializer inside of it, person serializer with nomination count. and awards an award serializer with some fields there. That's all you need. But how to make that fast, you need this I need all of the those prefetches to make that fast and to make that work with denomination count annotation. And how can we avoid that? Because that's crazy, that's hard to maintain. How can you completely avoid that And the idea is to have uh virtual models. Okay, that idea of uh

how can I define all this nesting, all the prefetches I'll need, all the annotations I'll need in a specific place of my code base. And we've built that with Django virtual models. So you have a virtual movie, which is has the declaration there that it is a movie on the class meta, and you have the directors inside of it. The directors are virtual persons, which are another virtual model that have nomination count inside of it. And the nomination count receives a lambda that will receive the query set and do something with it, which is the is this in this case is annotate, nomination count, and distinct. And then you have awards, which is

virtual award, another virtual model, but this time it's a nominations lookup. but a filtered nomination lookup. So you have Virtual Award, which is a nomination, but you have a specific filter there on the get prefetch query set where you filter for the winners. on of the nomination. So now it's not only a nomination, but it's a award, right? So it's the nominations that people have won. Okay, so if you have that, you can use those serializers without n plus ones or without missing annotations And we support all that I've told you before that we think we needed. If you declare a virtual model serializer, okay, and you pass the virtual model on meta

You can use it and it will automatically run all the necessary prefetches and annotations. Okay? Even nested ones. So you you to to use it you have to Use it like this, the virtual model serializer and the virtual model on class meta. And you have to go back to your base class and change it to be a virtual model list API view. If you cannot change your base classes, you can use mixings. We support that as well. Uh and if you like for example forget to declare directors inside your virtual movie We will show you an exception. Okay, so now you don't have to be careful because if your if your serializer is using a virtual model and you are trying to nest something, that something needs to be nested in the virtual uh model

as well. So if you forget, if you have a very simple virtual model that does anything, doesn't do anything, uh you see an exception during development telling you that you are missing a virtual model field. And that directors must be defined in virtual movie because it's a nested serializer inside movie serializer. So now you know the next step you have to make to reach performance with that serializer. So we are now warning uh during development time about missing prefetches and annotations for each serializer. Same thing for annotations. We'll show you errors telling you that you are missing something on your virtual model. We support serializer nesting, so uh

if you have an award serializer inside person You need to have that on the virtual person as well. So those things need to be aligned. Okay? And this works well because if it's if it's not aligned, you see errors during development. Okay, but uh maybe I'm just repeating stuff across uh my code base Actually not, because uh on the virtual model the idea is that you declare all the prefetches you may need. Okay, so if for example you have a person serializer that don't need the nomination count, it doesn't need that, you can have it on your virtual model

But we won't annotate nomination count on that serializer. Okay, so the idea is that you can use the same virtual model for different serializers. So you are localizing the you are putting out the pre-fetch logic and annotation logic on those virtual models like database views. Maybe I can say that, but it's not actually database views, just like query set operations. So you put all that logic there and different serializers can choose what logic they will use by just declaring fields. If you do that, it will work and you automatically prevent unnecessary prefetches and annotations. Okay, so what about the serialized method field? Because that's the hard thing. Can we support that? We can. It's a bit strange, but it works.

Okay. If you have this problem, as we saw before, has won an award, and I need to prefetch awards, how can I declare that I need awards? You can do something like this. Okay. Please don't get desperate. This will be common someday, I guess, in Python. Okay, uh you have this annotated thing. Uh this annotated thing is a built-in thing in Python where you can on the left side of it put the type. So the type uh The type checking will work just fine with that. We are not breaking type checking here. But on the on the right side, you can put any code you want. And the frameworks can use that code. Okay, we can introspect that uh

code and see what's there. So that's what Django Virtual Model does. If you we are telling that person needs a virtual field and that virtual field is called awards. So you can have any code you want on the right side and it we won't we are not breaking type checking here Okay, so if you do that, the framework will ensure awards will be prefetched for person. We can go beyond that. We can write like uh linters and everything that check for that. We didn't yet, but that's that that is possible as well Okay, so now we are keeping the serialized method field support and all works well for us. We solved many N plus

ones and missing annotation issues in our code base, our large Django codebases using Django virtual models. But it what if you like add the wrong hints? Because you're still human, you can error there, you can miss some annotation there. Let me try this other one. Oh, it's back? Okay. Uh if you add the wrong hints uh we will try to block all the queries inside serializers because the idea is that you have to handle it all on uh the Django virtual models the virtual models And if any query happens there, we'll show you an error during production. We won't break that in

oh sorry during development, we won't break that during production. Uh we'll show you an error telling that Some query is happening at a certain line and probably you are missing uh some uh hint there. Okay, there are other features for Django virtual models. We support mixings, as I told you, if you cannot use uh our base classes. We support joints, select relators. We support deferred fields for large fields that have like larger zones on it. We support annotations and prefetch relative to the current user. That can be done as well. You can pass down keyword arguments to prefetch and annotate calls if you need to add more custom logic there.

And there's more. Please check the documentation. If you find any problems or if you feel something is missing, please create an issue there and we'll see it. That's great, but maybe you didn't love that solution. Maybe you want something like maybe more organized than that, or maybe more opinionated than that. You don't want Django REST framework maybe anymore, you don't want serializers anymore. If you don't want Django REST framework serializes anymore, uh we wanted, that's why we had to build uh Django virtual models, because we had lots of logic in serializes and everything. But if you Want to drop your Django REST framework serializers, you can use other solutions. One of them is Django Readers. And the author of Django Readers uh basically he says that

How can I make n plus ones and missing annotations not a thing? How can I make developers solve that during the development time? And uh they built this uh function-oriented toolkit for efficient selection and projection of data in Django projects And selection preparation of this data is basically operating on the query set with annotations and prefetches. And projection is how how do you extract the field out of the query set items, the model instances uh into dictionaries. So it's like you those things need to be combined, right? Otherwise you may miss an annotation or you may miss a prefetch. So those things need to be combined and since they need to be combined, uh the key concept of Django readers

is something called reusable reader pairs. Okay, so it's a bit abstract, it's a bit confusing, but it works. Basically, if you need something like a nomination count, uh annotation that we saw before, you have to declare a pair. Okay, so the player has a preparation part, which is the first part of the tuple that annotates the query set with something. And the projection part uh tells uh Django reader how to extract uh that data uh uh from the model instances. So here you have a annotation and then you have an attribute gather uh type of thing with this produces a TTR So that's enough for nomination count. And you can grab that and you can put that in a spec.

Okay, so a spec kind of defines everything that will be done with the model instances. Returned by that that view. So you have, for example, here a name, which is the movie name, the directors, inside the directors, you have the name of the directors, and you have the nomination count. The nomination count needs to call the pair. And this will make uh Django readers to be able to navigate through that, do all the annotations that are needed, and then extract the data correctly. And then you can get that spec, okay, and you can combine with the list API view, and you just declare the query set there and you pass the spec. And now you have everything you need to extract that data efficiently

and without n plus ones, uh without missing annotations and everything. It's possible to combine that with uh Django REST framework plain serializers. You don't need uh uh model serializers anymore. Or you can combine that with something like Pydenic or other tools to serialize this data even more. But it it's supported out of the box like this. You can use just that. And the principles are that you should structure your code around plane functions. Uh and other principle is that complex business logic should be made by composing those functions. And by doing that, you can efficiently select and project a complex tree of related objects without necess unnecessary pre-fetches or n plus

ones. And that works. It works even for custom functions. So the the case that we saw before about the has won any award thing. You can declare the preparation part of that by using this code here that it's basically the filter of the winners for nominations, which are the awards. So you can use that custom logic here. And then you can have the produce part, the projection part to get that uh annotation that you just made and Do some logic with it. And now you have a pair of functions, a preparation and a projection, that is able to extract that efficiently out of your model instances from that query

set. So this works fine as well. Why didn't we use that? Why didn't we use uh Django readers instead of building uh Django virtual models It's very opinionated. It's not too familiar for Django Dev, so we didn't want to like add that to a huge code base and then eventually someone else would have to maintain that. uh and they wouldn't like it because it's not very uh familiar Django code. And we had way too much custom code in serialized methods and service functions And Django readers uh sorry Django virtual models allowed us to annotate all that code uh quite easily And conversely, Django Readers will require major rewrites. We would have to rewrite lots of our business logic because it's very opinionated about how you should organize your business logic.

Another solution that was discussed here actually in Django Con Bay is Django QSerializer. And they use that in production uh for uh on a startup in Brazil called Buzzer. And it works really well, I guess, because their site is super fast. So uh if you want you can try that as well. Uh you can you can check UDS talk uh Django form crew set to serialization later if you haven't already. Um And basically, you don't have uh Django REST frameworks anymore. Uh Django REST framework serializes anymore. You have your own uh serializers that you have to be very explicit about what you are prefetching.

and how you're going to serialize that and how you're going to nest stuff, how you're going to annotate stuff. You basically have to write more code, but there are some helpers there that can help you to streamline this. And it's actually better, more maintainable because it's very explicit. We didn't use that because uh that means no more DRF serializes, so it's the same issue. Major rewrites would be needed for us. Another solution, uh, some people won't like this one, is drop rest and go to GraphQL. Okay, I I don't suggest doing that but if you're like on the early stages of your API and you have lots of problem with nesting and everything maybe GraphQL is an option because GraphQL

frameworks usually have a dedicated layer where you pre-fetch and annotate those virtual fields. Fields that are not there stored in your database, but you have to do some logic to get them out. And there's Strawberry Django Plus. It's a still new project, but it's quite good because it supports all the things that I just discussed in Django virtual models, but focusing on GraphQL. It has a query optimizer to ensure prefetches when needed and preventing pre-fetches when not needed. It supports prefetch hints, it supports nesting, and supports custom code as well in methods. And how it looks like uh it's you declare like types.

Okay, so just as you need to declare the virtual models, here you have to declare types. It's not enough to simply use the The models you have to declare types. So now I have a movie type which is uh which has a name, a field from movie. You can just pass auto there. And it has directors, which is a list of person type. And person type has name, has nomination count, an integer, has awards This nomination count is available here on the get query set. I can annotate there, and now that will make a nomination count available for uh any of those persons returned by

GraphQL. And the person type supports custom functions as well. So for example, I can have person type with name, auto, and has won any award a boolean. And I can get that has one in award and implement that on the model, okay, as like a method there. It's actually a model property there, but you have to declare it like this. And I think this is great because the prefetch logic is very close to where you use the prefetch. So now you have a model property that tells uh you how you can get the awards think and right below it you use you're using the awards

We can go beyond serialization. Okay. Sometimes for doing uh permissions, pagination, filters, you can you may need virtual fields, right? And virtual models still need hooks to integrate with that. For example, we have uh in some of our code bases certain kinds of order. That needs some computation. For example, if I have a list of events, I want to order by the number of attendees of those events. Or going back to the example that we just saw, if I have a list of movies, I want to order by the nomination count or even the awards count of that movie, of those movies.

So those things are virtual, like uh you have to calculate that, you have to annotate that, right? And it would be great if virtual models supported that as well, for example, with Django filters. If you declare some of those things in Django filters, maybe virtual uh Django virtual models could read from that and check if you have declared all those annotations in your virtual model. And if you haven't, we'll show you an error. That's still not supported, okay, but it can be done, shouldn't be that hard to do. Permissions as well. Sometimes you need to compute something to check a permission. We don't support that as well. And you can go beyond serialization uh as well with Django virtual models if you pre-fetch fields from other services.

So this can be used for uh caching or API integration, stuff like that. There's a tool called Django Perfect UTUs. From the folks from Ruver. com. And it's great because you can hook custom code inside the prefetching logic. So you can do something like you can pre-fetch the Nomination count, but that nomination count is actually a call to an external API. And it won't be an N plus one call to the external API, right? You can do it in batch. Because it's doing you can do it at the prefetch level, for example. So this is very powerful for integrating uh stuff uh at the prefetch level. And it it works well for caching as well.

And my conclusion here is that maybe you need to use tools to be explicit about the data that you expect from the database. Otherwise you suffer for performance regressions all the time. We've been seeing that a lot while maintaining uh Django projects for the last nine years. Uh we solve something, we create a test to assert the number of queries, but then someone breaks that with another endpoint. Someone tried to reuse a serializer and now we are missing a prefetch. So I guess you we need to be very explicit about those sort of things. Otherwise you suffer from performance regressions. And even worse. Your logic can break. Okay, uh you may miss an annotation and that's now not available anymore for your front

end. or for your template and things will simply not work or will not appear for your end user and it'll be a hard thing to debug because there's a lot of change amplification involved in that. So you have to check multiple parts of your code base So I guess you need to be explicit and you need to localize that into a single part of your code base or a dedicated place where you can do the prefetches and annotations you need in order to solve that. So that's it. Thank you very much. Here's the link for Django Virtual Models GitHub Here's my uh Twitter handle, Vinta Software, in case you need any help with pre-fetches, please uh talk with us. And I'll be around tomorrow at the sprints if you want to try Django

virtual models in your code base or if you want to contribute with it. Okay, thank you very much