Custom Model Managers and QuerySets: Graduating from Django Beginner to ORM Master

Hi, my name is Josh Thomas and I will be giving a talk titled Custom Model Managers and Query Sets, graduating from Django Beginner to ORMester. Before I get to the talk, a little bit about myself and where I work. As I said, my name is Josh and I'm a web developer at the Westervault Company. I've been using Django professionally for around three and a half years, basically as long as I've been in my current position. I'm a member of the DSF and a member of Jazz Band. sometimes contributor to the various packages underneath that organization. Also maintain a handful of small, small, Django and Wagtail CMS packages. I graduated from the University of Alabama, Roll Tide , with a degree in civil engineering.

And I'm married to a wonderful wife who is currently at home taking care of our two kiddos right there. there and if you got babies that cute you gotta show them off. You can find my personal website at joshhomas. dev. You can also find me on Mastodon and that is my GitHub handle As I said, I'm the web developer at the Westervelt Company. We are headquartered in Tuscaloose, Alabama. We've been around a long time, originally founded in 1884 as a paper company. Since then, we've expanded and diversified our business all around this idea of environmental stewardship and resource conservation. Our big slogan is we are stewards of the land. Side note, we do actually have operations in New Zealand, and I keep on suggesting

they need a programmer on site dedicated to their part of business, but no one's taken me up on that offer yet. Um and Westerveld loves Django and Python and open source. We're financial sponsors of both Django and Python through GitHub sponsors, as well as a smattering of developers who maintain some of the many open source packages we rely on. Alright, so while we're here today, what exactly are managers? Well, as defined in the Django documentation, a manager is the interface through which database query operations are provided to Django models. Managers can have methods much in the same way models can have methods. The difference being model methods operate at a row level or an instance level.

Whereas manager methods operate across a table or across multiple instances of a model. If you're a beginner of Django, though, you may be asking yourself, wait, what exactly does that mean? Basically, if you've used the ORM in any capacity, you've used a manager, whether you've realized it or not. Here's the question model from the polls tutorial on Django Project. com. And we're going to query for any questions asking, is this talk going great? Yeah, it's it's right there. It's right there containing that object's attribute. That is your model manager. Out of the box, Django defines a default manager for any model you define. However, you can be a bit more explicit and define it yourself.

This this accomplishes the same thing that you get out of the box with Django. Okay, with custom managers, there are a couple different ways to define custom managers. You can have a custom manager, you can have a custom query set acting as a manager, and you can have both a custom manager and a query set and that's kind of how you would define it in code which I'm going to go into. Okay, first up the basic case where you inherit from models. manager This works, but is a bit limiting as we'll see later on. Next up, what I consider the bread and butter way of defining a custom manager. Instead of creating a manager, we're going to create a query set that inherits from models. query set. And then using the asManager

method provided by the query set, we're going to set the object's attribute. Last, the most advanced case, when you want to have both custom manager and query set methods. There's another way to accomplish the same very thing, but I left it out because you end up needing to duplicate. A lot of logic between the manager, the query set, and I never have actually used it that way in my work, so I'd skip straight to the from query set. Okay, so now you know how to define them. Now when should you use a manager versus a query set? At the most basic level, how I like to think of it is whether you want the methods to be chainable or not. So we've got the same question model, and we're defining a question manager

that has uh two methods, a published in future method and a questions about me method, because I'm feeling a bit vain today. When querying them on their own, they work fine. However, if you try to chain them together, you're gonna run into some issues What ends up happening is it actually raises an attribute error, and why that is is a little bit beyond the scope of this talk, so I'll just leave that to the reader. So how do we fix it? How do we fix these methods so they can be chained together? Well, I mentioned the query sets. Instead of using a manager, We can swap it for a query set inheriting from models. query set and change that objects, objects attribute, and it works like we would expect. Okay, so that really

doesn't answer the question of when to use one over the other. Basically, how I like to think of it is you have your typical CRUD actions, your create, read, update, and delete. And you would use query sets for all your reading and managers for everything else. You're creating, updating, and deleting. Now this is just my opinion and how I've kind of Settled on on a few after a few years. Um you may find a use for chaining creation or deletion methods, but I haven't seen it Okay, so that's the what and the how. Why? Why should you care about using custom model managers? Before I get to that, I do want to highlight this quote. Sean Enman gave a talk titled Mighty Model Managers, where he says they are a part of the core framework.

The documentation is spot on. They are really useful. And I feel like they are overlooked. He gave that talk at DjangoCon US 2016, and I think that quote is still largely true. For tutorials aimed at beginners, there is a lot of attention paid to the basics. For example, models, basic querying, views, templates, URLs, forms, the admin, etc. And rightly so. You need that foundation in order to be successful building good Django applications. There are also an incredible amount of intermediate and advanced tutorials out there. But as a beginner, it can be a little confusing. Which direction do you choose? Which direction do you go? It should hopefully

be obvious by the title and the content that I think model managers are a good um next step after the basics. Okay, so why use custom managers? Well they're awesome. They can greatly increase the readability of your code base. For instance, instead of on a views query set, writing the filter out directly on that query set , You can throw it behind a well-named manager method, and when you come back to it in the future, it kind of decreases the mental load and time to understand what's going on Encapsulation and testing go right with that. By creating small encapsulated units of logic, you can more easily test the various parts of your business logic. And without diving too far into the service layer debate, I find that between model methods and manager methods, I rarely find

the desire to define any custom service layer. And finally, the most important part, they're a great gateway into learning the ORM. The ORM is by far the biggest component of Django itself. And any amount of time you can invest into learning it will pay dividends later on. Because of the ability to encapsulate the more advanced parts of the ORM in small testable methods, It makes it so much easier to tackle what once seemed insurmountable. How do you eat an elephant one bite at a time? Side note, um, Jeff Triplett posted this on Mastodon, and Tim Schilling had a great follow-up post that kind of captures that sentiment of investing time into

learning the RM for those who can't read it. Django is clearly a database migration and data modeling library with a flexible web app on top of it. Change my mind. I agree. Credit to both of them. Alright, so before moving on, I want to do a little context setting. Why did I decide to give a talk about model managers at this year's DjangoCon? When I was first hired on as the web developer at Westerveld, one of the tasks I had was to modernize our lease management application. It is an old application that has been in production since around 2006. It manages all the tracks of land that we lease to hunting clubs. Around 800 clubs leasing roughly 1200 tracks.

And it was written in Cold Fusion. And that's going to be important in a second. The consulting company we were working with at the time wanted us to go with a stack that was. NET on the back end and React on the front end. I had just spent a few months building an internal application using Django, and I knew I wanted to work with it again. Somehow I managed to convince my boss to let me build it. Build the back end with Django using Django and Django Rust Framework. Now, if you've never worked with cold fusion, um the querying is some mix of raw SQL with some random cold fusion, variables logic mixed in. I'm sure if you use this every day, you get used to it and you become really good at it and efficient. But I found it difficult and pretty inscrutable to wrap my head around all the business logic scattered throughout the code base.

I do want to say I'm not trying to bag on my co-workers who wrote this code, mainly because they're going to watch this later, probably I think it's impressive enough to get any application running, much less one that lasts as long as this one has and is as important to our business as this one is. So here you can see just a few examples of what I was working with. They start off pretty simple, but they ramp up in complexity. rather quickly. Um until I saw this one and this one this one well this one almost broke my brain. Alright, so what did I do? Well, title of the talk, I relied on custom model managers. I went through and translated all the original tables to Django

models for the most part. And then for each SQL and Cold Fusion query, I created a descriptively named mount manager method with the original SQL as a comment to refer back to. I then just put my head down and slowly translated them one by one into the Django RM. I knew that even for the complicated queries, I could always fall back to the ORM's ability to just use Raw SQL. However, I ended up not needing to, thankfully. Alright. So with my remaining time. I'd like to try and show you a handful of the most common patterns I've used using real-world code examples I'm going to throw a lot at you and I don't expect you to understand or grok all of it, but I hope you'll be able to get at least get a general idea of what these methods do

and hopefully get inspired to use them in your code. Hopefully I'll be able to get through all these. So let's get started. First up for context. You would use this when you want to filter a query set on based on a model instance or some other context. Common pattern you often see out in the wild is the for-user method, where you only want to show the model instances related to that user. A slightly more embarrassing real-world example. My first professional Django application was an onboarding application to ensure new hires to the company had all the hardware and software they needed on their first day. Here is a method to get all the new hires that a hiring manager has submitted. Initially, the hiring manager responsible for the new hire

was a separate model before I really learned about the user model built into Django. I eventually started the migration to the user model, but um to this day I still haven't found the time to devote to finishing the migration. But since it's all wrapped up in that model uh manager method, that messiness is mostly self-contained. So when I actually do go find the time to clean it up, it's all right there. Next is the most common method I find myself writing: filtering based on a condition. So we have a tract. Tract has acres, it has a status, has this deer tract ID. And we define a tract query set with a few methods. One's responsible for telling whether a tract is leasable, that's whether it's active and

huntable. Also it one that excludes all the tracks that we own, but we do not have access to hunt on using this deer track ID, which I don't actually know what the numbers are, but they are saved somewhere. And also whether a tract is leased for the year. And you can see you can mix and match to your heart's desire. No, you can you can drop that is from the first method if you like. I just tend to prefer it with the is. All right. Create model and create model for context context are the only ones I'm mentioning that are only manager methods, meaning they're not really designed to be chainable. I like to think of these methods as creation with side effects when you need to create a model a certain way or with certain things always attached.

Here I've got a create user for club method to standardize the way a club user's a club's user is created, as well as adding it to the correct permission group. Django's user model is a good example of this as well. It provides a create user and a create super user methods that follow this pattern. Alright, sometimes you're going to need to filter models based on a date, a date time, a time delta, just basically a range of time. The within range method and the next two methods I'm about to talk about are great at that In our onboarding application, we send out surveys to new hires to let them know, to let us know how we are doing in the delivery of the hardware and software. that we give them. However, we don't want to send it right away when they start.

We want to give them a few days to sell in. This within start date method gets all the new hires that have started within a certain number of days. So we know that we need to send the notification or not. In that same vein, there is the greater than and less than value. And continuing on that same example with new higher survey. We don't want to send too many messages at once. So when sending our notifications, we check that they haven't been notified within a certain number of days. And if they have, we defer the notification till later. And here you can really start to see what I was talking about with encapsulating some complex ORM logic, you know, with this one method 's got some annotation and it's got case and when conditionals.

All right, this one's pretty straightforward. Set field and toggle field. You would just use this to efficiently change a single field across an entire query set. Let's say you've got a model with a status field that can either be active or inactive and set active and toggle status to what you expect. These are great, great to use in the admin. Okay, last one. I know this one looks like a lot, and it is, but this kind of method is the one I've overused when migrating our lease application. On a lot of models in that application, the relationships were not set up exactly how I would if I were writing it today as a Greenfield Django application. I found Annotation is a great way to build a bridge between these relationships where they were not defined.

In the application, we have a club track model that defines a club's lease on a tract for a year. And throughout the year, the tracks acres can be adjusted either through a sale or an acquisition of land. And when doing the end-of-the-year reports, the lease managers they only care about the adjusted acres for that year. And so that is what this uh mess is doing. It's grabbing all the adjustments, uh using a subquery, and once again, a lot of complex ORM code. But it's all right there. Subquery, outer refs referencing that subquery, annotations, case and win again, and then F expressions adding the adjustments to the acres if they're found Hopefully you can sort of get what's going on there.

Or at least. Anyway. I remember that Monster SQL query that almost broke my brain. Here it is. Here just translated to ORM. You can see it's actually calling another custom method on the invoice model within this method. And now instead of that multi-line brain-breaking query, You have this nice semantic readable testable method that you can understand basically at a glance, especially if you're working with this every day and you're more familiar with the application and models. Okay, so my hope in giving this talk was to give you a taste of real-world usage of custom model managers That said, I did have to leave out quite a bit. I actually had slides for all these, but they got cut for time.

I didn't mention modifying the default query set, which is one of the main reasons given by the documentation as to why you would want a custom manager in the first place. Or renaming the default manager. I just used objects every time in all the examples. Or any use cases with multiple managers attached to a model. You can also define managers for abstract models. Of course, it wouldn't be an ORM talk without a reference to SLECT-related and prefresh related to avoid those pesky and plus one queries. Basically, go read the docs. They're great, they're well written, they're understandable, and they'll go through all of this. Okay, just a few more things. Here are the various references that I've talked about. That first link is to the documentation for model

managers And the second one is the raw sequel. And then Mighty Model Managers, that's that talk given by Sean Inman back in DjangoCon US 2016. It's a great talk. Goes over a lot of the same things I've just gone over. He just approaches it in a slightly different way. And then the two Macedon posts. My slides will be up here at some point. Hopefully when you're watching this later, as of this moment, this is an empty repo. Okay, while I have the opportunity, I just want to say a few thanks. Firstly, anyone involved in creating, maintaining, writing about Django, thank you. My career wouldn't look a lot different had Django not existed. Probably not as rewarding or fun. Thank you, all the organizers and volunteers of DjangoCon this year. I'm grateful for the work each and every one of you do.

Thank you to the broader Django community for being so kind and welcoming to people of all walks of life. Come for the framework, stay for the community is real. Thank you to my family for everything, and thank you to listen for listening to my talk.