The (Python) Magic of Django: A Tour of the Codebase

So as it says on the slide, my name is Mark Smith, although I'm known as GD2K online generally for reasons that aren't as interesting as you might imagine. I'm a senior developer advocate at MongoDB, so it may be a bit of a surprise that I'm here given that Django and MongoDB aren't exactly designed to work together. But as a big fan of both MongoDB and Django , if anybody here is using the two things together and they have some suggestions of things we could do. do to help the two things come together or even if just generally you have some ideas how how we can bring those those two technologies together, I would love to have a chat with you. Around the conference. So that would be fantastic. So this talk is called The Python Magic of Django.

It's really an exploration of how Django uses some of the advanced dynamic programming and metaprogramming. features of Python uh to let Django do what it does, or let you do what you want to do with Django. So I've been kind of obsessed by all of these features like descriptors and metaclasses and things for for quite some time. It's possible that you may have seen a talk that I previously gave that has a very inaccurate title, which is Python types and metaclasses made simple, which as I say, it's inaccurate. A more accurately titled talk that I've given about these kinds of topics is uh another talk called Stupid Things I've Done with Python uh that I've given at PyCon Germany and PyCon Italia and I actually gave during lockdown at PyGoffin.

You may be looking at that picture and thinking that doesn't look like me. That lady actually translated my talk into American Sign Language, which I think is kind of cool. So the the agenda of this talk, we're going to have a little amousbouche looking at uh dynamically loading modules with import module. And then we're going to dive a little bit deeper. We're just going to kind of hunt through Django for various um Dunder methods and Dunder attributes. uh variables with a double underscore at the start and a double underscore at the end, that kind of indicates that Python is going to treat those in a special way. It's a sort of reserved namespace We're going to have a little bit of a talk about decorators. We are going to have a little bit of a talk about descriptors at more or less the same time. And then we are going to very

inadvisedly dive into meta classes um because I I I wanted to talk very fast for thirty minutes. So let's talk about dynamic imports. So when you set up a new Django project, uh one of the first things you're gonna have to do is after you've created your first app, you're gonna have to remember to add it to this list. I don't know about you, but I always forget to add my new apps to this list the first time and then it doesn't know what my models are and uh yeah it's it it usually takes me longer than it should do to remember that I have to do this. Um so obviously this is the default list of installed apps. Uh I've added my main app to the end of that. But bear in mind these are strings. They're not actually modules, they're not import statements, they're just strings. So I'm kind of although this is a Python file, when this is executed, something is taking those strings and using them to load Python code.

It's not just in settings PY. I mean there's a bunch of these modules that are defined as strings in settings PY. There's also this one that's created for you in your manage script. This is specific to my project. My project is called Tour and it's just given me a path to that settings file that we were just looking at under my under the beneath my tour namespace. So again, there's this string that that kind of tells Django that we want to load this module. Now, um if you're relatively inexperienced in Python, you might take that list of strings. This is not Django code, by the way, this won't even run. But uh you would loop through that l that list of strings and you would try and import them. But this won't work because that is a variable

And this is not a variable. I can't it that's a token of some kind and I don't know what it's called. It's maybe um a module literal or something like that, but it's it's the name of a module, not a variable. So this doesn't work with the import statement Fortunately, a few years ago, Brett Cannon uh extracted all of the import logic that was part of CPython into a pure Python module called Import Lib, which is distributed as part of Python. And that contains a function called import module. Import module is a function, it takes a string, and it basically does the same thing with the contents of that string as an import statement would. And then it just returns the module that it's loaded. So in this case I could create a dictionary and just kind of put the modules in the dictionary

keyed by the name of the module that was loaded. And this isn't super complex , but uh it is kind of fundamental to Django. So um when I was digging through the code base, I found a whole bunch of files that are all using import module. And notice that some of these these are have um occurrence counts at the end. So actually that config file is loaded, is is loading modules dynamically in six different places in the code. Uh and some are ones and twos. Uh and it this isn't all of them. This is just the first page. This is the second page of dynamic module loading in Django Uh looking at this one might say that actually Django's main purpose is to dynamically load code and then configure it and put it together. But this also Isn't all of it

because um import module isn't always used directly. There's a utility function called import string that just kind of it it messes with the module path a little bit and then passes it on to import module. And that is used about the same number of times or at least in the same number more or less the same number of files. So I just thought this was kind of interesting that Django is doing much more of this than I initially thought. I really thought it was just loading kind of um maybe models and apps and then everything else is being loaded normally. I would suggest from this that maybe that's not quite the case. In fact, overall 52 different modules in Django are dynamically importing code. So again, just a little interesting observation. So now I'm going to move on to what I've referred to so far so far as kind of

Dunder methods or dunder variables. I really wish there was a better term for that. They are often written like this. And some people might say that as double underscore init, double underscore Or uh sometimes in the past I've just not mentioned the underscores at all and I've just called this an init function. Um or I more often I tend to refer to it as a dunder in it, uh or if it wasn't in it, if it was something more like overriding a uh a an operator, I might call it a magic method of some kind. Uh in this case I might call it a constructor, but that's not true because it's an initializer. The instance has already been constructed by the time that init has been called. I'm gonna refer to these kinds of things a lot now and I'm gonna try and consistently refer to them as dunder. variables.

So I went hunting for some of them, and I have to warn you, there are regular expressions in the next part of this talk. So I used a tool called RipGrep, which is like grep. So it's a sort of it's a tool for searching through files. It's a lot faster than grep. It's got a slightly nicer, higher level interface. So in this case I wanted to search through a bunch of files. I only wanted to extract the variable names. I didn't want to print out the file names or the line numbers, and I only wanted to search Python files. It has a nice shorthand for that. Then there's a regular expression. It's horrible. It's also not entirely accurate in terms of looking up a variable name, but it was good enough for the kind of thing that I was doing. And then the long in there was uh that I wanted to look in the Django directory.

Um so when I ran that I got this list of a whole load of variable names, uh mostly dunder all and dunder new. It wasn't very useful. So I sorted it and then I extracted the unique values and counted the numbers of each instance, which gave me this list on the left-hand side. And again, like it was difficult to search through this long list and see what was interesting by frequency. So I wanted to sort by that first column, which is a number. Uh and orc is probably the right command line tool given the approach that I was using for this, but I can't remember how to use orc. So I wrote a Python file uh a Python script, which just kind of splits those values, converts the first value to a number, uh, and then prints them, sorts them and prints them out. So that gave me this

uh which actually also wasn't super interesting. I mean mostly that's constructors at the bottom. We've got uh references to class and things like that This also isn't accurate. So that's 1502 at the bottom. And then I realized that often these are referred to in the documentation and the comments. And I didn't really want to do that. It was sort of affecting the accounts. So I um actually not stack overflow and I found a script that removes all the doc strings and comments from the code. Could somebody just check that Daniela 's okay? Uh so I ran that as a preprocessor to rip grep. Um bet And that gives me actually a more accurate number at the bottom, 429. But it isn't this end of this list wasn't very useful. It was actually more interesting looking at the uh variables that aren't used so often

But it still became a bit like looking for a needle in a haystack. So I switched approach and I I I changed that this this sort of bunch of tools that I was using to count the number of magic variables in each file. And that was much more interesting. So it gave me the most magic file in uh Django is expressions. py Um and if I open that up, there's there's like there's some cool stuff in there. There's combinable. And this is this is really my wheelhouse. This is where we start to look at implementing methods that allow me to um negate this instances of this class or I can add them together using the plus operator or I can subtract them using subtraction subtraction operator. And build up these expressions that then you can traverse later to build say SQL or uh

may I say uh MongoDB query language perhaps. Um But there's a problem with this, and the problem is that Bass gave a talk about this yesterday. So I'm not going to talk about this file anymore because you've already heard enough about queries, or if you haven't, go and look up the video on YouTube So instead I picked functional, which is the second most magic file in Django, to go and have a look in there and see what was going on. So functional PY, there's a bunch of stuff in there, but I really like this because it fits on one slide and it demonstrates a couple of things that I think are interesting to talk about. That is decorators and descriptors. So class property is a decorator. I think I did remove the documentation from this. It does say it's a decorator. When you look at this thing.

It's really one of the internals of Django. It's not meant to be used by people using Django as far as I'm aware. But it it does a couple of interesting things. So let's first look at it as a decorator. The second thing is it's it's it's a descriptor as well But let's look at it as a decorator first. So looking at decorators more generally, when you apply a decorator to a function or a method. Um it's it's not entirely clear what this is doing. Like we all n well hopefully we know w the way a decorator That decorators do things to functions. The old way of doing this before this syntax was added to Python was that you would define your function And then you would overwrite that function by calling your decorator or wrapper. Wrapper's actually a better word in this context.

So you wrap the function um by by passing as an argument uh and then just replacing the variable name. So you're swapping it out with this new wrapped object Uh the truth is that decorators don't have to wrap anything. They usually do, but they you know they could return you something completely different. They could return a number if they wanted to, uh and then viewers async would just have a have a different value. Um so This is the the syntax we've just seen, like the sort of old syntax for assigning viewers async. And then when you look at the definition of class property. you can hopefully see it's it's a class, and that means when you call it, it creates a new instance of class property and calls that init method to initialize it. And that's just taking the function that's being wrapped, or the method that's being wrapped, and it just stores it in an attribute.

called fget. So we can come back to that later. Now let's look at this as a descriptor. So I know this is a descriptor because it implements Dunderget. Um Dundeget is just part of the descriptor protocol. If uh if you haven't come across descriptors before I really recommend you read this document. It's part of the core Python documentation. It's called the Python data model, and it describes how classes and instances and methods work. It's it it's actually fundamental to my understanding of of all the kind of weird Python stuff uh that I sometimes do. Um And lots of the stupid tricks I do kind of rely on techniques that are in this file. Plus it's official documentation, so I'm obviously allowed to do all this stuff Uh

so the descriptor protocol in summary, there's there's more to it than this, but it one of the things that it says is that when you access an attribute on a class or an instance. It will first have a look at the the value that you're looking at. So in this case whatever view is async is. And if it implements this dunderget method, then it will call it and return the result. So this this that that line at the top is effectively that line at the bottom. In this case, because our wrapper implements dunderget. So that's this method here. Now dunderget has, in this case, has been implemented so that it calls the wrapped method and returns the result. And so now if we access the attribute on the top line, uh what it effectively is doing behind the scenes, it is calling the wrap method

and passing it the class. Now this is exactly the same way that property works, except that property passes in the instance. So in this case, this is equivalent to a property But it's or it's more like a class method implement of uh instance uh of a property. So you access a class attribute and it calls a class method underneath the hood. Um Again, if you'd like to look at descriptors, I found out recently that Raymond Hettinger has written this long, kind of very practical, applied guide to descriptors. And I wish I found it before because it's really super useful. So I recommend checking that out. There are also a handful of other things in functional that are around laziness in Django.

So there is lazy, which is a wrapper for a method that will only call it if it's accessed if I remember I can't remember the exact details of that one. Lazy object and simple lazy object will wrap an object constructor and they will only construct that object if it's accessed if you try to use that object And so it means in many cases you've got an optimization for both processing time and memory if you've got lots of objects around that you're not actually accessing. Those are the main reason that there's lots of dunder stuff going on in this file is because uh lazy object implements more or less every operator that you can. So anytime you take something that's wrapped with lazy objects and you try and add it to something else or to subtract it to something else, then it will kind of inflate the object that it's

that's it's wrapping that hasn't been constructed yet. So those are kind of interesting. It's lots and lots of code. It's easier to wave my hands and kind of describe what they do than actually show you the implementation. So to recap where we are right now, uh at the 15-minute mark, which is actually better than I thought I was going to be doing, we have dynamic loading for loading um modules based on data, so based on strings that come from somewhere. We have uh decorators that wrap functions or methods and change their behavior. We have descriptors uh which change the way that um attribute access works um so that you can essentially execute code underneath the hood. um while it looks like you're just accessing some data. So now let's look at metaclasses. Oh it's a bit off.

So we all know what meta classes are. Meta classes are those inner classes that you define on models where you specify things like the name of the table that you want to store the data in. So you know it's only when I was writing this talk that I realized that meta in this context may not actually be short for meta class, it may be short for metadata, which I think would be a better name for this thing. Either way, this is not a Python meta-class. Perhaps I'm going to make the argument that it could be a Django meta class, but we're going to we'll talk about that a little bit later on. It is that that meta, that inner class is used by a meta-class. And again, come back to that in a moment.

So about 12 to 13 years ago when I was first using Django, I was just talking to somebody who knew more about Django than I did And at the time there were three metaclasses in Django, in the Django code base. And we were talking about the fact that meta classes are one of those things you should kind of minimize as much as possible. You should only really in in practical production code you should only use magic when it's giving you a benefit. So if possible you can use simpler techniques that that don't have the complexity of understanding and implementation that comes along with MetaClass. And the person I was talking to kind of theorized that ultimately it might be possible to reduce the number of metaclasses in Django to just one meta class, which I think we were talking about at the time was the metaclass behind models that would just register the model when that class was created.

So it was quite a surprise when I searched for this expression in Django and found out there's actually eight metaclasses now in Django. It was a fun surprise. And there's a bunch there. I don't know that Uh any of those are really used directly. They are they sit behind the scenes and implement functionality that's written about in the documentation and you're not meant to worry. about what it's really doing under the hood. You just implement things in certain ways, but usually by extending a class and you get extra functionality. Um there are another six meta classes in the test uh test code. I think some of these are testing the meta classes in the main code base, and some of them perhaps are just um kind of optimizing uh the the test code themselves reducing duplication. But before I'm not going to cover the any of the test meta

classes, but before I move on, I did just want to highlight PaulSpace. I didn't go as far as checking what that did, and I I'm kind of regretting that now I'm standing here. So we're not going to talk about horse base, we are going to talk about model base, which is probably not a huge surprise if you've come across model base before. So before I talk about the meta class itself, let's talk about models. So this is an implementation of a model, it's a subclass of the model um class, base class, and I've added a bunch of attributes here, a bunch of fields. It's a relatively standard kind of inventory style class that I can now save in a database. But first let's go and have a look at model and see the implementation of model. And in fact, just the first line of the model implementation says that model

has a meta class of model base. So we can look at the definition of model base as well. Model base extends type, which is not a surprise actually, because MetaClass is the default metaclass. uh is type, like just like the default class is object, essentially. It's the the top of the the hierarchy. And then uh just that that first method in there, the dunder new method um takes all the data that you need to construct a class. So it takes the um an initial kind of class implementation It takes the name of the class being produced, the superclasses of the class that you want to create, and a dictionary of all the attributes that you've defined on that class definition when you're defining that class.

So I'm just going to take a step back from this for a moment because I haven't explained what a metaclass is. So first let's talk about a class. One definition of a class in Python is that a class creates instances. That's what it's for, right? It there is some other behavior there. It's part of the inheritance hierarchy. So if you Try and access a field on an instance, it will fall back to the class and then go up the superclasses. But fundamentally the thing we're talking about right now is that instances are created by classes. To take that to meta-classes, metaclasses don't create instances, metaclasses create classes. And models are classes, right? You you get a class and it it comes with behavior.

And the meta class is the thing that provides that behavior. So we can kind of We can see what the model-based class is doing by removing it from the code for a moment. So I take an item and I've removed the inheritance from model, which means that now the meta class for this is just type. It's it's that initial type um meta class. I I should point out, like type is a terrible thing. It's one of the worst designs b design bits of Python. It does two things, and you've probably used it before to check the type of something. You've used it as a function, right? You pass something in and it returns the type that that thing is. It has a second use, which is its main use, is that it is also a metaclass. It's also the thing that makes classes by default.

So just that that overloading of this thing to do two completely different things really bugs me and it makes it really hard to understand what metaclasses are when fundamentally Actually, when you look at an implementation, they're not that hard. You just have to look at them as an as an analogy of classes. So now that I've got that little rant out of the way. Um we've got a normal class here with some fields attached to it, and if I load up the Django shell and look at what the value of ID is on that class, so I'm not instantiating anything. I get an auto field, which is exactly what you expect, right? That that is what I what I told it a ID should be However, if I then go back to the previous implementation where it does extend model and it has a meta class ,

now if I load it up in in the Django shell, I get this deferred attribute thing. The the evidence shows us that something is taking that class definition and changing it so the actual class that's created is something else. Which is not necessarily what you expect, right? You also get some extra stuff with item now. So it now has this underscore meta. field. That has a relatively small public interface. It has some private stuff in there that you shouldn't use. But it does have this public documented get fields method that just gives you it's not a list, it's actually a tuple, it's a a read-only list of all the fields in there. And while we're just looking at that, I want to highlight that each of those fields now knows what it's called.

Normally, when you instantiate an auto-field, it doesn't know it's being assigned to ID. But the this one does. And um the char field knows that it's called product. code, which again is something that the meta class is doing for us. It's kind of initializing each of these fields with the information each one of them needs to behave in the way that you need your Django fields to behave. And then finally, the thing that you probably always use on models is objects, which is a manager type, and is the way that you will interact with the database to get instances of your class. So the actual implementation of how this meta class works is really long. It's one big long method that checks the hierarchy of the model classes, it checks your if you're looking at an abstract model.

Yeah, there's a whole bunch of stuff in here, but fundamentally what it's doing is it's looping through the fields and configuring each of them and sticking them in the underscore meta. meta class. And then at the end it just registers this class with the app registry so that when you run commands like you know make migration it will it knows that this model exists um which is the like the the very basic thing that you need when you create a model. You need Django to know that it exists. So if you really want to know how Django creates and initializes models, there's this really good uh wiki post on the the Django project wiki. It was originally written in 2006, but it has been updated since.

It's still a few years old, so I don't know if it's entirely up to date, but it was really helpful when I was trying to understand this stuff. But I just wanted to kind of go back to what I was saying before about that underscore meta is created from the configuration data you stick in your meta inner class if you provide one. Otherwise it's just a set of defaults So effectively by the configuring the things you put in that meta in a class, you are metaprogramming, right? You are changing the way that your class is instantiated So it's still a meta class, even though it's a Django meta-class and not a Python meta-class, which is not necessarily like an easy thing to think about. But I think it's worth mentioning that you're still meta programming, you're still programming about your pro you're sort of manipulating the code of your program

when you configure one of these meta classes. And just a final point about metaclasses. There's a certain amount of subtlety to them. So I really simplified the definition of a class. by saying it's just there for creating instances. And then I kind of added some complexity to it by saying it's part of your inheritance hierarchy. One of the things that took me ages to realize was actually your meta classes are also part of your inheritance hierarchy. So if I was to implement a method on model base and then call that method on an instance of my model. It would first check the instance and see that there is nothing on there that has that name. It would then look at the class and the superclass until it got up to object.

And then it would say, no, I don't have it. And then it starts checking the meta classes. So it would actually end up calling a method that I defined on the meta class. So as well as them creating classes, they were also part of your kind of instance class meta class thing. And yet as far as I'm aware, yes you can have a meta meta class if you really want to, but at that point my brain kind of explodes and it's not use it's certainly not useful. So, yes, Metacloss is very powerful, also kind of subtle. Maybe overcomplex for what model is really doing uh with its meta class. So um there was a feature added in Python 3. 6 called init subclass. Um this is to stop people from writing meta classes essentially. The use case of meta class is nine times out of ten

is to take a class definition and just kind of register each of the fields and maybe register that whole class with some kind of registry. which is exactly what we're doing with model. And so potentially you could take all of that new method in the meta class and copy it into the model class under init subclass. Rename it to init subclass. And that might just work. There are a couple of methods in model base that are potentially there because you might want to call them via the instance class meta -class hierarchy. But I don't know whether they are or if they're used very often. And I'm not sure it's worth doing this, by the way. I'm just suggesting that actually you can get away without using meta classes in some of these cases. And the Django developers are very aware that init subclass exists because there are already two uses of init subclass in other places in Django.

So I just thought it was an interesting observation. Uh so I spent a lot of time playing with Python's More Magic features, but it was actually Django in the beginning that gave me permission to do that. It was very interesting using Django and then finding out how it did some of this stuff under the hood. Metaclasses, the theory of metaclasses made no sense to me But as soon as I started seeing what Django did with metaclasses, it was like, oh, that's what they're for. Um so I've learned a few things from Django. And one, which may not be completely obvious given the statistics that I showed earlier is you should keep the magic as localized as possible. Actually when you start to look at where magic is used in Django, you find it's really isolated to a couple of files. They're kind of fundamental modules like functional and expression.

And those are used elsewhere, but the actual magic is wrapped up in one or two modules. You should, if possible hide the magic from users. So don't make them implement meta classes or prov make a reference to a meta class directly because those are slightly scary concepts. Instead, you attach the meta class to a parent class and then people can extend that. Plus it gives you the ability to add functionality to the parent class instead of actually doing the meta programming in a meta class. Provide escape hatches. So there are various places I didn't show you this, but there are various places in Django where it has some default behavior, but sometimes it will check whether a method has been implemented on something it's manipulating, and it will call that instead Or there's a default implementation that it calls.

So you can change the way that Django actually manipulates your code if you need to in various different places. You can if you need to, if you if you're building a very dynamic framework or something, you can build your own meta programming framework. So like the meta in a class, you can provide a higher level implementation of kind of how you want things to be configured and manipulated. And I thought that was quite an interesting thing. That was a new realization while I was looking through Django to write this talk. And finally, um if your structure is complex and when people are writing new code to extend your library or use your framework um i and they need, let's say, an app file and a model file that have some default behavior, write a template for it.

Use Cookie Cutter, write a create a managed PY file that will automatically bootstrap people's own code bases. with no particular implementation and then people can extend that going forwards without having the initial burden of working out how to how to kind of bootstrap the project And so with that, 31 minutes, I do apologize. That is the end of my talk. Thank you very much.