Hunting for Treasure in Django by Sebastian Vetter

Speaker 1: Hello, um thanks for the introduction. Um yeah as Adam just said I'm uh Sebastian. I'm currently my main profession is a Django web developer. Um I've done some Python uh academic work I work for a company called Mobify in Vancouver in Canada, which I just uh moved to about six, seven months ago. And if you want to find me online, I'm usually found as Al Bashid. And the slides for this talk are going to be online on speaker deck. This is the bit. ly link. I'll have that later on the slides as well. Um but let's find some treasures. Um what I mean by that is we're all

Speaker 1: or most of us are familiar with forms and views and all the sort of good things that Django has to offer and the things that we use Django for But they are an old hat for those of us that have worked with it for a long time as well. So I'm trying to talk about a little bit more about um what the hidden treasures are And what I mean by that is basically when you look through the code in Django, you come across like utility functions and things like that. That are implemented in Django to facilitate all those other features, but they might actually be useful in your project or for your use case, and you don't have to write and maintain those little pieces of code yourself.

Speaker 1: Um the first question around that is where do you find that? And the way I found those was mostly through either looking at the Django source code itself because I wanted to try and understand this or that piece of code or something went wrong and I didn't quite understand what I did wrong so I had to dig into the code to understand it properly. Class-based view and their nested nature are a very good example for that. And obviously working with smart people that or talking to smart people that uh told me, oh, basically this thing that you were doing over here There is something in Django that already does that and you can just use it. And then there is uh

Speaker 1: my personal uh favorite source of information is uh funky bob that I had the pleasure to meet over and over again uh during my time in Melbourne. Um but he's like the the person to uh to ask questions on IRC and he's a great resource. Um but generally IRC is a great resource to ask questions And then there is sprinting at conferences and that goes back to like meeting smart people and talking to those people, learning what other people do and picking their brains So what I'll be doing is I'll I'll just pick a few things out that I've been uh uh that I found really useful. and point

Speaker 1: out like how I've used them and what might be a good way of uh for for you to use ZOLS. One of my personal favorites probably is um the cache property decorator So it is basically just a decorator. What it does, it's all in the name. It's uh a property on a class. The return value gets cached so you don't have to re uh compute whatever it is inside that function. So this is basically what it looks like. You have your class and instead of using the add property decorator Um there is a cache property decorator provided by Django that caches the result of the function And this can be useful for things like very compute heavy properties

Speaker 1: or a property that you expose that relates to multiple different tables in a database and you have to aggregate data of some sort or you're accessing uh another web API and retrieve data that way. So sorry, um as an example, let's take this uh class of a color object and we have um that object has a hex value And we have this amazing remote API that for any given hex value of a color gives you the name. And you expose that as a property, which means every time you have that uh you access that property You'll make a remote call to the API

Speaker 1: and this is what it looks like. You create your color, you access the name attribute, in this case the property, and it makes a request. And the next time you request that property It's doing a remote call to the API as well. Which is obviously uh not the most efficient way of doing it And then you got get to the point where you think like okay I can just cache that somewhere on the object. And one of the ways I've done that before is you have an under name on the same object and you check whether that is none or not and then you store it there and only return the cached value after that. But it can be so much easier It basically comes down to importing the cache property from

Speaker 1: uh Django utils functional and then just wrapping that property in cache property. And it ends up being exactly that. You're basically just doing that remote call once. And after that you have it cached on the property. And internally it's basically just cached in the uh double under dict. The only thing in this specific case to be aware of is query sets. Which is a thing that uh has bitten me once or twice. Uh you create a query set, you return that, and you think, oh yeah, that's cached, so I don't hit the database. But that's actually not what happens because the query sets are lazily evaluated, which means you're just

Speaker 1: uh caching the wrapper query the query set as a wrapper and not the actual database call. So if you're like in this case retrieve a list of objects You have to turn that into an actual list or into a set or something like that for it to be cached as an actual list of objects in in the cache property And that's it. Um and basically all you need to know is Django utils functional, import cache property. These links are basically on the slides for uh if you want to look into the slides and find some references. This uh references the docs and the sources.

Speaker 1: So this is our first treasure. A next favorite of mine, something that I've used um a little bit and found to be used a little bit, is uh The import string in third-party projects. What it does is basically you pass it a string that is a dotted path and that points to a class or a function and then it validates that that's a valid path and it returns the imported actual function or class that you point to. And in the case of the request library, for example, you could just pass in requests. get as a string. And what you Get back from the import

Speaker 1: string function is the function object and you can use that exactly the same way you would if you'd import requests and use requests. get So where is that going to be useful? Um well as I said in third-party applications where you're providing functionality to another developer and that developer doesn't have access to the internals of your library. Like there is only monkey patching to extend some very custom functionality or things like that that you could do. So in this case, it comes in very handy to say in the settings for Django, I can basically specify a function or a class that has a That

Speaker 1: complies with a specific API, and then I can use that instead of the default. So as an example Um in a web store a product is uh uniquely identified by a slug in that's used in the URL. And then you also want to provide your users that you have this web store framework written for to customize that slug. And because I'm a sucker for emojis, I want all of my slugs to be emojis. So I'm using this custom uh emoji slugifier And then internally uh as the person writing that uh piece of web store framework code, I can basically just take that string and say

Speaker 1: import string. And get back whatever that user has or that developer has written as custom code, past the values as defined by my API contract, and then return that value. There is something too um that I'm not taking into account here. Import string still raises an import error if there's something wrong, so you have to catch that yourself. But it's something that's really useful because you don't have to provide all those little um checks and things like that that you have to go through when you do this manually. That's all taken care of by Django So Django utils, module loading, import, import string is all that you need to remember.

Speaker 1: Um it's been called import by path previously. I think it was deprecated in one point six. And it will be removed in 1. 9. So there might be some differences in naming there And this is the second treasure that we found, that I found. Hope you. Um so The next one I want to look at is the lazy function and the um lazy object. And we've definitely all used it. Um and that's because Django settings are basically based on that sort of concept.

Speaker 1: I'm not quite sure whether that's the reason why the lazy object thing exists in the first place. But it's one of the most prominent places where it's used. And what it does is basically it provides a way of delaying the execution of an object. By wrapping that callable in another object. So as I said, it's used in the Django functions. And one of the reasons it exists is because the way Django is executed when you load it, some of the pieces that you might rely on in other parts of your code might not be available at that point. uh that you're executing that function.

Speaker 1: And one of the um best examples is the Having a reverse lookup for a URL in your on the either the class attribute level or on the module level of your code. Because that code is executed as at parse time and you can't really be sure that your URL config is fully loaded at that point, this might actually just return an empty string, and you don't really know why. So for this is a slightly bad example because there is already a reverse under lazy as well as some some other underlazy function in in Django. But it's a very easy way to illustrate the sort of problem.

Speaker 1: And one one of the uh problems with that Uh where I actually had to make or could make good use of that, uh I just came across recently, which is per-user um storage. I had a model and uh a file field on that But I wanted to store those files in a user-specific bucket. So I had S3 and because we don't have per-user storage in Django , Um it was basically a way of okay, I've gotta write a little wrapper and provide that. But S3, especially the um Django storage implementation. fell over at that point when I was starting to load my application. And

Speaker 1: it was like why? Well it turned out that um as the S3 storage would basically uh try to connect to S3 at the point in time when the file was parsed and then because I didn't have my credentials from the settings file at that point, it would just say like I can't really connect without credentials. So because I couldn't put it in any other place, the the easiest way to get around that was basically having those three lines of code in there. You write a function that returns the object that you actually want to have, and then you create a simple lazy object And a simple lazy object will only return and instantiate

Speaker 1: that object that you return in your function At the point when it's accessed for the first time. And then you can just stick your um private storage um object the lazy object into your um file field definition and it just works. So that saved me a lot of time. And once again you'll find those two in the utils functional, excuse me, uh the the Django utils functional module. And uh it's actually fairly easy to use them. And that is our third treasure.

Speaker 1: And then the last one that I'd like to look at is uh the request factory. That's actually one of the things that um Has saved me quite a bit of time in in testing. So it's not something that you would actually use in your day-to-day code that you're running in your application, but it's something that will actually help you During testing. So it's also fairly simple. It uh takes a URL and um The HTT sorry, the HTTP method that you're trying to use, and then from that it creates the Django

Speaker 1: request object that you're familiar with in your actual code as well So it behaves exactly the same way, but you don't have to go through a full um res request response cycle as you would do with um something like the client test case in Django. So the easiest way of using it is basically it's in the Django test module and you just import it from there. You use it similar to what you would do with um requests, for example. You have a get or a post and you pass in the URL and the parameters that you want to use with that. And as I said, one of the things

Speaker 1: that I've used it for is testing request-related code, for example, in views and something like that. Where you would have to mock quite a lot of uh functionality of the request so that it behaves like a request, when which basically then ends up you writing a lot of mocking code just to get that little piece of functionality when there is something that's already there that can construct it. So one very good use case for that uh in my experience is also writing uh middleware classes. Where you basically have those request responses that are getting passed in and processed in certain ways, and they can get fairly complex.

Speaker 1: So in this case you're basically you basically just have some middleware you've got your uh logic in the process request and it expects a request So you should pass in a request or something that looks extremely similar to what a request looks like. And based on that Sort of having the mocking logic in that can be quite substantial. So one way of getting around that is basically have your test function import the request factory And then having creating your request, you can even pass in query parameters or things like that. And then you just pass in that request or manipulate that request in some specific way

Speaker 1: so you're you know the type of data that you're passing in and you can test against that. And then obviously you proceed with your tests as you normally would. And that's basically it. That's the um you find it in Django test And it might make your tests a little bit faster, especially when you're sort of looking at the more unit type unit test type testing. And it saves you a lot of time in terms of mocking things And the treasure is now all yours. There is um obviously there is a lot of additional uh treasures in

Speaker 1: the Django codebase and especially Django utils is a trough of just like those little functions that clean up your strings and and all those types of things that are really really useful. And maybe just for your specific use case, but some of them aren't quite well documented in the Django docs So I would encourage everyone to just take a peek under the hood and look at some of those little functions and things like that. Just to see what is in there and and maybe there is something that you see, okay, I've actually written this in five places, maybe I should just use that

Speaker 1: And uh that got me quicker through my talk than I intended, but that gives you a little bit of time for questions. Thanks. Hey.

Speaker 2: Hello, there we go. Um so I have a question kind of like this. Um is there anything that isn't a Django treasure do you think should be? Like is there something that is there a pattern you can think of that should go into one of these places you're talking about in Django?

Speaker 1: Yes.

Speaker 2: What do you think it is?

Speaker 1: It's um I think the swappable models Are a piece of private API that I really liked because I had a sort of CMS type use case um where it was extremely handy from a library perspective to expose um or give the the user, the developer the ability to swap out a little bit of customized um functionality. without having to go through like multi-table inheritance or things like that.

Speaker 2: Right. Okay.

Speaker 1: So I think that's a that's a very that would be a very nice piece of uh Call to the eye. Any

Speaker 3: Okay, so you uh you mentioned there's there are these things hidden in in UTools. Some of them, admittedly, we've hidden them because we'd kinda like people not to use them because then they officially have to be supported. But for those that are good How do we get the message out there? Is it is it just a matter of dropping another line in the docs, or is it just going to get lost if that sort of stuff's in there?

Speaker 1: Um well, the ones that I've tried to pick here as well are the ones that have public documentation and are considered public API. So the documentation is there. Um I'm not quite sure what the best way would be to be more prominent about here is like cash property. I think cash property is actually one of the things that quite a few people know because um I think Daniel Greenfield Roy wrote a package that basically provides that functionality outside of Django. So there must be demand of using it outside of Django as well. Um and some people must know that it exists. Um I wouldn't know what it it's probably just

Speaker 1: like telling people more about it and like pointing out when you see somewhere in a in an open source package that someone's writing something that actually has existing Django functionality saying like by the way you don't have to maintain that you can remove those 15, 5, 20 lines of code. Thanks.

Speaker 4: So, uh probably a dumb question that everybody else knows the answer to, but I don't. So Um you mentioned early on one of the good places to find treasure was the Django IRC channel. Where is that?

Speaker 1: Um so the IRT IRC channel is on uh free node. It 's hashtag well hash Django for the Django room. Um and there is a gazillion um chat clients out there that support IRC There is uh it depends on like the sort of operating system that you're working on, what your preference is, whether you prefer working in a text environment or uh more UI type uh environment But there are I think if you're the best way probably is to just search for IRC on the in the Django docs and there is a reference there somewhere where to find it and how to set it up

Speaker 5: All right, there we go.

Speaker 6: Uh just wanted to share one of my little spots for hidden treasures which is um lots of the functions that are exposed to the template tags and filters are also really useful elsewhere in your code base that have nothing to do with display templates like you need to store Slugified values in your database for something you can easily go from Django default filters import slugify and use that against You know, first middle, last name, or whatever. And lots of those template tags and filters can be imported and used the same way outside of template land.

Speaker 1: Yeah, yeah. Thank you.

Speaker 5: Awesome, great, thank you.