path('/user/user.username:user/', view_profile) with Willem Van Onsem

Speaker 1: Hello everyone, my name is Willem van Onsheim and I'm going to talk about advanced bot converters or as you see a subtitle. essentially explains it already a bit. We are going to talk about how to define pods based upon models. Now pods are typically a bit The boring stuff in Django, I mean you do a lot of interesting stuff in the views, in the models, in the serializers, etcetera. And what do the parts do? Well The only thing they are really responsible for is routing the request to the right view. So it basically works as a traffic sign that routes requests to the correct uh city or the correct lane, right? But well, I think uh I discovered, not probably not as as first one, but a few interesting topics about podconverters I'm going to discuss here.

Speaker 1: And so there are essentially two uh topics I will discuss, model-based pod converters and automated overlap checking. Now we will see that the two topics are of course a bit distinct but still have something in in common. So the first topic will be uh model -based plot converters. And well, most of you Uh if you develop in Django, write pods and views all the time. And typically what you do is you define a pod where you, for example, have a certain pattern. And in that pattern you use a primitive type, like for example the str for a string that will then take the username. And in the view, what you do is uh you make a database query. Or you do something else with the username, but a typical use case is a getObject or

Speaker 1: 404, where you request the for to the database a user with that username, right? Now, the question might be, how does that exactly work? I mean, what does Django do behind the curtains to make that actually work Um well if if you were a developer in uh before twenty seventeen, you actually defined pods in a totally different way. In that uh at that time it wasn't pod, it was URL. And with the URL you define the rejects and in the rejects you could define capture groups. And uh then you wrote a view. And so what happened was uh the path was matched against all the regexes. And if one of the regexes eventually matched, what they did was uh the the view is triggered and the capture groups are passed to the view as parameters.

Speaker 1: And so that means that article ID, regardless what that was, and in this case it's clear it's a number-like thing, it will always be passed as a string. So in the view, you sometimes had to do a lot of processing to convert the string to an int. Right. And so, well, everything went well until in 2017 Shurchop Postmus introduced part converters, so a new syntax for parts. And a fun fact is I saw that uh change. Uh I was sitting on the first row when that change happened, why? Structure Posmus was my manager back then. And so uh I didn't have anything to do with it, but Uh well he discussed a bit what he was trying to do there. And so what he did was if you look at the Django source code, you have the Django URLs converters module, and there you have a class like str

Speaker 1: converter. And stringverter essentially has three items. It has a rejects, it has a two-Python function, and a two URL function And so uh what does these three components do? Well the rejects is injected as a part of how the pod pattern should look like. So in case of a str it's just a sequence of non-slash characters That can be arbitrarily long, but at least one character. And what does TwoPython do? Well, it parses the value that is captured by the rejects. So in this case it doesn't do much. It simply returns the value that it has captured, but you can use it for more advanced tooling. And the tour does something in the opposite direction. That is used when you, for example, use a reverse, a redirect, or the URL template

Speaker 1: tag. In that case it will first convert all the parameters it passes with with the corresponding to URL and use that to build up a pod. Okay? And at the end of the the the code you see that we register that convert as tr. Now Stri is of course not very uh uh advanced you would say. It's actually quite simple, but the nice thing is you can make custom pod converters. For example, I can here define a date converter with a rejects that has four digits, a hyphen, one or two digits, a hyphen, and finally one or two digits. And the twoPython function then parses the uh date uh string and it uses in this case the parsing module of the datetime

Speaker 1: package for that. And the two URL does the opposite. It first checks do we have a date object? In that case, we formatted this as year montate. And if it's not a date object, in this case we will just assume that it is somehow already formatted correctly. So we will return a value. And why is it useful? Well, we can register the pod converter as date, for example, and then we can use that as uh with between the angle brackets you see before the colon date sta. And that means That from now on it will take the rejects defined in that part converter. And so it will also parse the dates through the uh the parsing of the datetime module uh before passing the object to the view.

Speaker 1: So that means that if you see at the bottom the view, my date is a date object. It's no longer a string, right? And I think well it's a bit an underestimated feature of Django because you can use it to actually shorten a lot of of the logic. And actually uh when I saw that at a certain time in point in time I had an idea. Maybe it's interesting to do something more special with this. What if we would work with models directly? So how would we do that? Imagine that we have a podconverter that we here have named user podconverter. And the rejects is just an arbitrary string of word characters. And what we can do is in the toPython function, we just make a database query.

Speaker 1: We were asking, is there a user with that username as value, right? And if there is no such thing, then we raise a value error. And the to URL, which again does the opposite, it works in the opposite direction, first checks: do we have a user object? If that's the case, we return the username. And otherwise we assume we already have the username as a string. Now we can just like the date pod converter register our user pod converter as user. username. That makes it clear what model we are talking about and what field we are using. And then we can define a pod like profile user. username user. And so now what happens? If we now have a view that uh is is is uh linked from that pod.

Speaker 1: We now can use the user parameter as a user object. So for example, if we print user. emil, it will print the email of the user. So behind the curtains, Django will already have made the query. and knows for sure such object exists before even triggering the view. And another nice thing is that you can use it in the reverse direction as well. So if you have a template where you have a URL template tag, as parameter we can use the user object directly. So there is no more need for user. username, we can simply pass the user object and the pod converter knows Oh, I have to use the username of that user, right? So it makes the reverse shorter as well. Now of course we are not going to define a pod

Speaker 1: converter for every model, that would be a lot of work. But well, if you have the logic for one such model, you can easily generalize it to let it work for all registered models and all unique fields of all those registered models. And that's of course uh quite useful to then st start defining pods. And a bonus of this is that actually you can let this work with different pods. That have different models they are triggering. And why does that work? Because the pod converter has something special. If it raises a value error, it does not stop at the pod, it looks for other pods that are matching. So that means that we can here define a pod anything that will first look for a user with that username. And if that fails, it will look for a group with that name.

Speaker 1: I would however not recommend this approach. Why? Because there can be an overlap in the ID space. Imagine that you have a user with username admin and you have a group with name admin Then which one do you want to pick? In this case it will be the first one, but that just means that if you would later on in your application create a user with admin, all of a sudden there can be a different page that is showing up And a second reason why that's probably not a good idea is because if you have an entire uh if you have a long sequence of all those plots who are overlapping. It can create a lot of queries before it finally finds the thing you're looking for. So it will first look, oh, do I have a user with that name, username? No. Do I have a group with that name? No, and so on. So it makes the logic quite

Speaker 1: Generate a lot of queries. Now, the sad part of this approach is of course that we make queries eagerly. So imagine that you, for example, have a view that is login required Then even if the user is not authenticated, it will first fetch the article and then fetch the user. Whereas if the user is not logged in, we have fetched the article for nothing because the view will simply reject to to get triggered. And so we are doing useless database queries. But we can actually resolve that problem. And Django already does that in some parts of the logic. I mean, uh a lot of people use request. user quite often. And uh well

Speaker 1: you might wonder What if the view doesn't need request. user, then there are a lot of queries to the user table that ask the logged in user, and that generates a lot of extra work for the database. But that's not the case. So what Django does is it works with lazy objects. And what's a lazy object? Well a lazy object is a special class that has some sort of factory inside it. And uh that means that if you access an attribute of that object or you uh invoke a method of that object or you set an attribute or you uh access a property or whatever It will run the method wrapped inside it and use that and then act as a proxy to that object. But as long as you don't need any attributes, you don't need to

Speaker 1: call any method or whatever, it will not load uh the function that wrapped inside it. So that means we can use this simple lazy object to postpone the database query until we really need something from that object. So for example, if we would replace our uh code of our user pod converter with a simple lazy object instead of an eager query , We now can still define a view like profile. And for all we know, user is a user object. It is not, it is a simple lazy object, but it acts as a user from the moment you need something from that object. And so that means in this case, if we are if during business hours, uh our view will do a lot more work than outside business hours.

Speaker 1: Why? Because as long as we are outside business hours uh we don't print a user ID so there is no need to evaluate the user. And this thus prevents uh making queries that are useless in in the view And an extra bonus of this approach is we can subclass the simple AZ object and do some extra logic there, such that if we need the username, we don't have to do the query. And why do we don't have to do the query? Because it's already in the pod. If you visit profile my username and you ask what is the username of that user, we know it's my username. We don't have to do a query for that, right? An extra bonus is even we can omit the query when setting a foreign key.

Speaker 1: So uh as uh uh imagine that you here have a a view function where you want to link an article to a certain author. Well then the pod can look in the old way of doing it as an article ID which is an integer, the author ID, which is also an integer, and you want to link those together, right? And so what you can do is you fetch first fetch the article with that primary key, then you fetch the user, which is just the author, and then you set the author of that article to that author and you save the article. Now, if you uh develop a bit in Django, eventually uh you can see that there is a bit more optimized way to do that. And how does that work? You first fetch the article Then you don't need to fetch the author, you can simply set the author ID of that article and you save the article.

Speaker 1: And if we just work with our simple lazy objects and we do some extra logic on top of that, What you can actually do is uh work with an article, which is a lazy uh object, with the author, which is a lazy object as well. And from the moment we set article. author, it will fetch the article from the database, but it will not fetch the author, so only the article. And so that means that we save one round trip to the database in this case It however requires some technical details and I'm not going into details about this because well it it's quite complicated to explain in a 45-minute presentation, I guess. But well to summarize this approach, I think it's more transparent uh to actually work with model-based bot converters.

Speaker 1: It can reduce the amount of code. I think well not significantly in the sense that it will halve the amount of code of a view, but still uh most views start with some database fetches You can also make it if your field changes. So imagine that you have a model that's first an integer field for a primary key and later on a UID that it automatically converts the plot converters. So you don't have to Waste an entire day changing the patterns of your pods. And as discussed, it can reduce the number of database queries to the database. So what are the potential problems with this approach? Class-based views will need to be adapted to this Why they have a lot of logic like getObject, getQuery set, query set

Speaker 1: model, etc. So they are always focused on fetching one object from the pod, from the database into memory. That is of course also a bit a reason why a detail view is sometimes not very useful if there are multiple objects in the path, for example, because it only focuses on one path pattern. And another thing I think is sometimes problematic is that it's less predictable if and when those queries run. That is especially problematic if you update certain records in your view and depending on whether the query is evaluated before or after the update, you retrieve a different object into memory. So, well, I will give a very short demo about this. So essentially what's uh can everybody read the code? Uh no problems?

Speaker 1: Okay. What you here see is I defined two pods, one with an article ID and a username, and we want to link that article to the user with that username. Okay And the one below that is the the handler to actually link those articles. So it takes then the ID of the article, the ID of the user, and you can link them together So, how does the link article template exactly work? Well, it doesn't do much, it simply renders a template, the sample form, with the article and the user. And in that sample form What we do is we go to the link article with our article and user object directly, right? So without having to fetch IDs or whatever.

Speaker 1: If we uh load this in our browser, uh just give me a minute. Okay. Um Uh then you will see the form popping up. And so uh well They have zoom in a bit. So it has here generated a URL that is it's going to use. I only did that for debugging purposes. And what you see on the console, if you want, is it's uh well locked what object it's fetched from the database. And what you see is it's fetched the user from the database. Why? We need we have given it the username and we need the ID. So we definitely need to do a query, but it hasn't fetched the article from the database. So

Speaker 1: we here have saved one database query. And so what we can do is now uh actually trigger the logic with a post request. So let's link them. And if we take a look at the the the console log back again, it says look, I fetched the article and it has not fetched the user. Okay, so uh well for debugging purposes I well introduced of course a print statement to actually know when a certain object is fetched into memory. And as you can see the view Actually treats it as if article and author are an article object and a user object. So there is a lot of well extra logic necessary for this, but eventually you can let it work so that the number of queries is reduced. Okay.

Speaker 1: Um that's I guess the demo for uh this part. Um Okay. And so that's essentially the the first topic I wanted to discuss. Uh the second one is automated overlap checks. And while that's a bit of a different topic, we will see that it has some uh well uh overlap with with uh the first topic, which is of course a bit over ironic because we're talking about overlap. So, well, we already actually discussed it to some extent. Pods can overlap. For example, we here have defined two pods. One with a comment and a slug, and one with a comment slash add. Now if we're going to visit comment.

Speaker 1: add, what will happen is it will visit the first pod, right? And why is that? Because the text ad is a valid slug as well. And there have been numerous hours wasted on this kind of problems. And why is that? Because a lot of people think Oh look, it doesn't work. And they think it's the second view that actually is the problem. So they start introducing all sorts of print and debug statements, etc. And they then think it's the code cache or whatever. But eventually they find find out they are barking up the wrong tree. And and so uh well that takes sometimes a few hours to realize. It also introduces problems with, for example, redirect. Imagine that you do a redirect or a reverse or a URL.

Speaker 1: It will try to generate a pod based on the name of the view and the parameters you provide But that it's not said that that plot will really root towards that view anymore if the plots overlap. And so, well, that made me think. Um, as we discussed before, pods are actually compiled to rejects, right? So the pod converters we introduced earlier have rejects in them. And what we actually wrote With the comment and slug was actually a short version of this report thing where we introduce a rejects, right? It's only that the rejects is hidden in the pod converter and we don't have to deal with it. Now uh rejectors have been introduced by Stephen Kleen

Speaker 1: and um uh he the he also did a lot of research on them and it turns out that if you have The intersection of two rejects that's always a rejects as well. And you actually can calculate the intersection of two rejects. And uh that's not in the standard rejects library of Python, but there are packages like Greenery that can do that. So you can provide two rejectses, for example. So the first one is a lowercase A and then an any uppercase character And the second one is any lowercase character and an uppercase character between A and M. And what it can calculate is the rejects that only accepts strings accepted by the first two rejectses. And uh well the greenery package is not entirely perfect, but that

Speaker 1: has nothing to do with the rejectses fundamentally. It's just that they introduced all sorts of special syntax on top of those rejects, like word groups, etc. And greenery apparently have some trouble with answers, words, uh with uh well words, ranges, etc. So but you can massage basically an uh rejects such that greenery understands it. And so, why is that useful? Well, we can check if there is an overlap between those two pods. simply by looking if there is a rejects that intersects with those two reaches and if that's not empty, then we know there is at least one string that is accepted by both rejects. Right? And so then we can automatically detect that there is a part

Speaker 1: that well can trigger both parts, uh that there is well a pod in the request that can be triggered by both pod patterns. And uh that this means that there might be a certain problem. So how can we let this work? We first convert all the pods that we have registered to rejectes. We check then the overlap for every two rechexes and then of course we report potential overlaps. And essentially we can detect three problems with overlap. In order of more severe. So simple overlap is typically not that much of a problem. You can, for example, have here two pods where you want to, for example, like a certain post based on a slug. And a second part where you want to list

Speaker 1: all the posts of a certain author by another slug. And well, if you have a post with a slug author , Then of course there is a certain intersection between the two. Both parts could accept the part at the bottom. And in this case it's always the first and one that will accept it. That's not per se a problem. As long as there is no author with a slug like, uh that's fine. But that means that you would have to think, oh, the author should not be like, right? A more severe one is that the first uh that the second pot is contained by the first one. For example, here you have two pods where the first is a help and then any pot. And the second is a part

Speaker 1: that starts with help, about, and then any slug. And you might think, oh, if I go to help about foo, I go to the about page, but that's not the case. And regardless of what part you you hit uh well you you what request you are making the second view will never get triggered. So why? All the pods are captured by the first part pattern. And so the only way to resolve that is to swap the two pods so that the second one at least has some chance to capture certain content. And finally, uh we are actually back to an example in the previous topic, that is uh that if the two parts are equivalent. In that case, swapping the two is not a solution because they are equivalent. So you actually even if you would swap them you would end up with the same situation.

Speaker 1: So uh that's quite problematic and in that case uh I would advise to to find a way to make them non-overlapping. So how do companies sometimes do that? Well for example IMDB uses a prefix for this. So if you have a title on IMDB, it prefixes it with TT. Whereas if you have a name, so for example an actor, it has an M as a prefix. And that means that even if later on you have a URL where both the title and a name can be put in at least you know that the the the the if it's a title or a name right so uh what would be the pros of such a tool Well, it can prevent mistakes and you can also add it as a into the CICD pipeline.

Speaker 1: Um the potential problems with this is that You first need to fix the rejects such that greenery understands it. And uh well it's not entirely clear what greenery accepts and doesn't accept, so that will require a lot of work to rewrite the rejects. And sometimes overlap is acceptable, of course. So as we discussed with our author that should not be named like, for example, sometimes you say, okay, that's one of the conditions of an author that he should not be named like. And then you can implement that simply in the view logic. So Well, I implemented this in a function, well, in a management command named URL overlap. And so essentially

Speaker 1: what it does is it looks for URL patterns that are defined. I defined some extra URL patterns here at the bottom to of course trigger overlap. And if I then run the management command , What it will do is it will look for all rejects and try to find uh overlap between all those patterns. So what does it list, for example? Well it lists that we here have a part that's our article linked thing at the top. And at the bottom I have a wild card And it says of course, look, that's all very fine, but if you would have an article one and a user B, for example, the pod will be slash one

Speaker 1: slash B slash link. But that is also acceptable by a wall wild card where the wild card is then one slash b slash link. Okay, so there is an overlap on that That's not really an issue here, but if we move further down the list, you see that it has found all sorts of extra overlap, and eventually it has also found overlaps that are more problematic So just give me a minute to scroll. Yeah. Okay. So what I did here at the bottom was I made a pot pattern. Which is a wildcard, so essentially it accepts everything, and then I define the part unreachable. And so regardless how you try to trigger the unreachable pod, it will never happen.

Speaker 1: And uh it has detected okay the pattern with the wildcard and an unreachable overlap, for example with that slash unreachable. But since all captures of the second pot pattern are also captured by the first pattern, the second pattern will never fire. And so at the end it lists uh hints how you should reorder the pods such that all your pods eventually got triggered. Now I use this tool where I work on my workplace and we found a few of those overlaps. Typically, that's not because Uh you have an off day as a programmer and you don't see it, but what essentially happens is someone defines a plot and then someone else comes and he defines another plot. earlier in the sequence and well eventually certain views might not be accessible anymore.

Speaker 1: Okay. So that was the demo of the second topic. And well I'm implementing this logic in a GitHub repository Well, initially I had the idea that that would be finished by now, but I found some extra topics and IDs. So well it's still under construction, but I hope to finish it by the end of the year. And so that essentially concludes the talk. Okay.

Speaker 2: Earlier you had mentioned class-based views as opposed to the implementation that you had here. Can you go into more detail on how to implement this for class-based views?

Speaker 1: Well actually the the way how you could do that is simply removing the get object. So the the object is already in the URL parts, so in this the quarks, so to speak. And so that means that the get object becomes quite useless, so it should no longer take the ID parameter or whatever. It should essentially do nothing. So return none. Now that will not really work because a lot of the extra detail view logic depends on that. So essentially what I think is the most uh the best way to handle it is that you in a detail view give a name of the detail object that's already in the quarks and then you treat that as the object. And so uh that will also be work if by most of all the extra mix-ins that are built on top of a detail view, for example, because if if you would have a a a mix-in

Speaker 1: that says, oh, you you have the get object and I want to, for example, uh load related objects into into memory or whatever. You want to get object to really give you the object and not just none, right? So I guess that would be the the simplest way to fix it, but well uh Uh uh it might still uh raise some some some extra issues.

Speaker 3: Um and thank you.