Experimenting with file-based routing in Django

Speaker 1: Hello everyone, I'm Sage. I hope you all had a good coffee, cup of coffee. I'm a white tail developer at Touchbox. You can find me on GitHub and Twitter, anywhere else at LehmanH And today I'm going to be talking about experimenting with file-based routing in Django Before I start, I'd like to say thank you to my employer, Torchbox, for allowing me to be here. It's really nice to be here and to see all of you and to meet some familiar faces finally in person So yeah, um I'm going to start. Oh, by the way, this is my first time speaking on stage at a conference.

Speaker 1: So Thank you. And I just moved to the UK three weeks ago, so my mind is still all over the place. I hope I can do this talk right Anyway, let's start. In this talk, I'm going to be talking about what file-based routing is. how it works in other web frameworks, how it may be implemented in a Django, in a Django project, and possible improvements and uh challenges that we may face in implementing them and finally how you can use it in your project. But before I continue, I'd like to take a step back into

Speaker 1: 2020. If the year were 2020 and COVID didn't happen I would be standing here, or here , to talk about the new cross-database JSON field feature in Django 3. 1, which was just released in August. But that's not the universe that we live in. So we went on with our lives. Uh it was 2021. I graduated from university with my bachelor's degree and then I got a job as a front-end developer at a local startup back home in Indonesia. And On my job, we used Next.

Speaker 1: js, which is this React JavaScript web framework that apparently all the cool kids use these days. just like the other thousands of JavaScript frameworks out there. Um that was a joke. I don't hate JavaScript and I don't hate its web frameworks either. In fact I use it on my personal projects. But I wish I didn't have to make that clear, but I don't want to be on the front page of Hacker News So anyway, that's not what I'm here to talk about. So coming from Django to Next. js, I notice some similarities as they are both web frameworks

Speaker 1: uh and also of course some differences. But one thing that stood out to me the most is the fact that There is no something like a URLs. js or. ps because the cool kids use TypeScript. Yeah, there's not nothing like that. So If you have not used Next. js or the other JavaScript web framework, you might wonder, then how do you define the URLs in your web application. Well, they have this thing called file-based routing. What on earth is that, you may ask? Um is it just some tech jargon made up by the JavaScript community? No, it's actually a real thing

Speaker 1: and the name makes sense So with file-based routing, instead of defining routes in your code, for example, in your URLs. py The roots are defined by your code, or more precisely, by your code structure. So how does it work? I'm going to take Next. js as an example. So for example, on Next. js they they call it pages instead of views. And you have this pages directory where you can put your views or pages. And for every file in there named index. js, it will

Speaker 1: create URL pattern for you. So in this case I have a pages directory that contains an index. js. That will be the root root. slash um and then the next one is uh if there is a subdirectory in the pages directory like blog and it has an index. js directory That will also be mapped into a URL pattern. That's how it works. So yeah, basically the pages index becomes the root root and pages block index becomes slash block. And how is this possible? Because

Speaker 1: as we know a module can contain many functions in a single file, how does it know which function to call to handle a specific route? Well in JavaScript in order to import other functions from other files you have to export that function first. And this also applies to variables and other stuff And you can export multiple functions, but there's this one thing called export default function, which means that this function will become um this is what your module is represented when you import

Speaker 1: the module as a whole module So I guess you can imagine this in Python instead of you know from some module import some function, you can imagine it as like import, for example, my app. fuse. something as a function and you will get the function directly. So yeah that's how it knows which function to call for the page And it doesn't only apply with index. jsx files, it can also you can also use any name and it will automatically root those for you. So and it also works with nested

Speaker 1: files. And for dynamic paths, they call it in Next. js, things like the path parameters in Django. They use this syntax with the brackets to denote that it is a dynamic path. So for example, the blog slugg. jsx that slug can be anything. And there's also the in the unknown with the spread syntax that will capture everything beyond that point, so including slashes. That's how it works in Next. js. And there's also another popular

Speaker 1: JavaScript web framework that is pretty hot right now. It's Remix. It basically works the same way, but instead of using brackets, they use the dollar sign. And They also do some more advanced things with nested layouts, but I'm not going to talk about that. But at least it is a thing in Remix. And also Nux. js, which is like next, but it's for a few. Instead of dollar sign or brackets, they use underscores But basically the same thing. And also in Speltkit , it's a bit different.

Speaker 1: Instead of relying on the file names, they um they need you to explicitly create directories and the pages will be defined by a file called pluspage. spel. So the roots. need to be explicitly defined by directories. But again, it is a thing in spelt spelt gip to be precise. And the next thing is Gatsby. Gatsby is a React web framework that also uh integrates with GraphQL API. And in here you can see that it gets even more complex. It can hook up into the GraphQL API just point the fields.

Speaker 1: Again, I'm I haven't really used Cat B this way, but apparently it's possible to do such complex things But yeah, it is a thing and why is it so popular? Why are there so many JavaScript web frameworks I'm not finished that use file-based routing? Well, one of the reasons why I think it's nice is that the code structure it maps directly to your URL structure. So If you are modifying some code in the pages directory, for example, you don't need to look into your URLs Pi to find out which URL this file handles.

Speaker 1: And likewise, when you are reviewing the website, you don't have to look into the URLs. py to find out which code this URL is handled by. You can just follow the URL path in your address bar and follow that in your project structure. That's one of the nice things that I find about it. And another thing why this works so well in the JavaScript world is that because In order to import files in JavaScript, normally you can point to the module names with strings, and that's the normal way of importing things.

Speaker 1: So yeah the file names can contain all sorts of weird characters. I think you can even use emojis. But yeah, uh this is not a thing in Python because in Python module names need to be valid identifiers So you can only use alphanumeric characters with underscores and even then there are still some restrictions like you cannot start with numbers. But I think that's that's just the way the language is designed and it I I mean most server-side languages. also do that. So this is just a special thing in JavaScript. I mean there might be other languages that do this, but

Speaker 1: And so how can we have this in Django? Well, let's start with the file structure. So this is an example with Next. js and let's try at first just rename all the extensions to dot py So we have this. And I think with the index. py file we have a more suitable name for this in Python. which is in it. So it effectively makes all the subdirectories modules as well in Python. But the problem

Speaker 1: is with the dynamic paths, because we cannot we cannot really use square brackets in the module names Well, it is even more complex if we want to use how Django specifies path parameters with angle brackets. So how can we get around this? Should we turn them into double underscores? I think it gets ugly real quick and it gets long. So nah Can we somehow turn this into just the name of the path parameter and have some way in the code to define how

Speaker 1: To note that it's actually a path parameter. But I'm not going to go into that yet. Let's focus on the more simpler on the on the simpler use cases without the dynamic paths. And let's dive into how the fuse can be defined. So earlier I talked about the default function in JavaScript, and we don't really have export default in Python and since all functions are exported by default we can name it just default but that's not really nice isn't it

Speaker 1: Um, I guess we can also name it just few. But you know what? I'll do you one better. I'll use dispatch. You might see where I'm going with this, but let's save that for later. Just call it dispatch for now. And how do we hook this into our URLs. py? Because in the end, Django still looks for that root URL configuration, right? So uh we can design a function, for example, um this FsPath function. Just an example and we point the function to a directory that contains our views, for example, myapp. views

Speaker 1: And the way we design this function, uh we have many ways to define this. We can use maybe instead of dotted module paths, we can use slash like actual path of the directory. And instead of returning a single URL pattern, we may also return a list of URL patterns. So you have to unpack the list. And or maybe just concatenate the list with the URL patterns. But I'm going to stick with the first option there. uh function that takes the module path. So how do we define this FS

Speaker 1: path function? I have an FS paths function here that takes a module path. So the first thing, I don't know if you can read that But so the first thing that we want to do is we get the module object by importing the module path. And because this is inside a function, we ideally we don't want to use the regul normal import syntax because we need to specify the file name dynamically. So we need to somehow dynamically import the module And then we use getAttribute of the module to get the dispatch function

Speaker 1: or the view function. And then we construct the URL pattern by calling Django's path function. And we repeat the process for all submodules of the module that is given by the argument. So first thing get the module object. How can we do this? Python has a built-in library from import lib. It's built in so you don't have to install anything other other than Python of course, import module and you just point that to the module path with the dots and it will give you the module as an object.

Speaker 1: And use getAttribute. So yeah, just getAttributeModule dispatch. You get the view function. And then construct the path, just call path from Django URLs. In here assuming the the module path that is pointed to is the root root. I'm just going to name it index. And we need to repeat this for all submodules Now there are many ways we can do this. One of them is just by scanning the directories for files that end with. py extension But I

Speaker 1: like to treat the whole module as a package. So I looked into Django's uh sorry Python's standard library to see if there are functions that we can use to walk through the package. So first of all we will instead of just returning a path, we want to return a list of paths. I mean, we may not return an actual list, but we need to store them in a list first. And Just to be safe, I added a default for the get attribute here, none. And if it's not callable, don't add that to the list. And then we use the include

Speaker 1: function from Django to group all of the uh URL patterns into a single path function call And just to make it nice, I added an optional parameter namespace that you can define that will be passed to the include function so you can treat the resulting URL patterns just like an app so you can do like my app colon some view dot something And next. Yeah, there's a function in the Python standard library. It's called walk packages.

Speaker 1: that we can use to walk through the package. And it needs a path parameter that it takes a list That you can just take from the module dot double underscore path. That's just how Python works. And then you can add a prefix that uh that will get added to all the submodules in that uh in that module. So if you add a prefix like So if I put myapp. fuse, all the submodules will have myapp. fuse. something And then the wall packages function return

Speaker 1: a list of packages or module info that you can unpack into uh it's a name tuple That consists of three elements. The first one is a module finder and the second one is the module name. And the third one is a flag. that says is this a package or not, but we're not going to use that, so let's ignore it. And This is basically the same thing as import module, but it uses the module loader. So after doing finder find module name and then load module we will get the module object just like if we did import module

Speaker 1: And then we do the same thing. We get the dispatch function and because this is not the root root, we I strip out the prefix and replace the dots with slash to make sure that it's the correct route. And we use the uh the one that still have the ones that still have the dots instead of slashes as the name of the of the URL pattern And then I sort the list in reverse. This is just a quick way because If we want to have

Speaker 1: dynamic paths with the angle brackets, it needs to come last, otherwise it captures all the predefined routes. That should take precedence. So after we get the list of results, we just include that and return the path with that include And yeah, so this is how you can use it in the URL patterns. I added the namespace argument there, my app. And so what about path parameters? So apparently it just works because if we dynamically import the module

Speaker 1: Python doesn't really care. Python doesn't really care if the name is in is not a valid identifier. So it still works. But of course this has the downside of you not being able to import the few uh the the modules as you normally would. So Yeah, if we want to do the approach of just naming the file normally, like just slug. py I think one possible solution to this is by defining a variable inside the module that tells Django, hey, this is the actual root. uh the actual pattern that I want you

Speaker 1: that I want you to use. So it basically rewrites um what would have been generated by our FS paths function. So yeah, it can also work with integers. And but then doesn't this effectively defeat the point of FS paths because now you can just write anything you want in the path and yeah it can be something that's totally unrelated to the code structure, right? Well, that's just that's just the downside to this. So there's not much we can do about it. So as long as you follow the conventions, you should be fine.

Speaker 1: And yeah, it's just instead of doing this where we just take the take the model name module name to be the root name, uh sorry, to be the root pattern, uh we dynamically get a path variable if it exists in the module. Otherwise we fall back to the original method. And I guess one thing that we can do to make this better is that instead of allowing the module to completely rewrite the path, we can just allow it to rewrite the specific part of that path.

Speaker 1: So if you have projects slash ID dot py, this path variable will only override the ID. . py section, it still keeps the projects part. So basically it makes sure that but this is makes the implementation slightly more complex because you need to make sure that the children look up to the parents for their paths. So that uh it always always used the same um URL paths for the prefix. And yeah, so why did I choose dispatch for the function? Well

Speaker 1: you might notice that there's the dispatch function in Django 's class-based views. So I think it would be nice if we can do something like we just define the method handlers like get post We don't have to define the dispatch function and this FSPATS thing will automatically generate a dispatch function for you So this might be a good compromise between class-based views and function-based views. But yeah, I have not really tried this. And what about Rejects Paths? Well, I think

Speaker 1: one way we can do this is by defining an RE path variable and making sure that all the child routes from this point forward use re path. But this is just an idea So how can you use this in the project? But first I want to ask you do you really want to use this in the production side? Because again, this is just an experiment and I'm not even sure I want to use it myself, but at least it works. It's a proof of concept. But if you really want to I have made a repository here, but you can see that there's no package, so it's just

Speaker 1: a repository. The code is like really small, so you can just copy and paste it into your project. I would be fine with it. But if you really want to, there's A package made by Ed here that implements this in a slightly different way, but basically still the same idea. It's slightly more opinionated because it allows you to define your templates in your views as well. But it might just work for you, so you can check that out. It actually has test, so it should be working.

Speaker 1: And yeah, thank you. But before I finish my slides, maybe I can give you a quick demo of this just to make sure that I'm not making this up. So I don't know if you can see this but in here I also added an optional parameter prefix That lets you add a prefix before the URL pattern so it doesn't have to be on the root root. So if I can just show you this This is the root root, it's still 404 and if I do slash

Speaker 1: FS, it will show you that this is the root. of the URL pattern f with the file-based routing and yeah so something like that So yeah, um I automatically generate the dispatch function just for debugging. But yeah, so If you look at the code structure, I have the SDR username And it's just empty because I dynamically generate the functions. But

Speaker 1: Oops, okay. In here I, for example, I define a dispatch function. And For example, just showing you it actually works. I'm not making this up. So yeah, um let's Let's get back to this. Yeah, thank you. That is all from me. If you want to look at the slides, it's there and there are also links to the repositories. And my employer is hiring, so if you want to look at the job lifting, that would be fine. And we also maintain

Speaker 1: this open source Django based CMS called Wagtail so make sure to check that out. But that's it for me. Thank you very much.

Speaker 2: Thank you so much.