Take the goRe out of a DjangoReact stack

Speaker 1: Hi, hello. Okay, so uh my name is Nathan uh and I'm going to be talking about building and uh Architecturing uh Django alongside JavaScript applications. So I'm a full stack developer. Uh I work with Django and React on most of the project I work on. However, and despite the Not so funny name of this talk. Uh after a few months I really start to dislike it. Uh so despite the name it's not about React. Um Most of what I'm going to say is uh works exactly the same if you're using Angular, Vue, or Ember, any modern framework. Uh if anything it's maybe about Webpack, but we'll come back to this

Speaker 1: Okay, so just to get started by by show of hand, who here uses a similar stack with a modern JS framework alongside Django Okay, so quite a few of you. Um so I don't know about you guys, but I really find this exercise of setting up a JS framework alongside Django to uh uh really not being uh uh a trivial exercise I think the main reason is there's no standard dokumentad implementation guidelines or even arkitekturing guidelines. So I think it's something that we still can improve on in the Django community. So it's something I've done quite a few projects, and my goal for this talk is to take you through

Speaker 1: um some of the design decisions that we can make to try and build such an application, how it's gonna work in development and production. And in this talk I'll also try and show you uh what it ends up looking like in the code. So before I start a quick uh catch up. Sure everyone uh knows about this, so very quickly npm is the main JavaScript um uh package repository. It's it's both it's actually both the repository and the uh client for the repository, so it's equivalent to both PyPI and pip Webpack is the main build tool for modern

Speaker 1: JS applications, so it's what produces the bundle that's the JavaScript source code that ends up being shipped to the browser There's no direct equivalent for this in Python because we can just run the Python code. And Webpack Dev Server is a wrapper on Webpack that uh much like the Django auto reloader, uh looks for changes in the source code, recompiles and serve those files on the server. Alright, um so in this first part I want to go through Jen a quick recap of Django 's history and the tools we've had to handle JavaScript JavaScript files. And the goal here is also to set up a common vocabulary so we can all be talking about the same thing. Um so when when Django came out in 2005, um

Speaker 1: The um the main use for JS back then was uh to sprinkle on top of uh uh HTML pages that were otherwise generated by the backend. Um And uh for this we had tools like Django Pipeline and Django Compressor who came out in 20 uh 2008-2009. And their their main goal was to take this JavaScript source code that usually lived um in the static folders of Django. um minify it, uh generate uh render um a script tag in the in the template and that was it. Um And a few last later Django Unchained came out and that really messed off my chart. So if you were wondering where the spec is, that's

Speaker 1: Django Unchained. But around the same time you can see the red and the blue lines, it's uh Angular and then a couple years after React So the the modern frameworks that the mo the most common modern frameworks that are coming out in twenty twelve, twenty fourteen, I think. And those tool those frameworks need to much more potent uh build uh pipelines. They need integration with um The JS packages they need to pull in abundances from NPM. Um they need um To be able to work with uh image files, font files, CSS, etc. Um and so that's where uh Django pipeline and Django compressor weren't enough anymore and the front-end

Speaker 1: uh community came up with something called Webpack. Uh which solves all of this problem. Sorry And so a few last later in 2015 , Django Webpack Loader came out and it's the the first setup at integrating um uh webpack and uh Django together um um so the goal here was to really separate the responsibilities of Building the JavaScript for um for the browser and serving it. Before that, uh Django Pipeline, Django Compressor, and equivalent were doing all of this. From now on, we would leave Webpack in charge of building this bundle and and Django in charge of just generating a script tag that links to it.

Speaker 1: And so it transformed the development environment from a single process where Django is in charge of serving and generating the HTML and the JS. to a dual environment where we have both Webpack and Django running. And Django will still be in charge of generating the HTML, but then point to Webpack which would serve the JS. So something else I want to mention in this section uh is Emrico Augustin's blog article series from last year. because they really helped me out and admit on formulating um a lot of what I've been thinking about. So his blog article is about a series of uh a formalization of the requirements for building a Django JavaScript application. So it's very much linked to what I'm talking about.

Speaker 1: And the main thing I I I remembered from this is uh his taxonomy of JS applications. There's basically two kinds of JS applications or two uh two kinds of architecture. There's single-page applications and in hybrid applications. And I'll come back to this in a couple of slides. And in in his in his blogs, he also talks about production and development parity, authentication, cores, CSRF, etc. So I really recommend going them going there if uh If you're working on Django and JavaScript. Um so in in MRIX taxonomy uh a single page application is an application where the a the HTML is purely static and uh never changes

Speaker 1: So it's not rendered by Django, it can be a static file that's served by Nginx or Apache alongside uh the JS files. And in this context, Django is purely used as an API. And so typically in a production setting, uh the browser would end up um making requests to two different origins, the front end where it would load all the static files, HTML, CSS, uh JS images And uh the back end where Django runs as the API. And the counterpart is the hybrid application where um The HTML is sold by Django and and potentially as a as a template rendered by the Django documenting engine. And so you end up with an application running where the DOM is a mix of the HTML that was generated by Django

Speaker 1: and whatever the JavaScript uh framework, uh whatever changes it brought to this DOM Um and typically in the in the hybrid application the uh the the browser ends up only talking to one um origin where I would first load the HTML which would be rounded for it and then load the JS And API calls of course. So both infrastructures have pros and cons. There's no uh silver bullet, of course. And even even once you've been able to pick between single-page application and everybody application depending on what's right for you, there's still a lot of implementation details that will vary between implementations. Still a lot more of design decisions.

Speaker 1: So now that's exactly what I want to go through from um designing uh designing a Django and JavaScript architecture together. So before we go and start making random design decisions, it's probably better that we agree on a set of requirements first on what we want the application to be able to achieve. So of course those requirements will vary a lot between projects and the example I have here is something that's very typical for the project I work on. Uh but they are they are by no mean um they might not apply to your use case. Um around Having to having to separate the code base between front-end and backend, having to separate also deployment flow between front-end and back-end.

Speaker 1: I work in team where everyone is full stack, so that's fine for me. I know lots of companies out there have different teams for front end and backend and I think if that was a requirement for you you would end up with a very different solution. But hopefully I think the the the reasoning here will apply whatever your use case and uh um you should be able to use this on your own projects. Um so sorry My requirements are I want uh environment independent builds, so I'll come back to what I mean exactly for each of them in a minute. I want hot reloading and development, and I want development and production parity. So for environment independent build, uh first what do I mean by build? By build I mean um

Speaker 1: whatever code artifact you ship to a server when you deploy it. Could be a file archive, uh a Docker image. Um a binary a bit unconventional in Python but or a commit in a in a git repository for example. And why this is some why this is important to me is because it allows me to do version promotion. When I've had a a version running in one environment for a while and I know it it fixes the problem I intended to solve then I can take this exact build, this exact set of uh code and then send it to a different environment This is something that I discovered on Heroku a while back and it really stuck with me. So yeah, that brings it brings a lot of trust in what I'm about to deploy and I know I'm not going to break the next environment down the line.

Speaker 1: So how might I achieve this? Um the the the first uh the obvious answer is uh To not have any environment specific values harded in in the code. Yeah the main like the main uh what I'm talking about this section is very related to the 12 factor app. So if if you don't know about this I really recommend you go and read uh read the the 12 pages on 12factor. net. It's it's very interesting So environment variables, they're really easy to use in Django. That's not a problem at all in the back end. We can read them anywhere in the code and read them in the settings and then read the settings Um so Django is really easy to write in an environment in a benign way, as long as you have access to environment variables on your server.

Speaker 1: Um but in JavaScript it's a bit of a different story. In JavaScript it's a bit harder. Typically uh the um the main bit of config you would need in the JavaScript application is the API endpoint. Where where am I going to send my request to For example, in production I might want to connect to api. example. com over HTTPS, while in development I'm going to connect to localhost on port 8000 over HTTP. And so of course JavaScript doesn't have a concept of environment variable. We can't set environment variables in our uh users' uh browsers So as far as I can tell that leaves us with two options. Either we find a way to load the config when the application starts from somewhere. That probably means an API call and if we're making an API call to what endpoint,

Speaker 1: it's kind of snake-hitting its own tail. That's I that's probably not going to work. The alternative I can see is to inject the configuration inside of the HTML during rendering in um so in Django. So that means that our HTML has to be a Django template, so that means we're using a hybrid application. in in Augustine's taxonomy. Um and one way you might uh what what this would translate in uh in the template if this is the template for your uh JavaScript application you might want to write a global config variable in JavaScript with the set of this is an example but I'm setting for example an API endpoint that's actually just uh uh a context variable in Django template and also all the business logic values that are actually coming from

Speaker 1: the database. Right, okay, so that was environment uh independence. Um next, hot reloading in development. Um so hot reloading is this very powerful um Front end development feature that's very much like Django, what the Django through loader provides. It's based on Webpack Dev Server and as soon as I make a change in my JavaScript source code, my front end my browser refreshes, then I can see the change immediately in my browser. So it really shortens the feedback loop from writing a code and potentially a bug and then detecting it. So this feature relies on Webpack Dev Server, as I just said, and there's actually two levels to it. Live reloading

Speaker 1: where as soon as I make a change in my source code then it's it gets pick picked up and my browser refreshes So that means I do see the change in my browser, but I lose all the variables, uh values, I lose all of my context where I was in the application, etc. So it's a start but not great. And then hot reloading is the next step. It's where instead of triggering a refresh of the page, Webpack Dev server is going to send the new rebuild chunk of our WebSocket to the browser. And the browser is going to replace it in place with that triggering refresh, so I get to keep my variable where I was in the application, etc. Right, so let's start with hot uh hot reloading actually and it's going to be quick. Um so hot reloading uses WebSocket

Speaker 1: and Webpack Dev server and As I just described, the way it looks like is in development you'd load the HTML from Django, which would then point to the static phase in a pack dev server, and then every time I make a change to my JavaScript source code, then Webpack is going to ship those hot updates over WebSocket and my application is going to update itself. So I've got one big problem with hot reloading. And it's because I'm using Create React app, which is the main React burloplate out there, and it's broken. I won't go into why, if you're interested, then we can discuss this after the talk um but I'm waiting for uh feedback from the the folks at Facebook. If you are not using Creator Act app and you're using uh uh vanilla or a custom setup of a webpack dev

Speaker 1: server, then that might just work. So taking a step back and let's take a step back and just aim for live reloading. For a live reloading to work, um All I need is to make sure that my JavaScript source code is coming from Webpack Dev Server. That leaves us again with two options. Either when the application when when I load the page on my browser, it's able to tell which files I need on Webpack Dev server and dynamically load them in the page. Or Django is able to render all of the scripts in the page for me at render time and serve the HTML to my browser so it loses the files. And both of them work, both of them are fine. And to help us decide between those two

Speaker 1: solutions, one thing we could do is look at the third criteria, development, production, parity. So what is this going to look like in production? Well in production we won't have Webpack Dev server running of course. So we will have to use um We will have to be able to uh render script tags with the right URLs in the HTML. Uh and uh for for the development production priority requirement to work then we should probably pick the same thing in development. Um So now I have to be very honest with you and tell you that I've been doing solution one for a long time for absolutely no good reason. And just for the sake of completeness, I'm going to show you what that looks like.

Speaker 1: So it's probably not readable from a file, don't worry, I'll just uh skim through it. If you've got good eyes, you can see several if debug, if not debug, etc. Um so that's very bad for the remote production parity. And this this big chunk in the middle here is actually manually fetching a manifest with the list of all the files in Webpack Dev server and uh dynamically adding them to the document here. So it's very dirty. Django Web Pack Loader earlier, and I think that's where Django Web Pack Loader could help a lot. We could transform this and Nice animations. We could replace all of this with just two lines.

Speaker 1: the if and else uh and also the need for dynamically loading all of this. Um so as I've said I I haven't tried this method yet it's on my to-do list for the very short term Uh so I can't guarantee this is gonna work, but I think this is the right way to go. Um about the development and production parity, it's actually very difficult to achieve 100% parity, especially between um development and any other environment. Actually if you think about it, my my requirement around um production uh development um sorry environment independent builds is kind of uh UAT versus production parity or staging versus production parity. Because I want to make sure that they're exactly the same.

Speaker 1: But in development it's different because you always end up using different uh different tools, different uh servers. So it's difficult to reach 100% parity, but it's it's still something that's definitely worth aiming for. Um all right, so now that we've um gone through those requirements and we've got uh already a list of um design decisions we've made let's look at what it looks like in the code um So this is our setup, uh at least in development. And the the main piece of integration between um um Django and JavaScript now resides in the template for the HTML uh that's rendered by Django. And so here we end up with the uh

Speaker 1: the the main block here setting the the the the config for the JavaScript application that's actually rendered by Django. Um we have the two render bundle tags on the for the CSS in the top and the JS at the bottom. That's coming from our um um hot reloading requirement and development production priority requirement. And that's it. So you this is Built for React, so there's a there's a div with ID app, but you could really adapt this to whatever. Um uh framework you're using. And you could actually you could actually build many other things pre pre-built into the HTML and just load the uh the JS uh the JS application in the tiny part of the page. That's also an option here.

Speaker 1: Because we're using hybrid application, it gives us this flexibility around how much content do I want to be generated by Django and how much content do I want to be uh taken care of by uh by by JavaScript So of course once we have this template we need a view to render it. I thought I just showed this just to show where I set the where I define the context variable for rendering. In this example I'm reading both from the settings and from uh models in the database, but you can pull in the information to configure your front-end application for where wherever that makes sense in your case. Um and finally we need to mount this view on a URL um which uh needs to be generic enough so that we're going to set

Speaker 1: s uh uh serve this HTML for every page of our JavaScript application, every URL that's supported by your JavaScript application. Right. And um in the front end it's uh even simpler. In the front end there's no need to make any change to our JavaScript at least. The the the one tiny change that we need to make is to make sure that our uh Webpack configuration is able to uh output um a manifest. So a list of all the files that it just outputted last time it ran so that uh Django can then read this and so and and link to all the files in the HTML. So that's it's this one line here, the uh the bundle tracker plugin. Um and finally I thought I'd show you what my deployment

Speaker 1: uh uh look like so I use Docker but if you use something similar you still have the main steps which are first building the JavaScript application so you you output a set of um JavaScript, CSS images, whatever, and also a manifest file which contains a list of all of those other files Then copying this over to a static directory that's uh controlled by Django, which Django has access to, and then finally running collect static and from here You know that the select files are available to be served and Django Webpack loader will know where to get those files from. So he can links to them. And that's it. So that's it's um it's not but it's by far not a complete example.

Speaker 1: There's many other things that we probably would need to go through when designing a Django drive application. Things like uh authentication, uh core, CSRF, although I wouldn't say that those last two are interesting topics, but they're in the list anyway. Uh SEO, progressive web app support, etc. And this most of these are actually topics that were covered by Augustin in his in his uh article series. So if you're interested in this, I again recommend highly recommend reading them. So that's it for me. I think in conclusion I 'd say that we can still improve the state of the documentation of on how to integrate JavaScript and Django. I

Speaker 1: don't think that Django is missing any feature to make this work properly. It's more about building the documenting the the the known state of the art for this to work and and giving examples for both code and and architecture patterns. Um Right. And I'm also really curious to know how you guys manage your own JavaScript and Django applications, so feel free to come and talk to me after. Thanks.

Speaker 2: Thank you, Nathan. And do we have any questions from the audience? As usual, please line up. Anybody? No. All right. Thank you, Nathan.

Speaker 1: Thank you.

Speaker 2: Oh wait, one one question. We have questions from the internet?

Speaker 3: Oh there we go. Now we got mic. No? Oh, is that one? Oh, there we go. If you open the floor for questions, there's always a question. Um in your example code there, the um where you had the settings being rendered in Django and then the rest being One of the pieces that's in there is uh uh like this list of opening hours or something like that.

Speaker 1: Do you mean this?

Speaker 3: Yes, that's the one. So you you've got you have your opening hours being rendered as something from template. That's not going to play well with the hot reloading, I would assume. Is there is that uh it's like an example of something that you you maybe want to try and push to an API or is the other thing.

Speaker 1: So this set of variables are s Do you mean hot reloading or live reloading?

Speaker 3: Sorry, live reloading, sorry.

Speaker 1: Live loading. So on live roading every time I every time the the browser is refreshed then the HTML would be re-re-compiled and rerun. So those variables will be reset every time.

Speaker 3: Okay.

Speaker 1: And Django re just rerun down the same exact template every time.

Speaker 3: Right.

Speaker 1: So I don't think that would be a problem.

Speaker 3: Okay, but if but and then so on the JavaScript, on the hot reloading, um That so once the page renders once the settings are only ever done on the page reload, but you can modify the JavaScript and then that will that won't pick up any changes there.

Speaker 1: So I don't think it creates problem with this because this variable is never reset, it's always defined globally. But it does create problem um especially I think it's especially it's so hot -related works really well for React components, for example. For every everything visual it's perfectly fine. It works less well when you change a bit of logic somewhere. I couldn't explain you why. I'm I'm not an expert by far on hot leading.

Speaker 3: Right. Okay. But it does have limitations. in what you can put in that kind of block.

Speaker 1: I think this is fairly flexible and I'm I'm pretty confident that this is not going to be reset by either live loading or hot reloading. You can just see this as the main configuration object for your JavaScript application and and and put anything you want in there.

Speaker 3: Sure.

Speaker 4: Very quick question. Uh first of all, I'm not a full stack, just a back-end engineer, but uh our team has some pro had some problems previously about the uh Django local uh um dop on server not uh running over HTTPS. Did do you have anything to

Speaker 1: with HTTPS

Speaker 4: Yes.

Speaker 1: Um

Speaker 4: you didn't have any problems in your React app at all.

Speaker 1: No, I'm not sure uh could could say lots of things about HTTPS. I'm not sure what exactly was your issue

Speaker 4: We couldn't serve the development server or HTTPS and uh React at some point has some problems maybe Cross site.

Speaker 1: I don't know. In development, I I don't use HTTPS in development and this has never been a problem to me. And when once you're in production the Webpack dev server doesn't matter anymore because you just output a set of static files that are always going to be served. So I I personally create a Nginx container for them and it and they are served over HTTPS. So maybe your issues could be around um Uh requesting non-HTTP resources from a page that's uh served over HTTPS. I know there are limitations in in browsers around this.

Speaker 4: Maybe probably. Thank you.