HTML-ivating your Django web app's experience with HTMX, AlpineJS, and streaming HTML

Thank you so much, everyone. It is, gosh, it's an honor to be here. This is my first Django Khan. Thank you. Thank you. I imagine many for you as many of you as well But I've been a longtime appreciator of the community. And I'm so thankful to give this opportunity to contribute back because I want to elevate the experience of your Django app. So to talk about elevating something, we need a benchmark, right? So I'm going to start off talking about spas And of course not these spas specifically, but single-page applications. If you're not familiar with the term, it's a prevalent architectural pattern in the web today.

You have more than likely used one The idea is when you visit a web page, among the things you download is a custom JavaScript application. This JavaScript application handles all of your interactions. or can handle them. So it listens to your mouse movement, your mouse clicks, your keyboard, and if an event pops up, it can say, oh, I can handle that for you, browser. You don't have to worry about this. And so it'll if it when it needs to communicate with servers, it'll do that over APIs. And then based on the data that you do in the app or the data it gets from the API servers it can change the HTML of the page that it's on. And sometimes even change the URL. So it may look like you're on another page, but you're actually still on that same single page.

Now, this talk is not about how bad single-page applications are. They are great for their use cases. A wonderful example is Gmail. I've talked to many people who think Gmail is the best email application out there. It just happens to live in a web browser. But there is a concern across the industry about the overuse of the spa pattern. For example, the large consultancy ThoughtWorks once said, spas incur complexity that simply doesn't exist with traditional server-based websites. Too often, teams are skipping the trade-off analysis, blindly accepting the complexity of spas by default, even when business needs don't justify it

Furthermore, some developers aren't even aware of an alternative approach because they spent their entire career in a framework like React. And I imagine many of you here today are using Django as an API server for such an application. So I'm also an organizer at PyRVA, the Richmond, Virginia Python User Group. And I for years I've talked to people who are excited to learn Python, but they feel torn because they feel like they need to learn Django, or not Django, React or some other JavaScript framework in order to make websites. And my heart kind of goes out to them. Additionally, most Python jobs for web developers mention a React , a JavaScript framework as a requirement for the job.

So how did they become so popular? I submit they became popular because when they came on the scene, they had a better experience. Specifically, every interaction you had with the with a single page application did not result in a new page loading. This attracted a lot of people to them and they started creating dynamic and engaging interfaces. Additionally, as I kind of mentioned before, you could update the interface with small payloads from the server And then any data that was on the page, once you change the data, it could update the HTML just as a result of changing data. So spas

have had an advantage in the user experience, but this is no longer the case. Traditional websites can be just as friendly to users, but much easier on us developers to maintain. So I'm going to expire, I hope to inspire you by this real life story from a developer named Taylor Hunt. He made an incredible five-piece blog series that inspired me to write this talk. And it's incredibly relevant for you today because of the new capabilities of Django and Python. His first blog post was entitled Making the World's Fastest Website and Other Mistakes. And in this five-piece series, it chronicled his story of being a developer at a supermarket chain.

He knew their web app needed a better experience. And the he wanted to really show other people what the customers of this chain felt, what they experienced. So the chain sold phones, uh sold cell phones, and one model was much popular than all the other ones. So he bought several copies of it and um gave it to people and recorded them as they accomplished a task. So I'm going to show you the video of some of these people. The task that he asked was to load either the production web app, uh the native app for this the supermarket chain, or one of the competitors, Amazon. com or Walmart. com.

Once it's loaded, you would search for a product, specifically eggs. you would add the first result to a shopping cart and start the checkout process. So I'm going to play that for us now. And you'll see there are little white dots that appear on the interface. This represents users touching the device. And you know, sometimes you can even see there's a lag between when they touch the device and when things actually happen. So each of these take a significant time to accomplish this task. But you'll see here like Amazon is essentially taking the lead. In just a moment is search results are appearing. And they don't necessarily have the best search because you'll see the first item is actually a pizza crust mix instead of eggs, which

You know, Amazon. But it's the first item in the cart, or first item in the result, so it's going into the cart. Amazon is going to finish first at just at 59 seconds. And you'll see that in just a moment The native app is the slowest to start, but it's going to finish second at a minute and 21. You'll see here Amazon just finished. The other two websites lag behind, with the production app taking almost four minutes for the person to accomplish this task. Four minutes. Four times as long as Amazon. com But Taylor knew it could be better, and he had a vision. He wanted their website to be so fast

that it's fun on the worst devices and networks our customers use. So to do this, he thought, okay, let me do some research, and he heeded some advice that he found in 2017, which was for optimal website performance, establish a budget of 130 kilobytes. For all your HTML, CSS, and JavaScript. Essentially that first download needs to be 130K or less. Taylor realized there's no way he's going to get the production web app into 130 kilobytes, right? This is a big production app. So he needed to start over. But to be taken seriously, he realized this demo app needed to fulfill some business requirements that aren't obvious.

The business required third-party JavaScript frameworks to collect data, specifically analytics and advertising information on the users. And those frameworks took all this budget But he figured, okay, let me experiment and see if I could just kind of open up a little bit more budget. And he, after a bunch of experiment experimentation, he found he could get 20k more to still have really good experience. Now the production web app used React. js and Redux, and both of those together were more than twice as budget. And after considering all of his other options, he realized the spa pattern just would not fit in this situation. With so little room, he thought.

Maybe if I sprinkled HTML with just enough CSS to look good, and if I had any room left, some laser-focused JavaScript for the pieces that benefit most from complex interactivity. So Taylor had a lofty goal. He had a very sharp approach. And I can't wait to show you his result. His demo app is on the far left, and the other four are what you saw before. Watch this. As you can see, the demo app is much quicker to load and interact with. It's using the same phone, on the same public internet, hitting a server co-located to the production app. This is not on his laptop.

This is real life as close to real life as you can get. And it's finished in 20 seconds. 20 seconds The demo app, it's important to say that the demo app uses all the same APIs to get the data onto the screen and gathers the same data about the user. And at the end, we'll create the same order in the back end. From a business perspective, this is a functional equivalent. From a user's perspective, it is a drastically elevated experience. This demo app is a multi-page HTML-driven web app that performs better than a native app. And the best news is you can use Django today to create an app like this. And so I want to do that with you.

I want to show you how you can have a better-than-spa, potentially better-than-native experience with just Python and HTML. To do that, let me introduce myself first. My name is Chris May. I recently started as a lead software engineer at Source 7. But before that, I served as a Python technical coach, which was an incredible experience because I got to join teams and help improve the way they write Python, they think about Python, they communicate And it also gave me a time to really think about how do we code? How do we write Python? How do we do things? And I also have an interesting past that kind of helped me with a lot of this. I started my career as a graphic designer. And it's there that I learned how user experience could drastically transform any project.

I mean we've already seen it in Taylor's example. So I've been building websites since websites since 1995, and I fell in love with Python in 2007. And my passion is to help other people enjoy Python, enjoy their websites, enjoy their life as much as possible. But with that, now I want to tell you how you can enjoy improving your app. So let's make an elevated experience. I have five components that I think make great experiences. First, remove whole page refreshes for every interaction. This is not to say remove every whole page refresh, because as we saw in Taylor's example, if you're fast, you're good.

But for every interaction, we don't need a whole page refresh Next, use small payloads from the server to update the interface. Third, update HTML as a result of changes in data This might seem a little familiar. Fourth, empower rich on-page interactions. And fifth, be fast. So we're going to kind of dive into some of these. To kind of show you how you can take advantage of these experiences, I want to start by telling you another or the experience of another developer named Caleb Porzio. Caleb is a developer who worked at Titan, one of the best PHP shops in the land. And he built many websites for their clients.

And on an episode of his podcast, No Plans to Merge, he talked about his experience. He dove deep into spas, like many of us, and just really enjoyed it. Until some point he realized there was a significant cost and complexity with all the projects that he was making spas with. And he realized, you know, it's gonna serve my team better, it's gonna serve my clients better if I By default, I choose a traditional web app and only choose a spa when the the needs call for it. And so he started doing that and was doing well, but every project he would start creating a traditional web app And then he would start feeling this gravitational pull pulling him back towards a spa. And he couldn't figure out what's going on. For years, he would create traditional web apps before spas

became a thing. But every new app project that he would create, he would just start feeling maybe I should just go ahead and make it a spa. And he's trying to figure out why. Was there a watershed decision where if you stay to one side of it, you can create a traditional website, and if you go to the other, you'd create a spa? It turns out after a lot of research he realized there was one seminal decision. And it's if you decide to return JSON from the server. Because at that point, you need JavaScript on the page to receive that request. You need some kind of JavaScript templating to put that into the page. That reduces the friction. Maybe you want some like state. management and it just removes the friction to becoming a spa.

If however you return HTML from the server, you need much less JavaScript on the page to handle that. With this insight, he created a framework called LiveWire, which many Larevel developers use today to create great experiences. But of course, we're not live uh Laravel developers, but there is actually a package called Django Unicorn that has a lot of the same fun uh philosophy. So shout out to the people who who maintain that But as a whole, Python developers and Django developers have adopted a different framework called HTMX. HTMX, HTMX. Handles communicating the server. It's it's huge in the uh many people have heard it and and enjoy it. Um

and and it has been growing in popularity. HTMX handles specifically communicating with the server from the page. And it gives you the upgraded experience of updating only part of your page from the server at the cost of just 14K. I don't have a ton of time to talk about HTMX, but later this week, Mario Munoz will be talking about how ubiquitous it's becoming and a number of great resources you can have during an online talk. So I would highly suggest watching that. But with the HTMX at your side, it can enable you to accomplish the first two components of elevated experiences. To talk about some of the other ones, I'm going to move on to another framework called Alpine. js , which Caleb

Poisio actually made to support LiveWire. Alpine is great because it it gives you the ability to focus on the in-page experience. So if you want to create modal uh components like modals and accordions. If you want to make dynamic form fields, or if you want to recreate a spreadsheet program inside your your website, you can do it, no problem. It's reactive, so any data that you have on the page that's associated with HTML, when you change the data, it'll automatically re-render it. It's an incredible little incredible framework. Gives you rich interactions with little JavaScript. And with those two Frameworks at your side for less than 30k, these enable you to experience, give the experience a spa

to your users. And that's great for your users, but what about us? I don't want to recommend something that's going to be hard for us or painful for us to adopt. So I want to talk about developer experience a little bit. While both of these frameworks have a JavaScript interface that you can interact with to customize their behavior, they're actually optimized to use HTML attributes. Which is a really interesting feature because it gives you this thing called locality that some of you may have heard Carlton Gibson talk about. The idea is like if you're creating an HTML component and you're just you know writing out its structure and what data is in it. When you need to talk about how to customize its behavior, normally you'd have to go open up a JavaScript file and maybe make a new one and figure out what to name it and then you know write in JavaScript and make sure it compiles and all these things, right?

Well with HTMX and Alpine, instead you can stay in your HTML file and describe the behavior you want right there. It's really incredible. Especially since if you are able to like buy in on this, you can potentially remove your JavaScript build system, which I know for many people I talk about is like one of the selling features right by itself. It allows you to write it mostly in Python and HTML. It's much easier for maintenance. You just have more fewer languages to deal with and you give faster iteration cycles. Personally, these two frameworks and one other one that I use and love, Tailwind CSS, really brought back the joy of web development to my life.

I hadn't even realized how much it had faded over time. And it's not just me. I know there are several people who've been clapping about HTMX over here. I've heard many people on podcasts talking about how much they love HTMX. I also wanted to give one more testimony. At DjangoCon Europe last year, David Guillot gave an incredible talk about how his company transitioned from React to HTMX. And I want to share two slides with you. First off, if you haven't seen this video and you're interested in HDMX, you should absolutely go to that page and watch it because it's amazing. But after showing his this the video of what's possible, he talked about what are the trade-offs? What's the negative and positive consequences of what we've done?

First with regards to the user experience He said, the negative trade-offs, there were none. On the other hand, the positive trade-offs was that they had so many new possibilities Specifically, by removing React, they can now deal with the native browser, which had more capability to render large documents, which is exactly what they do. Next, he also talked about web performance. He shared some performance metrics where HTMX version was a clear win. Specifically, and it's kind of probably hard to see at this distance, but on the top left it shows the time to interactive for the Django and React version could take up to six seconds to render Whereas the

Django and HTMX version would take up to two. That's a two-thirds improvement in performance. Speaking of performance, I want to talk most about this last component of elevated experiences, because perceived speed may be the most important consideration of experience. Maybe not quite trumping the rest, but it could. And there's one hidden cost of single-page applications, and that is the time to boot up. Tim Cadlick researched the HTTP archive, which is essentially a place where they're just trying to archive as much of the web as possible into a database and save kind of some data along with it. And he found that half of all websites built with React

take over 10 seconds to render on a mobile device. That's what that little bar in the center of that big blue box shows. And obviously, it's not just React, right? Every framework has has issues with this. But what really kind of knocked me off my feet was that one bar all the way to the right of that big red, uh big blue box. That's the 90th percentile. That means that 10% of all sites with React that are built with React take over 25 seconds to load on a mobile device And this can lead to significant user experience issues. In fact, we've already seen one. If we go back to the production web app, watch what happens when the user searches for eggs.

Preparing the JavaScript takes so much of the phone's resources that the text input lags. You'll see them type E G G S G G S and only E appears. Additionally, I don't know what happens in this experience. I'm assuming the user must have hit submit, but the JavaScript's going to clear eggs from the search bar. And it's going to submit the search request in just a moment. So what it's done is erase what the user has put in and submit an empty search request. So now it's looking at over 360,000 items to summarize and put into the page Which you'll see here in just a second. Takes a little while sometimes. But this means that the user will have to go back up to the search bar, type in eggs again, do another search.

This interaction sidetracks the user by over a minute and a half. Let's compare that to the demo app, which I'm gonna have to explain beforehand because it happens so fast. It appears within two seconds, but it actually won't finish loading because the user is going to search for eggs and you'll see that little bar at the top is going to kind of shorten because before it reaches the whole fullness, it's going to start loading the search results. Just like that. This app does something that I think is truly important. It delivers critical elements as quickly as possible and does not prevent the user from accomplishing their goal. As soon as the search bar is visible, it's hot.

It's interactive. It's able to be searched. And I want us as Python developers to do this. Because as we today, the vast majority of us do not create websites that do this. We actually treat data as the most important component. So if you think about a traditional web app, when a request comes in, the first thing we think of is what data do we need? Let me go to the database to grab some, maybe call an API, pull out something from the cache, and once we have all the data together, we kind of stick it all together into a context and we hand it off to a template renderer. That template renderer then you know grabs a template, tries to find what other ones to stitch together and starts building out this HTML string and inserting data as needed.

And once this string is completely full and finished, it sends it down to the user. This is good, but in some cases suboptimal, and it occasionally can hurt the user experience. What I propose is to, when the request comes in, start sending a template down as soon as you can. And just send pieces of it down. And if there's a piece of data that you don't have, wait for it If it especially if it's not too long in coming down the pipe, then render the HTML when it's there and could keep sending pieces down the pipe until it's done. Now conceptually I imagine this kind of makes sense to you, but you don't maybe the words make sense, but you don't really understand. So I created a website to demonstrate this. This has a homepage that has four recommendations on it.

I also created a recommendation engine that can take up to five seconds to make a recommendation. So if we were to visit this web page, I'll click load page now. And so now we're actually loading the page. But we don't see anything because the recommendation engine is kicked in and is making the recommendations. And we won't see everything until now when they're all done. So what if we could allow Python to send as much as possible down the pipe before the recommendation engine slows it down? That's what this is going to show us. Click the button here and we already have the header. And we will continue loading until the rest of the page loads once the recommendation engine is finished. The thing that's great about this is I'm kind of zoomed out

so that you can see this most of the page, but most users are probably going to be the bottom of their their browser might be around where the free shipping button is. So they could be completely unaware that their web page hasn't loaded yet. And they can still interact with the site. They can click the shop now button that the marketing people would love them to. They could search. They could navigate to another page or interact with their shopping cart. This streaming that I just showed you is all without JavaScript. This is just Python. Now what if we could send each recommendation down the pipe as soon as it's ready? That's what we're going to see here. I click to load the page and we're streaming HTML. This streaming HTML is a technology that's from 1997.

So every browser knows how to do this. It's optimized for this. We're not taking up extra resources. Finally, we can add a little CSS that essentially says Given a container element, does it have four items in it? If not, render the CSS. And this will render skeleton elements. Skeleton elements are shaped like the eventual content. They orient the user faster, hint at what to expect, and prevent the page from jumping around as it renders. Just like this. Now some of you might say this is kind of a ridiculous example. Who would have a recommendation engine that takes five seconds to make a recommendation? But the truth is, many times you can't speed things up.

You might have an API that's slow, or worse, like maybe you need to call an API, get the results, call another one. And network interference and all sorts of things can slow things down. You may not be able to control it. So this is an important pattern to be aware of. Next, I want to show you an example of this in the wild, in real life. You may be familiar with the website GitHub. When you drill into a repo , let me explain what happens first. You'll see that you can click on any file name to drill right into the file. At the same time, GitHub is a is asynchronously loading details that are nice to have. Let's play it.

So once again, GitHub is allowing critical elements to appear quickly, and the site doesn't prevent you from accomplishing your goal. And this is a great pattern. And I want to show you how to do this in Django. So I'm about to show the code that powered the previous example that I made. I imagine it's a little bit harder to read, so I'm going to zoom in and get rid of the imports. So when a request comes in to this, it goes to this function. In this function, the first thing we do is grab the template home. html. We then return a streaming HTTP response. Many of us probably are not familiar with this. This is actually not new, but in Django 4. 2 there is a new feature to this, which means it's asynchronous

and can handle an asynchronous iterator. That yields byte strings, memory view, or strings as its content. And so we're going to create the content by calling generate async on a function on the template. And if this looks a little weird, hold on one second. To generate the template, we're going to pass in data, a data context. So in this case we're saying the recommendations are going to be generated by this function, which is this function. And it's not shown, but this function essentially takes customized recommendations, which is a list of dictionaries, and will fake a slow recommendation engine by waiting seven tenths of a second and then yielding an item. then that item gets rendered in HTML and gets sent down the pipe.

This works in Django today if you're using Jinja templates. Because unfortunately, Ginja's march towards being fully asynchronous is not yet complete. Its teplate engine cannot create uh generate asynchronously yet. But I'm sure it won't take long. It probably let's say a year maybe. I d I don't want to put uh words into a maintainer's mouths. I'd like to contribute, but we'll see. But before you know it we'll be able to do something like this or even better. But today, in October 2023, I want to give you some patterns that work with Django templates. The first one, we're going to take the template and split it into pieces and yield each part down to the user.

And when I say yield, it just essentially means send it down the pipe. It's going to create this experience. So just like the fourth option I showed you earlier. This view uses three templates. Home. html, which is the blue one. It extends shell. html so it comes along for the ride. And then item. html is what renders the recommendations. So let's look at the code that powers this. And again, let's zoom in. So when a user visits this page, it comes into this function. And once again, we return a streaming HTTP response. And we the content that we want to generate to stream down is generated in this function.

The first thing we do in this function is grab home. html. And uh we render it to string and then split it. This is really weird. This is not something we normally do. So let me open up home. html to give you the idea of why. Somewhere in home. html we have this section where if we were to hand it recommendations as a data context, it would render them as normal. But if we don't pass in any recommendations, these three lines just don't render at all. And what we want is some way to say like, okay, split everything that comes before and after of this so we can insert our recommendations. So I included a little comment tag so we can use it as a way to split everything into two.

These two pieces go into these variables, and as soon as we have them, we yield the first one, which means we send it down the pipe to the user. Then for each recommendation we grab item. html, render it to string, and yield it. Send it to the user as well. Then when all four recommendations have been rendered, we yield the rest of it. That code produces this result by splitting templates into pieces, yielding each part. Now splitting templates, you know, it can be complicated pretty quickly. It's not the most elegant pattern, but it's an option. So I wanted to show you another option. This time we're going to render a view as we normally would.

Then we're going to subscribe to our server using a technology called server sent events, leveraging HTMX to do so. And so the idea is once the slow parts are ready and are finished rendering, we'll send it down the pipe. This is the experience we're going to see. we're going to produce. If you're not familiar with server-set events, it's another old technology, this time from around 2004. And if you're again if you're not familiar, it's kind of a simpler version of WebSockets that I'm really excited to be able to take advantage of with Django 4. 2. So let's look at the code that develops this, that creates this, and zoom in once more. So this time when a request comes in, it comes into

the index function. And it will render the home underscore SSE template as if it's a normal page. So when it gets sent down to the pipe, I'll show you the code actually. Let's start with the code. So this is in home underscore SSE HTML. When it gets sent down to the user, if we don't pass, if we do pass in recommendations, it can render them you know as normal. But if we don't pass in recommendations, it's going to render this div. This div has a very interesting role to play. It controls the server sent event connection with HTMX. And so there's two sides to this. On one side, like when the page loads, it needs to reach out to our server and say, hey, I'm going to listen to you on server

any if you have any server-cent events, I'm going to listen to you. Once an event comes in, it also needs to control the behavior to what to do with. And this side I'm going to talk about first. So when a message comes in, HTMX looks at this attribute on this div and says the target of this is going to be the element that has the ID home recommendation, which is this element here. So what HTMX would do is with when a message comes in, it's going to replace the children of this item. At least by default, you can override this, but by default it'll delete the children of this element. And replace it with the new content. The child of this element turns out to be our div that we've been talking about.

And by removing this div, it actually closed the SSE connection, which is kind of a nice way to kind of clean itself up. Now let's talk about the first side where we're reaching out to our server. When the page renders, HTMX will see this attribute. And say, okay, I need to establish a service event connection by reaching out to our server at the recommended SSE endpoint. This is a normal URL that goes through Django's URL patterns and ends up at this function. This function returns another streaming HTTP response. But this time we change its content type to be specifically for server-cent events so that our browser knows this is a long-lived connection and to handle it as such.

But otherwise it's the same. We need to generate content and we do that in this function. This time what we're going to do is collect all the recommendations into a list and join them at the end. So for each recommendation we render it to a string, but then we strip out any new line characters. And we do this because the server sent events protocol denotes the end of a message with two new line characters. And with that We've generated this experience by rendering a view, subscribing to ServerSend events with HTMX, and then having Django send them down the pipe when they're ready. Now these are not your only two options.

I mean I only have 45 minutes to give a talk, but I wanted to show you some that could kind of fit inside the talk. But the the organizers of the Pi Hat stack organization on GitHub have given me a space to put the code that I've just shown you up on there. And I want contributions because we are all much smarter than I am. And I think we can create better patterns that can power us and maybe even inspire Django to do new things. So please, let's build something better together. Finally, I just want to leave you with a reminder of how to make exceptional experiences in Django. Number one, use streaming HTTP responses to stream critical elements to the user as quickly as possible

to support them in accomplishing their goal. Two, use HTML fragments to update parts of the page with HTMX. Leverage scope-down frameworks like Alpine. js to power rich interactions. And that's it. I hope this inspires you to make your app better. And thank you for having me. I especially want to thank the Django maintainers and contributors for making such a great framework for us. And thank you to the organizers and everyone who has made this talk and this possible. Thank you very much.