Building a Large-Scale Wagtail Hosting Platform – Vince Salvino

Speaker 1: Um so my talk is about building a large-scale Wagtail hosting platform. And uh this is not just limited to Wagtail. Of course, Wagtail is just Django, as uh I love that line. And you know, so this is a Django hosting platform and we also have WordPress and a few other things based on our own client support needs. But you know, the exciting part about it really is the wagtail part. So um so let's just dive right in here. The aviary, um you know you have your your Wagtail site and uh maybe you like Wagtails so you've You've built two different Wagtail sites, and you want to put them in this nice pretty cage,

Speaker 1: and you're going to host your two Wagtail sites. Uh but what happens when you have, you know, three Wagtail sites, four, eight, sixteen, thirty-two, sixty-four, you know. your your uh aviary will look like will go from this to looking more something like this uh pretty quickly once that infrastructure gets out of control. And you know, that's kind of what we you know, we build a lot of client sites. Most of the sites we build are small, so we build small marketing sites for for our clients, but we have a lot of them. Which is the opposite a lot of shops will have one giant site that they work on. We have many small sites, so the hosting of them is kind of a little bit of a different challenge than what the typical setups are.

Speaker 1: So before we talk about building any kind of Wagtail hosting setup, let's just talk about the common ways that people do it. And I'm going to exclude like Heroku and Divio for the sake of this because those are pre-built platforms and they're they're great ways to do it. But uh let's I just want to talk about the kind of the traditional ways of hosting uh really any Django app. So the first one is uh the old school. If you've been a you've been around the block, you've seen some stuff. If it ain't broke, don't fix it. You know, this is your standard Lamp stack, right? The Linux machine, maybe Apache or Nginx with Mod WISG , you know, MySQL or Postgres, probably running it on localhost,

Speaker 1: you know Python, you're just installing it from the system package manager like any God-fearing blue true blue American would do, right? It's all gone downhill since 2. 7 anyways. Just being cynical there, but um this is your your old school sysadmin talking right here. So um, you know, this way works great. I've used it, you know, it's a great way. You can throw this up on a $5 digital ocean box and it'll get the the job done for a small site. So that's one setup. Another setup is the captain. This is a great quote from Moby Dick here. For there is no folly of the beast of the earth which is not infinitely outdone by the madness of men. And I think that accurately describes Docker.

Speaker 1: So once again, just being cynical. So this is the Docker approach. Everything is Docker. You can put Python in a container. You know, it's running a WISGI server that's in a container. You've got a front-end web server that's talking to that. So your Nginx or your Apache is also in another container. your your database is in another container and because you've got all these containers going uh well the containers don't really have any kind of persistent files they're um They're more like static. So for your media files, you are using S3 or some kind of object storage or something. And this is a pretty typical setup once again. You might have a few more containers, you know, maybe you have a Let's Encrypt container or you have some other process management container.

Speaker 1: So to host a Django site with Docker, you you might be looking at you know running four or five different containers and and doing that kind of a setup. So the even more um I'll say modern approach is uh what I call the pioneer. Uh and here's a I think a fitting quote from the Wright brothers or Orville Wright. With 12 horsepower at our command, uh we considered that we could permit the weight of the machine with operator to rise to 750 or 800 pounds. And I think that's a great metaphor for some of our web development apps where you're like, oh yeah, I've just got this little Django site, but now I'm going to add this integration and this integration and this cloud service

Speaker 1: and this build and this everything And pretty soon you have an 800-pound web app just because of the sheer amount of infrastructure involved. Once again, this has its pros and its cons. There's, you know, this solves a lot of problems, it introduces different problems. But this is the total cloud-based infrastructure. So you're using, you know, you are heavily relying on continuous integration. you know, container registries, deployments, network gateways, reverse proxies, load balancers, certificate issuers, all these this big web of cloud services. you know, database services, Redis, uh, Memcache, elastic, um, Elastic Cache Storage services. Yeah, I'm getting my tongue twisted here saying them all. Probably a Kubernetes in there at some point.

Speaker 1: So you get the idea of this setup. It's a very flexible, it's a very dynamic setup. You can scale infinitely. But uh you know you're basically using everything but an actual VM. So this is kind of the total opposite of the old school approach. So, okay, so those are three different ways that you could kind of build your hosting setup. And like I say, I've used all three of these and they have their pros and cons. But that's for one app. So now let's scale it up and see what happens when we have a hundred Wagtail sites. So 100x old school. Now we are managing a hundred virtual machines because this was the virtual machine approach, the lamp

Speaker 1: stack approach. So okay, that's not bad. We can manage a hundred VMs. There's um You know, there's very mature solutions. This problem has been around for a while. Ansible Chef, there's all kinds of tools. Um it is slow because bringing a V Up VM up or down can take uh several minutes. It's you cannot instantly provision a VM, um not very easily anyhow. Most of the cloud services, you know. you have to spin one up and wait for that task to complete and etc. It's it's not an instant process. The other downside is uh because I'm you know hosting s these wagtail sites, I would like to benefit from an economy of scale if possible. And uh the with the VMs you don't really get that. You know, you pay if you're paying $20 or $30

Speaker 1: for a small VM If you're running a hundred of those, you're still paying $20 or $30 per VM. There's it's a it's a linear, there is no cost savings for doing more. So even if they're running at 10% memory because there's a low traffic, you're still paying for all that VM power. So that's uh that's what a hundred old schools would look like. Um a hundred X the captain Okay, so this is the Docker setup. And remember, each Docker, each app, each Django Wagtail site is probably using Docker containers. So a hundred of that is actually equal to 400 containers that you're managing. And once again, media files are still the same problem.

Speaker 1: So now you're running a hundred different S3 buckets. Which uh, you know, for anyone who uh likes uh cores, you know, cores rules, have fun with that. Solutions such as uh Kubernetes or Swarm are of no use to you because those solutions are pretty much are uh primarily designed for running a hundred copies of the same app. So you know I'm Netflix and I need a a hundred dashboards spun up to handle my load Or I'm Google and I need a million Google search backends running. But what if I have a hundred totally separate Wagtail sites with totally separate databases? You know, that's not really the problem that Kubernetes is trying to solve.

Speaker 1: So these orchestration solutions are uh you know they're they're just not the right fit here. Economies of scale. Yes, you can get some economies of scale out of containers because a container runs essentially one process or two processes. and uh it uses a small amount of memory and you can kind of benefit from the shared memory of the host. Um you know so if you're running three different containers and one needs more memory the host can give it more memory. can take memory away from other containers if they're not using it. So you do get some uh economy of scale here. Um The third approach is 100x the pioneer. So this is the cloud service approach.

Speaker 1: Say you're using a dozen different cloud services to host your app. You're using storage and uh proxies and gateways and uh you know continuous integration, git repos. Say you're using a dozen per app. Now you're up to twelve hundred uh different I'm calling them management subjects, just 1200 things that you now have to manage for all hundred of your apps. So now some of these cloud services are designed to scale in that direction. So you know know managing one is just as easy as managing a hundred. Not all of them scale that way. You know, it just really totally depends. You're you're really getting really deep into webs of infrastructure at this point. Like I say, you're going to have thousands of different things to be keeping one straight from one client or one website to the next.

Speaker 1: Economies of scale. Usually not because once again similar to the VM approach, you know, they charge you per request, they charge you per gigabyte used. So If you're hosting a hundred sites, it's not going to be any cheaper than hosting one site because they're they're metering you know per per thing that you use. So it's a very fixed pricing sort of. So okay, so those three different kind of common options, you know, what what is the best way to host 100 Wagtail sites? Well the answer is uh you know none of them. So the you know Django is a lot different, uh Python rather is a lot different than PHP.

Speaker 1: You know, the the hosting scenario is just totally different. You know, this problem was solved with PHP a long time ago and with WordPress, you know, you know, mass WordPress hosting is is very common and very cheap. because though those scale problems have kind of been solved. But with Python it's a little different because you've got WISGI, you need more, you need more infrastructure involved than you do with PHP. So To figure out what the solution is, let us examine the anatomy of a wagtail So this is kind of the main parts of running a Wagtail site. We're going to go all the way from the very bottom, the bedrock, all the way up to essentially the internet.

Speaker 1: You've got the machine, the OS that's actually running the computer. You have Python, you have above that, you have the Django Wagtail stack. Then you have your custom code, kind of on top of that you have WISGI, which is this you know kind of a black box for most web developers, but it's translating between the web server, you know, and the internet and the Python code. So let's just dive into this here. The machine, you know, what does the machine give you? Well, it actually gives you quite a lot. You get cron jobs, you get a fast, persistent file system. Files don't uh just go away when you shut the machine down. They stay there forever until you totally delete it. Um

Speaker 1: you get strong user and file permissions, you know. Permissions and security and stuff have been baked into operating systems for such a long time that those tools are rock solid. You also get super low overhead. You know, it doesn't cost much to run a machine. A stripped-down Linux distro can run in a few you know, less than a hundred megabytes of memory. Um and for the for that hundred uh megabytes of memory you get all those things, file permissions, uh storage, jobs, all kinds of stuff. Python, this is kind of obvious what Python does, but Python has one downfall that is often heavily debated at PyCons and whatnot, and that is the Gill.

Speaker 1: or the global interpreter lock. Essentially, without getting too deep into the hood of Python, the GIL essentially means that If you want to run two two um you know if you want to handle two Django requests or you want to run two different Python things at the same time, you have to take your code, create a copy of it, and run two copies of your code at the same time. There's no overlap, there's no efficiency between them. You're basically, you know, if you want to run two Python processes of your code, you're doubling the amount of resources that takes. And there's a whole, you know, multi-threading is a whole thing, but for the most part of Django, Django is heavily kind of a single

Speaker 1: threaded because of this gill. While that is a generally speaking a downfall for a performance, it can be a benefit because we are able to predict resource usage. If you're running two processes, you're using twice as much. If you're running four processes, you're using four times as much. It does make it predictable, kind of a silver lining. Python itself is also pretty low overhead, you know, less than 10 megabytes to run a Python process. Which you know for us old schoolers you say, oh, 10 megabytes, what a waste of memory. But uh when you look at some of the stuff out nowadays uh that uses you know hundreds of megabytes just to run an interpreter, um you know, various JavaScript frameworks and whatnot, uh, you know, they they've just got a lot going on.

Speaker 1: So Python is pretty efficient in comparison. Now we we get up to Wagtail and Django. These numbers are purely just empirical generalizations, but uh You know, running a Wagtail site, just a base-level Wagtail process serving some pages, is going to run about 100 megabytes of memory. It might be a little less, it might be more depending on what you're doing, but just on average, kind of your standard Wagtail site is going to use that much per process or per uh per site that's running Of course with Django, because of the whole Python Gill situation, you get one request at a time. Django cannot handle two requests at the exact same time.

Speaker 1: It does one, it does them sequentially And with Wagtail sites, assuming you have a Wagtail site that's pretty heavily filled out with content, which is a more of a real-world scenario. You might be looking at a total end-to-end time of 100 milliseconds response time. Now, realistically that might only be 20 milliseconds of CPU time or something, but You know, from end to end it's 100 milliseconds. Once again, these numbers are just totally just kind of observations. It will vary wildly from site to site. You also have media files in Wagtail, which Wagtail and Django cannot serve media files. They do not do that. It's a feature built into Django and Wagtail.

Speaker 1: that they cannot it's it's kind of weird when you think about it because they they have this feature that they cannot actually give you in a sense. So you need something else to serve those. So that's one of course they can serve it, you can make them do it, but uh under normal circumstances you you really should not. The other thing about Wagtail, and this is certainly worth saying, is that it is relatively well-behaved and uh bug-free for the most part. The same cannot be said for other frameworks, especially when you're talking about hosting them on a server environment and uh you know a framework decides to just start consuming memory sort of out of the blue and it's before you know it, it's up to six, seven gigs of memory.

Speaker 1: for serving uh you know a single page on a site and you're like what the heck's going on uh you're not really gonna get that with Wagtail itself. Um it's it's pretty predictable, it's pretty uh well behaved And now we get into the custom code. So this is what you're writing. This is what your app does. And this is total wildcard, you know. Every app is totally different. The developer can be allocating a lot of memory, a lot of CPU. They might be accessing the file system. They might be writing to var log because uh, you know some uh logging module never got updated and it's still hard coded to the vars log path or something like that. They might be making network requests, they might be downloading things or uploading things, uh

Speaker 1: APIs. You know, you have no control over this. Uh this is the developer , this is totally within the developer's control. Naturally, because of this, this is the really scary part as a hosting provider because you know if this part gets hacked or if it's malicious, it could totally wreak havoc on your system because it has just full access to everything. The next layer up in the stack is a WISGI. This stands for I believe Web Server Gateway Interface. And this is essentially what translates Python into Internet and Internet back into Python. So we all know the request object in Django, you know, self

Speaker 1: dot request or just request. You know, that object essentially is getting created by WISGI, which is taking the HTTP request, the text, the packets of TCPIP, and turning them into a nice Python object that you can use. So um because of this, it's you know almost always boilerplate, right? Well we never really touch our WISGI files. We use something like UWISGI or GUUNicorn because we don't want to write our own WISGI server. Web developers do not want to do WISGI. It's a chore. It's not something that you or is it really helpful to you at all? But because it is that thing that is kind of running your app or handling

Speaker 1: your app WISGI is actually the WISGI layer has an immense source of power to control the resources, to manage processes, to do it can do a lot of stuff. So a very fine-tuned WISGI server can actually totally change the behavior of your Django or Wagtail app. And then finally, the topmost layer of the stack is the web server. This is Apache or Nginx. Once again, this is usually boilerplate. You know, you're mostly As part, you know, your website is not going to be affected, you know, the the design and the function of your website is not going to be affected by you know how you configure your Nginx.

Speaker 1: As a developer, you don't really want to have to do this. You'd rather just have it work. This layer once again is also customizable because it can Do all kinds of things like upstream caching, proxying, you know, it this handles the SSL has to be configured correctly. So there's a lot of TDM involved in this and a lot of customizability that is really uh kind of unwanted but necessary. So okay, so knowing all of those parts and kind of cross-comparing that with the three different approaches I outlined Here's what we ended up deciding to do as the best way, kind of picking the best parts of each. So this is our setup This is what we ended up deciding on after trying many different ways of doing it.

Speaker 1: We like the operating system, we like the machine. We don't want to go totally serverless because we actually get a lot of good stuff from the server So we wanted to keep that operating system, files, you know, all that good stuff. On the other end of the spectrum, the web server, once again, this is something that can be configured relatively boilerplate. It's not running your code. It's not doing anything that's related to your code. It's simply passing, you know, passing from one layer to the next. So this is also something that we have a very you know custom customized version of and it runs on the machine. All the good stuff in the middle, so stuff like Python and WISGI, you know, once again

Speaker 1: This we've kind of condensed them into a single reusable container image. So the Python and the WISGI is managed actually not by the developer but by the host. So um, you know, when it's time to do a Python uh security update, you know, 3. 7. 6 comes out. Um We can push that one container image and manage one container image and that can go out to all of our client sites and they can all get the Python update without us having to touch the code for any of the sites. So that's a huge management kind of tedium win for doing lots of these sites. Same with the WISGI. That can be really fine-tuned and then set and kind of forget.

Speaker 1: And it will ensure that all of our sites work really well and we're not having to reconfigure our platform or write YAML configs or what have you for every single site that we do. In between is the custom stuff. This is the good stuff. This is the stuff we care about. What version of Wagtail, what version of the PIP packages, and our custom code? And really, that's all I want to do as a developer. I want to set my requirements. txt. I want to write my code, and that's the only thing I want to have to manage. I don't want to have to manage anything else. So those are actually not part of the container. That's your code that you store wherever you want. And those files actually live on the persistent secure

Speaker 1: reliable file system provided by the OS And they kind of get loaded into the container at runtime. So the Python environment is sort of separated from the actual Python code, is a way of thinking about it. And this approach has worked really well. So, okay, so we start with this, we scale it up, and we can now run many Wagtail sites. Here I have what, eight sites. We can run eight separate, totally separate Wagtail sites, totally separate code bases, but they could all be using identical configurations and identical versions of Python. And if we We push out a Python security update, all eight of our sites are going to get that update immediately without us touching them. Of course, there's a ton of security involved here to keep these things isolated.

Speaker 1: I'm not even going to get into the security just for the sake of time, but um now we scale and we scale. And we scale. And uh this scales pretty well. I'm going to scale even more and zoom out now because that's just one layer. So that's the VM, right? We've created this hybrid VM. lamp stack uh container you know kind of hybrid beast that scales really well and is really easy to manage. But that's just we're talking one server. There's still kind of that third approach with the cloud services. We're going to now add in the cloud services that really glue the whole thing together.

Speaker 1: So let's add in geographies and cloud services. We can use a cloud. We're built primarily on Azure, but this can translate to any cloud provider regions. We can run our servers in the regions. We use backup storage in the region and encrypted key vault. So like all the environment variables and everything for your app are treated as secret level. uh secret level information and they're stored encrypted in a key vault um and they get loaded at runtime, decrypted and loaded at runtime. So that's being handled by a cloud service and some of these the backups are taken and snapshots are stored for 180 days in a cloud-based backup service. If a machine totally goes down

Speaker 1: or if you yourself accidentally delete your files or you know whatever worst case scenario, we've got 180 snapshots to choose from that you can pull it back. We can immediately pull it back and load it onto a machine very quickly in a matter of seconds. So that's a cool that that's some of the cool cloud services coming into play. But but we still can't manage it. This is still just a huge beast of infrastructure with uh no, you know, we need it needs to be managed. So now comes the management layer. We've actually created a agent that runs on every machine, and the agent can totally control the containers, it can control the OS, it can control everything that it needs to control. And we also have the dashboard, which is kind of the stuff that I've been talking about

Speaker 1: beta. You know, we have a beta going on and you can try out the beta, which is the beta is primarily the dashboard, the infrastructure is pretty solid. But um and now we can talk to things so the dashboard could say hey I have a website I have a client website It's located in this region on this machine. Go to that machine and tell the agent to do something. Tell it to apply a security patch. Tell the agent to you know, reboot or reload my Python code or uh pip install for me or create a super user for me or something. And the agent will do it. And that's all controllable from the dashboard. We also plan to make a command line tool as well, but uh

Speaker 1: so the agents can also talk to each other like a real agent, like an actual person would do, you know. The agents can talk to each other. So I can go to agent, say my my client site is hosted on a server and I need to move it to another server to do maintenance or I need to clone it into a staging environment or I need to move it from one region into another region. I can uh you know tell the agent, hey, I want you to to move, you know, you talk to this other agent and move it to their system. And the agents can work between themselves to to do whatever they need to do to sort it out. So that is our our management infrastructure. Once again, uh the security, I'm not even going into here. There's a whole layer of security isolating every little part from every other part.

Speaker 1: Everything is 100% encrypted, in motion, at rest. The networks, you know, everything totally encrypted. Nothing, not a single site can spy on another site. They're totally isolated. So how did we build all this? You know, how did we how did this happen? Um the answer is uh magic. Just kidding. A lot of Python, wizardry, and a lot of Python. You know, this entire thing is pretty much built in Python. There's a little bit of Go code. We wanted to use Go more actually because it's exciting and it this is a basically the ideal use case for it, but it's never a good idea to give a bunch of Python developers

Speaker 1: you know, say, hey, let's do this really important thing in Go, uh, a language that you don't understand. So we stuck to Python and Python actually has really great bindings for doing OS level stuff and containerization stuff. And You know, Python has really amazing bindings into everything. So I'm just about at time. Like I said, this isn't beta. This is the end result, just so you could see what what it actually looks like is we we kind of Like like I say, WordPress has done this for a while, so you have sites like WP Engine or even GoDaddy now is coming around and Kinstar and all these different WordPress sites that it's like Go to a dashboard and click a button and you get your WordPress site. You know, log into the WordPress admin by clicking this button, those kind of things.

Speaker 1: And um that's not really something Heroku does. Heroku is more focused on giving you the infrastructure, kind of managing the infrastructure. That is awesome, but we wanted to do something more specific for our needs of Wagtail. So we don't just want infrastructure managed, we want everything including Wagtail to be managed. So I can just say, hey, upgrade my Wagtail, or hey, upgrade my Python, and it just does it without having to worry about it. So that that's really our need, and that's how we can very efficiently manage many different Wagtail websites without having to really do much development at all on them outside of their initial creation. So um anyhow, that's the end. I'll open it up to questions.

Speaker 1: Uh by the way, I'm on Twitter. I have not really been tweeting. I've just been on Slack, but uh That's my Twitter at the bottom in case you didn't notice. Feel free to connect or whatnot. So yeah, let me go back to my Zoom window here. Um

Speaker 2: thank you Vince. That was really great.

Speaker 3: Thanks, Vince.

Speaker 1: Okay.

Speaker 2: We're we're a little bit over time. We're just a minute over time. So and there is some discussion and and questions, but if it's right with the events, I suggest that we you you handle the questions on Slack.

Speaker 1: Yeah, sure. We'll do Slack and I'll be around as well after after uh lightning talks and stuff too. Happy to talk to anyone. So thank you.

Speaker 2: Thanks, Vince. It's really great.