One Thousand and One Django Sites

Speaker 1: Hello. Thank you. Thanks everybody. Um Before I begin my talk, I did not come alone. I brought a friend. This is Django Pony. And you can um uh whoever donates to the DSF first will win this Django Pony, and you can talk to my colleague Thibaut and Sarah. They're both wearing the bright green shirts back there. So if you would like to have a Django Pony of your own, you can donate to the DSF. Okay, so uh 1001 nights. Actually, we're going to be talking about 1001 sites, specifically uh Django sites. So Uh and what I'm going to do is tell you a story here.

Speaker 1: So, once upon a time, as all good stories start. There was an agency named Code Red, which is where I work. Building many Django and Wagtail sites, they must host them in a manner that would be pleasing to clients and developers alike. As the sites numbered in the dozens, then the hundreds, the magical lamp stack became burdensome. Weary with maintenance, fatigued by three-letter acronyms from the Almighty Cloud Providers They set sail for a new platform. So this will be a story of adventure, monsters.

Speaker 1: wisdom and what we've learned on our journey to hosting uh a thousand Django sites. So the background is um you know I work at uh Code Red And we have a Django and Wagtail hosting platform, which is relatively new. This is Coderad. cloud. And recently, as of last year, we crossed a milestone of hosting over a thousand Django sites. uh Django and Wagtail. So it's very exciting and I just wanted to share kind of the journey. And just real quick shout out to Wagtail because I'm a core contributor to that. And We do a ton of wagtail. So if you if you're not familiar with what that is, this is what it looks like. It's basically a Django, uh CMS built on top of Django. So it's pretty much

Speaker 1: the same as Django for hosting purposes. So the first uh story that we're going to look at is Sinbad the Sailor. And this is about how we found our technical architecture. So in the beginning, uh Sinbad is a young man, he's looking to make his way in the world, and like many famous great leaders of antiquity, uh he decides to use Django. So armed with Django, he seeks out fame, fortune, and riches, going to use Django to do this. On his very first voyage, he develops a few Django sites for uh clients, and it's a great success. It's a perfect fit. Setting sail to host his first site, he chooses the traditional

Speaker 1: time-honored uh lamp stack, right? This is the Linux Apache uh MySQL or Postgres and um P is uh instead of PHP we're going to use Python. So uh the first voyage, the seas are choppy, the Linux is unforgiving. But the Django sites run admirably. An excellent choice, he exclaims. But Sinbad has yet to encounter his first real challenge. And that is the den of pythons. So as more sites keep being added and created, uh Sinbad quickly learns that each has Wildly different resource usage,

Speaker 1: different memory space, CPU time, different Python interpreters, different Python versions. All of this has to be managed and restricted Uh and it's a very difficult thing to do on one server. Uh and it has to be different for every single site. They have to be separated. So how do we tackle this challenge? How do we tame these pythons? Well, to start out, Sinbad employs the tools of his forefathers. And that is mod WISGI , subprocess, virtual ENVs , various system quotas, right? So you can set this up, you can isolate uh different um uh websites on an Apache server using um you know WISGI and virtually

Speaker 1: NVs and it will give you a very basic level of isolization. or uh isolation, sorry. Um, but it still proves to be quite complex of a challenge. Um, you know. Apache processes can be interfered with by, you know, the other Apache processes of one site can consume too much resource on the server. for another site. So now you're looking at just running separate servers for all of these sites. And you know even with these tools, it's still uh kind of a difficult solution to manage when you have more than one site But in the distance they spot a great whale. Seeking safe haven, Sinbad discovers the island of Docker.

Speaker 1: At first, it appears to be a land of ease in development, the rich fruits, everything is good. But in production, the island begins to rumble It turns out that this was not actually an island, but it was just the tip of a great big whale. And the whale in disguise destroys Sinbad's boat and much of his crew. And I just thought that was a perfect analogy for Docker. So I had to use it. Eventually Docker is tamed and put to good use. So it basically what happens here is You know, Docker is pretty easy to use and it's a good solution, but when you start managing many different websites with Docker, you can still run into some of the same problems, right?

Speaker 1: How do you configure, you know, are you just managing a bunch of images? Do you need to get into container registries? You know, um having all of these sites, um, okay, getting ahead of myself, but uh, you know Having all of these sites just can create even if you containerize them in Docker, you still have the problem of now instead of managing a bunch of Apache, you're managing a bunch of Docker containers. So You might be thinking to yourself, what about Kubernetes? But uh Kubernetes is more of a tool for uh if you are running many copies of the same site. So I have a bunch of workers, uh maybe I'm Netflix and a new episode of uh something, you know

Speaker 1: White Lotus or whatever, uh something drops and it's like, okay, everyone's watching that, so I need now I need to double the number of Netflix workers I have to handle the traffic. That's a great case for Kubernetes. But if you're an agency and you're building many different websites, you have to have those totally separate. You know, that Kubernetes is not going to help you with with with running uh different things like that. So um we just ended up going for a completely different approach, which is using Docker purely as an environment. So uh we have actually a single image for each version of Python, and that is it. Every single customer uses that exact same image.

Speaker 1: and their code is directly mounted into that image at runtime. So we've basically created a like a perfected Python environment for every different use case. And then all of this code can be managed separately and you know using a reusable environment. So that has actually worked really well. It's automated by an agent that's on our system. But There's still the problem, you know, once you get to the next level up in the chain, so now you've got all of these managed, but now how do you manage all of these different code bases and all of these different servers that are needed To run this uh Docker monstrosity. So uh we've ended up creating an actual uh agent that runs on the operating system on the host.

Speaker 1: This is also written in Python And this is able to manage all types of Docker containers. It's able to manage resources on the system, the file system permissions, everything. So now when we want to queue up a new uh environment, uh we can make a call to the agent and say, Give me a Python 3. 9 or Python 3. 13 with Django installed 5. 2 on the server with the highest amount of resources available with X number of workers and the agent can find the server that meets that criteria, spin up a site, and pip install everything into that new environment that it just created. So this is kind of

Speaker 1: our goal was always to make like the world's easiest Django deployment or just the simplest Django deployment that's basically just pure Django. I don't want to have to write YAML. I don't want to have to write manage a bunch of Docker scripts. I don't want to have to, you know, do any of that kind of stuff. So right here you see A very basic Django site, right? There's a manage. py, there's a requirements. txt. Like this is pretty much what you get when you run Django admin start. Or in Wagtail's case, Wagtail start, and it gives you a similar boilerplate. So having done that, uh let me just show you kind of what we this is what we ended up with. So I've got my default Django

Speaker 1: site. I'm going to run CR deploy, which is our command line tool that that kind of orchestrates all of this. So it's going to connect to the first available server. It's going to copy that code up to the server. It's going to queue a deployment, which is based on this per version of Python that I specified. And uh in a few more seconds. You can see that it is now getting these deployment logs. So you can see the output of pip install. You can see it running migrations. It just collected static and now the site is done in uh 19-20 seconds roughly. So that is our total end-to-end process. We can deploy a Django site in about 20 to 30 seconds

Speaker 1: from zero to a fully working site with SSL and everything running on the internet. That was like a huge accomplishment for us, and it took us many years to get to that point. So yeah, awesome. Thank you. Thank you. So but uh so that's kind of the technical side of things, which is what you know we as developers usually focus on. But there's more to running a website than just the infrastructure, right? Anyone who runs a site will will know that you start running into uh some similar problems that Aladdin ran into, which is Your wish is my command. Business challenges that come along with managing all of these sites

Speaker 1: So for Aladdin, he starts out trapped in a cave. He receives a visit from his first client, who is a magical sorcerer. The sorcerer promises to reward him with riches. A philadin can help him with a difficult task, and that task is actually fetching a magic lamp out of the cave. But soon, trying to complete this task, he finds himself trapped and stuck in the cave. So if you've ever engaged, uh you know, if you're an agency or a freelancer, you may have found yourself uh trapped in a contract. unknowingly. As Aladdin soon learns, web hosting requires an immense number of what I

Speaker 1: call assumed services, right? So things like 24-7, 365, uh, you know, having a 99. 9 % SLA. You know, is that something you really think of when you're you know initially hosting a website? Um There has to be disaster recovery, right? What happens if the site goes down? How do you handle that? Do you have a plan established for that? And part of disaster recovery is not just going down, right? We usually think, oh, uh, you know, AWS, uh US East One went down, we better fail over to another region. But that doesn't happen very often, right? The disaster recovery that happens every single day is somebody makes a bad deployment.

Speaker 1: Or a client deletes a bunch of stuff off their, you know, from their database, or they delete a bunch of images or something, and they're like, oh, I didn't mean to do that. Can I get them back? And a lot of times, you know, that is considered a disaster because it may cause an outage or it may cause part of the site to be broken. So do you have a plan for having backups or snapshots that you can easily roll back data or roll back code when something happens like that? So to have very minimal downtime And a lot of times you need tools for this. So how about upgrading Python versions? Right? If you're just running on a regular Linux server, uh usually that Linux ships with one version of Python, you know, and uh especially something like Debian

Speaker 1: or Red Hat, they're a little bit uh slower to adopt new versions. So how do you upgrade from one version to the next? So there's just a lot of things like that that come with hosting a website that you don't initially think of until you know things start to go wrong. In Aladdin's case, he summons the jinn or the genie. He quickly learns the financial burden of providing these services. It grows exponentially over time. More and more maintenance, you know, is required on a day-to-day basis. The only s solution is to summon uh superhuman levels of efficiency. And in his case, that is the gin. But uh we will use automation instead. So uh the things that we automate uh to reduce the amount of maintenance uh day-to-day

Speaker 1: are uh things like creating servers, creating containers, you know that's through the deployment process that I showed you, creating databases, all that happens automatically, creating SSL certificates, updating DNS, all that stuff happens automatically. But error detection. Right, so if some if a site uh if a monitor detects that a site is uh no longer responding, uh sometimes there's some quick fixes you can do, right? You can restart the services, uh you can Is it a network issue? Is it a database issue? Or is it throwing actual errors like 500 errors? And that's why it's considered down, you know? And in some cases, you can just kind of automatically heal that by saying, oh, it's using all of its available memory or all of its available CPU.

Speaker 1: Let's just kill the process and restart the server and then the site only has to be down for 20-30 seconds and you know no one really will have to notice it it will kind of heal itself. So tools for doing this so that people aren't having to constantly do this on all of these different sites. And things like security. Monitoring traffic is a huge one. You know, bots, and I'll talk more about bots in a little bit, but things like that can also uh take down a site or just destroy the performance and now a person has to get involved once again and this stuff happens on a daily basis when when you have more and more sites. So the last story that I will tell is that of Alibaba and the 40 thieves.

Speaker 1: And this is the spammers, bots, and uh malicious uh behavior that can bring a site down to its knees. So it all starts with open sesame. Alibaba sees that a large portion of web traffic is evil bots. Attempting to brute force URLs. Here's a fun fact about Wagtail, and some Django sites are susceptible of this as well. You can actually quite easily uh take down a Wagtail site by just hitting it really hard with a lot of 404s. And that is because um For example, XMLRPC. php. That's most likely going to 404 on your Django or Wagtail site.

Speaker 1: Right. But that is a major, like number one vulnerability in a WordPress site is that specific URL. So guess what all these bots are going to do? They're just going to start hammering XMLRPC. php to see if they can find it or they can make it do different things or they can try and exploit a vulnerability. So your site is probably going to get a huge amount of 404 traffic on that specific URL. And in the case of uh if you're using a content management system It's probably going to look at all the pages in the database, say, oh, none of these pages exist. Let me check the redirects table, look at all the redirects in the database, and it has to look for all of them because it's not going to find it and say that doesn't exist. Let me check the Django URLs.

Speaker 1: No, that doesn't exist. Okay, now now I've just done all of these really expensive operations to determine that something doesn't exist and it's a 404 So um, you know, 404s can be quite expensive actually, and when you have uh bots just hitting them nonstop. you could see 100% resource usage all being dedicated to just serving 404s. So these are just some really odd things that you find that you know bots can inadvertently take down a site So Alibaba must find a way to stop these legion of bots. But how do you distinguish bot traffic from human traffic? Alibaba writes a sentinel middleware in the reverse proxy to watch all traffic identify patterns and block the bots.

Speaker 1: So this is where we're just having to get more and more complex. It's almost like a war against you know bots versus uh just keeping the site up. So in our case we ended up having to do something like this. It's sort of like a rate limiter, but um mainly it just watches traffic, looks for certain patterns, and when something seems to be falling into a really bad pattern where it's making too many requests uh in a certain you know way We will just temporarily block them for five minutes. And a lot of times that's enough to just make it go away without causing any harm to someone who may trigger it with a false positive. But that was only part of the problem. The worst part, arguably, is the bad humans Uh the 40 thieves use

Speaker 1: fraudulent info to open accounts and abuse and steal resources. So at this point in age, a real email addresses are meaningless. Uh real phone numbers are meaningless because you can generate a Gmail account with a Google Voice in, you know, 30 seconds. And those are both real valid things that can be used for two-factor authentication and You know, so so those really don't mean anything. People can use that to bypass terms of service, right? And every website, if you have a sign-up form on your website, People are probably signing up with fraudulent info to try and steal your content or abuse whatever services you may be offering. In our case, we ended in this was, you know, we had a cap

Speaker 1: show, we had everything, but these were like real people clicking through and just creating hundreds of accounts and uh doing more bad things that I'll show you in a second In our case, we ended up finding a really great service that I'd like to share with you called MaxMind, and they have a fraud detection service where you can basically pump all of this information into it, like request uh browser, you know, user agent, browser info, as well as their you know email address and they will analyze that and kind of cross-reference it with you know, known databases of people who are being malicious and actually score the result and say this is a 15% chance of being malicious. So, you know, this uh is is nice and you can use this. Uh I am not a robot on some requests, but you can't use this across your entire site.

Speaker 1: And even more so, uh, I am not a bad person. This doesn't exist as far as I'm aware, so it becomes you know even more difficult to solve these kind of problems. So Like I said, we used uh MaxMind and they have a MinFraud service that can be used to detect fraudulent user info. So this was a really huge uh help for us. And just a couple more left. So the last problem, this is a newer one that started happening this year, and that is AI training. So this is just kind of a look at this is a 30-day snapshot of some traffic. Um and you could see in you know in comparison to the the graph there's roughly about 10% of traffic at all times is bots. And These are normal bots, you know, Bing, Google,

Speaker 1: crawlers, SEO tools, what have you, and they're fine. They they don't you know abuse the site, they follow the robots. txt, they're pretty well behaved, right? So uh an anomaly that we started seeing recently is a new bot that showed up. And uh if anyone here works at meta slash Facebook, I have a few words for you. But uh we actually were having some customers get fully DDoSed. By Facebook, by their training AP uh bots. So we were seeing roughly 60% of traffic was just coming uh non-stop from like hundreds of different IP addresses all for AI training.

Speaker 1: And for some sites that had slower, you know, large amounts of content or slower, you know, big databases and things like that that weren't optimized. It was uh just completely taking down the site because they couldn't handle the volume of traffic. You know. So um the reason you see dips in that is because we had just had to systematically block them uh temporarily for certain clients to keep their sites up, you know, because it was just totally unexpected traffic. So uh but you know they keep finding new IPs, they keep coming back. So this is like a new Challenge that we have yet to totally uh figure out exactly how to work around. These challenges keep happening every single day So and the very last one that I'll show you is uh actual code injection attacks.

Speaker 1: This is another one that happens. Um so can your manage. p file uh manage. py file download binary executables Can it open an HTTP HTTP proxy to evade law enforcement? Can it run a Tor network? Can it mine Bitcoin? Uh yes it can. So Python manage. py migrate, right? This is something we all run every day. But the question is, what is in your manage. py file? This is actually a real example which has been slimmed down, but um You know, this seems unassuming, right? Manage PY Django's command line utility for administrative tasks. This is actually downloading Node.

Speaker 1: js and unpacking it and running it on the server. And uh it could be used to run all kinds of nasty things. So this is not something that should be happening on a Django site. So this is you know one of the many uh things that's when you talk about bad people getting in and just trying to abuse services, code injection is a big one that you have to look out for. So uh anyhow. This is just kind of a summary of some of the things we've uh had to deal with, hosting 1,001 Django sites. Thank you very much.

Speaker 2: Thank you for your talk. I was wondering how you'd respond to customers that let's say have these Node. js Intact injection scripts into stuff that makes it to your environment.

Speaker 1: Yeah.

Speaker 2: Uh if they end up having a vulnerability, what are the actions you take? And how do you work with the customer to overcome them?

Speaker 1: Yeah, great question. So it in our case, those are almost always coming from the customers themselves who are uh, you know trying to do something bad that they shouldn't be doing. Code injection is uh has has never happened at that level. for legitimate customers because they are deploying their code, you know, from a version control system or something. But you know, it's just kind of a reminder that if you if you allow people the ability to upload their own uh code, they you will get some very nasty code. So we just uh monitor processes and things like that on the servers and if an an anomaly comes up uh then we can uh immediately you know either contact the customer or suspend the account.

Speaker 2: Thank you.

Speaker 1: Yeah.

Speaker 3: Hi, thanks for the talk. I noticed you mentioned that you heard what you serve WordPress as well as Django websites. Yeah. Could you talk talk a little bit about how Managing the two of those is different from the perspective of someone who owns this a platform like this?

Speaker 1: Yeah, definitely. So um we actually manage them quite uh pretty much in the same way. We use that similar environment approach for both WordPress and Django. So the Django sites have a kind of a kind of perfected golden Python environment. And the WordPress sites have sort of a golden PHP environment that's also pre-configured for that kind of site. Both of them get the exact same permissions applied, so they get locked down and specific file system permissions, um and then the code is loaded in at runtime and the process starts. So they actually behave pretty much the same. Um and then we, you know, constantly are doing patches for PHP and Python. And our plan is also to add, you know.

Speaker 1: uh maybe Node. js or Ruby or some other uh frameworks in the future. Uh we just started with Django and WordPress because those are the vast majority of sites that we're working on.

Speaker 4: Hi, thank you for the talk. Uh could you manage uh could you comment a bit on uh how you manage uh high availability in this case? So basically if you have one site and then you spawn like more of the two. Increasing number of plans. Thank you.

Speaker 1: Yeah, high availability. Um so high availability is one of those things that we kind of think about as In some cases, it's over engineering, right? It depends on the the customer. So we found that actually for most customers 99. 9 % is a perfectly adequate ability availability for, you know, within their budget, right? So in that case, we have a process that we can we can pretty much be guaranteed within that. If they need higher availability, then that's something where we're doing multi-systems or multi-regions, and it's a very similar process to that, but it obviously does double the amount of resources that's required on the back end. And we also uh have customers have staging sites as well.

Speaker 1: So it can triple or quadruple the amount of resources for the high availability. But yeah, it's actually quite a similar process. The same environment image is used in both environments and the same code like requirements. txt is put into both environments. Um so yeah, and then it can be load balanced uh or or have a failover type system. Yeah.

Speaker 4: Great, thank you. Um

Speaker 5: given that Docker itself is not usually a security primitive, um how are you um isolating around Docker to um secure uh the users from each other.

Speaker 1: Great question. So everything within our system is like really like isolation is kind of the first uh principle that we follow. So everything, each uh site will have first of all its own uh Linux user and its own sort of truded environment on the host. as well as its own um truded file system and that's basically where uh trude is basically where slash equals there Directory, so they can't get out of that. Then beyond that, we have a Docker being run as limited users for that specific user and as well as an isolated network for every client. And then within the container itself, they also have an isolated user within the container. So there's many different layers

Speaker 1: of isolation involved And uh it's worked pretty well for us so far. Yeah, and then for the the biggest uh most secure uh sensitive ones will still be on a dedicated server. as well.

Speaker 6: I wanted to ask uh do you only uh use Docker like Docker Compose with the uh PostgreSQL um service so that you have a whole stack or have you just one Database for all the Docker containers?

Speaker 1: Yeah, great question. So we're kind of in the process of Dockerizing databases. Right now we use high availability databases and have those kind of isolated on a dedicated database server. And then those are isolated on that server. So they're kind of separate right now. But we're we've been exploring, kind of coupling them more with a Docker-compose type scenario. But we haven't done that yet because mainly because of resource constraints, because databases need a lot more memory and uh I. O. you know, than a website does. So putting those on the same server doesn't always make sense because the server is not you know optimized for different things. So yeah, that's been kind of a work in progress.

Speaker 6: Okay, thank you.

Speaker 7: Okay, thank you.

Speaker 1: Thank you.