Real World Django Q&A

Um

Speaker 1: so yeah, we're here to talk about Django and the Real World, which is sort of the Mostly issues about scale and performance using within your Django app. Before we get started, I'll have the panel introduce themselves. I'm Jacob Birch, I'm an engineer at RevSys.

Speaker 2: Hello, uh I'm Honza Kral. I I was I sadly no uh don't work with Django anymore, but uh I was an engineer at Whiskey Media that uh later become Berman Braun, so I have uh experience with running uh running larger content websites.

Speaker 3: I am Frank Wiles, I'm the president of Revolution Systems and we help companies with scalability and performance problems around Django.

Speaker 4: Peter Baumgartner, founder at Lincoln Loop, and we both build large-scale Django projects for clients and also help them out of performance jams as well. I'm Andrew Golwin. I'm a senior software engineer at Eventbrite and we sort of have a big like transactional rather than content kind of issues with scoping. Uh

Speaker 1: so before we get started, I'll just sort of lay out the format of this. We're gonna try to do 15 to 25 minutes of um me asking these guys a bunch of questions. really common questions that come up. And we're hoping to have a longer 15 to 20 minute Q<unk>A session for you guys to ask any questions you want of these guys. I want to start. Problems of scalability and performance are often sort of ton-in-cheek labeled as happy problems. If you're having a lot of data or you have a lot of customers and you have a lot of bandwidth, a lot of traffic, uh these are usually good problems to have Because of this, engineers and data architects and software architects will often prematurely optimize, even worse, worry about premature optimization. And so what advice would you get people before they even run

Speaker 1: Start Project? What should they actually spend time on and make sure they get right the first time? And what does it matter until they actually have the happy problems And so yeah, well, Peter, you can start.

Speaker 2: Well, I would say early on you definitely want to optimize for developer productivity. Um, your goal is to build features and and get this thing out in the wild as soon as possible. Um

Speaker 5: the the things you want to avoid are making technology decisions that are gonna you know shoot you in the foot uh later on. So um I I would say you know You don't need uh, you know, if you if you think you're gonna scale, you don't need to grab some like crazy esoteric database that uh, you know, you you think is going to solve your scalability problems. I would say the probably the biggest thing is to stick to the kind of known tried and true solutions. Don't reinvent the wheel and keep in mind and you can scale uh with bigger boxes for a pretty long time until uh your your scalability problems become code problems. Uh so um yeah Yeah, you know, use stuff like Postgres and uh you know

Speaker 5: a good WSBI server or an Nginx or Varnish. Um these things will get you really far. So uh I would say don't you know don't sweat it too soon and don't use uh But don't pull in some crazy technology just because you think you're going to need it.

Speaker 2: One hundred percent agree and I would add to that get into habit of monitoring system uh systems and looking into how things go. Once you move into production you should should always have record how things were, how things are now, and where do you spend most of the time or what what component is giving you the the most problem. Even if you start with absolutely standard stuff. You should have this in place so you can later identify what's the problem and you'll never have to sort of guess and assume that this will help you or that that's a problem and that's the component that you need to rewrite. And

Speaker 4: sort of feeding into that as well, like I've seen a lot of code that was written to be very clever and then six months, let's be startup, six months online. the people or the the sort of the knowledge of that has moved on and you go back like this is suddenly a massive technical debt. So like I would heavily encourage simple solutions within well-known problems and well-known technology because If you want to bring in people or bring in sort of XML contractors, you want them to know what's going on as fast as possible. By giving them this strange STO environment with special new modules and a new database type you made yourself, you're taking yourself your own hole at that point, I think.

Speaker 3: So everything GitLab said is absolutely true and and you should absolutely do those things, but I think that it's kind of a natural peak inclination to want to figure out what's the what's the fastest way to do this. So I often will think about what happens when. Is there a clean, easy way to shard this data when we get there? Let's game that out. Let's make sure we don't write ourselves into a corner so that it makes that impossible twelve months down the road. Or you know what, if I denormalize this way I will be able, you know, my I know my access patterns are going to be like this, and if I denormalize this and I use Redis, then boom, I I've got a whole bunch more performance. I don't actually go write that code. I just kind of whiteboard and game out. what what would I do when I get there? And so that satisfies the the geek in me that wants to know like what we would do.

Speaker 3: And then we also kind of have a plan for when we get there, but we don't set up you know technological roadblocks that make everything slow for the early development.

Speaker 1: So I want to talk about the other side of things. When you have a site or an app that you You did those things, you said it right, and now it's against the wall. Uh it hit the fan and it um it got slashed dotted, it got fireballed, and all of a sudden uh Things are going well. But before we do that, I was talking to Hansa yesterday and we wanted to make sure we sort of clarified a key vocab issue that comes up a lot when you start talking to people. um about performance or scalability and that's what is performance and what is scalability. They're often interchanged with they all they often get using conversations with My website is slow, how do I make it not slow? Or I have a billion rows of data and I can't fit it anymore. Um so Hans, since we talked about it, do you want to address that first?

Speaker 2: Yeah, so uh performance is how fast can uh how fast can you serve your content? And that's very important for user experience and for general like operation side And it it helps the scalability, but it's not the same thing. Scalability is the question, will I be able to serve Ten times, hundred times, a thousand times the amount of transactions, people, or content based on my on my current stack. So that's the scale I will it scale to uh serve more And it doesn't necessarily uh have to be the same as performance. You can have systems that are that are super fast but don't scale at all. And you can have systems that are fairly slow but will scale infinitely.

Speaker 2: For example, if you chose to use S3 as your as your database, it will scale almost indefinitely, but it will never be really fast. So anything to add to that?

Speaker 4: I think it's also sort of a difference in the approach. So certainly the Novemberite performance for us is sort of like, you know, we get a 5-7 % -10% speed gain if we're lucky out of the main rendering flow and that that buys us a few months of headway. But the real sort of future planning comes from scalability. Like you know, we need to pl plan to serve ten, a hundred X people. A 10% speed increase is great, but it's only a short-term solution. Like you can't get by growing forever on just a series of small increases. You have to be able to have that big switch. And so you know, like when we we did a thing with Django's road expression URL server gained us five, ten percent, which is fine, but then you know the server resources we gained were gone in like two months from that, because we grow at that speed. It's important, I think, to sort of see them as different kind of things. Sometimes you need performance gains, sometimes you don't.

Speaker 4: Cool.

Speaker 1: Um so yeah, let's go back to websites that R actually need some of these sort of early gains of they either got really popular or uh all of a sudden they have a lot of customer data that they need to store right away and um all they're observing is we have this cool app and now things are slow uh now the website is crashing. Um what do we do? Uh what are um Where where should people look first when this happens and what kind of questions should they be asking themselves and where should they look to fix their problems? So

Speaker 2: as I said earlier, into their monitoring, they should they should know they should have been monitoring all along like and seeing it get to get to that place so they will know which part is the problem.

Speaker 1: No, this is jingo in the real world, so let's say that isn't true.

Speaker 2: Yeah, so uh so in that case experiment and try, try to identify the the the piece that's giving you trouble, replace that. And the important part is if you replace something, for example, you In the real world, we found that we have a problem when Google hits our site and they try to paginate to the 100,000th page. We literally had 100,000 pages And uh it was killing Postgres, uh obviously, because it was sorted by three different three different ways. It was a huge data set So we replaced that with pagination with the help of Fredis. So we would use Fredis to paginate and then just use Postgres to retrieve the rows. And then we went and saw where else can we use this pattern to alleviate the uh the pain that we have?

Speaker 2: So once we f once we fixed one instance we say oh now we have a different tool. What what other uses for this tool we also have where it might help us. So this is also always a second step that not many people actually actually do

Speaker 5: Yeah, I would say if you don't have good data, just uh going in and looking at the load on your servers, uh, you know, is your database server overheating? Is your app server overheating? Figuring out where the hotspot is, uh, just on the big scale, is it, you know? of my database is my application and then uh you know attacking it from that angle. If you can scale up vertically, that's uh you know a a quick, easy win. Throw some money at the problem, throw some hardware at it. Uh otherwise you know sometimes uh really kind of quick and dirty caching solutions like uh database query cache if your database is uh getting hammered there's um stuff like uh let's see we use johnny cache um cache machine cache ops django cache ops there's there's a few out there that will just uh you know uh

Speaker 5: a few lines of code uh can take off a a ton uh of pressure from your database. And then the same goes if your if your app server is overheating, yeah you can add more app servers or you throw a cache out in front of it, put put varnish uh in front of it, and just have it cache all your anonymous users. We were just talking about this earlier. It doesn't need to be for hours, it could be for five minutes, and that might be enough to alleviate a lot of pressure

Speaker 1: Cool. Um so uh so far we've talked about um sort of the ecosystem around a Django app. We've heard uh Redis mentioned, Varnish mentioned. uh memcash mentioned. Um is there anything missing in Django up to Django 1. 7 uh that you feel should be uh in core to help each other growth. It's sort of always been Django's mandate to be very easy to get started and possible to scale out. Do you feel there's anywhere in Django that is either not in core and needs to be sort of smoothed out along that way?

Speaker 4: Um so there's a couple of things I have my eye on for a while. So I know um Hans as well, the the cue abstraction is one of Hans's sort of ongoing uh uh things. I'd also love some kind of um message bus abstraction Um because in particular Mentbrite, as we're putting things to separate services, some kind of way like if we had had that in there in the first place and said that you know these things like Casselery is is great for sort of moving tasks off, but it's not enough like cache and validation and stuff So like I would love to see some kind of decent way of sending messages to multiple targets in Django with say like hey we have made new events we can evaluate new pages we have sold the tickets and we can evaluate accounts and stuff Like that to me is almost the missing piece, at least for the things we're facing now. Um I'm not sure we'd help a small site, but perhaps it plays into like you know like live data binding, like you know you see

Speaker 5: I think there's some improvements that could be made to the cache framework in general. That's a piece that we seem to override a lot. One thing like uh on a high traffic site if you're using the cache framework quite a bit, um you tend to see uh these spikes as as your cache invalidates, um, spikes on uh you know your application. application and then you know everything gets cached and it goes back down and you get into this kind of wave. So one thing we do find ourselves doing is just adding like jitter into the cache timeout. So you know a cash timeout varies by 10% and you don't have all your content invalidating at the same time. There's a couple other things I'm uh slipping my mind, but if anybody has any notes on that.

Speaker 3: I know the trend is to try and take things out of core and put them out, uh but I think that's something a little a little more instrumentation in decor. So things like if if run server as part of you know the logs showed you how many queries were run. Something that Django Debug toolbar does great, but something that was built into the system We could have a little bit better probably testing unit testing tools around some of those things. I know I end up writing like a, you know A certain num of queries is less than this amount. Like I I don't need to have it know that it's exactly twelve. It's it's it wasn't in one six. Yeah. There's a certain number of queries, like that it's exactly twelve, but I just want to say, hey, you know what? I only want to have this fail if it jumps over fifty. Something like that. We could get a little bit better around those

Speaker 3: those kinds of things. Cool. I

Speaker 2: just like to say that uh a lot of the things that have been said here actually don't need to be in in Jennifer. For example what Peter said with it adding adding jitter, it's something that you don't have to k worry about at the beginning and once you see this behavior You can just grab an off-the-shelf cash backend from GitHub from PyPI and just and just plug it in and it's a drop-in replacement. It's also something that you don't, for example, use in development or in in your staging environment. So those are the ways how how you can progress. You start with a with a na naive Vendel on Django and then you uh start replacing components, for example, for session storage or for the caching. Uh we also had our uh our own uh cache backend which actually does like uh the thundering

Speaker 2: herd protection that it will randomly not return the cache value once so it will get regenerated in the background instead of just uh having potentially multiple threads trying to do the same work and so on and so forth. We also expose a lot more because we use Redis as our cache. So we expose a lot more functionality there. Uh instead of just the increment that that the cash backend has, we uh we do more. And that's something that you can introduce later. So it's Oh, to tie to the previous questions, like you don't have to worry about this in the beginning, but know that these are the places that give you a lot of uh a lot of potential to do better.

Speaker 5: Yeah, I I think instrumentation in general is um something like even production instrument instrumentation. Uh uh I know um you guys at Lanyard are using some kind of like backwards methods to get into you know how many how many queries did this request fire off and then you know I want to store that data or I want to send it to Stats D or something something like that um so so I can track that uh performance over time. Um it would be nice to to have some some better hooks into that uh information in production.

Speaker 4: Yeah, so the the thing I think you talk about tiki bar um I imagine or the remember the the the remember it has something c bar which is similar to what you described where like we track the queries you run, that kind of stuff. Um There is definitely a lack of hooks in Django to have a easy third-party drop-in for implementation. And one of the things that now migrations is much more done um I'd like to look at is having those hooks in there so like you can say like okay I do want to see every query that comes through in this list or I do want to like wrap this in a timer and that kind of stuff. Like I'd like to know how long my URL resolving takes or how long my database calls take, how many how long the cache calls take. So if we can have some easy solution back that doesn't increase the overhead, which is the key problem, because Django gets unfortunately a little bit slower every release, then that would be a good thing to look at

Speaker 1: So this will be uh the last question before we open it up to the floor. Um four years ago, actually, not in this room, I was on the other side of the river, but I moderated a panel uh on those SQL solutions at DjangoCon. Um and it was such a popular topic that it was a plenary session.

Speaker 3: It was everybody. Um four years since then of of wonderful NoSQL pro progress. It barely warrants a mention in this talk. Why do you think that is?

Speaker 1: Was the hype real? Yeah, I'll open it from there.

Speaker 4: First of all, the term no sequel is objectionable to me. Thank you Um so yes, I mean there was especially at that time the prevalence of things. So like MongoDB is by far and away the most prevalent thing at that point in history. I think what's happened since then is that people have gone through that process have seen that what they're exchanging what they're giving up in exchange for that kind of you know document storage is some things that you kind of rely on later on when you're scaling like things like indexing and easy changes and consistent data formats. And also at some point it's kind of become more of a shoot. Like you know, Postgres now has pretty much first class support for schematic stuff.

Speaker 4: So I think it's kind of a combination of that. And then also Things like Redis and and Cassandra and React just became normal parts of the low infrastructure. Like the the stuff the stuff that proved itself just became almost assumed. Like I I still see talks about that stuff at conferences where people are like, well, you know, obviously we use Redis and we use Postgres and we use use you know Elastic Social Cyber because those are the three very complementary things. Like they can't do each other's jobs. You can't you know you could write a search engine in Reddit if you wanted to, but That's crazy talk. So I think that's kind of like the we've settled down almost on the sort of more decided set of bit of work. Uh

Speaker 5: yeah, I think the you know the talk at the time was, you know, how are we going to get uh the admin to work on MongoDB or Redis? Um yeah, I think we've now learned that you don't there's no need to do that. You don't need to use these as your primary data store. Um for most people, most scenarios a relational database system is what you want to use for your primary data store, but there's all sorts of peripheral needs where those types of databases come in really, really handy. Search and sorting and uh the kind of stuff that relational databases don't do as well.

Speaker 2: And also like if you use something else than relational data store , you don't want to treat it as a relational data store, so there is no reason to pretend to hide it behind the same API to use the the Django query set to query Elasticsearch or Redis. That that in that case like The different data store gives you nothing because you cannot use uh its entire power, you're limited to what a what an what a SQL database could do. So it doesn't make much sense. And people have realized like the the ultimate promise of the NoSQL movement, I'm sorry, was like choose the right tool for the job. So if you have a search problem, use use a search engine. If you just want something quick and easy and

Speaker 2: very easily updatable, use Thredis. If you want something persistent and reliable, use Postgres. And like you can mix and match those.

Speaker 4: Just one one adjunct to the admin conversation. Um one of this year's GSO projects has been about exposing non-model things to the admin. Um so yeah For example, there has been a um I think it was Rust Gil student who was doing like they had they had Gmail backed onto the admin. So like some progress has been made in that kind of area separate from the whole sort of NoSQL movement as a as a whole thing but you know it's it's it's a different kind of thing. The admin is a whole nother topic and a whole other paddle, I think.

Speaker 1: All right, so we'll open the floor to questions. We've got about a little over 20 minutes for them. So any questions you have for these guys, please come up. The microphone is on my right, your left. Oh, we have both mics. I'm sorry. I'm totally wrong. Either side works.

Speaker 6: Hey gentlemen, thanks for the awesome panel. I'm wondering, I I think Without exception, we heard about uh elements of the app stack and their implications in performance and scalability. Uh I'm kind of inclined to ask the same start project question that started this panel, but for the deployment tooling. Is it better, do you think, to use Do you use the lit like a company starting today, a small project starting today? Do they do like Jenkins from day one and Ansible and Docker from day one? Do you use the hot the whatever is hot and new the day that you start and what are the risks of that becoming a tradition and and so on?

Speaker 4: So I I think this feeds into my my one of my first responses that you should use whatever is common and isn't gonna get like you know at some level these are all component tool, they all have some little common aspect. Like I personally prefer Docker, um that's as more personal preference. Like I designed a thing similar to a while ago. But I think whatever your team, whatever gets in your way the least is fine. Because like deployment generally you can change later on. It's not sort of fi a fixed constant like You can change points system usually without downtime because it's sort of ancillary to your main project. So I would say whatever whatever you feel most comfortable with as a small technical team, I would go with that Um obviously excluding things like don't you choose bash scripts, you said real tool. But like you know, between Ansible and Puppet and Chef and Docker and Jenkins I

Speaker 6: I'm inclined to to fire back that that uh you know Ansible playbooks and Jenkins config are fairly substantial bits of logic.

Speaker 4: I I'm not sure that they're you know, but you know at some point You're saving time on those in the first few deploys and then everything after that is always payback. So I think don't do you know don't have the gamblers fallacy. Don't think because you you've invested so much effort into it you can't change. A deployment tool saves you time pretty much from day two or three. Because like you you like it's not only how fast deployments are, but it's the confidence in deploying, it's the speed of fact that you can just go hit a button and just sort of go and get some team be like this is fine. Like I'm not sort of stressed and typing commands and like accidentally removing files. So like it it it's more than just speed and and time investment I think.

Speaker 7: Thanks so much. Um I you mentioned monitoring several times and I'm wondering what sort of monitoring you love. And a particular question also about monitoring is um If you recommend keeping things like the Postgres logging on in production or not.

Speaker 5: You know, getting started uh New Relic is pretty fantastic. Uh Grand, you can Send me that twenty bucks later. But I mean honestly, like for for the amount of effort it takes uh to to to get there. Um that's I I think you know for people starting out that's that's what you should do. Uh at some point there's information you're gonna want that New Relic can't provide and there's other tools to do that but um

Speaker 7: Like Munin or something like that.

Speaker 5: Yeah, yeah. Yeah, Munim, there's uh Kibana and Elasticsearch, there's Grafana and Graphite, uh there's you know a a million tools out there. I think those are you know I think I think there's a couple of different things. You you want to know your what your logs are doing. So that's uh you know kind of Kupana Elasticsearch thing you want and then you want to know the the numbers. You know how How much traffic am I getting? What are the response times and all that? And that's more of a kind of graphite and grafana or you know any of the other tools there. Yeah, I don't know.

Speaker 3: Yeah, I I think that the it depends on the size of your company and whether or not you have op staff exactly which you pick. If you have dedicated ops people, then they're going to have decent ideas about what they'd like to use, and you just need to kind of tell them what you'd like to be able to see or what things you think are important. If you don't, use a service. You know, there's hosted graphite, there's keen. io, there's all sorts of places that you can push this data. And all it is is a little bit of setup config and and you know, a hundred bucks a month or something um for to to your organization to get these services up and running and uh that gets you started. Um it I um I think it took me more than a full day to get a fully working graphite

Speaker 3: set up following tutorials online. And I'd like to kind of think that I could set up a Django app pretty quickly, but it took me like a whole day. So it's easy to get into a time suck. Setting these things up and then abandoning them and going, you know, we'll do that later when we really need this, we'll set it up later. Just use a service or something and get the data at some point somewhere that you can look at it. What

Speaker 7: I think the Postgres logging, like I know there's a lot of great Postgres analyzers out there, or or maybe they're not. I was wondering what you think.

Speaker 3: Just don't log it to the same disks that the box is using using for data in indexes. So use syslog, ship it off to another box, and then like run PG Badger over it or you know shove them into Kibana so that you can look at specific queries and patterns and things. But yeah, the the biggest thing is just don't have the Don't have the Postgres logs going to the same disk that your data's on.

Speaker 4: One tip with Postgres logs I want to just sort of bring up is that um at Landed Element right we uh insert the URL of the current page in as a comment into the SQL that we run So even in the logs we can see exactly where that that SQL query came from. So our slow query log says, well, these took 30 seconds, but they're from this URL. And so we can exactly pinpoint where things come from, which is a cool sort of middle set that someone is missing in standard query logs.

Speaker 5: I think uh full full logging is good for uh you know if you if you need like security or you know you have a breach or something and you want to be able to go back and say you know what what exactly ran. But a quick thing you can do if you're on a recent version of Postgres is enable PG stat statements, which is basically like a running slow query log. can do all sorts of stuff uh and it um yeah it's really awesome. So uh if you're if if what you're after is what queries am I running the most, how slow are they, you know, what's taking the most time, uh PG statements will get you

Speaker 2: I'm sorry I need to start with the disclaimer. I actually work for Elasticsearch, which is currently the the the Weapon of choice for storing centralized logs. So I'm gonna advocate really, really heavily for uh centralized log. Now the reason why I advocate for centralized logs is uh those information are great, but you need to take them into context. So you need to s know like so I've had this heightened load on the on the database Did I did it also mean that I had more load? If then it's okay. But if I had no more queries on no more uh HTTP requests on my app servers, that means that something is wrong. And you need to sort of be able to correlate this. And that's where centralized monitoring and logging and everything can give you much more

Speaker 2: than each and individual think, analyze, even if that would be in more detail. Sort of to have this overview and to be able to identify patterns. So as Frank said, yeah, use a user service. throw all the logs somewhere, visualize them, and over time you will learn what what was the most useful. So if you then grow Big enough congratulations and you can you can build it yourself with just the most useful thing and then you can start adding your own information For example, with logs, it's great to in in uh in index additional metadata. So structured logging, if you if you log anything, add all the information is there in there Is was this request by an

Speaker 2: authenticated user or not? Where did the user come from? And all this stuff that will give you more information. And even if you just store it on disk so you can uh backload it later or you will do it only if you run into problems, like even then to have this data it's it's invaluable to be able to identify where the bottleneck is or where the quickest win is.

Speaker 3: I think one mistake to uh real quick on on logs that people make is they worry about how long they're going to keep them in a searchable state Way too much. I mean uh I need to refer to logs multiple times a day uh with different clients. I can't remember the last time that I needed more than four days ago. The only reason I needed three days was because the issue happened after five o'clock on Friday and I'm just seeing it on Monday. And the reason that I need it for four days is sometimes I don't get to that email until Tuesday. Um but like I've never even needed it a whole week. And so people worry about storing the last 90, 180 days. We were gonna keep a whole year in there. And that just slows down every query you're doing. Which slows down your ops team and your ability to look at stuff and you know sure

Speaker 3: offload that data be able to load it in if you need to get to it, but you probably really only need to actually have indexed the last seven days of data in most cases.

Speaker 8: Yeah, that's awesome. Can I ask one quick photo what was your name by the way? Uh what was your name? Sorry.

Speaker 4: So that again?

Speaker 8: So what was your name rather? Uh

Speaker 7: and um yeah, uh your your comment about putting in the URL was that from the Django or M you're passing that as a comment in the C

Speaker 4: well so um one of the hacks we do to get instrumentation is that we wrap the cursor inside Django. and goes back in so we can add times around it. And so as part of that, we're already overriding it so we can just add a comment at that point. So we have some horrific things. There's a request thread local, some other horrific things that you shouldn't be doing.

Speaker 7: And then do you have to inspect the call stack to get the method name?

Speaker 4: You shouldn't do this as as good as the re the result is fantastic, the methodology is not so nice. I want this stuff in Django, so I want to be able to go in there and delete our horrible monkey patches and and be like, we're much more pure now. But at the moment it's more sort of just like grungy, but it works. Okay, thanks so much. Good.

Speaker 5: So had some

Speaker 8: small ideas about you know code improvements that could go into Django. Where do you think there could be documentation improvements? There is like Uh one of the frequently asked questions or some place buried in the documentation about small optimizations, but I don't even think they touch a single thing that you guys said in this panel. I mean there is the caching framework, there is HTTP caching. but nothing that really glues those all together. And I know, you know, Peter, you're working on a book about this, which is an awesome resource for the community. But you know Where where could Django be better about helping at least point people in the right direction for some of these things?

Speaker 4: I mean th this is a problem sort of it it starts very much at this at a tutorial which is I think the part point now the point does exist but for a long time it was neglected a little bit. As you start with Django, there's this wonderful narrative thread of the tutorial, it brings you to all the ideas as a brief slide. It shows you like this exists going later, this exists going later, and then at some point it just drops off and goes, and here's the reference docs. And of course I'm sure five, six years ago they were much smaller, but now like you know even as part of migration building added like another like a couple of like twenty-thirty pages of docs. And so reading all of it becomes very difficult Um I'm not sure what the best way to expose that is. Um perhaps there is a call for some kind of more like you know advanced reading topics or just sort of like a very brief summary of these other things that it

Speaker 4: like for a long time people didn't know South existed. Like South was this assumed thing in Django, but nothing on the website that it existed because It wasn't an official project. And so there was this weird thing where like every conference would be like people were like, Oh yeah, and used to do south on the stage, but it was never any set anywhere. And that that fixed itself eventually. But I I don't know, it is a problem. Um I think at some point there is some aspect of community involvement or I think books and things as well are very valid. Because like dots are two things. They're a learning resource and a reference resource. And Django is very good at the latter. Like if I want to find out what a field is, I can find it almost entertaining. But as a learning resource that there is an improvement to be made

Speaker 5: I think one hard thing is the JO docs are not opinionated, and I think that's a good thing. Um they uh but I think when you get into this stuff it's it if you're learning I think it's really good to have an opinionated resource like you don't want to know that there's eight different WISGI servers that you could possibly use. You want somebody to just say, this is the good one, you should use this one. And I don't know if that's Django Doc 's place to say things like that. So yeah, I mean maybe it It's third party resources that you know say, you know, I mean I I've yeah, we've written a book that that's doing basically this, but I I think it was born out of uh I you know we felt like there was a need for a an opinionated resource that isn't

Speaker 5: uh doesn't belong in in the in the Django docs. The Django docs should be pretty agnostic on on how you how you use it and everything. So um yeah I think that's a challenge and and trying to cover all the bases is a is a monumental task. And then you're you know I think it was a big deal to get you whiskey in to the Django docs. And now somebody has to maintain the UWISGY and the Apache. And I don't know if there's a you know G Unicorn. one and all this stuff. So maintaining all those different versions and then you drop the the new user in and you say, well you could do this or you could do that. Have fun. So yeah I think I think that's the challenge in it.

Speaker 3: For about five minutes at DjangoCon EU, Russ and I talked a little bit about putting in um like some automated checks. um that we've made absolutely no progress on and I said that I was going to work on it, but I just now remembered that I'd said that. So so um But hopefully in the next couple of months I'll get that done. But no, that looks for the very simplest common things, like you know, you're storing your sessions in the database. Just a a management command that you can run against your existing settings and say, well, you probably shouldn't be doing that That's the

Speaker 8: default. That's the default.

Speaker 3: That's the default, is is to use the database, right. But that's appropriate for getting started. But it's really easy, and I've even done it on large production sites. I've forgotten to change that. And, you know you can get twenty percent better performance across the board just by swapping out and saying use Redis or Memcach D. Um so there's lots of little things that are easy to forget, you know, whether or not you're you know using a cache template loader and things that are not that are all in the docs. They're in the reference that tells you exactly that this will get you better performance or this This shouldn't be used in production, but there is no quick easy. Let me do a quick health check on my settings and see that, okay, I'm at least not doing the top ten things that are bad. You know, I'm I might need to do ten more other things that are never gonna, you know, that are gonna be very site and project specific, but at least I'm not doing the ten, you know, easy, easy to fix things.

Speaker 3: Ross.

Speaker 4: Sorry, I I can't imagine the wine situation at Changakonie Hugh and have had anything to do with the fact we've forgotten what we all agreed we were going to do. There are there's multiple places where you can be attacking sort of performance and scaling issues. There's always the code stuff that you know use an order one order one algorithm, not an order n

Speaker 2: squared algorithm, and that sort of thing. of thing. But then at the at the DevOps side, at the beginning you sort of you did make the point about these are quite often problems you'd like to have and you don't necessarily have to deal with them. There are vendors now who are out there like Heroku and Gondor and whatnot that sort of talk about the DevOps and we'll just make the DevOps problem go away and if you have load problems, just turn this knob and we'll charge you more money and everything. will go away. How much of that is snake oil? How much of that is convenient when it's small, useful when it's large, and how do you determine when you cross that boundary?

Speaker 4: As someone who's written a service like this, So a lot of the advantage in platforms like Heroku and stuff is that their very architecture forces you to write scalably. So Heroku forces upon you several different constraints which you wouldn't have by default on your own system. Which are things that make more scalable. Like things like lack of a writable file system, this is a scalability thing. Things like your things aren't guaranteed to be on the same guarantee to be on the same machine. Like you know, they they enforce holds and such constraints like the way the way things boot up and run that Naturally makes the whole thing like horizontally chargeable to some extent. So I think that part is not snake oil. I think the term not make it fast thing is a little bit like that, especially because some problems aren't solved by just adding more servers uh especially ones that are constrained by sort of cross-talk and cross-tabers

Speaker 4: access. In particular, like so Heroku Postgres, I think, is almost the better product from them, because like that actually does do a lot of stuff with Postgres. That is very useful and tricky to do yourself, like things like having um followers and that kind of stuff. But similarly like you know you can't just at some point expect to make your website magically faster. If someone says they can do that, don't trust that they're selling you a lie Like some part of it might be true, but the entirety is probably not true. It's it's it's always hard work with all we millionaires probably.

Speaker 3: Yeah, I mean if if uh if if those services really could just dial off the the knob. Um Peter wouldn't have a reason to have written a book and and I we wouldn't have a lot of clients. Um no, you can get really, really far. It just gets very, very expensive. And at a certain point, the ROI, um both in terms of kind of the flexibility to use the tools you want to use on the ops and dev side and that just the sheer cost of those services make it make it cost effective to you know Dedicate a person to look into these things for for three months or hire somebody like us to come in and and and do stuff for them. Um it you know You can get very, very expensive very, very quickly with those services by just dialing up the knob

Speaker 5: Yeah, I mean I like the i if you properly you know once once you're beyond a single server you have you know which is again like this is kind of what is enforced by Heroku. You know, you have your database server, you have your app server, and you have your static files getting served someplace else. Going from that to auto-scaling your app servers, which have no state in them. is pretty easy. So going from one server to, well, maybe not one, but two servers to five, to ten, to a hundred, that's easy. But all you're doing is pushing your pressure lower down the stack. So uh, you know If you're if you're scaling with two servers and there's no pressure on your database and then all of a sudden oh my app servers are overloading and I need to add 50 of them now, that same database isn't going to handle

Speaker 5: that traffic probably. So you you pushed it down your stack and now you need uh either a bigger database or multiple databases and and that's not as easy a thing to just turn the knob on. You require some kind of logic and stuff there. So yeah, I think you know on your your stateless layer auto-scaling makes a lot of sense and works, but it it doesn't get you uh all the way there.

Speaker 4: And sort of on the pushing things down, like you can go further. Like at some point there are sites where the network engineering It's a critical part. Like sure, your absolute is going to have the throughput, you have all the files, but network engineering, like the the IP addresses, the unicast, the bandwidth, the throughput, the topology, that's the bit where the pressure is. And so like you know, you can't expect one solution to fit all problems. Like you're thinking of You've got things like Netflix is a very different kind of problem from a web project example. Like we but they serve static content that's very easily shottable. We have to do like transactional checks and sell an exact number of tickets and then stop. Um but we have very spiky load, networks have very particular nice rampy load and So you know these they're different challenges. Like if all websites were the same, I suspect that this problem might be more solved, but it isn't, which is kind of fun for us in a way

Speaker 4: I'll throw one more on there. In the big data space, there's a big problem about knowing exactly what big data is. You know, I I have a big data website that contains up to one gigabyte of data. There's a similar sort of thing here with performance where people say, you know, I've got a really, really high-traffic website. I've got to deal with like four or five customers hitting me three or four times a second. Where where do you could have draw the line of the the the sort of sites when w what do you think of as a high traffic site that you actually need to start really seriously thinking about these problems

Speaker 2: versus you know, okay, yeah, you might have a bit of traffic, but you really should better do that on just a single server and not worry about all this other stuff.

Speaker 4: So I think it depends on the type of site. So I've run I've run a site that got four thousand requests a second before in one server because it was a very simple form static. At the same time, the sending of requests in a command write takes an entire cluster because it's a very different kind of load. So I think it's not necessarily the number of requests a second. It's a lovely number to wave around. It's not that important. Like you know I'm sure CTNs get a lot more than anybody else does. That's you know, very different kind of load. Um although it's also a difficult job. I'm not saying they're easy to write. Uh but I I I think it's the case, it's almost a t a a team Like the point where the growth and the speed start out out outstripping your ability to code up to it easily, I think that's the point where things change. Like where a small team of a few people is no longer enough to keep up with an increasing load, where like you're constantly firefighting and that's

Speaker 4: almost for me more the turning point where suddenly like you need an ops department, you need a separate sort of like QA department or whatever. Like I think that's the breaking point in in my head

Speaker 2: Yeah, so again as as with monitoring the resource utilization or your machines, you can even monitor or pretend to your team or your ability to code. Like at some point you you you're you're doing okay and it's linear and at some point you can see that you're not coping fast enough. So at that point you need like a quantum leap You need to you need to change either by hiring more people, using different services, using different technology, or or something. So you can identify those points and unfortunately no one can tell you what the right solution for that would be uh unless you give them a lot of money. Uh but that 's sort of that's sort of uh this metric will identify like where where it needs to happen.

Speaker 2: If you see that you're still doing okay and you you still have time to develop new features, you're okay. You don't have to go looking for anything else. If you see that I'm still able to do that, but one month from now, based on the curves, I will not have any free time to actually develop the features. You need to do something. And it's it's a pure and simple business business calculation, like do I want to spend more money, do I want to spend uh hire more people or do I want to hire a consultant or do I want to spend time and investigate a better solution? Maybe a change of architecture, maybe change of features, whatever.

Speaker 5: Yeah, I I think there's very, very few sites that get to that phase that you're talking about. Or should be at that phase. Um I think a lot of people like to think they are or like to think that their you know their site is this unique flower that can't be handled by any tools that you know normal men have made. and women. And um uh it 's uh learning more about your stack and your tool set can can buy you so much time. Um you know you you find that one knob in Postgres or in Varnish or whatever that you turn and it's like, oh my gosh, you know, I just doubled my throughput, you know, just by learning how how my tool set works. So, you know, I would say

Speaker 5: you're you're at that point of, you know, scale is a major issue when the standard toolkit no longer serves your needs. uh you know and you you really do have to you know invent invent new things to to make it work and and I don't think there's many people that or many uh websites that really get to that phase where the standard toolkit doesn't

Speaker 1: Cool, and we're out of time. Thanks everybody.