Scaling the database - using multiple databases with Django

This video features Jake Howard at DjangoCon Europe 2026 in Athens, Greece.

Scaling the database - using multiple databases with Django
0:28:44
Published July 20, 2026
7 views

TALK: Scaling the database - using multiple databases with Django by Jake Howard

As your Django application gets popular, you'll need to scale up. Maybe you need to run more web workers, more CPU cores, or even spread across more servers. This works great, up to a point. Your database is likely at the core of your application, but whilst it's easy to throw more resources at a database, it's much harder to scale to multiple. After a while, you'll outgrow a single database. Maybe performance isn't the issue, and instead you need the rock-solid reliability which can only come from multiple database servers. Django supports multiple database connections, but leaves it up to you to manage how to use them and which queries to send where. But, how? How do you split your data between multiple databases? How do you tell Django which to use when? If your infrastructure already has replicas, how can you use them effectively.

More information: https://2026.djangocon.eu/speakers/CQZJUD/

Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope

#django #djangocon #djangoconeurope #djangoconeurope2026

Transcript

5,115 words · auto-generated · en Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:03

Speaker 1: So great that you make back again to uh the presentation stage. Uh and um I have the honor of introducing Jake Howard. He's a senior systems engineer at Torchbox. He's the author of Django Task. We've been talking about your work already today. Great to have you here. And also, I'm not only looking forward to hearing your nice voice right now in this talk. Everybody Even those people watching from home can hear more from Jake in today 's news uh newly released Django Chat Podcast. The new episode is out today and it's great to have you here. Big welcome to Jake.

0:45

Speaker 2: Thank you very much. So, what does it mean to scale? Well, it simply means we don't have enough stuff. We want to turn a dial and get more stuff. Scaling generally comes in two forms when we're talking about application development at least. We have horizontal and we have vertical Horizontal scaling simply means making more machines, whether that's more pods, more dinos, more containers, more instances of Django Vertical scaling on the other hand means we want to give the things that we have more resources, more CPU, more RAM, faster networking, more room for that Django to grow and expand. When we scale our application, we generally scale it horizontally.

1:31

Speaker 2: Two machines of half the size can generally serve more requests than one that is bigger anyway. Vertical scaling isn't perfect due to other bottlenecks anyway. And with that you get more resilience. If one of those machines fails, the other is still there to handle the requests. But when we do that, there's still only one database. Most databases are very good at using all of the resources given to them, as I'm sure many people in this room can attest to. But those resources only go so far What if you've outgrown a single database server or you need the reliability which you can only get from multiple instances? Hi. I'm Jake. I'm a system senior systems engineer at TorxBox, which involves a lot of dev, ops, DevOps, and everything in between. I'm a member of the Django security team. I'm a member of the Django Software Foundation.

2:17

Speaker 2: I'm also on the security, core team, and performance teams for WagTal CMS, and I exist in many, many places on the internet. A database is not your typical application. It has persistent storage, it requires disk space to save that on. Applications with persistent storage usually expect it to be fast. Databases are no exception. You can't just centralize your database's storage on S3 and expect it to work. It won't. In the web world, if we need more applications to read more files from disk, it's fine. You can't just run more Postgres instances with the same storage and expect them to just work. Postgres expects to be the only one talking to its storage. But for that price, databases bring incredible speed, flexibility, and features.

3:02

Speaker 2: Now, we can't run two isolated instances of Postgres, as they won't have the same data. And we can't point them at the same storage since they'll conflict with each other. But people run multiple databases server all the time, right? How do they do it? Well one way of doing it with one way of doing it with multiple databases is called sharding. Sharding is a way of doing horizontal scaling, but by dividing up the data instead, sometimes by table. maybe putting users and groups separate to pages, for example, in your content management system. But you could also do it by organization. For example, moving some of the larger organizations onto their own database if they need to keep doing complicated reports and keep using up all of your system cycles. But it's not without its drawbacks. Making sure that you draw the lines in the right place. They're really, really hard to move later if you do divvy

3:49

Speaker 2: things up incorrectly. You don't necessarily gain reliability by doing this, and you definitely lose automatic foreign key constraints when you do. Instead, I'm gonna walk through something much, much simpler. Um remember the word simple as I go through this. I promise it's simpler So we need multiple servers and they need to be completely separate, but with a little bit of magic that kind of sinks in between them. So first off, let's talk about how the sync would work roughly. The easiest way to handle the syncing is with primary and replicas. Each one is a completely separate database and a completely separate installation of your database engine of choice. For me, that's Postgres Most workloads tend to be read-heavy rather than write, especially for things like a content management system, which is what I spend a lot on my day building. Read queries can then be spread across multiple replicas

4:37

Speaker 2: and only the writes need to be handled by that single primary If you have more reads, just run more replicas. This is much less complicated than having multiple writer instances. It's much less obvious what to do when there's a conflict. The primary database pushes updates to the readers whenever changes are made. And they get visible within just a couple of milliseconds. The replicas don't allow any rights, so they never get out of sync. It's an entirely one-way relationship between the primary and the replicas, and it has much fewer surprises. Now, we could have Django connect to eek database, but that doesn't scale very well. Four instances might be hot fine to handle manually, but what about 40? If one of those servers goes down or gets restarted, Django will still try and send queries to it.

5:23

Speaker 2: This results in either degraded performance or outages. So instead we do like we do with web requests. We need a load balancer. For web requests, there's things um there's things like Nginx, CADI, Trafic, or Traffic if you pronounce it wrong, and the like. For Postgres, you've got things like PGBancer or PG Dog if you particularly enjoy Rust. The load balancer sits between Django and the replica servers. That means Django just cares about two connections. It has the primary and the load balancer. Load balance has been useful for quite a few things. If a node goes unhealthy for whatever reason, it's taken out of the pool. Django doesn't notice. New instances are automatically added when they're provisioned, maybe through automatic scaling. Again, Django doesn't notice either. Queries just magically get faster.

6:09

Speaker 2: This is all great. Most la mm most load balangers will intelligently split queries rather than round robin rather than round robining, perhaps maybe taking response time into account. If you're using a managed database, it may already handle this for you. You could add the primary replica here and have four instead of three because four is bigger than three to try and gain an extra reader instance, but you really shouldn't Handling rights is resource-intensive enough. You don't want the primary database to do any more work than it absolutely has to. Now, leaving the world of infrastructure, let's talk about Django. This is DjangoCon after all. What do we need to do to make this actually work with Django? Well, we now need to tell Django how to connect to our databases, both the primary and the replicas.

7:00

Speaker 2: In the settings. py there is a databases dipped, which is where you configure your databases. Most projects will look like this. They have a single connection to your database. Fortunately, Django natively has support for multiple databases. There's a reason the database setting is a dict after all. If you want another database connection, just add another entry and you get all the magic useful connection handling that Django has out of the box. For convenience, we're going to have the writer connection or the primary instance be the default. Just for the sake of convenience. Um if something isn't aware of the other connection, it will still work just suboptimally, which we can then trace down using query logs in future to try and root things properly. But how do I choose which database to query? As you're used to, queries all go to the default database backend

7:48

Speaker 2: by default But how do we tell it to use the replica? And most importantly, when to use it properly? Well again, Django has us covered. Don't you love those batteries? So we change this using the queryset. using method. Then it'll use the connection we want to. In this case, we pass replica, so the query is sent to the replica backend. Now I'm sure you're one I'm sure you're asking as you're reading this, won't that r uh won't that require rewriting every single query in the entire system to explicitly use the replica where we want it, both in our code and in third parties? Well, yes. But actually no. Enter database routers. A database router takes your query and then routes it to the correct database. A router which handles

8:34

Speaker 2: database queries might sound quite complicated, but fortunately it's not. All you need to do is define a class. There are four methods you can define for a router. If you don't define them, it just uses the default behavior. The first two are the most important for implementing this. We have the db4 read method. It gets called to find the alias for read queries. Similarly, db4Write gets called to find the alias for write queries. These methods do exactly what you might think. We want read queries to always go to the replicas, so we always return the value replica, which in turn gets sent to the load balancers and replicated between all of our instances, all of the infrastructure level. We want rights to always go to the primary, so we always return default.

9:19

Speaker 2: Not so helpful in our case, but still useful to know. There are some extra methods and some um these are all called with hints. So if we want to use a specific database for users, this is where we would do it. This is how you would implement sharding. Django handles identifying whether a query is a read or a write for you, so the router then knows how to deal with this properly. For those of you who are good at maths, you remember I said there are four methods, but I've only shown two. That's because there are actually two more. The first one is called Allow Relation, and it's used to check if Django can actually create a foreign key relationship between the two instances. In our case, if a model is using either our primary or our replica, because it's the same data, it's okay. In reality, this won't actually get used much, but if we were adding another completely separate database to our system in future, it's useful to have our router be aware of things.

10:12

Speaker 2: The fourth method is allow migrate. This says whether an app and a model can be migrated using the given connection. The replica is read-only, so it doesn't make sense to allow migrations here, so we return false. Returning none is the same as not defining the method, so it means use the default behavior, or that this router doesn't really care. Um and there you have it, a fully formed database router in around 17 lines of code. takes us from being able from using the default to being able to use our database replicas purely and efficiently. Django again makes using our router almost as easy as it was to write it. Much like anything, it's just a setting away. Django is now aware of where to send queries. Internally, it knows which methods to call. This doesn't work for raw SQL because that's far, far more complicated.

10:57

Speaker 2: You'd need to handle that manually. But if you're using the RM, Django has you covered. Read queries get sent to the replica, which they'll be nicely handled fast and scale well. Any writes get sent to the primary, which then propagate out to the replicas The primary instance isn't handling anything complex like reporting, so it's much less congested. Writes can happen faster and propagate out from there down to the replicas. Separate read queries can be scaled across read replicas, much like horizontally scaling a web server. So we're done, right? Of course not. Nothing's ever that simple at scale. Building complex distributed systems is one of the hardest challenges developers face. So I want to cover a few of those challenges I faced when trying to actually implement this.

11:43

Speaker 2: There are many caveats and challenges to using multiple databases. These are just the ones I've hit. Sharding almost certainly has its own challenges. The first issue I found is around good old database transactions. Consider this code. You create a user, which Which runs an insert query, and then you count the users, which runs a select query. And so, assuming the database was empty beforehand, there is one user in our system. This all makes sense. What about this? It's the same code, but it's just run inside a transaction. What does that print? Any guesses? A lot of people saying zero. You're right. Um this is not a bug.

12:28

Speaker 2: It's technically correct, the best kind of correct. But it's not what we expect. So why does it happen? Well, the second query, the select, is run against the replica, whilst the insert is run against the primary. Replication only happens on committed data rather than replicating the contents of an in-progress transaction. Therefore Until the transaction is committed, the replicas can't see the data. Much like how other connections to the same database can't see uncommitted data. Because that's functionally what these are. They are completely separate connections. So what's the solution? Well, there are a few, as always. One might be to carve out areas of the code base which write and force all of those to use the primary.

13:15

Speaker 2: For example, everything in your content management system must always use the replica. Maybe any post requests must always use the primary But that's inefficient and brittle, since there are plenty of reads that get caught up in those requests, which don't benefit from being accelerated by our lovely pool of replicas. Another solution, and the one I ultimately went for, is to just check whether we're in a transaction. If we are, then use the default connection. It's less efficient, but it's the only way to ensure consistency. Django's RM conveniently makes this trivial to check. There is a transaction. getAutocommit method, which returns whether we are in autocommit mode, which basically means we are not in a transaction. So we just consider that in our router with a very quick check and we're good.

14:02

Speaker 2: Next up, Django itself. This isn't so much of an issue, more of an optimization. Because we're doing this in the interest of performance, we want the replica to be used as much as possible. Django gives us quite a few tools for doing this doing things performantly and properly when it comes to database queries, but sometimes it's not perfect. For example, Uh where does this query go? Well, the get part of it should go to the replica, because it's a read. But the all create part of it is a write, so it needs the default connection for the primary. So which does Django choose? Any thoughts? It goes to the primary, always. Um the reason being is that the get part, and this is directly from a comment in the source code, the get

14:48

Speaker 2: needs to be targeted at the right connection in order to avoid potential transaction consistency problems. This is fine and good. We don't want consistency problems. But sometimes we know better. Django has to do what's best for everyone rather than what's best for just me. Not that I haven't not that it has any personal vendettes against me. So how should this be done? Well all we need to do is make get all create aware of the replica. Chances are the get path is far more common than the all create path, so we optimize for it. First, we try a simple get. If all goes well, that's all that's needed. And that uses the replica. If that fails, we fall back to the regular getAllCreate method. This avoids needing to actually re-implement the bits internally

15:36

Speaker 2: to getAllCreate. Django leads heavily on well-tested conflict resolution to do this, and so we want to use it as much as we can. This in turn uses the default connection as I've already mentioned. Unfortunately, it will end up resulting in a duplicate get query on the unhappy path, but hopefully that path isn't called too often. Once the record is created, other requests can then easily retrieve it from the replica, just as we want. Now, walking the road less traveled. So I discovered a few bugs. Oh, there we go. The final problem: walking the road less traveled. The majority of Django users only have a single database connection. And that's good. Chances are you don't need multiple and you definitely don't want the complexity that comes with it. Django is very, very well tested and takes the stability of its features very seriously.

16:25

Speaker 2: But not all software is perfect. Sometimes if something doesn't work, it might be Django. And in my case, it was twice. So, bug number one discovered from this work. This is a screenshot taken from Django's issue tracker, Track. The ticket has a hilariously long title because that's what I needed to describe. It might be the longest in history. If someone from the ops team can confirm it, that'd be great. When running tests in a configuration like this, you need to tell Django that the replica and the main database are actually the same thing, so it doesn't try and create two. This is done using the mirror setting in your database settings. Then Django's transaction test case runs your tests outside of a transaction so you can handle the transaction yourself. There's a setting for the um there's a setting on this test case called serialized rollback, which serializes the models in the database before each test rather than just nuking all of the tables.

17:17

Speaker 2: It's slower, but it's really useful if migrations if you have database migrations that create instances, for example, bootstrapping default models. The issue is, an issue that I rather helpfully found, is that when using serialized rollback and mirroring, Django would try and read your actual databases, not the second ones it creates for your tests. Which leads to really rather weird behavior in CI where you see Django create all your tables and then complain none of them exist. Conveniently, this was fixed in Django 5. 2 and has been backported to 5. 1 and obviously is in 6. 0. So you never have to worry about this, you never need to share my pain. Book number two. Chanka has the concept of generic foreign keys. Rather than a column always pointing to the primary key of a specific table, you can add an additional column which tells it the model.

18:07

Speaker 2: The White Tile contract management system uses this quite a lot internally. Django helpfully gives you a generic interface over this. So you don't need to think about the models that are being used, much like in a normal foreign key. And again, much like a foreign key, you can access the other side of that foreign key relationship on the destination model. The problem is that relationship does really weird things. If you try and read from it, it uses the replica. This is great. If you try and update it, for example, instance. relationship. update, it also uses the replica, which is not great. Unlike the issue with get or create, where it was just inefficient, this is actually a problem. Our replicas are read-only, so we get an error whenever we try and do an update. Unfortunately, this is still an outstanding issue.

18:52

Speaker 2: There's a PR, but it's not been touched for a while. My intention here isn't to point out holes in Django when talking about these issues. It's to show that getting this right is hard, both for developers and for those using Django. Django is really, really complicated. The fellows can only exercise so many cases. It's important that we us as users help them out with some real-world cases. Synthetics can only get us so far. If something looks weird, poke it a little bit deeper. If it still looks weird, make an issue on track Now, working around the issue of generic relationship uh generic relations is a bit of a bogged, but it is possible. Um again, it is more functionality inside of our database router. Unfortunately, the source code of the source of this bug is deep down inside the RM, so it's a little bit hard to patch around.

19:41

Speaker 2: Instead, we do it slightly differently. Any model which uses a generic foreign key needs to be forced to use the default connection, for now at the very least. So first we discover them and then we query for them. And then if we query for them, we force it to use the default connection. See, this is where having the model is actually really, really helpful. On my project, it's only used in a few small models, so the performance penalty isn't huge, but I'm really hoping this gets fixed soon so I can remove the bunk. So let's put this all together. This has been quite a really broad conversation about Django and RebLEP because touching both the infrastructure and the code. And so I've glossed over a lot of details as a result. But the important bits are still here, so let's recap. Our infrastructure has multiple databases, one primary, which handles all of the writes, and some number of replicas, which handle the reads, in this case

20:30

Speaker 2: three. with queries being balanced between them using a load balancer. The primary automatically syncs writes over to the replicas so they all see the same data. The exact details of how this gets implemented in that actual syncing process will vary based on whether you're using a cloud provider or whether you're rolling things yourself. If you have a systems team, make it their problem, much like the rest of my team did to me. Django in turn is given two connection URLs, one for the primary directly and one for the load balancer that sits in front of the replicas. Both get configured in the databases setting. The rest of the handling is configured is confined to the database router, which routes then which routes traffic on queries in this case to the correct backend. If we're using a write query, we want to always use that default connection.

21:17

Speaker 2: If we're in a transaction, always use the default connection. If we're trying to query a generic relation model, always use the default connection, at the very least until this bug gets resolved. Otherwise, let's use the replica. Almost nothing else in our code base needs to change But if we know it's going to be mostly a read-heavy code path, we can avoid some of Django's built-in things for an extra bit of performance, much like we did for get or create. The router acts on any RM query, whether that's in our code or someone else's. So we add a router and all of our code paths get updated. And with that, with just a few small code changes later, our application is more fault tolerant, faster, easy to scale, or

22:03

Speaker 2: easier to scale, and just a little bit more fun to administer. Thank you.

22:16

Speaker 1: Thank you, Jake. So I'm sure there's many questions. Questions.

22:24

Speaker 3: Thank you. Thank you for your talk. I was triggered a bit by your first example about the transactions. What if if you are outside of a transaction, do a write and do a read again? Will the uh Will it be automatically uh already in in the second database or is there a chance that you're reading an old value again

22:47

Speaker 2: There's yeah, there is a small chance. So the so I mentioned a bit that kind of the the replicas sync their data from the primary That is not instantaneous. We haven't quite cracked sort of faster than light syncing yet. In theory, yes, it is possible for you to do an insert, immediately read that back. outside of a transaction and get either incorrect data or data not existing, things like that. It is in theory possible. In practice, I've not seen it, but it is 100% possible. The best thing to do about that is just keep your transactions kind of as high as possible if you are seeing those issues where you are expecting to write data and immediately read it Wrapping in a transaction, at least in this setup, would help, but potentially in other use cases you might want a transaction anyway to make sure that this lump of code is internally consistent.

23:41

Speaker 2: Is

23:41

Speaker 1: that right? Other questions? No. Um

23:52

Speaker 4: some databases might have to um have a synchronous uh replication option. Is this something you've encountered and tried, or is it uh not fast enough for uh getting the replicas up to date? So surely you um waste some time in waiting for the commits to propagate uh across all replicas, but um would that work against the replication lag?

24:17

Speaker 2: Yeah, you can replication lag is always going to be a problem like I mentioned. There's always going to be an amount of time where things don't work to when they do. In practice, I've been running this project at least on AWS with RDS There the um that kind of the lag is sort of single divided milliseconds, which for at least for our use case for rights happening and then being propagated out is fine for our use case. Other use cases it probably isn't. And so in those cases, you're either going to need to try and do as much as you can to optimize that sync latency. whether that's with changing how the replication works or things like that, or you're gonna need to kind of re-architect things so you're not immediately writing something and expecting it to be there immediately. It's kind of a at a certain scale

25:02

Speaker 2: you need to do things slightly more asynchronous. That's not a plug for Django Tasks, but also please use Django tasks.

25:14

Speaker 1: So maybe a question for me if there's nobody else uh asking, sorry. Um how do you test this?

25:20

Speaker 2: How do you test this? Um It's quite difficult. You can do this with unit tests. So I mentioned one of the bugs that I found was in unit testing. So you could have multiple connections. to Django, you can point them at the same database and check that you get things properly. Testing the router, in the end, I did that kind of synthetically. So we had the router configured and then I basically the use cases that I mentioned. Using the transaction test case, you can kind of point your code exactly how you want it to run with transactions, with linking to other models and things like that. So you just kind of have to think of all the use cases that might come up that might break. And then hope that you've written the use cases and the test cases to match that in a much similar way that you do any other um

26:06

Speaker 2: any other unit tests Things get a bit more complicated when you're trying to unit test how this behaves with like the actual infrastructure. That's a bit more complicated. The only way I found you can probably set this up locally and run logical replication locally We didn't bother, we just went with we've probably caught all of the bugs. Let's get it into our staging environments. Let's have lots of people try and throw at it there and see how it behaves. So it's not perfect. Testing and production or things like that is generally bad, but sometimes it's the best we can get.

26:37

Speaker 1: Thank you. There's a question now. Finally.

26:45

Speaker 5: Hi. Thanks for your talk. Um About the uh database replica lag, um we have a similar problem and it was more of a problem. Um I don't know how it fits into what you you're proposing, so uh yeah Take it if you want. But we um we we we set a property on on the request that basically pins the remainder of the database transactions to the write as soon as there's been a write.

27:16

Speaker 2: Yeah, doing it by request, as I mentioned, like is a really good way of doing it and saying like these views, these requests, things like that will always use the primary. That works. The problem is that You end up if that ends up being sort of a large proportion of your requests, then it starts to sort of you lose the performance benefits. Um it also doesn't necessarily work. In theory, it's much rarer But if you're using third-party views for whatever reason, you no longer have quite as much control in there. So this was on a client Wagtail build. Technically I have commit access to the Wagtop repo, so I can go and poke at it. I'd probably be fired if I did that, so I'm not going to. Um but at that point you're kind of you can poke things internally if you have full control over the request, over the views, sorry, it works.

28:01

Speaker 2: But sometimes having things in the router is really nice because it kind of separates that concern out. You have a thing that deals with playing with your queries, and then you have your view code. The main thing that doing it in a router was really beneficial for me is it meant I could deal with the intricacies of when this works, when this breaks, right at the tests. The rest of the developers on my team can work exactly as they have done before. And other than being slightly careful around transactions, can move at their normal momentum. And for us, that was really valuable.

28:32

Speaker 1: Thank you, Jake. Great talk. I learned a lot. Thank you so much.

Presenters

Conference

DjangoCon Europe

April 15 - 17, 2026

Athens, Greece

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos by Jake Howard

More videos from DjangoCon Europe