Where did it all `BEGIN;`?

This video features Charlie Denton and Sam Searles-Bryant at DjangoCon Europe 2026 in Athens, Greece.

Where did it all `BEGIN;`?
0:29:03
Published October 26, 2026
248 views

TALK: Where did it all BEGIN;? by Charlie Denton & Sam Searles-Bryant

Until 2013, Django's atomic context manager didn't exist. This talk will explore how we had to manage database transactions -- as well as save-points, rollbacks, and commits -- before the ergonomics of atomic and what changed when it was introduced. By understanding what tools Django users had in Django 1.5 to make their code atomic, durable, or both, we can understand what problems atomic solved and how it made writing Django applications better. Fast-forward to 2026 and we all take atomic for granted. But what problems might it have for modern development? We take a look at where atomic can cause confusion and suggest some ways to handle that in the future using django-subatomic.

More information: https://2026.djangocon.eu/speakers/KZTTYE/

Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope

#django #djangocon #djangoconeurope #djangoconeurope2026

Transcript

4,654 words · auto-generated · en Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:03

Uh

0:03

Speaker 1: we've met Socrates, we've met Plato today, now it's time for Democrit. We are staying in the history, um finding atoms first in philosophy, then natural science, where I personally found that a very successful model until we found the subatomic model Atomic particles, quite a few of them. Um yeah. To talk about that or something similar, I assume. Um I would like to introduce to you Charlie Denton and Sam Sells Bryant, both from Craken Tech. Give them a warm app welcome please. Is this your screen on? Well

0:47

Speaker 2: you can sit down there. Oh

0:48

Speaker 1: can we have the screen up there too? Yes, and now the philosophical questions of all. Where did it all begin? Stage is yours

0:59

Speaker 2: Good afternoon. This is Charlie. I'm Sam. We both work at Kraken. You've met a number of our colleagues already up here today the last couple of days. We work on a team called Developer Foundations, which means we treat other developers at the company like our customers, and it's our job to make sure they have the tools they need to do their work easily and safely. We're uh the two of us uh are going to be kind of handing off between us to talk to you today. Um we have two bugbears. Um one of them is uh too many SQL statements. Which is particularly bad at Kraken given the scale that we run at. And code that looks like it does one thing, but actually it does something completely different. Because that's confusing and it confuses us. So today we're going to go through

1:45

Speaker 2: a history of of uh transaction management in Django. We're going to start off by explaining what transactions are, just to make sure we're all on the same page. Then we're going to go through how transactions have been managed in Django right from the beginning up until uh today. And then we're going to describe some of the problems that we still have at Kraken with managing transactions in our application. and show you some of the tools we've we've come up with for dealing with that. Okay. Hi.

2:13

Speaker 3: Are you hearing me well? Okay, I suppose I'll project. Um transactions are an SQL concept. Okay. Oh, you hear me now. Great. Transactions are an SQL concept. Uh they represent uh committing multiple changes at once or not at all. This concept is atomicity, which has been mentioned and we'll come back to. An important thing to understand in transactions in Django is there is no such thing as a nested transaction. Transactions begin with begin. You can make multiple statements and you commit them to the database with commit. If something goes wrong in the middle, you can call rollback.

3:02

Speaker 3: to roll it back. We're going to demonstrate that to you so that you are familiar with it. So let's show you two connections to the same database. On the first one, we will create a table. And we're going to uh see we've create table heroes. We've got heroes with names. We're going to begin a database transaction and insert a couple of heroes. So we're gonna insert Achilles.

3:49

Speaker 2: Oh, you can't see it. Sorry.

3:52

Speaker 3: Oh, hello.

3:52

Speaker 2: Tell you what,

3:53

Speaker 3: that says Heracles. On this side you can see we have select star from heroes and we see what's there. On the other database connection When we do the same select statement, there's nothing there because we've not yet committed the transaction. So if we go back to the first foot the first stage, commit, and we'll see that those heroes are now available on both sides Okay. And just to quickly demonstrate a rollback. uh we will begin a new transaction, insert more heroes, or a new hero. It's a shame you can't quite see that JSON is spelt

4:39

Speaker 3: J S O N there. So select star from heroes and we'll see that that exists on this side with the open transaction. It's not going to yet exist on the other side. And as we realize our mistake, we will roll it back and it will disappear from both sides.

4:58

Speaker 2: Well, if I can spell.

5:00

Speaker 3: Almost.

5:01

Speaker 2: There we go. Live typing.

5:04

Speaker 3: One more select. Yeah. Okay. Life demo. Thanks. Um Okay, so a little more to say here. Oh, we need to flip that.

5:20

Speaker 2: That would be nice if we could have our. There we go.

5:24

Speaker 3: Great. So there's a little more to understand about transactions. As we said, they begin with begin, they're committed with commit. SQL has two modes. It has a default mode where the begin is implicit. And it has auto-commit mode where the commit is implicit. So you do not need to insert and then follow it by a commit in autocommit mode. You simply insert. Insert uh yeah, let's let's move on to let's move on to um the history in in Django.

6:02

Speaker 2: So we're going to start talking about Django, and that's going to involve some Python code. And so just to make sure we're all on the same page, one of the things that's going to come a lot here is context managers. A context manager starts with a with statement, and then there's some code that's indented inside that block that is run inside the context. There is some code that's run when you enter the context and there is some code that runs when you exit the context. That exit might be because you run out of code in the indented block and you're just moving on. It might be because an exception was raised and that gets handled outside. the context and the exit function might choose to do something with that exception which we'll see a lot of. These context managers can also often be decorators on a function. Uh and most of the ones that we show you today, in fact I think all of the ones we show you today

6:47

Speaker 2: can also be uh functions. So whereas we've got with some context inside, we could have inside as a function decorated with the context, which means we wouldn't see the context. manager in our code. We haven't done that here because that would mean function calls and definitions and that would be far too much code to put on one slide. We've also tried to highlight the important bits in purple. We like purple at Kraken. Um just so that when there's a bit more code on the page you can see exactly what we're trying to talk about. So Django. In the beginning, there was no transaction control. Um Django did everything for you and it emulated the auto-commit mode that Charlie was talking about by putting a commit after every save or delete call. Uh this meant that every change you made was immediately committed in the database, but there was no transactions.

7:36

Speaker 2: You couldn't group changes to be visible once some of them were done. It also meant that every time you called save, you got at least two statements being executed, which we'll come back to in a moment. So this was great and we had this for a little while until July 2006, so nearly 20 years ago, we got the concept of transactions and we got the ability to turn off that auto-commit emulation. Turn off that that immediate commit. And we've got a whole suite of tools to manage that. And we're not going to go into all of them because we've only got half an hour. But the most important one for us is this commit on success context manager. Commit on success does what it says on the tin. If you exit the context successfully by running out of code, it will commit. If an error is raised or an exception bubbles out of that context, you get a rollback.

8:24

Speaker 2: And this is great. This allows us to make several statements altogether and then commit them at the same time. But there was a little bit of a problem with this, and it was called out in the docs at the time. And that problem is when you have multiple commit on success contexts nested inside each other. So in this case, we've got our commit on success on the outside, followed by creating an author. And then we've got another commit on success in which we create a book. This is okay when the inner commit on success exits, we commit. Which means the whole transaction is committed and we've got our author and our book visible to other connections in the database. But it does mean right at the bottom there, we've got another commit which does nothing. because there's nothing left to commit.

9:11

Speaker 2: You might be okay with this. This is not ideal, but we can accept it. But it becomes a bit confusing when there are errors So one place there could be an error is at the end of the outer commit on success. And at this point, we'll try to roll back the changes, but it'll be too late because the inner one already committed them So that error doesn't really do what we expect it to do. We've got some code here that looks like it does one thing, but actually it does something else. And if we have the exception on the inside, and I've wrapped this with the try accept to make it clear it's only being handled by the inner commit on success. We raise our exception and then we get a rollback which rolls back the whole transaction, the commit, the author and the book, which means we get to the end where we think everything is successful and we try to commit

9:58

Speaker 2: We don't get anything. Our database is empty because we've already rolled back the thing that we thought we were going to commit This is particularly sneaky when, as I said, this create book function is doing some indirection and you can't see that there's another commit on success there. So reading this code, you don't know that create book might be committing, it might be rolling back, it might be leaving you with no data at all. You don't know what's going to get committed at the end of your uh your outer commit on success block. Thankfully, that problem is solvable. And that's what Charlie's going to talk about in a moment. But just to recap, we went from having no transaction control at all. To having a bit of transaction control, but we had this commit statement after every save and delete, and we had uh

10:45

Speaker 2: no ability to undo some changes without blowing everything out of the water.

10:51

Speaker 3: So I introduced you to the default SQL mode and the autocommit mode. In Django 1. 6 We moved from using the default mode to using the autocommit mode. So inserts, saves became automatically applied, and now you could automatic now you can apply your context with a begin statement which must be done explicitly. That was enabled by deprecating commit on success and replacing it with atomic. Atomic looks very much like commit on success, except it explicitly calls begin, which is necessary now that we're in auto-commit mode. It also goes about solving the nesting problem, which it does using save points.

11:39

Speaker 3: Save points are another new concept. So a save point allows you to roll back to a specific point in the transaction and continue the transaction as though nothing went wrong. This is like in a video game, where halfway through the level you can save your progress and something goes wrong and you start off from where you were. We previously showed you this slide. It showed a lack of nesting support using commit on success. Let's show you what it looks like with Atomic. With atomic, the outer block uses a begin statement and the inner block now creates a save point that you can roll back to or just move past.

12:26

Speaker 3: If all goes well. If you've got an exception in this inner block now, when the exception's raised, you roll back to the named save point and you can continue committing only those changes which you're sure about as you move forward. Let's recap.

12:46

Speaker 2: So we've gone from having no transaction support at all. to having a bit of transaction control and then in 1. 6 we got partial rollback and we've got this explicit scope on our transactions And this is almost everything that we have today. Atomic should be very familiar to anyone that's been trying to manage transactions in a Django project. But there's one thing left before we quite catch up to the modern era. Now we might have some code at some point that we actually want to happen outside a transaction. For example, in this case, we've started a transaction, created a book. saved that book and then we want to tell the author of the book that we've we've saved their book. And we want that email to be sent outside a transaction because

13:31

Speaker 2: if we roll back the transaction, we can't really get back that email once it's gone out the door. An obvious way to do this is with on commit, so you can register actions to happen sometime later whenever the transaction commits. But sometimes that's not appropriate. If you had more code under here that needed to use the return value of notify author, for example, you'd need to do it now, not sometime in the future. So this looks good. This this looks safe. If I have a function and I I come across it and I look at it, I go, great. Notify author is happening outside the tran transaction. Only this is a lie, because there's another with atomic somewhere else in our code that's wrapping this. And that might raise an exception after We've sent our email, which means that that

14:19

Speaker 2: begin that I thought I had and the commit that I thought I had before notifying the author were actually save points. And something else might roll back the book. I mean it doesn't get it get saved. The missing property here is called durability. I want to know that that book is going to endure into the future, no matter what happens, and it can't be rolled back by something else. So there's a solution to this. Five years ago in 2021, we got durable as an argument to atomic. And that was in Django 3. 2. And that allows us to say this atomic block must be durable. Once it finishes, nobody else is allowed to roll it back. How that works is if you try to call it you try to enter that transaction that atomic context inside another atomic context, you just get an error at runtime.

15:05

Speaker 2: And that should get picked up by your tests. And if it doesn't get picked up by your tests You'd expect it to get biddy pricked up pretty quickly in uh production. So the recap. No transaction control. A bit of transaction control. In 1. 6 we got atomic, which gave us partial rollback and explicit transactions. And then we got durability five years ago. And that brings us up to exactly where we are today.

15:29

Speaker 3: Except We still have the problem at Kraken of too many unnecessary queries and code that doesn't do what it looks like it does. We're going to run through how Atomic still presents us with these problems sometimes. In this case. . where we are rolling back the inner atomic block, but not catching the exception. We've created a save point that is not used. We roll back the save point and then we roll back the whole transaction. So whether we created the save point or not, really there are unnecessary queries. Django did offer a solution to this when Atomic

16:16

Speaker 3: was first introduced, which is the save point equals false flag. Unfortunately, we found generally speaking this gets forgotten. So we still have too many queries in the majority of cases. As previously mentioned, there's also the problem of ambiguity. You come across an atomic block, you don't know just by looking at the code whether it's going to create a save point or if it's going to create a transaction. And depending on the state of the code where this is called, uh, the same piece of code could do different things. depending on how it's called. It might be wrapped in a transaction and it will create a save pointer or it might not, and so it'll create a transaction. Finally , there's no

17:01

Speaker 3: nice API for saying make sure there's a connection open or a transaction open or make sure that there's no transaction open. Um you can do this kind of statement, assert not connection in atomic block or assert um in autocommit. Um those Work, but they're not very nice and they don't play very well with tests, so you'd need to make transaction test cases which have their own costs. So we've split atomic at Kraken into multiple parts. We've got transaction, which creates a transaction. We've got save point, which creates a save point. Transaction, if it's already in a transaction, will raise an error. Save point

17:47

Speaker 3: will fail if it's not in a transaction. We found that most of the time when you want atomicity, you want things to play together. You don't actually Want to necessarily define the scope of the transaction. So transaction required simply says these things must run together, but we're not going to define the scope of the transaction here. We don't want to create a transaction. And we've also got durable for saying that you must not be in a transaction. So where we had this before. Where we say our outer block is durable and must create a transaction. Our inner block must not create save points. With our new API.

18:32

Speaker 3: Instead we have with transaction, which creates a transaction, and with transaction required, which simply enforces that the code inside must be run within a transaction. This is we found transaction required is used orders of magnitude more than we actually need save points. So all of those places we had atomic blocks. We've now not got the queries. We've reduced our queries a lot and the code is a lot clearer. It's easier to reason about this code. Because we don't need to worry about the external concept context. And it applies a pressure on us when you call a function that has transaction required on it to make sure you're managing your transaction properly

19:18

Speaker 3: Now we've released these tools open source as Django Subatomic. They're available for you to use. Uh they 've been in production at Kraken. We are running almost six million lines of code. Uh six sorry, almost sixteen million lines of Python. And we haven't fully rolled it out, but it's working and it's working well and it solves these problems. Sam, would you like to wrap up?

19:48

Speaker 2: So again, in conclusion, we've gone from having no transaction control at all to gaining the concept of transactions in 0. 95 to having the ability to partially roll back those transactions and set their scope. getting the concept of durability as part of transactions. And then at Kraken, we've got this explicit control over the different parts of our transaction management with Django Subatomic. Now this whole history uh has been worked on by people from across the Django community. Um Charlie and I have been doing a little bit with Subatomic. Um but we are indebted to everything that has come before us, especially in figuring out this history. Two particular cases I want to call out are Emeric's talk at DjangoCon 2013, where he talked about the changes that were being made to introduce atomic, which I think he worked on up to that point.

20:39

Speaker 2: And uh a lot of our history has been derived from uh from that talk. There's also uh David Sedden who uh used to work at Kraken uh and was responsible for building some of the foundation of this stuff internally and complaining loudly on the internet about the lack of durable, um which was then picked up by some good people that did the work to put it into Django.

20:59

Speaker 3: By Lily, I think

21:01

Speaker 2: Exactly. So Django Subatomic uh is uh released. It was released yesterday At version 1. 0. It's been used in production at Kraken for a year or so. Over a year. Over a year. In you know, a hundred different deployed environments, 16 million lines of code. It does all sorts of things beyond what we've just described. It's got some tools for managing on-commit callbacks, and it plays very nicely in tests. Which is a whole can of worms we don't have time to get into. But if you're interested, come and talk to us at lunch or at the sprints tomorrow and Sunday. There are a few things that it doesn't do in tests. that we're aware of. For example, managing deferred constraints when you exit a transaction context is a little bit hairy and we haven't quite figured that out yet.

21:46

Speaker 2: But for production stuff, it is feature complete. We'd love for you to try it out. We'd love to hear about whether you think it improves your code, whether it's uh its API is easy to use, whether it's self-explanatory, whether the docs are good. And if you'd love to try it out over the we'd love for you to try it out at the sprints and come and talk to us and let us know what you think. Thank you very much

22:11

Speaker 1: Thank you. So they ask questions. I can see I can see you. See it first.

22:21

Speaker 4: So shirely hanging in the air is can we have this in Django Core?

22:26

Speaker 3: I would love to get this into Django Call. There are some complications that we haven't yet solved. There would be a bit of a namespace clash with Subtomic, for example. We have begun writing a proposal for getting it into Django Core. But as I said, we've got some unsolved problems that we've got.

22:50

Speaker 1: Thank you. Nice. There's a question up there.

22:55

Speaker 5: Uh does this play nicely with if you're still using atomic blocks or any gotchas if you have old and new code? mixing if you're trying to migrate.

23:05

Speaker 3: If your atomic block is the outermost block, in production code it's gonna be fine. You're not gonna get the guardrails that this offers. But you'll get the expected behavior. In tests, this project offers some more stringent controls that can be turned on and off that won't be possible if you're using atomic. We're still using Atomic at Kraken and we're rolling this out slowly. So they do work in concert together. You just won't get all the benefits.

23:41

Speaker 2: To elaborate a bit, um one of the things that uh it does in tests is emulate real-world conditions. If you have a lot of tests in in a Django project will be wrapped in a transaction. Which means if you have things that should happen on commit, they'll never happen because you never commit the changes in a test One of the things that the sort of the transaction context does is in tests it will emulate leaving a transaction and it'll specifically call those on-commit actions and and What else? Uh which is really helpful and that's what doesn't work if you've got atomic contexts kind of mixed in with it. That can be a little bit it doesn't always know whether it's leaving a transaction or not because you're you're doing mix and match. There are settings that you can turn on and off in test

24:27

Speaker 2: to manage that behavior that make it a little bit easier to transition. And that's a whole different talk, so we'd love to talk to you about that at lunch or at the sprints. Um but we we we thought about putting it in here and then decided we didn't have time.

24:39

Speaker 1: So you hear the sprints and I'll be looking into the readme to know more about this.

24:44

Speaker 2: Absolutely.

24:48

Speaker 6: Thank you. Besides atomic, uh Django also has uh the option of turning it in the settings, uh connecting the um the transactions to request handling does that also work well with the subatomic?

25:05

Speaker 3: We don't have that turned on at Kraken. If you did that with subatomic, then because the outermost block would be atomic, you wouldn't get the automatic. on commit callbacks as as you'd expect. Maybe there's a way of tying it into the requests, but uh we haven't looked into that.

25:33

Speaker 4: Uh thank you very much. Uh are there any database restrictions on Or does it work with any Django supported backends?

25:42

Speaker 2: We test against Postgres and we exclusively use Postgres at Kraken. So short answer, I don't know, but we don't have any reason to believe it would be

25:54

Speaker 3: we intend to expand the tests to more types and everything in the code base is is using Python. There's no special SQL in there yet, but that might change if we have to support Deferred callback. Sorry, deferred constraints. I'm not quite sure, but I expect that we'll find a way of making it work for everything if we can.

26:22

Speaker 1: Thank you. More questions? We have still some.

26:26

Speaker 3: Should should also say, especially as we would like to get it into Django, we would like to support all databases. Absolutely. Yeah. Sorry.

26:35

Speaker 1: More questions. Yeah, there's one right in the front row and then I walk.

26:42

Speaker 2: Hopefully a friendly question.

26:44

Speaker 4: Hopefully.

26:53

Speaker 7: Are there any other particular benefits? specifically maybe have if you're using Postgres to removing save points?

26:59

Speaker 2: So having too many save points is a problem because you've got uh statements being executed that aren't necessary. When you have a lot of nested save points, Postgres can handle that to a point, and there's a point at which it changes the way that it's keeping track of that and becomes very inefficient. This is a problem we had at Kraken and this was actually what started us down a lot of this road was when you're nesting save points down a long way , things get bad. Mm.

27:26

Speaker 3: It stops processing things in memory, I think, and starts com putting stuff on disk.

27:30

Speaker 2: Yeah. I'm sure t that was a leading question from Tim, who I'm sure knows probably more than we do, about the exact consequences of that. But we'd we'd love to talk about it again.

27:42

Speaker 1: Okay. Thank you. There's there's oh there's more one more question. Great.

27:50

Speaker 8: Perhaps leading on from Tim 's, um do you have any tools or what tools do you have to kind of interrogate the code base to see when this is starting to look to be appropriate to start using this stuff? So when you've got nested transactions. Do you have like linters, that kind of stuff maybe that shows this?

28:08

Speaker 3: So internally to Kraken, we have wrappers around these tools that measure uh that that record metrics to show how we're using them. That's not built into subatomic, but you could create similar wrappers that would uh then record you could record a depth metric, for example. And you could do that same thing just using atomic, which is how we started here. We wrapped atomic, replaced that throughout our code base, and added metrics to just see how it was being used, and that's where we started.

28:45

Speaker 1: Okay, thank you once more to Charlie and Sam for this interesting view into the subatomic world. Please give them a good applause.

28:57

Speaker 3: Okay.

Conference

DjangoCon Europe

April 15 - 17, 2026

Athens, Greece

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos from DjangoCon Europe