Django 5.0: Elevating Experiences with Server-Sent Events
Published July 10, 2024
This video features melhin at DjangoCon Europe 2026 in Athens, Greece.
TALK: Django Task Workers in Subinterpreters: Single-Server Django Applications Without Process Overhead by Melhin Ahammad
Deploying Django applications with background tasks typically requires managing multiple processes and queues. But what if you could run everything in a single process without sacrificing isolation or performance? This talk shows how Python 3.14's subinterpreters enable a practical pattern for self-contained Django applications.
More information: https://2026.djangocon.eu/speakers/PSQY8D/
Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope
#django #djangocon #djangoconeurope #djangoconeurope2026
Automatically transcribed, so expect mistakes in names and technical terms.
Speaker 1: And with those announcements out of the way, it is my very great pleasure to introduce Melvin, who will be talking to us about tasks. He's a experienced developer and longtime tinkerer interested in making deployment simple. something we can all get behind. I for one am very much looking forward to this talk. So over to you.
Speaker 2: Thank you, Tim. Hi. It's weird, isn't it? You always want to start the talk and you're always thinking what is the first word? You say hi or you go with a bang. Anyway, back to the talk. So uh there's not a lot of uh Django talks about sub-interpreters, and I just thought, yeah, I just thought it's interesting and curious. So I'm gonna try Django task workers in subinterpreters. Intro, I'm Elhin. Been working as an engineer for quite quite some time. I'm mostly uh back in dev and also do things in infra. Oh sorry about that. Oh. Yeah, okay.
Speaker 2: That's fine. Yeah. Um I also work at Kraken Technologies like some of the other speakers. I'm not going to pitch Kraken because Tim did a really good job yesterday. Um I like uh exploring stuff mostly around self-contained, easy-to-reason systems. And before we start, I want to make a small context around. So Anthony Shaw's Python subinterpreter's talk was something that made me curious about sub-entrepreneurs and I was thinking can I also do this uh with Django? Can I do task processing with this? I wanted to combine two features that we have, the newer ones, one sub-interpreters and Django Task that was also in introduced.
Speaker 2: And I also wanted to see if new primitives can solve familiar uh problems that we have Oh, I'm not going to go deep into benchmarks because some of the talks that I'm going to link towards the end of my talk have done a decent job at explaining it You have to have Python 3. 14 and Django 6 to actually use some of the things that I'm going to talk about. Before we start with the how, let's start with the why. So a lot of our Django applications in general have deferred processing. And the reason for that is you don't want to block the request and response cycle.
Speaker 2: You want to be your app you want your app to be responsible. Your workflows are always not linear, like the talk yesterday you saw, there's always something that you have to wait on. And you can't wait on the request response cycle. And also you need to offload some of these CPU intensive workloads that you have. That's interesting. Okay. Um This is an architecture that basically you would kind of follow. So you have like a Django app server, you have broker worker and database.
Speaker 2: Um and all these black boxes are different components that you have to actually uh deploy separately and also maintain. Let's look at the components. Sorry about that. Just have some trouble with my Oops. Ah, okay. What? Quickly go back to the slides. Yeah, anyway.
Speaker 2: Yeah, let's look at the deployment options that we have. Um so if if the sorry, sorry about that. Um going back to the components that we were talking about, right? So one of them is the Django server that you have to deploy. So you have Unicorn, you have Daphne, Uicon, Hypocon. And Unicorn has been there for a while. Daphne and UV UVCO supports ASCII , HyperCon also has ASCII, HTTP2, and multiple things. I'm going to use HyperCon as an example, and the reason is that HyperCon has like a programmable interface that I can actually use to
Speaker 2: deploy this. And task processing framework is another interesting thing that we have. So we we have Celery, we have RQ, we have Django Q. I've there are also multiple of these. I've just taken some three different examples. Celery being the one that also is mostly used. It's a mature stable framework RQ is predominantly based on Redis and if you need latency and if you already have Redis in the system you can use that. Django Q kind of uses the database if you have that. Now all these help us to differ processes and execute them effectively, provided they run independently and detached from the web server.
Speaker 2: The cost of running a separate server is you have broker, you have the worker, you have the web worker, and all the deploy all of them have to be deployed separately and monitored separately. You have a lot of moving parts Um and every extra process is another thing to configure and reason about. I'll come to the real why. Feriverse excites me. It takes me back to the fundamentals of internet, um where it was decentralized and interoperable. I wanted to run my own Feriverse server without much hassle and I wanted to do it in Django. I don't want to run I don't want to run a separate worker. I don't want to run a separate broker. I just want to run a single server that can handle all the fan-out
Speaker 2: workflows. and can be just deployed on the edge. And yesterday we heard about the proof of life, the proof of online life. And if you want to run your own social media server, you could federate your online proof of life. I want my server to be like this. I want my deployment mechanism to be like this. I everything in one single Django server. I have the app, I have the internal queue, I have the worker. So what I'm going to do is going to combine these things. So I'm going to have Hypercon workers, Q and Django task workers all within one process. And when let's look at these components a little more deeply. Before we talk about subinterpreters, let's talk about
Speaker 2: What was there before sub-interpreters? So multiple interpreters was there in Python from 1. 5, but they shared significant global state. And because of that, the JIL was not exactly locked at the interpreter level, but more on the process level. And then in uh 3. 12 uh we started getting the actual per interpreter GIL being introduced. That was in uh that was part of PEP 684 Multiple interpreters were introduced in the standard library in 314. Before that they were actually an extra package, called extra interpreters.
Speaker 2: Both of these together started enabling us the way of actually having multiple Python interpreters in a single process. And then you have some structure like this where you have the main interpreter launching a bunch of sub-interpreters and then there's um message passing happening between them and the objects are actually copied. And this is one of the features of isolated state. So you have um separate uh built-ins and imports. The objects are actually copied so that you don't have any mutable state bugs. And the message passing also happens primarily between data structures like queues
Speaker 2: and memory view. And those kind of sort. This is not saying subinterpreters are the best. This is just a s comparison right just to keep all the primitives together. I've excluded free threading out of this for now, but just the other primitives. So you have the threads which are in the same process but they have they are not parallel there's no true parallelism, there's isolation uh they lack isolation and they but they don't have any IPC and then you go to ISYNC IO then you also have uh they run in the same single thread They're cooperative multitask tasking. Then you have multi-processing where there's actually true parallelism.
Speaker 2: The only difference is they're not in one process and the they have to communicate sometimes via the OS pipes And subinterpreters are just predominantly in one process and they don't have to go over to a different process. This is how in 3. 14 you can actually start using interpreters. And this interface is kind of familiar to us because we have process pools and we have thread pool executor for a long time. Ha Haki stole my thunder yesterday, so I'm not gonna go into detail, but I just wanted to give a lot of a little bit of um preface on Django task, so kind of introduced into the framework in Django 6, provides a unified way to define and run background tasks, abstract survey, implementation details.
Speaker 2: Allows switching between different backends, which is kind of interesting because then you can use the same task with different backends using the same task definition. Your code doesn't actually change You can also do different things by running locally, running with subinterpreter backend, which doesn't exist yet, but uh switching to a database backend. I don't know if it's actually okay to do that, but I tried this by just running it in two different dynamically running background workers. So you could just put it into two different backends and if you're running two different background workers who are picking up stuff, then they just basically run in two different things. And I'm combining both, you could see that the main interpreter and then there are like these groups of web
Speaker 2: task sub-interpreters. And there you can see f what they do is they actually All communicate via queues, so you can see the worker queue, the shutdown queue, the exit queue being created in the main interpreter and then just passed on to the sub-interpreters. And along with that, uh you also have to replicate something because you also have to listen to the main process. So you have like a spawn thread that just listens on the shutdown queue And another interesting thing around it is the exit worker. And I'll come to one of the interesting things around uh error handling because the main interpreter Doesn't actually know if the sub-interpreter prematurely exited, and you have to then coordinate this.
Speaker 2: And this is something that I'm also using in a sense. you could have this kind of message passing across all the things in the uh in the architecture uh my own weird implementation of subinterpreter q backend. Um I just basically You can just basically inherit the base backend and then uh override the NQ. And since you've already passed in the queue, you can now start using that. Now passing in the queue is something that That is still work in progress. I've done a small hack where I'm just setting a global state which uh I can take it within the Django application because when I start the Django application using HyperCon, for instance I don't actually pass uh
Speaker 2: the queue as an argument. And then yeah, of course this is uh the way basically just to configure stuff. And your task definition just looks like this. Uh it's just a a CP intuition C task and then it's just basically the NQ. You have to write your own task worker. And there is a lot more configuration. I'm not just going to go into details around it, but It might it's looking like this. So you get the work a Q and then you're infinitely just working uh waiting for it based on an interval that if you want to provide And finally, when you combine all this, you get something like this. So you get uh a group of sub-interpreters,
Speaker 2: group of task workers, all have communicating with a queue Uh the main process launches it. So there's just one main process in this particular case. It's quite interesting because when I started looking at how can I run different um web suh web sub -interpreters I came across this interesting problem that I had. So I was looking at Unicorn and trying to understand how are these multiple processes binding to one single process. So if you have run local run server and if you run it in multiple of your tabs, it will say the process. actually is bound bound the socket, the port is actually bound to uh one process. And Gunicon does an interesting stuff around pre -fork model or something like that where it just
Speaker 2: um forks from the main and there are these all these single contained applications that we have. Uh uh the sequence diagram is just for me to talk, but it it the the thing is at the end of the day the kernel then starts deciding. uh which uh pro which process to actually give the request to. And when it comes to sub-interpreters, now it's a little different. We don't have pre fork over there, right? It's all basically one process. So you could technically just bind to the bind to the port, but there's a slight issue. Um because when you try to bind to the same port, they're all sharing the same uh file descriptors down on the bottom you can see that
Speaker 2: and when one of them fails or closes the whole socket disappears. So you You can't actually um do this, so what you have to do is you have to do some kind of aliasing around file descriptors to then actually make this happen I'm going to show a small demo because it's a little hard to compress all this information into one single demo. I'm going to run a A law um um I'm just going to run a log. Sorry. Yeah. I've just recorded the log too. actually help me explain what is going on. And what is happening over here is uh to my original uh
Speaker 2: problem To my original problem of trying to you know deploy Fediverse. Um I've I've spawned like four of these instances, and here you can see The moment one of them spawns, there is the task worker, there's uh web worker, there's uh also two task workers. And all the other colors are just different instances. I just wanted to replicate like a system where they're all talking to each other. And here you can just say so a message, right? So you're creating a message, you're just writing hello world. And Now you could just
Speaker 2: immediately see that the task has been enqueued and then you've also started to broadcast it to other um uh uh other listening instances and you those instances are also now performed in different tasks. So the annotation over here is this is picked up by the task the second task worker. And this is actually picked up by the first task worker. And uh here also you could see that the web was actually received with the first one. So the second one is still there. So you have a combination of predominantly two ser uh two processes for web and two for task processing.
Speaker 2: What this kind of enables is you can also then start linking workflows together. So you have you can store the self-pro uh post and then you can actually broadcast your followers which is then NQ 'd and here you can use your dynamic stuff where you can put it into a DBS base queue or an immediate queue or however you want. And then your flow then becomes uh like a uh like a workflow. Error handling is what uh I wanted to come to. So you see a lot of cues and because the failures are a little isolated, it's not completely isolated. uh because sub-interpreters are not processes.
Speaker 2: Uh but still you would have these weird issues where you just see things getting handled hanged. Startup sequencing is in important because tasks Have uh so if there are no tasks to receive, then there is a bit of an issue over there. And Q is the main main coordination mechanism because now you go from Synchronous sequential to this uh CSP styles thing. If you just have a single server, then you It's low, it no broker, tasks are basically in memory and fast, but cues are basically lost on restart and there's only a single server But you could also run the same setup if you
Speaker 2: just switch to the database backend. You could just have Uh now the same tasks, uh task workers are listening in different servers. So you can also start scaling things linearly if you want. There is limitations in subordinators, technology maturity being the main thing, uh, because it's still evolving. The object sharing is also evolving. Serialization and deserialization is also something that is uh that can be improved. Um there is an on there's not a lot of monitoring tools for these. So if you want to check which subinterpreter has uh a memory leak or something like that, it's a little hard.
Speaker 2: And that leads us to the performance unknowns that we have. Basically, there's uh these kinds of things of what is the overhead? What is the memory usage? Is something that we don't actually have. If you are running Django, Postgres, and any of the pure Python dependencies, you could experiment with sub-interpreters and task workers together in combination. There is an issue in NumPy about supporting subinterpreters and also PyO3. also has the same issue which means Pydantic also would have it because of the core things. And the main blocker for them is because the C
Speaker 2: extensions that they have haven't implemented something called a multi-phase initialization. This basically separates the global state and I'll also link that towards the end. It's still a question of what happened to my Freddie verse surfer. I just wanted to do that, but I just started got distracted by exploring things. Which finally get ended up being uh at a talk. I want to build something similar to Sucker Punch, uh which is in in Ruby. Uh I want to lean on the solid frameworks like hype account, Django and the management command, but build a simple wrapper unless someone convince me
Speaker 2: convinces me not to do it. I'll share with you uh the very bad no test code that I have been working as a reference for how to how this works. Just closing thoughts. So subinterpreters unlock a new kind of primitive for deferred workflows. You could still run things if you're just using pure Python and Psychop G2. Uh also watch the ecosystem. Um when I when I started to look into sub-interpreters, Django was not supported and the reason was not Because there was anything wrong wrong in Django, it was the date time wasn't supported. At some point SQL Lite was the only thing that you could use. And now at the level I think with Postgres
Speaker 2: support, there is something that you can actually start Experimenting when your workflows are not that complicated. And who knows, maybe I'll build a small wrapper that you could just easily port your simple Django apps to this So you don't actually need to know all the things that are there. Before I go to my last last uh last slide, there's a QR code which links to a GitHub page which has a lot more articles and talks about subinterpreters and the performance benchmark, you would find that uh interesting Thank you.
Speaker 1: Thank you very much, Melon. Um I think we have some time for some questions. Um if anyone has any. Yes.
Speaker 3: Have you tried running uh a database worker in a sub interpreter?
Speaker 2: Yeah, yes, yes, yes.
Speaker 1: And did did that did that that go well?
Speaker 2: Yeah, that's so you uh okay, maybe I didn't really put that in, but the uh the multiple server work Configuration is basically database backed. So you can so you can just start with the database backed first in a single server, and if you want to Scale scale it linearly, then you can just add another server and then it'll just work. On top of load balance, of course, but still.
Speaker 1: Great. Any other questions? Yes.
Speaker 2: Oh Jake.
Speaker 4: Don't don't be too concerned when I put my hand up. Um I think this is fantastic. Do you think there's scope for these kinds of things getting included into Django Core rather than them sitting outside as separate things?
Speaker 2: For sub sub-interpreters, I have to see where this goes because of the dependency issues. If they were not there. I think it's an interesting thing because all our workflows in general are synchronous and sequential. It's easy to reason about. So when you move to free threading, for instance, that becomes a little different. And with subinterpreters, you can just defer this a little bit.
Speaker 1: Thank you. Any any other questions?
Speaker 5: Hello.
Speaker 2: Sorry?
Speaker 5: Hi. Thank you for the talk and for the work.
Speaker 2: Yeah, sorry. Um
Speaker 5: you just mentioned free threading and uh I was wondering if you can tell us anything about how free threading inter intersects with or interaffects uh the the issue of sub-interpreters and how that affects your work here
Speaker 2: So predominantly the free threading or no gill kind of is a completely different concept to this one because here you still have the gill, uh, but it is not um locked to one process. So that's where the difference comes. And with free threading, things would be a little different. Because as of now when we run Python programs uh and when you use thread threading you have the protection of the gill to an extent. The moment you have no gill, then And a lot of those things you might have to keep in mind. So there is a talk from Yuri. I'm sorry I don't know the last name, and I've also linked it in the in the in the slides. where he has a GIF
Speaker 2: uh uh meme where he says the free threading is for the like the It's a complicated thing, so it's for the mega brain or something like that. So it's it's a l it's our it's a different concept in general.
Speaker 1: I think we have time for one more question. Two two hands go up. We'll do two
Speaker 6: Hi, thanks. I the thing that's one of the things that interests me about subinterpreters is the possibility of kind of deferring a mes memory intensive operation. I know like an image processing thing. I don't necessarily want that in my memory the whole time. Yeah. Is that a use case that you see Working well. I mean you mentioned that we don't really know the memory profile.
Speaker 2: Yeah, yeah. So um there has been some work around Trying to so the the talk that I mentioned about um Yuri for instance where they're trying to do that too. So It makes a lot of sense, right? One of these mutables I'm sorry, immutable big objects, if you can share them and then differ it to the OSC score, then that kind of solves all a lot of problems that we have. So it's uh it's in in their git, which is kind of dormant for two years. I I don't know exactly what is going on, but there's also another structure called memory view Um so it's still evolving. I'm hoping that something like that comes. So that's also something that everyone wants, right?
Speaker 2: You you just want to differ, you just want to load one, so you don't have All these multiple processes all loading the same models. I hope that comes. So yeah.
Speaker 7: Yeah uh a question regarding the um decommissioning of the sub interpreter if a sub interpreter has finished its work is there a reliable signal that The work has ended, that it's crashed, that it has shut down, or whatever reason it has ended. Um
Speaker 2: in our case it's a little different, but when you normally run uh sub interpreters like any other thread pool or process pool executor, you always has have this futures that gives you whether it's completed or not. So that's actually a signal. But in the way that I am doing it, it's a little crazy, but uh the way I am doing it is they are all long-running processes. So they if they exit uh then we might have to do something like what Unicorn does is it um it monitors them and refreshes them. It only comes so I've implemented something like that when the startup doesn't work properly. So
Speaker 2: when you know you want to run a migration and you know have a conflict or your database has gone away or something like that, I've saw s seen all these subinterprets just hang. So now this one sends back a signal so in a queue saying that the main process kill all the other sub-interpreters and then the main process sends the message saying that finished we we're not doing anything this time. So yeah.
Speaker 7: Okay. Thank you.
Speaker 1: I think that's sadly all we have time for, but thanks once again, Melanie.
April 15 - 17, 2026
Athens, Greece
Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026