Understanding Celery to maintain your sanity - Ashwini Balnaves

Speaker 1: My name is Ashwinny and my talk is called Understanding Self. To maintain your sanity. I had the idea for this talk after we had a critical issue in production. I work as a full-time software engineer at a company called Capiche. And I was fairly new to the company at the time, but I had read the docs on how to use celery. I'd even uh written a few few tasks, but now that things had gone really wrong and nothing was working, I needed to know exactly how everything worked so that I could jump in and see what was happening and try to fix it quickly. That was a very stressful way to learn how celery works. Especially since a lot of the online resources that I found focused on which methods to call and how to get started

Speaker 1: I would recommend to anyone who has celery in their stack that they invest in the time to understand celery before it goes wrong, which is what I hope to help you achieve with this talk. So the first thing to know is that Celery is an implementation of a distributed asynchronous task queue. In my case, we have these very long batches of work that need to be done. Like I was saying before, I work at a startup called Capiche, based in Brisbane, Australia. We process huge amounts of customer data in order to uncover customer insights. This means uploading and handling CSV files with hundreds of columns and millions of rows, and updating large natural language processing topic models.

Speaker 1: So we have a lot of work to do that takes quite a while. But why do we want to do these tasks in the background as opposed to in a Django view? After all, doesn't Django have everything I need? Well, the thing with doing things in the background means that you can respond back to your client very quickly. Otherwise, you're going to make your client and ultimately your users. wait for pages to load or for actions to complete. This is going to consume a connection in your Django app and limits the responsiveness of your application. And if that doesn't convince you, I found out the hard way that browsers have timeouts, so there's a hard limit on how long a request handler can take to process a request

Speaker 1: anyway. So if we have celery doing all this work instead of doing it in our views, we also get another huge benefit, which is we decouple that work from the Django app. Decoupling gives us two main benefits. The first is to horizontally scale independently of the main Django application. For us specifically, this means I can add more salary services to our Kubernetes cluster without impacting the Django app at all. This also gives us the ability to restart the Django app, for example for a deployment, without having to wait for tasks to complete. For us, this is very useful considering some of our tasks can take several hours.

Speaker 1: There are many other reasons as well. When I was putting together this talk, I asked some of my other developer friends what they use celery for. And there were many other reasons. Most of them were also using Celery for long-running CPU-bounded tasks, but other reasons include managing resource contention and parallelizing work as an optimization. So why celery? I use celery because it was part of the stack that I inherited. A lot of us as developers don't work on Greenville projects mostly. of the time. And due to Celery's popularity, a lot of Django applications involve Celery. It is by far the most popular of all the task queues. I think its popularity comes from the fact that it's a very mature project.

Speaker 1: It has many integrations with different frameworks, message queues, and databases. It's included in many tutorials, and Markets itself is very simple and powerful. I would agree that it's very simple for simple use cases, and it also has very many powerful features, but with those come quite a dramatic increase in complexity when using it in more advanced situations. Specifically when you start changing some of the default values for the configuration settings. There's so many levers that you can pull, it's really important to understand what you're doing. But it's by far the most popular. But there are alternatives, such as the ones that I've listed here, in no particular order. So

Speaker 1: what is it actually doing? Let's talk about some definitions We're going to go over each part of the system. There's going to be a little bit of code, but our main focus is going to be how everything is working together. So a bit more of a systems view. First, we have the client. This is the service that wants to submit work to be done. And in our case, this is the Django view. Then we have that work that actually needs to be done and in Celery these are called tasks. Finally, we have the workers and they are ultimately what executes the work. all the tasks. You can see here on the right we have a little bit of code. We can see that tasks are defined as Python functions

Speaker 1: in a tasks file within our Django app using the shared task decorator. So the shared task decorator is unique to Django apps. So when you see Celery tutorials that are using other frameworks other than Django, you might see a slightly different decorator, which is the app. tast decorator. This is a little bit of a side note, but because Celery app um celery only has one instance, uh enable to expose all the sub-modules or the sub-apps in general To that single Celery instance, we need the special decorator. Then we have the client, and the client imports the tasks and calls them using the delay method. So I've taken this code snippet from the Django Official Docs, one of their very first examples, and this is the simplest way that you can call a task.

Speaker 1: Celery has a lot of powerful features, especially when it comes to the ways that you can call tasks There's the ability to link them together so they execute one after each other, or we can send them all off in a group. We can even attach callbacks to that group, but I'm not going to go too much into that. There are some really good resources online. Search for what Celery calls canvas design. And then we have workers So a worker spawns subprocessors in order to execute the tasks. The number of subprocessors defaults to the number of CPUs available on the machine on which the worker is running. But if you want you can change that with a dash

Speaker 1: C argument. In our example here we have three workers, but one of the big value of what Celery has to offer is that it is distributed. So these workers can be, and in fact you usually are on different machines. Each one will have as many subprocessors as is available on the box on which it's running. So there are three workers , three worker boxes in this picture, but there could potentially be more subprocessors actually running. So, there's a little bit more going on. How does the client actually get the task to the worker to execute? Well, we use a broker or a message transport.

Speaker 1: This is the way that the messages get communicated between the clients and the workers. The client will submit a task to the broker, and the workers will subscribe to the broker and fetch tasks off of its queue. Once a worker has processed has executed a task, they then write the results of that task into the results store. So let's talk a little bit more about the broker. Just as well , the documentation and a lot of tutorials use the word broker and message queue into Interchangeably, which I will make confuse me a little bit at first, but we're just going to stick with broker to keep it consistent. So there are several services that Celery can use as a broker

Speaker 1: RabbitMQ is the service that Celery docs put up to you first, but Redis is also a popular opinion, and Redis is what I use. When the task is submitted to the broker, it's given an ID and it's stored in a serialized format. Celery supports many different serialization formats as well. But you might also be limited by what serialization format your broker supports. In my case, we use both Pickle and JSON. Then there's the results store. So also the results store is used interchangeably with the word results backend. We're going to be sticking with store. I use the Django ORM, which has a nice integration called Django

Speaker 1: salary results, but as you can see, there are many options that you can choose from. Django Celery Results even gives you an interface through the Django admin panel. You can see here where each row is a completed task that's been written into the results backend. So now it's time for a story of how things go wrong. And with a tool so configurable, it probably will. So there are quite a few stories that I could have chosen to tell. For example, we had problems with growing memory requirements in Redis. because I was passing the entire contents of a file through the task parameters

Speaker 1: because I was treating tasks as if they were regular functions. I didn't realize that the function parameters were being serialized and stored in Redis as well. We also had a lot of fun when we found out that workers will pre-fetch a certain number of tasks. That means if you have tasks that run for a long time, which we do, a worker could potentially fetch a batch of them, let's say three. And it will go through and execute each of those while perhaps there are other workers that are available. So let's say it's it's fetched three and the first one takes long time. Tasks two and three are sitting there waiting to be run, even though there's other workers available.

Speaker 1: So if you do have quite long-running tasks The suggestion is that you run with the flag dash o fair. That way there will be no pre-fetching of tasks and workers only grab them one at a time. We've also had fun when um tasks have failed to change as expected because there are immutable signature types that behave differently to mutable ones. This is because tasks created using the immutable signature signatures, don't require the task previous results, and if you use the linking method and examples in the salary docs, it doesn't actually wait, and they're not executed sequentially. A lot of these stories are a result of me not understanding how celery actually works, because since everything was working when I got there, I found the documentation to be a little overwhelming, so I didn't really feel the need to.

Speaker 1: I assumed that everything was set up optimally and it was running fine. But what I found out was that Celery's default configuration settings are set up for a high frequency shortish amount of tasks. What I mean is a lot of tasks that are taking a shortish amount of time As our company began to grow though and gain more and more customers with larger and larger datasets, our tasks became longer and longer. We needed to change the default configuration settings. away from what the default values were. However, because celery has so many levers you can pull, we started to get ourselves in trouble as we changed settings without realizing the impact of what it was gonna have on the whole system.

Speaker 1: We didn't have a comprehensive understanding of how all the levers impacted each other But by the way, I would say I still don't have a comprehensive understanding of how all the different configuration works within Celery. But I kind of glossed through those stories and some of them probably didn't even make much sense if you're new to celery but we're gonna go through one in a little bit more detail. Um one Which I was referencing in the beginning of this talk that made me have to learn a lot about salary. So, how did it all begin? Well, the defaults for salary are configured to have

Speaker 1: a lower reliability than what we really wanted as a team So we turned on the task axe slate to increase reliability. Well, what does that mean? I'll tell you. It means that a worker can acknowledge the message from the broker before or after the worker actually executes the task. So the client submits the task to the broker and the worker pulls off the task and acknowledges it and then that task is no longer in the broker's queue. While that task is executing on the worker. However, the non-default behavior Is, which we turned on, is that when the task is sitting in the broker's queue

Speaker 1: and the worker pulls off the task, It waits until the execution of that task has finished before sending the ACK. So the task is sitting in the broker's queue while it's executing and only gets finished when it's done. This means that if something happened to the process while the task was executing and the task was never acknowledged, then the broker would retry the task again. The idea here is that we would be protected from data loss Due to work of failure. Everything seemed to be going well, but then we kept growing, and the tasks were getting longer, and our tasks started to time out. We could tell this because, well, none of our tasks were completing, but we could also see the timeout in the logs, and we could see that the default was set for 300 seconds.

Speaker 1: So we figured we could simply up this time limit and our tasks would complete. We knew that most of our tasks took in the order of hours and so we set it to 24 hours, even though it was a little overkill, because we knew it would be definitely longer. long enough for even our longest task to complete. All seemed well again until We discovered through a customer report that nothing was working. In our UI, the new file uploads were being marked as queued and were not being processed. This is actually another pretty good tip is if you can surface the state of your tasks within your UI to reflect the state that they have in Celery, that can be very useful.

Speaker 1: In this case, we could see that they were being queued, but not being executed. This was not a good situation to be in. The pressure was on to fix the downtime in our system. So we could see that the tasks were being sent by the client, but where could the blockage be? Was it in the broker? Were the workers not picking up the tasks? Were the workers themselves failing in some way? Well, income salary CLI. So this is a great tool to be able to see what's happening within Celery. Celery Cli has many great commands, but rather than reading the docs, I would recommend you have a play with it to see what it's capable of

Speaker 1: in some sort of environment that's safe. We're going to have a look at just how useful it is in solving our problem. So the first thing we needed to do is just get some visibility on what was going on. And so the prime command for that is inspect active. I've included a little example here of what the salary workers would look like if they had no tasks. But back in our story, this is what the result was to inspect active. You can see that our salary workers are very busy. And not only are they busy, They're all busy with the same task.

Speaker 1: This is actually a response, well uh like a sanitized and simplified version of what the actual command's output was on that day So they're all busy with the same task, which was very confusing because not only was it strange to have them all executing the same task. But I could see that that task was masked as successfully completed in the results store This must have meant that the broker is keeping the task in the message queue and allowing workers to retry it and pull it off the broker again and again. So why aren't the messages on the broker being act when we know that the task was successfully executed

Speaker 1: Well , there's a little bit of a clue here, which is that the acknowledged is marked false. So we do have we had confirmation that the task was in fact sitting in the broker, and the new tasks that needed to get executed were just piling up behind it. I really want to know what's going on, but because it's our production system and it's currently not working, we had to look at fixing that before investigating what could possibly have caused this. Celery CLI comes to the rescue again. Now, I wasn't sure at the time what the difference between revoke and terminate was.

Speaker 1: And it's still kind of hard to find this information. The monitoring and management guide has a lot about the um ability to examine tasks but not so much control them. And even the command line interface docs don't really specify exactly what everything does. That's why I suggest if you can play around and use the dash -help option, that's a great way to learn about the capabilities of Celery CLI. So revoking a task versus terminating a task. Revoking a task, what it actually does is it adds that task's ID. to a set of task IDs within the salary worker's in memory set. So whenever a salary

Speaker 1: worker pulls off a new task from the broker 's message queue, it checks whether it's in that set or not, and if it is, it won't run it. As you can see, if you're already running it, it's already way past that check. And so revoking a task is not going to affect a task that is already running. As a side note as well, because this set is in memory, in the salary worker, not in Redis, our broker, any restarts mean that the tasks will effectively be unrevoked. And if you want Your revoke to be persistent, you have to set up that set to exist on disk on the machine that your worker is running on. But that's a side note.

Speaker 1: So what we want to do is terminate. So we terminated um our task that was running on all the workers It killed all the processes that were executing our task. And it's important to note that it does actually then acknowledge the task so it's no longer queued. This is dangerous though, and the reason for that is that it kills the process, not just the task. This means if there is anything in memory on your celery worker that is important, such as prefetched tasks, they could go missing. So what was happening? Well, after we managed to unblock the system, we found out that the reason

Speaker 1: Um for this problem was that the broker can't wait indefinitely for the acknowledgement to eventually come. As you can see I've got here the broker's thinking, how long should I wait for my acknowledgement before I decide that something has happened to the celery worker and the acknowledgement is never going to come and I need to retry the task. Well, it turns out there is a setting in Redis for this. It even turns out that there's a caveat section under the brokers section of the salary documentation. But none of us in our team had read the entire documentation in order to know this. So in the end, we set the visibility time out to 24 hours as well.

Speaker 1: Under a time crunch, we got a lot of value out of Celery CLI, but There is a wonderful graphical user interface that you can use called Flower. And this just gives you the same functionality as Celery CLI, but a much nicer interface And with the benefit of foresight, you can set it up. If you're using a distributed system, adding tracing and having good logging is also incredibly valuable I actually cannot recommend it high enough. Our tracing system, I love. We use honeycomb and it's dramatically increased my ability to gain visibility. over what on earth is going on within our system.

Speaker 1: But that's another talk. I hope that this talk, though, has helped you with your um will help you with your celery issues and will give you a foundation to draw upon when you're reading the celery docs. Thank you Oh, just for the sake of the recording, I'll repeat the question, which is the Django community is moving f towards using Django Q. What are they missing out on? Um, I'm probably not the best person to ask this question to because I haven't had much experience with Django Q. I think Really, one of the main focuses that I wanted for my talk was for people who were in a similar situation like me who have inherited a project with uh

Speaker 1: celery already in it. And maybe didn't feel it necessary to learn the underlyings of how it worked because it was all just um being happy and and chugging along um successfully um until something goes wrong in um a way that requires you to jump in. So I personally haven't been able to look into the other task queues very much. Are there any other questions? But

Speaker 2: What were some of the reasons that caused you guys to start changing more of the uh background settings that you were warning against

Speaker 1: Yeah, so um I guess it's important to context for this conversation is the startup that I work in uh work at is quite small but we're really starting at the scaling up phase of our journey. So our engineering team at the moment is three people um and um what that means is for us we are still finding product market fit and what we were testing with and what we thought our the kind of data that our customers would have would be much smaller. We thought we were going to be dealing with um a couple of hundred rows in a spreadsheet. sort of thing and the code was written with that in mind. That's what we were testing against.

Speaker 1: And then as we started to work out that enterprise level customers were um much more of our ideal customer profile, what that means is they came with way more data than we ever expected. And so our tasks and everything were set up. without that really being in mind and they were really long long running and ultimately we had to support those sales as they came on and that meant just tweaking things as we go to try and make uh the current system that we have work for it because we don't had like six months to um really make sure that we were being really precise and careful with everything. It's kind of uh one of the joys of the startup lifestyle.

Speaker 1: Did that answer your question?

Speaker 2: It did, and I've got a quick follow-on to that then, which is do you have any insight then on on understanding when you should be looking at actually you know your Django code or you know whatever whatever celery is running with or looking at celery for the optimizations.

Speaker 1: Yeah, that's a great um that's a great point. Um I think I think for us, a lot of it has to do with profiling and trying to work out what's the quickest way to to get these gains. At the moment a lot of we're we're focusing a lot on performance work and a lot of that isn't even in the um Django level or the It's more um the shape of our data and making sure it's coming out efficiently out of the database. So

Speaker 1: even now we still have quite long-running tasks and um we uh just making it work, I guess, until we have time to optimize those as well.

Speaker 2: Thank you.

Speaker 1: No worries. Tim asks, do you have any recommendations for any tutorials on learning celery? I actually do. And there's actually uh quite a few good um common gotcha articles and I think I'll I'll link them in the Slack if anybody else, so everybody else can uh have a look at them. I came across quite a few while I was researching for this talk as well.

Speaker 3: I would love that too.

Speaker 1: No problem

Speaker 4: Hey, thank you very much for your talk. Great talk. Um I have a question regarding uh flower. I didn't get to play around with it too much, but I noticed it only works if it's already been running for a while, like it's keeping its own lock or something Do you know where it stores it and do I need to worry if I keep it running a long time that at some point something will just store too much?

Speaker 1: Um I'm not totally sure and the reason for that is because uh we have it set up um in our local environment at the moment, so we have um it running as a service uh in our Docker Compose for our local environment, but we haven't managed to get it um for our production system Yeah. I don't take my own advice apparently.

Speaker 4: Cool. Thanks.

Speaker 1: I did see in the Slack that a lot of other people have had some experience with flowers, so it might be worth having um a discussion there.

Speaker 4: Yeah, sure. Thank you

Speaker 5: Hi, hello. First of all, thank you for the great talk. Perhaps this is kind of a new question, but um When should we start uh considering using uh salary tasks instead of crumb jobs? Thank you.

Speaker 1: Oh that's a good question. So Celery does have a quite robust periodic tasks functionality as well. You run it as a separate service called CeleryBeat. Um I have had some pretty good experiences with that. We use it pretty extensively in in our environment. And I think there is a lot that the periodics tasks framework gives you. You can configure it in the admin, Django admin as well. You can manually run tasks. you can um disable them and enable them from the Django admin and that's one of the my like

Speaker 1: biggest pros that I found with using that.

Speaker 5: Yeah that that sounds really cool being able to configure it from the the admin. Thank you.

Speaker 1: No problem. We did actually we did actually just uh just yesterday have issues with um with that though, because we had set up one Kubernetes cluster. And then an upgraded Kubernetes cluster. Um and they were sharing the database. And we were planning on uh running them both at the same time and then switching over. But we forgot that uh our periodic tasks were reading from the database and so we had two sets of uh celery tasks going at once. And it was only by pure pure coincidence that we had not set up the uh GCP permissions correctly on the second uh clusters nodes so all the uploads failed thankfully otherwise we would have been cleaning out duplicated data um all day yesterday

Speaker 1: Hi

Speaker 6: hi Ashwini, yeah, yeah. I think uh Yeah, uh adding on to uh to the question about Crown, Cron and Speaker, I think Cron is more used where uh we can relate to our system level I think salary is used where you can relate to the application regarding. Maybe there's a task and Python application where you want it to run. I think cron wouldn't be able to figure out what to do. It is basically cron like scheduled task for a system level and salary is mainly used for scheduled tasks for the application level So I think that might help add up to the question too.

Speaker 1: Yeah, that's a really good point. Thanks, Noah. Thanks for that.

Speaker 6: Yeah.