Serving Files with Django

Speaker 1: Okay, uh hello everyone, uh welcome to my talk uh serving files with uh Django. Let's uh get up the slides. Okay Maybe at first a little bit about myself. My name is Jochen Wester. I'm a freelance software developer from Düsseldorf. I'm a long-term Python programmer. Um I learned a I learned uh about Django in 2013 and uh have used it in a lot of uh projects since then and um But I'm also doing a lot of data science and machine learning uh stuff. Um I'm also co-host of a uh German uh

Speaker 1: speaking podcast about Python called Python Python Podcast for which we didn't uh choose uh commercial podcast hosting uh provider um and we didn't uh use uh use an off-the-shelf uh podcast hosting software in PHP or something like that. Um Instead I wrote um uh uh podcast hosting application using Django. Mainly because I'm not very clever. And that brings us to the topic of this talk because the main purpose of a podcast hosting application is to serve audio files to clients. And um

Speaker 1: yeah, this is what you can expect from the talk. It's not about managing assets, the parts of your application that are not source code. But instead just serving files from your application server while everybody is saying that this is not the recommended way to do it Um yeah. Um so um the goal of of all of this uh would be um if uh I would be able to saturate the gigabit Ethernet connection I have on the uh application server at my hosting provider. If I I would be able to saturate this this connection with audio file downloads Then I would be happier and would say, okay, this is

Speaker 1: this is uh solved for me. And um Yeah, let's just jump into it and do some practical experiments to see if it's possible, whether it's possible or not or not. Um Yeah, um uh for for this benchmark I'll use uh gigaby connection between um the MacBook running this presentation and uh Mac Mini. I have 125 files, each uh 10 megabyte in size, so they should saturate a gigabit connection for about 10 seconds. Uh if it's a lot l a lot more than uh than ten seconds, then it's probably uh then

Speaker 1: we didn't saturate this gigabit. If it's a lot less, then physics uh is broken or time. The code for this benchmark is available via the link on the slide. Okay, I just fire up a terminal. Let's see. Um oo uh I just start uh just start an application server. Pretty standard WSGI GUNicorn server with three worker processes to serve those 125 files. Um nothing fancy, yeah. And

Speaker 1: then I'll use this uh uh Jupyter Notebook. I forgot to clear the output to run the benchmark. Jupyter notebooks are great. I use them for my data science projects all the time. But I also really like to use them for Django projects because uh you can run uh Django Shell inside Jupyter Notebooks and then you can have your All your models at your fingertips you can try out stuff, just uh prototype uh some functions and then if you are ready copy them into your real source code. It's pretty amazing Uh okay, let's let's uh do the benchmark. Mmm

Speaker 1: okay. An arrow. Just scroll on. Nothing to see here. Okay, um the next cell starts in the benchmark, so I'll switch over to the terminal again. And we can see okay, oh, there's a lot of stuff happening in the server log file, and then we see okay the gigabit connection is saturated. Looks good. Oh, and it's done. Didn't take much longer than ten seconds. Maybe exactly ten seconds. Really cool. Hmm, that that wasn't difficult at all. Why are people uh saying that you shouldn't do that?

Speaker 1: Looks like it has worked really uh really great. Piece of cake. Yeah, uh I recorded the start and finish timestamps of all those file downloads. And um yeah, let's plot them against time and bandwidth and um Yeah, this uh leads to a really interesting uh image. Uh this is uh looking a little bit weird Each line stands for one file download, but we see they are not really being downloaded concurrently. We know we have started all the downloads at the same time, but they didn't start at the same time. Pretty much started one after each other.

Speaker 1: And uh only three at a time, and they used a lot of bandwidth. So this wouldn't work in a uh environment where um uh Your clients have uh pretty low bandwidth connections, yeah. When you live in Germany, you know what I'm talking about? We pretty much only have low bandwidth connections here. And uh okay. Yeah, so this is not good. I I uh Explain in more detail what's going on here later on, but um yeah. So maybe we just need to uh Uh run

Speaker 1: a different is uh different application server. Okay Let's uh start UV-Corn, which is based on libuv. This is uh really fast event loop implementation, it's also about powering uh Node. js, uh it's battle tests that it's really fast. It's a nice abstraction above all those low level operating system. This calls you need to do uh You need to call to uh have fast networking, like e-pol on on Linux. And uh yeah. Okay, and I also had to start two workers because otherwise this wouldn't be possible to saturate a gigabit. Okay, let's start the benchmark again.

Speaker 1: Benchmar client again. Okay, we see yeah gigabit connection also gets saturated again, but nothing happens in the cyberlog. Maybe because the quests are still being delivered. Huh? Yes? No? Let's have a look at the uh timestamps from start to finish. Yeah, and we see all the Now all the downloads start at the same time, use one megabyte of bandwidth each, and then end all at the same time. And then this is the image we want to see. It's looking great So this is pretty much if if if I'm able to do this on my server, yeah, I'm I'm I'm I'm satisfied. This is this is this is great.

Speaker 1: So um yeah what did I have to do to make this work? Not really that much. Um I had to modify the standard uh Django file response a little bit because Usually it would just uh use a blocking read from from the file and you can't do that because if you just read from a file then you are Server process will block and you can't do anything else. And here I have this dunder annext. method uh which uses AIO files library which maintains the thread pool for doing uh

Speaker 1: um AO uh IO operations on files asynchronously uh to enable this asynchronous read from the file. And it's the same same mechanism, same library as used uh as in this file response class of uh starlet the uh framework uh which powers fast api. I did some initial benchmarks with fast API and what I was pretty impressed with the uh speed of of fast API and then I So okay, ah, it's uh solid. Ooh, ah, there's there's this file response. And then I tried to steal just the file response class uh because I thought, oh Uh it's also an HGI server, maybe I can just use it in Django.

Speaker 1: But it's not really possible because The HGI handler is a little bit different. The HGI handler in Django iterates over streaming responses. For example, this this line for part and response calls DunderNext, uh the DunderNext method for each part, but does uh it does it synchronously. So you can't await uh um an asynchronous read uh in a in a synchronous uh function call. So yeah, it's not possible to serve a file asynchronously without modifying the SGI handlers. So I did that. I um edit the code marked with this green

Speaker 1: greenish background And um I I check for an attribute uh named Thunder A Itter and um if it's there, then I assume okay this It's probably an async file response, and then I use async4 to um call dunder a uh the anex methods asynchronously. And then I can use a way to read from from the file, and then I can also asynchronously send back the data to the client. And those two modifications were all that was needed to do to make this work and have uh high concurrency file summing.

Speaker 1: Um yeah. So yeah. Uh those two images again. Okay. Uh yeah. But maybe there are other options uh when you have uh concurrency is a is a key thing here. And um but maybe I think oh is not the only option you have uh to to reach this goal, so um I made a slide to um give an overview uh of a different approaches uh in decreasing uh resource usage. Uh Uh you can add more machines, yeah. This would also uh increase your concurrency. Uh you can put more cores in your machines, you can start more processes uh um

Speaker 1: yeah more threads or use async io so Um and those lightweight options might be the most suitable. Depends, but probably yes. So, okay, let's have a closer look at why it's a little bit difficult uh to achieve with uh processes. That would also explain the image we saw in the first benchmark. Let's say we have only two workers and uh four clients requesting four files at the same time. Then only client one and two will receive an response immediately. Client three and four have to wait, because worker one and two

Speaker 1: are busy and can't do anything else. And not until worker one is finished has finished serving client one and becomes available again, and then then client three will receive also a response and uh A little bit later, client 4 also will receive a response, but until then they have to wait. And maybe they just time out and uh user the user will see an error message. Just bad. So you probably need uh n workers to serve n clients concurrently. We can test this assumption. by just uh starting one hundred thirty

Speaker 1: uh processes instead of three And see if the image changes. We need to wait until all all workers have started. Takes pretty long because G went uh the Journey convocers are big. Okay. Now they should be there. Then C, yeah. Gigabit connection also gets saturated. Okay, this looks nice. And now we see the same image for the process based uh concurrency as for the uh async based. So it's possible to serve a lot of requests concurrently using processes.

Speaker 1: Uh it's just not that efficient. Um yeah, next option uh for example an additional G Unicon worker process uh needs about 30 megabytes of main memory. So if you want to start 1000 processes, you need 30 gigabytes of main memory and yeah, you get other problems too. So it's not not really efficient. You can also use uh threads, threats are more lightweight, yeah, an additional thread would just allocate about 8 megabytes of uh main memory. And this memory is just virtual memory. Um it's not resident. Depends on what your thread exactly does, but It's probably a lot less memory usage than uh starting a new process.

Speaker 1: And on a decent recent Linux uh system you should be able to start Yeah, about ten thousand threads without problems, and this should leave a lot of headroom uh for concurrent requests. We can Verify that by starting a G Vend uh Unicorn worker with uh 130 threads. So this started pretty quick compared to processes. And then we'll just measure it. Hmm. Okay, yeah. Gigabit connection already saturated again. Nothing in the server log. Ten seconds. Okay. Let's have a look at the graph. Yeah.

Speaker 1: Looks fine. Okay, uh yeah, threads are also working. Great. And we yeah, of course, have also async. io. Um Adding an active uh coroutine to event loop uh costs only about two kilobytes of main memory, so this is really really cheap. But you have uh but but adding coroutines is not really free because at some point um The scheduling overhead for your coroutines will dominate um the uh all all of the things your uh application server process does.

Speaker 1: So I don't know the exact number, but if you At uh thousands, tens of thousands, I don't know, coroutines. At some point um your application server process will be busy just scheduling those coroutines and uh calling. signal handlers, choosing which coroutine to run next, stuff like that. And not being able to serve any more files or doing interesting business logic stuff. So but what you can do is uh um start two processes for each call you have and then limit the number of active uh coroutines to let's say 100 or something like that And then distribute the low

Speaker 1: the the requests over those cores and processes and uh event loops and um then you'll probably have a really high throughput networking system. Um Yeah, so okay, it seems like it's possible to serve files uh using your application server, but um Maybe the established best practices are good enough. Why do it? Okay. For small uh files like assets. There's and and files that are now on on application start, there's this really um great approach using a combination of white noise and a CDN.

Speaker 1: White noise to handle the caching headers and initially solve the files file using send file, and then CDN to reduce the latency by moving the files closer to the client. There's a great um talk about about that uh um assets in Django without losing your hair from Jacob Kaplan Moss from DjangoCon 2019, if I remember correctly. Or you can use just an dedicated web server like Nginx. Would also work. For uh user-uploaded files, this approach with white noise doesn't work anymore because uh it needs access to the files at application start time and yeah.

Speaker 1: uh user uploaded files are not really uh known at application start and um yeah for them you need uh some kind of an object store and maybe a cdn to serve them to the clients or Maybe uh also a dedicated web server. Uh you shouldn't uh write those uh uploaded files into the file system where your uh application server lives because uh adding user controlled content uh to the server where uh to the filesystem of the server where your code runs, uh it's a pretty dangerous uh thing to do. So you need some kind of an object store and then. It'll be fine. It's um yeah for for the Python Podcast host uh side we also use uh

Speaker 1: Amazon S3 and um Cloud front as a CDN. Works great. Um so no need to serve files uh using your application server. For now. But what about serving private files? Well, yeah. The thing uh I see uh people doing uh in the wild usually is using proper authentication authorization for your for jungle views and then Um serving file URLs URLs to files to private files which are not really ri protected. If you know the URL, then you can access the file.

Speaker 1: And yeah. If you ask them they say ah they have a cryptic name and we put a hash in the name and then uh nobody will be able to find them. Oh Yeah, but they could get leaked via some log file or it's it's it's not really it's a little bit uh uh ugly uh yeah for my taste. Uh better option would be to use dedicated uh dedicated server like nginx in front of your application servers so um The file requests gets sent to your application server, then your application server checks authentication authorization. sends back an empty file response

Speaker 1: with a special header telling nginx okay this request was valid and then engine x will replace uh your empty file response with the actual file response containing the file. This works, but Yeah, you now have two systems uh controlling access to your files and uh one is maybe tested, one not. Yeah, and if you use a content delivery network, you have the same problem. It can't really access your database and uh check whether permissions are um given to view this file, yeah so you have to work around this using signed cookies or signatures in URLs. You can add a little bit more stuff in into them

Speaker 1: like this URL is only valid for 30 minutes or it's only valid uh for clients uh accessing the file from this IP address. But uh it's a little bit like hiding hiding your files because it's not really checking whether you are author authorized or not. Um yeah, it's uh and the main problem I think is that you have to put a lot of trust in your into your systems working well together, which you usually shouldn't do because Um if someone messes messes up your uh Engine X config, then your Django application server test uh application tests won't tell you about it. So you

Speaker 1: think everything is fine. Yeah, no no one can access private files, but it's not true because your nginx config is broken, but you won't see it. Um yeah, and same same is true for for the CDN uh solution because there are times when you don't when you cannot test. uh your application using the real CDN. Yeah, when you're on the plane or don't have network connection. It's just impractical. Additional things you can do uh when you serve your files uh by your application server is uh putting advertisement into your podcast episodes or uh track users uh stop uh stop stop

Speaker 1: listening to your podcast if you make a really bad joke Um yeah, I don't like those use cases because they are evil, but um Hmm, as a thought experiment, it's not that bad because for example the advertisement use case it's it would be quite complicated to do this with the CDN because If you have five spots where you are able to insert advertisement uh in your podcast episode, then you have to generate five different files. Uh six, one without uh advertisement and one uh which each spot filled and then uh upload all of this files to your CDN and then

Speaker 1: um modify your feed to uh get your modified uh uh audio files delivered and then count uh uh put information in the log files of your CDN on how many times you have delivered a special file and then get this information back into your application server so you can stop uh uh messing with your feet uh f feet uh because you have uh delivered enough uh spots this would be this is a mess but if you just would add spots dynamically while serving the file. You can just stop after ten thousand spot playouts and then say okay it's fine. And you won't have modified a single audio file.

Speaker 1: Just dynamically while serving. Yeah, maybe there are better use cases because I I don't like uh advertisement, but yeah. There's also uh interesting point. Um for biggish files, uh C DNs are not really uh Cost efficient, uh maybe because Yeah, they are great if your files are small. The s the smaller your files, the better CDNs are. Because Um your site's performance will profit the most if uh reduced latency is uh is is uh measurable for for the user.

Speaker 1: So if uh C uh your CSS files and JavaScript files uh arrive quicker, then it will speed up your site and this will be really good. But if you have to wait a few seconds, uh because it's a big podcast episode, then uh it won't matter if whether it's fifty fifty milliseconds. Uh Faster or not. So it won't change the u the user experience. But it's much more expensive because you pay by gigabyte traffic. For for uh for example, the the Python Podcast, uh small podcast, um, but it generates a few hundred gigabytes of traffic each month, so I have to pay twenty to thirty dollars um just um

Speaker 1: for the uh CDN traffic. But um yeah. And I could have uh rented a decent application server for that price. Um and it wouldn't be uh traffic wouldn't cost me anything. For the application server. So maybe it's great to serve uh small files uh uh using a CDN, but if your files are bigger uh maybe solve them using your application server. Um so Okay, oh already So we are at the conclusion. Mm Yeah

Speaker 1: All methods we tried to uh improve our concurrency uh in serving files did work. This is great. But you still have to look at your use case to uh choose the right method. And um there are some some things, for example, to consider. Uh if you are um using some asynchronous mess message queue you want to uh talk to while serving a file uh and uh it has an async interface and you have want to await it then you will pretty much have to use an ASGI server and an async view because otherwise you can't call await. It's not possible. Um

Speaker 1: But okay, one more thing. If you are just want to increase um the concurrency of your file responses. You can just maybe the easiest thing to do would be to use uh A G Event worker in Junicorn. Let's start the benchmark again. You'll see uh it uh also saturates uh the gigabyte gigabit connection nice and uh it's also Yeah, really good concurrency. Perfect.

Speaker 1: Well. Yeah, and this was the WSGI interface and the standard Django file response. And uh files were served asynchronously and concurrently, um disp uh in spite of using this traditional uh WSGI code and um file response class And this is possible because uh Gven does a lot of really scary things like uh monkey patching, a lot of stuff in the SendEd library, like uh in the socket module, replacing all the read and write chords with uh own variants. And you have to l be a little bit brave, but if the only thing you want to do is uh serving files, hmm

Speaker 1: Maybe it's okay. And then you can isolate uh weird things that happen if you mucapatch everything, maybe by just uh splitting the traffic uh at your load balancer in file uh requests and the rest and the rest goes to the uh synchronous uh geunicorn application server and all the file requests go to the event uh g event worker. And then You don't have to do uh a lot, and you'll get pretty decent uh file serving capabilities for your jungle application server. Okay, well um I think um that's it.

Speaker 1: Thank you very much for your attention. Um I hope you uh still enjoy the conference and See you soon No, no, yes, yes.

Speaker 2: Um yes, I have a a practical question for if you want to use your approach of by serving through uh through Django itself. Is there like a way to if you have like multiple workers to like dedicate some of the workers to the rest of your app like the normally I mean the like navigating pages etc etc and only then on specific workers only then for the file handling uh do you have some recommendations about that

Speaker 1: Yeah, also uh the the way I would do it is uh to split the traffic at the uh you you um also need probably uh um reverse proxy before your application servers doing SSL offloading and load balancing. And there you could split the traffic between file uh file responses and uh all the other um uh uh other requests and um uh maybe you can just for the file responses run an application server on some different port and then do only file responses uh with an g event worker something like that and use a normal sync worker for all the normal short running requests. Yes.

Speaker 2: Okay, thank you. Hi

Speaker 3: Jochen.

Speaker 1: Hello.

Speaker 3: I I have a question. How how far do you think you can go with this? How many gigabits can you serve? with this setup. I know that you've tested this on uh one with a bit

Speaker 1: connection.

Speaker 3: But how far do you think you can push this?

Speaker 1: Uh uh yeah for my for my uh okay I have some uh only anecdotal evidence I have Only a a little bit of anecdot uh anecdot for this, but uh usually one core would be able to uh one one I uh I five uh Intel I five would be able to serve about 60 uh 60 megabytes per second. So uh with uh with eight cores I would think it's would be Probably possible to to uh saturate four gigabit, but I don't know because I don't have a four gigabit link. Um it's uh also what what really surprised me was thus uh that my M M1 uh MacBook Air is uh per uh on a single core. It's much faster than uh my um I also served uh tried serving from my um

Speaker 1: maxed out uh uh MacBook uh Pro uh uh uh uh I9 uh I9 and it um um yeah it's about I I think seventy megabyte per second and the M1 is on one core by uh at one hundred ten uh megabyte per second, so it's a lot faster. And I I wouldn't have expected that.

Speaker 3: So you you're telling us we should buy more hardware.

Speaker 1: I don't know. Depends from from where you want to serve your files because if you want to do it from home, maybe but uh hosted M1 variants are uh uh relatively expensive. So Maybe it's uh cheaper to just buy more servers uh that are not M1.

Speaker 3: All right. Thank you very much. There are some questions in the chat. Uh that people

Speaker 1: Big send file. Yes. Yeah, the the question uh this was uh um also addressed in the talk and I I also couldn't uh really follow it. Uh but uh and then I learned uh I had to uh drop the quality a little bit and then it worked. Um yeah, um using Xen file is uh is is uh a lot better than just uh obfuscating the file names. But um yeah it's uh also not that uh um clean because it's it's it's not not that easy to test against. Um you can't always have the full uh your full architecture uh running when you uh just want to ru uh to to to run your application tests and uh but if you want to test your complete system and uh

Speaker 1: Yeah, using Xand file means that your Nginx is now part of your application and you have uh you have to test uh its configuration to be sure that uh no one can access uh your protected files then you would also have to run an Nginx in uh for a a test time or uh for in development time, which is kind of unpractical. And also it's not really possible to uh may probably won't serve files from from the file system, but from from some S3 compatible uh object store. And um yeah, that means probably running some kind of a Minio Min IO uh server. And you also won't uh uh won't uh uh uh really uh

Speaker 1: wouldn't really do that in in a in in the for development. environment. So I think it's it's yeah, it's it's it's maybe the best option. The uh the the closest one to serving files directly from the application server uh uh in in um uh uh respect to uh security but uh Yeah, it's also a little bit impractical, but it's definitely possible. And uh what what uh what would be really interesting for me is uh to see whether um engine X is faster uh um serving files from menu, for example. Uh which would be really interesting, but I I I doubt it because um it's only um if you if you just um copy uh some blocks of data from one socket to another, then it shouldn't matter if it

Speaker 1: if you do it in Python or if it if you do it in in in C because uh yeah uh the operating system is uh doing all the hard work and you are just telling the operating system what to do. You are just orchestrating the operating system. So I I I it would surprise me if Nginx is a lot faster. It would be a lot faster if it uh can use sendfile, but I think it's not possible to use sendfile in this scenario. Ah, the link in the code. It's um to the code is in um on the uh slide title practical experiment. I uh can Make it visible here? It's a code for this benchmark link.

Speaker 1: Uh should work.

Speaker 3: Is that linked on your on your talk page?

Speaker 1: Yes.

Speaker 3: Excellent. Thanks.

Speaker 1: On the on this practical experiment uh thing. Oh yeah. No, I have oh I'm I'm double. Uh let's see. Okay. Yeah. Uh header, test the header response in development. Yes, of course. You can test uh if the header was set correctly But you uh you have to trust your Nginx configuration. Yeah. And what's also not that uh not that uh uh cool is that Uh your engine X now has to have access to all your users' private files, which is if there's some security problem with Nginx, then uh yeah, you're exposed.

Speaker 1: It's it's just a little bit more uh attack surface. Yeah. The question is uh some people just um copy the the private files to a public uh uh space web space where everyone can access us and use uh maybe hashes of the contents of the files as file names. But yes, um this is this is what variants of this is what most people do. And um I uh I think it's it's not really uh uh really good but because uh those URLs which are generated. Yeah it maybe if you do it temporarily, hmm But um those URLs can leak somewhere. Maybe in a log file, maybe on the client side, w when uh

Speaker 1: someone clicks on a bad email. or if someone shared this those links uh via some instant messenger and then all those people getting access to those URLs are able to get access to the files which is Uh problematic in my opinion. So yeah, maybe it works, maybe it's it's enough to just obfusc uh obfuscate maybe uh uh the the files, but and uh this is what most people do, but um Yeah, I'm uh I I I wouldn't feel very comfortable if it uh were my users and uh my system.

Speaker 3: Yeah I think another another item on that list is caching because uh if you have these files accessible somewhere they will be cached on in many places. on the internet and they'll just leak uh in in like in the middle. And that's probably not good.

Speaker 1: Yeah. But if you if you use H HTTPS

Speaker 3: Yeah, okay, yeah. In that case they they will be uh they will be quite safe. That's true.

Speaker 1: Oh. I hear some weird audio. Ah, okay. It's oh sorry, it was just loud spawn being loud.

Speaker 3: Well I'm going to log on. Thank you, Johan, for your talk.

Speaker 1: Yeah.

Speaker 3: See you soon.

Speaker 1: Let's see when we yeah, can meet each other again. Bye bye.

Speaker 2: Thank you. Bye.