Files in Django by Josh Schneier

Speaker 1: Awesome, great. Um hi, I'm Josh Schneier for Files in Django, and I actually just made a Twitter account for all of you guys, but I left my phone at the Airbnb, so they've locked it because I didn't give them my phone. But it will be Quack DuckHello, which was the first thing that came came into my mind. So this is files in Django. So just quickly about me and the talk. I've been using Django for a little while, one three. Getting in the custom user model was awesome as I recall, as was uh native migrations. Um, and just keeps getting better. So I maintained Django storages for the last couple years. I gave a lightning talk about it. Um I'll mention Django storages. But this talk is not about that. But I have a lot to say if you're interested. So this talk will proceed in three parts. It's an overview of the file API

Speaker 1: provided by Django, static, media, storage, that sort of thing. of thing. Some options slash recommendations for deployment to production and you know some caveats and things to be aware of. And uh we'll writing a storage backend will actually be demo storage backend. Um it's IPFS, which is quite exciting, and I just got it working. So hopefully it works here. Um so let's go. Uh so static and media files. Uh sorry. Uh part of the reason I I'm giving this talk um and it's on the beginner track is because I had a serious problem uh knowing or understanding or grokking the difference between static and media files. There's all these settings, they're both files, you know, static root and And static URL, media root and media URL. Why are there two? And Django was one of the first pieces of programming I really

Speaker 1: got into. four or five years ago. Um so the basic uh difference between them is quite straightforward. Static files are files that are everything else that power your application. You know, not everything unfortunately can be Python. only. But CSS, JavaScript, image files, and media files are are files that are uploaded by the user. They come from It's unc untrusted content. Static files, you're you're verifying, you know, there it's the defined at collect static time. Media files, profile pictures. What have you. So I think part of the reason that it was a little confusing for me and I think for other people, because I have I used to run an agency and I've onboarded a lot of junior developers and I've noticed uh

Speaker 1: pretty consistent confusion is they are both use the same storage API to store them, especially uh for for local for um development and oftentimes for production. But there is this very important distinction that has important security uh concerns anytime you're dealing with untrusted input. It's bad. Um so about static files. So static files really means the contrib static files app, uh which ships is the it's it's enabled by default when you do start project. Um it's got a bunch of settings. These settings are they all culminate in how collect static works. So Essentially, what you're working with is you have your files in in

Speaker 1: your static uh under static in in your local development, all your dependencies applications. I don't know, debug toolbar and and whatever other admin and third party applications have static files that they're including. Um and You need to include those or else you'll get links breakage. Nothing will work. Um so So yeah, um so it all comes together with collect static. Uh essentially the static file storage executes all of your finders looking through all your directories and placing them into static root. Static URL is just the endpoint that hits to tell Django that hey, this is a static file. Um if anyone was at the USGI talk uh a point that uh yesterday, a point that was made continuously was

Speaker 1: You know, the way that static files are served in development automatically is through run server, which is also a static files management command. And it's it's very insecure. It's never use it for production. And it it's nice because it just works automatically. So that's static files. Media files. Media files has a whole lot more settings. Um and this is largely because you're dealing with things that are coming in from the internet and there's things you want to control, things um that are important to think about, you know, denial of service attacks, that sort of thing. Excuse me. So like I was saying, media root and media URL is confusing for me at first. They're analogous to static root and static URL.

Speaker 1: In so much as static root is where collect static places all of your static files at the end, and media root is where your media files end up These are different because you really don't want someone to upload a static a media file that has a path of your static file that will then get served to all of your users. Maybe the someone uploads jQuery JS and suddenly their JavaScript is executing in in your context if there was a fallback. So there it's enforced by the framework, these have to be different. Paths, they have to be different URLs. For whatever reason, that uh it's not actually clear to me. This is something I add to every project. Um taken from the documentation, which is of course fantastic. Fantastic, the best documentation I've ever seen in a software project, really.

Speaker 1: And it adds serving of media files to your using the same same tool, same tooling, same view as static files. Um to your development. And you can see it's wrapped in settings. debug because never use run server, especially never use run server for serving your uh s media files uh in production. So the third major piece of this that unifies them is the storage API. And it's a pretty straightforward interface. Hasn't really changed that much since 1. 0, modulo, you know, the time zone awareness and a couple other things for max length main. And and such. And the core ships with file system storage, where you know, when you're defining your media route, your static route.

Speaker 1: You see all of a sudden you upload a file all of a sudden off of your media root. You probably forgot to git ignore it and then you went back and added it to git ignore because you really don't want to upload your you know random cat photos that you're using for development, which I I've done before. So there's a fairly straightforward interface. Not all of it is required. And uh yeah, so I'll I'm gonna demonstrate uh bit or or show a bit uh a novel one, um Django Sorges, which is probably the most popular uh package that uh uh uh implements this interface for various backends. Basically is a wrapper around third-party libraries that and this interface. So

Speaker 1: generally speaking, files and core. I was actually not 100% sure where to place this slide if it should go first or last. Um, but I I think that it's kind of unlikely It's not super common to work with the file directly. Um, but what it actually is shipping you is a a Django core files file, which is just a very thin wrapper that does things like content chunking. So usually this is the sort of thing that gets uploaded and you're working with you're working with an image field or you know imit field dot file. You're actually working with a file file file. True story, that's in my template. Uh so and I just want to mention that you can specify which storage to use instead of default file storage. Default file storage is what controls where things

Speaker 1: The storage that is uploaded to images and file fields by default. There is one caveat I'll note or one annoying bug, which is that I find that you don't want to use use the same file storage in development and production. And if you specify separate storages, maybe you need different ACLs or you need different And that's usually not what you want for production. There's an open issue to um turn call uh storage into a callable so you can you could pass your own callback function here um which would which would solve that issue. Um people have opened many many issues on on storages about that Okay, so that was the API overview, and now I would like to talk

Speaker 1: static file survey. Um so we definitely shouldn't use run server. It's been driven in a lot, it's a warning in the docs a lot. Uh so so what what what is to be done? What's next? And of course, we want all the nice things that enable our websites to be fast and performant, things like caching headers, things like um hashed hash files. So there is They're unique. Things like uh Gzip compression, um, minification, which plugs into collect static, uh actually, like you usually use something like compr Or whatever, if you have SAS, you want to transcode it to CSS and minify it. Um so one option which I think is Totally valid is to use a reverse proxy. Whatever you're using for your reverse proxy anyway.

Speaker 1: Gunicorn says you need a reverse proxy because it's vulnerable to slow loris, so you use a rexy like Nginx um to handle this for you. And there's there's a lot of goods about a lot of good things to say about this. The main reason, not the main reason, one important reason you'd want you want to do this is to reduce load on on Django. Python is slow. Even though it's using send file, it's still running through Django, requests that could be better served, you know, answering. Doing LRM queries and looking at APIs and all sorts of context switching that Nginx is highly optimized for this. Um cons, you need to have access to a said reverse proxy. I've deployed a lot of applications to Heroku. it's popular. Um and it's not not really a thing that that you have access to.

Speaker 1: I've also done a lot of work with AWS and I've I've I've done Apache entro and excuse me Nginx for this. It's it's quite fiddly and sort of best practices change. Uh it's hard to get right and easy to get wrong, which are definitely not the same thing. It's not gonna compromise your website probably. But it'll get slower and you won't really realize it because you'll think I I configured it well. I'm done. Um so there are other options. Another possibility is white noise, which is a fantastic package. It burst onto the scene from my perspective two or three years ago. Before that, a lot of people used DJ static and it was sort of, I think people probably Use run server at some point, I'm not sure.

Speaker 1: But uh white noise is is uh is you know I was gonna list about a bunch of possibilities here, but I think sometimes choice is dangerous and white noise just works. It's very easy to install. It ships with all the best practices, including gzip, brotly, hash files, caching headers. It's pure Python. Brotly is a new compression algorithm from Google that's based on like a dictionary of common things. for comp uh beating Huffman encoding for Gzib. And there's actually a open ticket for Django to somehow integrate white noise into uh core, which would be really great because right now it's you need to go third party and white noise is pure Python and just works. It's It's quite nice. Um so

Speaker 1: one of the drawbacks of this of course is you you do end up hitting Python. You know, you do it white noise it's ships it's Django middleware that sees the, you know, the it starts with static uh static URL and and returns the file. So a common thing to do is if your site is small and low traffic, then this is fine. It really, it's not that important. I think there's a lot of premature optimization about Um and it you know white noise sets the right headers. So but it's quite nice to put it behind a CDN content distribution network. So AWS has CloudFront, there's Akamai, there's a lot of them. Basically they have servers everywhere that are closer to your users. They'll request a request will come in for the file. If they don't have it, they'll grab it from your origin server, in this case from Django, via white noise and they'll cache it there at the

Speaker 1: at the edge. So it's very fast, very nice, especially if you know your servers are in Virginia and you have users in high Hong Kong. And one thing that's common, which I actually do not recommend at all, is surveying static files out of S3. It's really common, people said, you know , You know, they use S3 uh storage from from the Django storages and they set the different location locations and and that's just what they do. It's it's very brittle. Um it's like you have to put your Amazon API key somewhere to So you can actually go ahead and deploy. It doesn't, you need to keep some sort of manifest, or else it'll it'll upload the same files every time, or it's making a lot of uh exists API calls. Um So I I actually honestly think that if you're using a CDN with white noise, it'll it'll just work better.

Speaker 1: That being said, for media files, you should definitely not use your own domain. There's a big security risk. People can craft all sorts of fun, malicious content that you know has the file extension of JPEG. But isn't JPEG? It's it's some sort of flash or who knows. There's a million of these attacks. Um Google put out a nice paper saying, you know, this is why we host content from a different domain. Um and you'll see a lot of sites do this. You know, the You have example. com, then you have um uh static dash example. com, but not static dot, because subdomains still can execute in the same security con uh same context. What we're really re uh relying on here is the same origin policy. Um so This is the sort of thing where I think Django Sorge is or similar

Speaker 1: S3 is perfect. Um you still use a CDN because I have noticed that or I think I think S3 is fine. Uh for I've I've worked on a bunch of sites that use our heavy e-commerce. And when you load a full page, S3 is really not meant to be a content-serving platform. It's like a in a fantastic object store, but for serving content It's just I don't know if they're doing IP throttling. There's a lot of it bottlenecks hard. Um so then I put a CDN in front of my S3 bucket and it just it works um really nicely. Um I will say I had a really, really, really fun bug with S3. and course headers and image tags and loading the same image via JavaScript. So you can ask me about that sometime. I had to call up my friend who works on Chrome. Cool. So now let's get to the fun

Speaker 1: fun part. Yeah, fun. So um like I said, I maintain Django storages, it's a bunch of backends, Azure, SFTP, whatever. Um so I thought let's write a storage backend. That was way too ambitious because I just banged that out in the last two hours and I barely got it done. But um so IPFS is the interplanetary file system. I love the name. The lame the name alone would mean I'd do this. But um you know it's a distributed peer-to-peer content. version of mutable storage, every buzzword imaginable. If you saw the file coin initial coin offering, this is these guys. So I just wanted to show off very quickly and then I'll take questions what it looks like. So let's see. So here I am running an IPFS daemon and I quickly

Speaker 1: threw together a demo which has very standard Yeah. Which has very standard model. I think that every everyone should have this model at least maybe hidden and not admin accessible. And I wrote uh please do not use this anywhere. I wrote I wrote what must be one of the most insecure storage backends ever on an IPFS subprocess-based. Um so IPFS source content immutably by its hash. Um so I I r I I uh read through the docs and quickly put together an IPFS-based storage backend.

Speaker 1: Um so This is what this looks like. I would like to go just to cat parties and I have a couple of images and I don't need to s you know terminate with cat because I already got that. So if we just start with party. and I go to Party Cat and I do save, then indeed you can see that this is now saved in my IPFS uh backend. So I have an IPFS node running locally Um if you want to talk about crypto and blockchain, please do not ask any questions about it, but we can talk about it outside or over whatever we're doing later.

Speaker 1: So Questions?

Speaker 2: Content delivery network between your S3 object and um the uh people you're serving it to. Could you elaborate on that some more?

Speaker 1: Sure. So the general uh principle which I went over quickly would be that you then your your links um You'd you'd you'd set a settings where your links or your um static URL rather is is the in the case of um CloudFront, it would be like in you know, cloudfront. net slash whatever. Um and so all of your links that would serve to the user would be those would be links linking to cloud. So when you actually when the user clicks or views or downloads, you know, loads that image tag or what have you, it would go to CloudFront. Cloudfront would check It's local cache. If it doesn't exist, it would go to your server and hit your server, which would return that, setting all sorts of nice caching headers saying, hey CloudFront, since we've hashed this, you know, cache it for 30

Speaker 1: years. And then it would every time that that uh request came back to to to CloudFront, it would already be there, so it wouldn't even have to go to your server.

Speaker 3: So thanks. Um for uh separating uh media content, user uploaded media from the actual static files, does it also include uh Like PDF files or or docs that user upload, you shouldn't save them or serve save them on your own domain.

Speaker 1: Yeah, jet I mean any any user content I would say is J I mean PDF file doesn't it's really hard to validate a PDF file is a PDF file. You know you could read you could use libmagic to read the header, but just because it ends in PDF doesn't mean doesn't mean anything, basically. And PDF is is binary, right? It's not in

Speaker 4: Okay, so great talk by the way. Thank you for that. Um for media uploads, how do you stock? um like different users from uploading or overriding other users files.

Speaker 1: Yeah, um so it dep it depends. Um It there is a for example, there one of the um API methods in that interface for storage is is get available name. So in that your storage backend could check, does this name exist? Maybe you want a user to be able to upload their own file. Um in that case I'd say you probably need some domain logic in front of it because in that case the business logic gets complicated There's a there's a file there's a setting to do this uh at least in like the botto back end, but it's just it's just changing get available name. Um for a lot of files, for a lot of stuff it's it's really nice for me to I'm able to like you know go down directories, so I use the um

Speaker 1: upload to link, which can take a callback to say like hey, put it under this namespace, which I you know I don't let the user control. I say, Oh, it's based on the user as ID. In which case it's almost like they have their own separate file thing. Um there is some trickery, which is why it's always good to use a library where you could imagine someone could upload a file that starts with dot dot dot dot and then suddenly they're doing a directory traversal. And it's dangerous. Um so yeah, that's why this sort of any user stuff uh sorry, security related stuff is usually not the best idea to write your own. I I think there's too much like nobody should write it in the community. But I think it's it really needs a lot of thought put into it.

Speaker 2: Do you have any recommended uh patterns or best practices around access control of files that people upload.

Speaker 1: Yeah, so um I just was d looking at this for for a couple clients. So for the cloud stuff, it's pretty normal to make you know your your button For an S3 's uh case is private by default, the ACL. And then you have to have um signed URLs. Uh but anyone with the signed URL, which you we can include, include a timeout can get it. Um past that, you really end up having to do something where you put because at that without that, you know, that's something that that Amazon is mediating with you. the inner the provider I'm most familiar with.

Speaker 1: Past that you end up having to just put your own view in front of it, which you know it gets much slower because then you need to do the logic yourself, then you need to fetch the file and then you need to serve it. Um whereas if it's coming directly from Amazon, you know, you generate the link, you give the link to the user. And you know, you could you could go to that directory until cacals come home if you don't have the the the hashed But there is, I mean there is, you know, I have my users uh the links expire after an hour. I've already had multiple people say, hey, it's broken. Like, well, I could make it never expire, but then you know if the link gets out, it's never it's not private anymore. Or depends on your trade-offs that you're willing to to look at.

Speaker 5: Any other questions? All right, thanks, Josh.