The evolution of a RESTful Django backend

Yeah, so it's great to see everybody here and it's nice to be in Portland where I live and I love the city so it's I'm glad you guys get to uh experience it. This talk how I envision this talk is giving some insights from real engineers in the production systems that Safari Books Online has deployed about various Rust uh our use of various web services libraries to build Django APIs. And it's sort of an evolution because You know, this we started kind of with one thing and then we tried something else, and now we're doing this other thing. And it's changed over time and we have a variety of libraries uh in production right now. So we'll kind of talk through uh you know, three or three or four of these things

and what I'm what I'm gonna do is give you some uh real words from from different engineers. Um not just me and not not necessarily like Safari's position, but um what we've experienced, like the good things and and some of the bad. But at first I have a question. Are any of you creators or authors of a Django Web Services Library? Okay, good. Good. So tone down the the the stress a little bit but uh so we will be talking about some cons and it's not a not a judgment necessarily. Alright so quick introductions uh before I get into it So a little bit about you, who I think you might be for this talk. You're familiar with Django and REST.

You want to hear about some real-world uses of Django, like I just described, uh Django Web Services Libraries to write APIs. And quickly about me at Safari, I'm a senior software engineer. I write front-end code and I write some back-end code and mostly back-end right now. I also like to write text editor plugins for some reason. I don't know what it is for different editors. So there you go. And a little bit about Safari since they sent me here and I got to use this. uh nice uh PowerPoint template to make this presentation. Uh you know it's sort of you might know us by the the longer form, Safari Books Online. We're kind of throwing around Safari now, it's a little bit shorter.

And just sort of our catchphrase, you know, is we're we're bringing the best uh books and courses to life. So we want to give you guys access to videos, books, whatever it is you need to learn. Come come and and look at our site and learn. But a spoiler alert, if you need to jump out, I want you to leave with something from this talk. Um so I will tell you the end of the talk before we do the entire talk And that is that we use Django REST framework for our new APIs. And we'll get into some reasons why. There's a long list of things we like about it. Uh but if you need to leave with something and I'm sure you've already heard this many times, this probably isn't new, but Jenga Rust Framework is is good Uh so let's start at the beginning, right? And this is in

some ways like the beginning, maybe it's the beginning of your prototype. We can also look at it as before we did, we before we use a library for doing our web services. We use some plain views and I've also used some prototypes as well, as I mentioned. So let's look at some pros of using just vanilla Django views for your API. We won't spend a lot of time on this, but you guys you may know or may not know, they're good for returning like snippets of rendered HTML from a template or something really simple like you want to just expose a read-only API for some JSON or XML data. So all good. They give you complete control over the structure and output of your your uh your data and you can do whatever you want in your view function or class-based view to get this done.

Um and a little helper I find nice if I'm doing a prototype and I want to use vanilla Django views is JSON View. It's actually called Django JSON View. It basically gives you this little decorator you put on top of your view and uh it it helps manage like the Serializing, deserializing the Python data structures that just JSON. It's just like handy, you know, cut down some of that boilerplate. But of course, there are some cons, like why would we have all these library frameworks if there weren't some nasty things about using vanilla views? Um you wind up with a lot of boilerplate. I'm sure if you've written if you've done this, you know, you know, you you wind up with a lot of stuff. A lot of it is around validating input data for over the line. It's a lot about um Setting up the deserializ

uh serialization and deserialization for multiple content types and so forth. Um having total control can lead to inconsistency of your responses. across and within projects. So maybe you have multiple apps that have APIs. You might have developers doing stuff differently in different apps and suddenly, you know The errors you get back when there's an error are slightly a different different structure and the clients have to code differently. Blah blah blah. It's just not that fun. And you might end up with multiple views. Well you will end up with multiple views for different content types that help manage, you know, I want my stuff back in XML or JSON. Uh it's not so easy if it's plain Django view. You might have to do some extra stuff All right. You

already know all this. I want to look at Piston, mostly because it's what we used, I think, as our first real like web services library. Probably everybody knows you know piston isn't the greatest choice at this point, um not to malign piston, but we'll talk about it because this is a history, this is an evolution. Uh and and so we'll talk about pros of Piston. There's probably more than this. This is not very nice, right? But the major pro if you were to delete, use it today, is you can get a a simple API up and running. Probably any library can do that, but um There you go. I have a lot of cons, so keep this short. So we ran into some problems with Piston. Uh number one problem, of course, if you've used Piston, it's really not being maintained. It's kind of a dead library at this point. The last

Release was on PyPy was in like 2011. So uh you know we we perceive that the community is drying up. We still have APIs that were written in Piston, but we're probably going to migrate them over to Django Rest framework. And I think we have migrated some over to TasePy and we might mind up migrating those to DRF as well. Uh so real problems like one of the real problems we had is there's not an explicit serializer really for your uh your API classes. So it kind of makes it you know it's a little bit sticky to manage the uh or to mangle the output of the data. So if I want to add in some you know dynamically calculated field in my in my JSON that I'm returning. uh a lot of that logic winds up inside of this one monolithic API class.

Uh and that's just it's it doesn't it it's not good for maintainability. You know, this is a problem that actually is in a lot of different libraries, um, but Piston is one where we found it the first time. Uh and there's no pagination out of the box. So Piston didn't ship with uh the ability to paginate results set your results set in in the API. Turns out that's pretty useful if you are returning lots of lots of items in a list. Um And there's more, right? There's other ones, but why should we really spend a lot of time on Piston at this point? Um I think Piston was a great library. When I first started using it, I was like, dang, this is way better than writing vanilla views. You know, this is awesome And it paved the way. So props to piston. The next thing we settled on was Tasty

Pie. So What are some pros of TCP? People are still thinking, well, do I use TCPi? Do I use DRF at this point? Um so what can TCP give you? What did it give us? Um it gave us pagination out of the box, which was awesome. Uh and then so it was a little bit a little bit better, right? There was uh there's a couple improvements over Piston. One of them was uh it was easier to mig to to uh change the output data we are sending back because TCPI has this thing, these these methods you can override for hydrating and dehydrating, which is another, you know, it's just another for serializing, disrupt deserializing your data structures. So that was nice. It was an improvement.

And we found the resource object that you subclass to be relatively straightforward in terms of its APIs, you know, interacting with it. We'll talk a little bit more about it, but on the surface. Pretty straightforward. And then one of the cool things about TCPI, which is still cool, is that uh it will generate a schema based on your API automatically. So if you have a client that can do something with that schema, which is sort of a big if, but if you have one, then that's pretty helpful to do client-side validation, then you're also going to do your server-side validation as well. And it handles setting up the URLs for you, which is kind of a double-edged sword, right? It's like that's nice. It's fast. Uh it is not necessarily

Pythonic just an automatic URL thing, um in my opinion, because you know I kind of like having you know my URLs pretty explicit, but a lot of frameworks um do that for you. You know uh Rails does it as well. Um and these other libraries like Django Rest framework have a have a router. Anyway, pros. So a couple cons. We kinda ran into some big problems using HCPi and they weren't really things we expected. Uh at Safari, a lot of our back-end APIs are like uh it's like a pipeline of content, right? So we get a bunch of stuff from publishers, we get lots of material from them. A lot of it is like EPUBs and PDFs. Some of these files are pretty large, even when you chunk them. And TastyPy didn't handle like binary

types. very well. There's an open issue I linked to. I didn't want to just be like, ugh, this didn't work that well. So I I put a link in here if you can if you get access to the PDF version or not, you can check it out. And then I think this is more like um we end up having to do some custom stuff, yada yada yada, to get it to get a file upload working against the API. Um If you read that issue, you know, there's like, oh, I got this fork and you can use it, you know. So one of the problems we ran into was like high memory usage uh on a receiving a file upload because it wasn't doing any kind of streaming. It would just buffer the entire thing into memory and they were uploading huge objects a lot of times.

So another thing was that we ran into was uh it it turned out to be, while the resource object was easy to use, it turned out that it was not so easy to extend lots of parts of Tasty Pie in our in our opinion. And again this is multiple engineers give me feedback on this so it's not like we all had the same problems, but this was a perceived difficulty. Um and the same with piston is and this I've seen this is you generally end up with a single API class that's kind of responsible for doing everything. Like it does the it handles all the custom serialization, deserialization logic. and a bunch of other you know fields that control like authentication and stuff. So that is what it is, you know, but it's it's generally a lot of code in one place.

responsible for a lot of different things, which causes a smell about it uh when you have to maintain it. Uh and uh uh one of one of the engineers told me that Tasty Pie had too much magic, which you can take with a grain of salt, because actually this this person uh uses a lot of generator expressions, which if you've had to maintain are interesting to uh to debug sometimes. Um So another one, this is kind of an interesting point, right? So a big thing about uh web services libraries is You know, you have all this validation that you need to do. Like you need to give, like we want to give the client a schema so the scheme so the client can do some validation before it uses any requests. To send any data over at all. But you also need to do all that same validation

on the server side when you receive data. And uh Piston, I think, and TastyPie both approach this by using Django's excellent already existing code to do input validation, which is the form lab the form code, right? So You would create a form and you check your impound data against this form schema and this form acts like a schema, right? So you can say, oh, this is this is not typed correctly, yada yada yada. Send back uh however you want. But that's kind of weird um i for us. Like we don't really like using forms to validate Like JSON data coming in from an API, it's not specialized to that purpose. It's they're kind of specialized to the purpose of handling like form inputs, right? So one of the things like say you want to output

The error messages that the form generated from validation because of the validation problems, right? You have to do some funky stuff like get in there and Kind of get the uh get the there's not a clean API to get like simple text out of the former sometimes. Or maybe there is now, but there wasn't. I think there is actually in s 1. 7. But A little bit odd. Alright, so we were in these problems. We didn't know what to do. There's so many options. And then uh we had a new engineer come to us. And we asked him, hey, what do you think about Django Rust Framework? Like, is it cool? And uh to my knowledge, he told us that he thought it was cool. And because he's a new engineer, I think we wanted to really look at his input

because of new people on the team and the audience, good. So we took a look at JGORES framework. And turned out uh it was awesome. It was actually really awesome. So I actually had to break up list of good things about Django Bros former into two parts because there were so many things we liked about it. And I didn't really want to leave any of these out. I actually had to leave out a bunch of stuff out. Um and I put this one at the top. So the ordering of these is based on me. It's sort of arbitrary, right? But uh the number one thing about Django West framework that I really love that we have found is how well documented it is. Like you can go to the site today. You don't need to attend to talk about Jenger. I mean you should if you're interested, but you don't need to read anything but the docs to really get up and running and also Beyond that, to really go deep into understanding how part of parts of it work.

So it's very well done, and the creator uh is very attentive on Stack Overflow form, asking questions. It feels good. And you know, you are everybody's er, you know, one of the first things you see about Django Rust Framework is, how's this awesome uh HTTP REST API browser? You've probably seen this already, right? But Django Rust Framework, it creates this HTML view that you can access as a human being that shows you information about the API. It's awesome. I don't know if it's really number two on my list, but it's it's something that everybody likes when they first start using it, right? But some real stuff, like what do what are we what did we really find useful? Um out-of-the-box pagination and this filtering feature it has. were really useful to us as we were upgrading some of our old APIs.

And we upgraded a bunch of APIs from TCPI to Django REST framework. And this came in handy. Uh and as just as an antidote, not a little as a bullet point, but as an antidote, that API I'm not saying it's T C Py's fault But um there were a lot of pieces of that API that we upgraded that were very poorly performant in in memory. And you could look in New Relic, we get the alerts, you know, and you're like, oh God, spew, it's just crushing the server. Like So much RAM is being used by this stuff. And we try to fix it and you know we get somewhere and here and there. That doesn't happen anymore with our Django Rust framework uh version. So It's more complex than oh we switched to Django Rust framework. We didn't have any problems anymore. Um we made some changes based on what we learned wasn't working, uh

but also I f it's my perception that performance is a little bit better in Django Rust framework. Uh and so back to the bullet points. It uh it supports tokenoth out of the box, uh which is just nice if you want to get up and running with uh token auth. Like yeah, you just drop it in, it's gonna start working. But there's more. Uh there's more. So uh it is very easy to customize output. From your APIs, including resource relationships. So in REST, right, we have the we have this thing called the resource and they relate to each other. Uh and it's easy to set that all up and get it the way you want it with URLs or IDs or whatever with these explicit serializer classes. So the thing about Django Rust framework is you end up with a bunch of composed

objects, right? So you have your thing and it has a serializer, it has a filter class, yada yada. It's not just one big class that does everything. It's composed into pieces that make sense, that are separately testable, reusable in different contexts. Uh and one of those things is a serializer. Okay, and here's a cool thing. So server-side validation. You're gonna have server-side validation different stages. Um a lot of people who will use Validation routines in the model classes to do validation. So a great thing about Django Rest framework is when it's creating something over the API, it will call full clean on your model. So you can stick all your business rule validation into full clean. and use it. And Django Rest framework will use that. You don't have to abstract it into something else and then that you know

use that in your API and call it from your full clean or whatever. It's just you can put it where it sort of belongs and it will get used, which is nice. And sort of bouncing around here, but the community itself is very active, very helpful, we have found, which is very, very important. Obviously, and you know, more about like beauty, right? We perceive it to be elegant, and that's important. This is Python. Um Our perception of beauty in our team is actually a high criteria kind of important thing to us. And looking at the code, but also looking at what we got out of it, you know, what does our API code actually look like? We find it to be elegant, we find it to be extensible,

and that makes us happy every day. So plus one. And sort of related, your API representation ends up being pretty concise. And it's, you know, this is I didn't write this bullet point myself, but you know what you end up with is, like I said, it's a bunch of composed objects. So It is concise in the sense that here's this one API. It looks pretty small when I look at the high-level view, right? And then I'm going to go down and I'm going to look at the individual composed classes like the filter. or whatever and I and you're going to expand out into multiple classes, but it's nice to be able to see a concise overview sort of at the at the top level of the object hierarchy. Class hierarchy, we'll say actually. So um

but I am not going to say that it's the best thing on earth. You know, the everything has a couple flaws or whatever. These are flaws. We found some cons. Um so one thing that we really miss about TastyPie is there's no built-in schema generation. So uh clients can't auto-discover that stuff. And we have to maintain our own schema, which is pretty common um when you have an API. I guess. Uh but I I I don't really like it. I don't like having to go in and like, oh I changed the thing in the API that oh right, I gotta go in and update the schema file and yeah That's fine. We have a client that uh can read this schema and knows what to do with it, our own like in-house rest client.

So that's a thing. Maybe someone will release some package that does this for us someday and I won't have to do this anymore. I hope. Then there can still be lots of boilerplate. So it's an interesting thing about this, right? I'm pretty sure there's a bullet point I either removed or put on every single one of these sections, that there's bullet boilerplate. I think that's just programming like you. Actually have to write code with no matter what library you choose, right? So take it with a grain of salt. But exposing filtering mechanisms we found can sometimes be convoluted, and I think There's different things going on here, right? It's Django Rust Framework gives you this thing that is the ability to filter results in the URL.

And it kind of feels to me, it feels a lot like your uh filter parameter you're gonna pass in to the Django ORM, right? But it's not really the same. Um and there's a little bit of cognitive dissonance there, I think. Uh And you uh you kind of have to do some stuff to expose these different types of filtering options within your URLs. So it can be a little convoluted And somebody informed me that uh multi model fields are read-only, which I don't think is gonna be a game changer. I think you can, you know, probably by default, yada yada, you do some overrides and you can we do set. uh related fields and related objects, but you have to do some you have to actually write some code, I think. Terrible.

Um and some of us like the way that routers are implemented. And if you're not familiar with router , router is is this thing you can use to uh like say you have um This API code you wrote for a book or books, we'll say. It returns information about books, and you can use a router object to like automatically generate some URLs for it. I don't really have a problem with it, but you know, we're we're a diverse group of folks. I want to give the voice, honor the voice of the engineer who did not actually like that, how they were implemented. So uh I'm almost out of time here, I think. Uh we'll do a quick summary. So sometimes you can get away with a plain Django view I often when I'm prototyping

will use just straight up views and then uh if it turns out we're gonna do something, like maybe we go back and and add it. You do it right with a with a library. Um Which I gotta stop doing I think like uh it's uh it might be easier just to start with the library. The Django Rust memory is pretty simple to set up, so Um and I want to re-emphasize that I don't think that piston is bad and even that you know Tasty Pie was bad. Um Piston got us rolling, you know, it was Library that I started with like back at when I worked at Dark Horse for a while doing uh digital publishing. It was great, you know, it was a good start Sort of paved the way. Um Tisy TastyPie was a great improvement over Piston.

It gave us all these nice features. And you know everything depends on something else, right? You can't have you can't get up here without Starting here and all these libraries sort of build on what we learn. Um Django Rust Framework, I think, has learned a lot from TastyPie, has learned a lot from Piston. and other stuff. I'm not I can't speak for the author um to say oh I know how he how he figured out you know how to do this. But I think we learn and we are using Django Rust framework for new APIs because we think it's great. So that's the takeaway. I am free after the talk to answer any like questions you might have about any of these points or to uh or to chat. So thank you very much.

Yeah.