Building a REST API with Django & Django REST Framework

Thank you for coming to the presentation. Thank you to SAS for sending me out here. Uh so I've been with SAS since January of this year. That's really when I first started using uh Python and Django. Um so uh what I'll talk today about is just some of the pitfalls I encountered in building a project and uh how we overcame those and hopefully that will help some of you out. So when I joined SAS, uh the project I had was to build a REST API. They were looking to automate uh firewall rule edition So I was working for a network team that's part of a larger uh IT group. But they were looking to automate this because it was kind of a pain point in terms of time that they were spending on it. There are requirements for to build an application that would support direct API calls and provide a self-service portal through our just internal web.

And they were they had a preference for Python, and that was because some of the switches that they were getting had Python preloaded on those, and they just wanted to go ahead and start building an expertise in that language. So they already had the input for this application defined. We had a customer, source, destination IP address, destination port, and then we had an optional business justification where a user could say this is why I'm requesting this. Alright, so I had to select a starting point coming in. They had the Python preference, but they pretty much gave me free reign. They said, look at what would be the best way to build this REST API, and we'll go forward in that direction. So I started looking at Django because it was something I had heard of. It was something I was interested in learning. I just hadn't had really a chance to do it before then.

It was Python-based, uh, and then also the built-in admin interface. It looked pretty and it looked really nice and it looked like you could get it with very little work. And then in doing some more research, I stumbled across the Django Rust framework. which offered the browsable API, which also looked very pretty, looked like it gave you a lot for very little work. I did have someone recommend looking at Flask. And uh to me it just seemed really complex getting into it. If I looked at it today, it'd probably be easier for me, but at that time I didn't know a lot about Python. So I started doing some prototyping with Django, Django REST framework, got something up and running fairly quickly, and I was able to demo that to our management team and then our internal customers.

and it was well received so we went forward with that. So in in learning how to use Django, Django Rust Framework, I I had to go through all the tutorials and things. I think several other presenters have covered a lot of this material, so I won't go in depth. But you have your models. This is a way to define database tables. You have serializers. For our application, we use JSON as our data format. And then for your views, you're taking in a request, you're doing something, and then you're sending back a response. You have your view sets, which can be collections of related views. And you have your URLs to map the addresses to places and your routers and then settings, of course, where you can configure the application. For the data formats, you have different renderers available.

So the one we're using is the JSON renderer. There is XML and there's some other ones out there too. And of course the browsable API renderer. That's a big part of our project. So for our application, we've gone through two iterations at this point. Both of those gone into production environment. The most recent one went in last week And we've definitely hit some pitfalls along the way and worked our way around it. And that's what I'll get into now. So one thing to consider when you're building a REST API is for your views, who do you want to access those views? You can certainly use the user and groups table as a way of maintaining membership just of the application and the views themselves

from an application standpoint. Django does allow for you to set whether or not you want to automatically add a new user, and if you choose to do so, you can even customize how that user object gets created. So in this case someone tries to get to your application, they try to log in. If Django says, hey, I don't know who you are, then if you've got this set to automatically add them, it'll do that and build it the way you want, or build that user object the way you told it to And there's default settings for that. For us, we didn't want that. If someone came to our application and we didn't know who they were, well, we just didn't want them to use the application. So we we turn that off. Another way we were controlling access to our views were to just check the user data from the request. So the request comes in, and I think it's request. user, you get an object for whoever's accessing the application.

or that view. And so then even if someone's not logged in and you have a view that doesn't require a login, you can get an anonymous user object through that. And You can then build in logic into your view to check who is this user and what do I want them to be able to do. For us, well we in our second iteration we added uh LDAP authentication. So we're actually querying Active Directory. to say, okay, this is the user that's accessing our application. Do they belong to a sp specific Active Directory group? If they do, okay, you can use the application. And if they don't, then we'll return an appropriate response code instead. Uh so uh for disabling a view, that's something you could also do. Uh one one thing is if you're writing your own custom views

Uh so maybe you've written a git view and not a post view. So that may take care of itself if you don't already have a certain view out there. Uh for us we're using the view sets. uh so that we just have all the REST API actions already available. Uh that meets our internal company REST API standards. Uh and it also gives us uh The intent is to give us some growing room. Like right now, we don't support a delete method, but we have it in there because eventually we may. So if someone calls delete on any of our views, we're just going to return a 403 forbidden and just say, hey, you can't do this. But depending on your application needs, you could certainly return a different type of response code. Alright, so as we got to iteration two, our requirements changed, and of course

this impacted our underlying logic for our application. Uh instead of accepting a single request on a firewall change We were now taking a list of requests. And then instead of taking just a single IPv4 address for a source and destination, we now had to take that, also IP ranges, and also subnet masks. And in addition to that, instead of just one value, we may have received a list of values. And then similarly for destination port, we could take a single port or a range of ports or a list of any combination of those types of input. And then business justification. One of our internal customers came to us and they said, hey, we really think that this needs to be required. Now we we don't want you to process anything unless we have that data there.

Someone's entered something in So we had to build that back in. And then uh so in in iteration one we built our our own uh web form to self to serve as our self-service portal. And just used uh Bootstrap JS for that. But in iteration two, we had one of the company standard uh IT groups provide our front end. And this group was called IT Change. So ITCH had its own own user, appropriately named ITCHE. So if uh if our view was called, we could get that user and say, hey, this is ITCHange calling us. And if they call us, well, we have to keep a record of every firewall change that we actually make, and we need to tie that back to a user. So we can say, hey, this is ITCHange that called us, and ITChange

has to provide us with a user value so that we know who actually submitted that request. Now if someone directly called our API, we're just going to already have who that user is, so we don't have to check for that value. But for uh IT change we say, hey, are you IT change? Did you give us a user value And if so, we keep going. And if not, we return uh I think it's error 400, which is a bad bad request or bad data. All right, so uh validators. Uh we're taking in all kinds of input now. In iteration one, uh we're using basic things. So we were able to use an existing validator for the IPv4 address. There's some text validation. I think we just did a simple integer validation for the port numbers. Uh when we got to iteration two, that didn't quite work for us.

Uh we had to build in some regular expressions so that we could check to see the data that we were receiving if it matched up. You know, was this a range of IP addresses? Did this IP address come with a subnet mask? Uh what about the port numbers? Was it just a single port number? Was it uh one through three, something like that? Uh you know, it's very small, but one through a thousand even. So we had to write our own validators for that. And one of the things we also were able to do was to just take and group validators together. So for your models uh you can declare an actual validator for things. And I think it just points to one or at least that was my understanding when I was going through uh the development. So I'd point it to a single validator that was a group, and within that single validator, I could call other validators.

And for those If if any of those throws a validation error, because if you have a problem you want it to raise a validation error, but you can kind of put in logic. Maybe you have five validators that are part of this group validator And as long as the data comes in and passes one of those validators, it's okay. So you just handle uh any exceptions you come across or anything and If you deem it invalid, you return that validation error and you can pass in data for that so that you're telling your user, hey, this is why it was invalid, and then they can go and adjust whatever it was they were given to the application. uh so they can have a successful request on their next attempt. Alright, so for the data going around in the model, uh one thing that was kind of a pitfall for us was in iteration one, we we're just going with single IP addresses

We knew at some point we may bring in ranges or subnet masks, but we we didn't put a lot of forethought into that. We just, okay, we're we're handling an IP address, we'll deal with that when we get there. Uh we we should have put more thought into it, could have planned a little bit better. So what we ended up doing in iteration two was building a bit of an umbrella model. And this may not be a best practice, but we found it useful within our group and our purposes. And this is just a big model for our API entry point, which says this is all the type of data, all the types of data that we may need now or eventually, and we put some of that in there. Now if it's something we needed eventually, we could make it an optional field, at least for now. But we we never save this model to the database. We just simply use it for when we're accepting input.

And then we have subset models that have the actual data we're saving to the database for our record purposes. Another thing to keep in mind is what do you need to tell the user after you've completed an action? So they've called your API, you've done some logic behind the scenes, what do you need to give them back? For us, uh You know, record keeping is a big deal. We need to be able to map who's submitted these changes, why were they done? So we associated an ID with that. So our request comes in, we do all our logic behind the scenes, then we write all the records to the database, and we associate that group of actions with a single ID. we give that to the user. So if the user has a question later on, they can come back and they can reference that ID with us. If they just come back and say, hey, we had a problem,

you know, your your service gave us a 503 service unavailable. Well we we can't do a lot with that. Um but I'll uh well I guess that wouldn't apply there, but if uh they put in a request and they say I put this request in, but I can't reach uh uh the destination IP from that source IP. What's going on? Well if they give us a request number, we can go look it up and see what was actually done on the back end for that and see if we have any logs associated with that. Was there an error? Did we capture something? We have that point of reference. Another thing to keep in mind is the error data. So for the user, we want to gracefully handle any errors that may occur. And we just want to give them, uh right now we just give them a 503. Hey, service is unavailable. You should try again later, or you can contact the support team.

On the other hand, for our support staff, we really need a lot more data than that. The user doesn't really care. If it's not working, then that's all they need to know. They can let us know, hey, it didn't work. But we need to go back and figure out what went wrong. So we tried to do two things. One of those is we try to s we capture all the information that's available at that time. Put that into an email and send an email to the support staff so that we can hopefully have a almost real-time um Alert of that error. Another thing we try to do is to log the error information, that same information, into a database. And so then we have that record in there and we can go back and look it up. Now if both of those methods fail, we have something catastrophic going on, but at least we tried.

So So in going from iteration one to iteration two, we had to start accepting multiple input types of, as I mentioned a moment ago. So what we were aiming to do is we could have done an entirely different API call to accept these different types of formats. But as part of our iteration one deliverable, we already had internal teams that were looking at our API, learning our API, starting to write automation that would utilize our API. So we want it to be as minimally impactful to them as we could be. So the one one big change was having that business justification, whereas before it was optional, now it was required, but that that was hopefully a minor hit to them. But we we uh basically redesigned our existing API call to support this uh these multiple inputs.

And we did that by taking the request that came into the view and building logic in to examine, do kind of a pre-check of the input we received. So one of the first things we do is say, for the request that came in, is it a list? And if it's not a list, we add it to a list. If it is a list, we're okay. And then we take for the IP addresses, the ports, we say, are these list? And if they they are a list, okay. If they're not a list, we add those, put those into a list. And so then all of our logic that's going on beyond that point can treat all the input as if it is a list because we've guaranteed it is. And that just made writing that uh the logic further down a lot easier Another thing that we didn't have in iteration one

that would have been great was to have a version with our API. Because then going to iteration two, we could have just had uh our customers call us and give us a different version number and we could have broken out logic to handle uh that input differently. So we did version our API in uh iteration two or yeah iteration two Uh when I was versioning it, I came across two schools of thought on how to actually version it. One of those being to put a version number in the accept header and another to be was uh putting it into the URL. The problem we found with if we went forward with the URL is if we have some address slash V1 and then we move to a version two. Well, if we want to maintain backwards compatibility, we now have those two addresses to deal with.

So we've got to keep V1 working, and then we've got to keep maybe V2 working. So we really didn't want to go that route. We went with just the HTTP accept header. It can come in. If a user doesn't give that to us, we assume, hey, we're going to use version 1. 0. And as we go forward and increment from here, we can then branch out the logic in our code based on a version number we actually receive. Alright, so for the save process uh for the model, you can actually extend that uh in iteration one. I I'll chalk this up to just learning. I'd put a lot of logic into the model save method. So it was doing a lot of things, checking the data before it actually saved the data to the database. So we took and extracted all of that out in iteration two

, and our logic is uh done in a in our view file and then a separate uh logic file. But you can override the save method if you need to. Now, as we were doing things in iteration two, we had to save a lot more to the database. Uh we were breaking things out into uh permutations of combinations of all these IP addresses since we could now receive these lists of input. So as we started to save these entries to the database, we immediately saw performance impacts. It was just taking a lot of time to save all that data. And we were trying to throw large quantities of data at it for the input to do kind of internal load testing. So what we found was we could use Bulk Create to have all of these objects written to the database as part of one transaction.

The problem we had with that was that we had a lot of foreign keys. So these were other objects that really needed to go into a main object, our main request object. So what we ended up having to do was a combination. For any type of object we were saving where we were just going to have maybe a handful of those. uh and they were foreign keys, we just individually save those m uh models. And then for our bigger request permutations, Since we hate we could have a multitude of those, uh by that time we already have our foreign uh objects, uh foreign key objects, because As we've saved those, we've gotten either a foreign key ID back or we just have the object itself. And we can insert that into We basically build a big list of all these other objects we need to save, throw the foreign keys in there as we're building those objects, and once we have that big list.

we can call bulk create on that. And we immediately saw the performance issues go away from that. We may I've learned some things here at DjangoCon that I can go back and look at and maybe we can optimize it even more. But that for us was an immediate uh savior. Documentation considerations, of course the built-in browsable API is very powerful for doing that. Our internal customers have found it very valuable. It allows them to play with our API. We can give them a sandbox system and they can just have at it. So they've really liked that. You can also use it to see the different data formats so they can see the JSON response that we're going to give them and they can go back and write automation on their end that can handle that and process it. I looked at Swagger, which is very similar to the browsable API.

It's much more beautiful in my opinion, but when I looked at this, it was probably in a March or April timeframe And they were having some browser incompatibility issues. I think those have been resolved. But I was still wanting to, for some of the actions, it just listed every REST API action. And I wanted to go in and customize it and say, well, okay, you can call delete on our API, but we really don't want you to right now. So I don't want that to show up in the uh in that web view. And I couldn't find an easy way to remove that. So it's something I'll go back to and look at. And it may be a lot easier to do that by now. Another thing we do is uh we have uh our own internal team wiki, uh which is accessible from anyone in our company.

Uh but we or document our API there, we're putting out uh all the different use cases. So if you give us a successful request, this is the type of data you can expect back. If you give us a bad request, here's what you'll get back. If you have an error, then this is what we will give you back. And that way all the teams that are building automation based on our API have that available for them. So some other lessons I learned along the way is we did have some legacy databases, and I think another talk has already covered some of that. So we're running MySQL on our for our database and uh We found that we could run the inspect db command and it would actually take all the pre-existing uh database tables and generate model code for that.

The problem we found there is we were looking to not only pull this in as uh for the models, but also so that we could manage these legacy databases from our admin interface. And we needed to write to it as part of our request processing. So if we were to save one of these models using the automatically generated code, The Inspect DB had made uh the auto increment field and integer field in the Django model code Which meant that Django did not automatically give us an ID back after we saved that model. So the quick fix we found for that was to change it from an integer field into an auto field. And then we could save that model, we get that ID back, and we can can continue uh our processing using that.

And that that may be improved in 1. 7, I'm not sure. We were using 1. 65. So another thing, once again, we were using MySQL, still are. We started seeing issues where it said MySQL server has gone away. Server was there, we could access it, there's no problem with it, uh that we could access the database. This just seemed to happen, uh it didn't We we couldn't find any reason behind uh oh we had a connection open too long. It just seemed random to us. Uh and the information I could find online The appropriate way to handle this was to just close that database connection before you tried to uh have any database transaction. I I didn't like that response. I'm hopeful that maybe that will change eventually, but that was the way it was at that time.

And we put that in there. So we had uh Database, uh maybe database. closeconnection was the command. Uh we put that in there and then we do a model. save. The issue went away. But then we ran our unit testing and that broke because it just didn't like that you were trying to close the connection to the test database. So we ended up going to our settings file and we set uh setting in there unit testing, true or false Uh so when we run our code uh we we go based on that and when our y uh so our code logic will actually say If we're not unit testing, close that connection and then do a model. save. Now if we're unit testing, it just ignores that and we just save the model. And that 's worked for us so far. Another thing we're doing is using the coverage model and that way we can run our unit test

and we can actually have some nice H2Mail created where we can look at the code coverage. We can go see what lines of code we haven't uh actually uh test it and we can go back and try to hit those and uh we've successfully maintained eighty percent code coverage for all of our production iterations so far and we'll we'll try to keep that higher but at least eighty percent is our minimal. Some other things are we had this IT Change is providing our web form. And they're using JavaScript to build their form. and it could not work with our code uh until we added certain HTTP response headers. It was really easy to do, just in your view code, you can add logic to add uh data to the headers, set the headers. So we did that, no problem.

But uh we encountered yet another issue. So we had uh our application is available, it's running on its own servers, running Apache. Uh We have to support two different uh methods of accessing our application just based on our own internal limitations. Uh so ITCHange has to call our API through just the server address. And then any other user in our company has to call it through basically a proxy. And when you're trying to run both of those together on the same system, you start to have some problems. So for us, we tried configuring our settings files. We tried to have multiple settings files that could uh They basically inherit it from master settings files.

And then based on the server we're running on and based on whether IT change called us or just some other user called us. we would then select the appropriate settings files, set those settings. Well, what I found with running Apache, and this is something I'll deal with when I go back. is once the first call is made, those settings are set. So if someone else comes in, the settings didn't change. And so I've got to go back and look at that. So we ended up having to uh manipulate our settings files a little bit more so that a single settings file worked. regardless of whether ITChange called us or just some generic user. But uh another thing we hit there was the browsable API, as great as it is,

uh it was automatically generating some of the links that we had and it was generating them for our actual direct server address and we didn't want that for our general users. They were the ones that were going to actually use the browsable API And IT change wouldn't. So we wanted the general users to have the link that went to the actual correct address so they could still use the API. So we had to go in, customize the template, we had to customize the router for that that actually generated the link. But the good thing there was it was all customizable. It was just learning how to do it. The fourth script name there, that was uh We basically had our internal site. sass. com slash our application. So you had to set our application as force script name to get that to show up in the link.

All right, so in closing, uh I found Django, Django Rust Framework to be a very easy entry point for myself. I I was definitely a novice at the time. still am in some ways, but it was very easy for us to get something up and running quickly. We found it to be very customizable as our usage needs advanced. For me, my management team was more than willing to give me the time I needed to explore and figure out how to get things to work. So that that's definitely a key. If you have the time to put into it, you can ac you can customize it just uh to no limit it seems Alright, so I've got uh email, Twitter if you want to follow up with me. I've got my slides posted. They've been on on there the whole time. Um any questions?

I don't think it's a good one.