How to Solve Django's Top 5 Enterprise Headaches

Good afternoon everyone. My name is Matt Mackay and I am a developer of Angelus Fritulio. But before I started Atwilio, I was doing consulting work in large enterprises across government, nonprofit, and commercial organizations. And I discovered that there were a lot of things that came up on every project, drove me nuts, were incredibly frustrating, and when I heard that there was going to be an enterprise-related track at DjangoCon, I thought This would be the time for me to vent my frustration into a presentation and then also hopefully help you guys in these situations. to solve some of these problems. So like a good consultant, I've decided I was going well I decided I was going to whiteboard this out, but there's no whiteboards in here, so I just drew this ahead of time. So this is my whiteboard version of my presentation.

And I've also drawn some pictures to go along with this. And I am going to use the term client uh to represent the sort of boss figure, the the person that is driving what the the feature requests are. So for people that are Staff, uh it's probably just the boss. Uh for me in a client consulting relationship, I'm just gonna say client, so just so you know what it is. And so the things that that I would get requested all the time from clients Here's the top five things that I've on every project and it would drive me nuts. The first one was integrating with Active Directory. It's everywhere. Uh it literally is in every enterprise and it was always at the last minute, oh we're getting ready to deploy to production. Uh this integrates with Active Directory, right? I'm like, uh yeah, I I

well let me just check on that for a second. So I would make sure that I would integrate with Active Directory beforehand right instead of doing it right before production. The second thing is passing security audits. How do you go about convincing security that this crazy framework called Django, which they always seem to pronounce d the Django, is not something that is any more inherently insecure than the Spring MVC framework that they're used to doing audits on? The third thing is ingesting legacy data. So in any enterprise, there are existing systems. How do you interact with those existing systems or take over existing databases that you are now putting a new front end on? The fourth thing is securing dependency installations. So we now in the Python community have the ability to do pip

install whatever framework, pip install Django. And we're so used to doing that, but in in enterprise environment, sometimes they think, well that's that's strange. We don't control that. that repository of libraries. We need something else. We need to secure that connection somehow. And then the fifth thing is correcting Uh this isn't something necessarily specific to Django, but it's something specific to Python and any sort of dynamic language. It's amazing to me the number of conversations I've had with literally CIOs or executive level people that are like This dynamic typing thing. And I'm like, this is not something you should be concerning yourself with as a CIO. But it comes up over and over again and there is a certain way. that I have, through much trial and error, been able to convince many people on the executive level that this is not something that they should be concerning themselves with.

So the first thing is the boss man, the client guy says active directory integration. Let's make sure we can integrate with our active directory system So when I hear Active Directory, I'm like this my eyes just glaze over. And I'm thinking like this is something, some enterprisey thing. that like there's an enterprise diagram that some enterprise architect drew by hand over a course of a year uh to just like figure out where all the systems are and I just like that's not what excites me as a software developer This is something that I just want to do, I want to get it over with, and I just don't want to think about ever again. I just want to do it once. And actually, Act Directory is actually not that complicated. By and large, there's a few different setups you can have, but by and large what they really say when they want you to integrate with Active Directory is

look, we've already stored emails, first names, last names, a bunch of information about our employees and the people in our organization We just want to make sure that they don't have to like plug that information in again. And so that when they log into their system, they don't have to constantly re-enter that information. They're logging into our Django app for the first time and it says, welcome, Kate. You thank you, you know, here's here's the application. They don't have to punch in, like it doesn't ask them like who are you? What's your user ID? Like that information you should be able to look up. Unfortunately, Active Directory actually supplies that. So the way that you would handle this is uh so we can look at it as any sort of back end, as if we were doing uh a model backend, which is the standard one that we'd be looking up for the user uh in the database. So we instead of having a model backend, we can have an active directory backend.

The way that we would do this, fortunately, is with uh with Python LDAP. So any slide with a green background is a library that you can install. So we can do pip install Python LDAP. And this takes away some of the complexity from us. But how do we actually integrate this general Python LDAP library into our Django application? Fortunately, it actually doesn't require that much work. First thing would be to add some things in our settings. py file, just like if we're using any sort of dependency. Now, I don't want you to have to worry too much about the details of what this looks like. First off, my handwriting is pretty terrible. And second of all, this is just a very abstract sort of pseudocode way of looking at it. At the end of the presentation, I will give you all of the resources that I mentioned today. I wrote a blog post specifically for this talk. It has links to all the presentation

uh all of the resources I mentioned in the presentation, all the libraries. It even has the source code to the presentation. So you'll get everything at the end. Just Can you just read or just look through this stuff and and remember and that you'll be getting the resources at the end. So the way that we look at this in the settings. py file is we have a DNS name, which is our with our server for Active Directory. We have a port number that is associated with that in general, that is 389. We have some search fields. These search fields are really what we're looking up. So we want to know what the email address is of the user who's logging in, who's using our application. What is their first name, last name? And this is going to be dependent upon your organization, what they decide to store in Active Directory. So that is one question you're going to want to ask ahead of time.

What do we store in Active Directory? Do we just use it for basic user information, or do we actually have roles and different privileges that are stored in Active Directory that we should be we should be using? Or do we need to rebuild that in our application? And just because you have this information here doesn't mean you can't also have some other username, password, or some other authentication scheme. This is most of the time, at the very least, just used so you don't have to have people re-enter their information. With the LDAP URL, um most of the time you'd construct that using the DNS aim and the LDAP port. I just hard-coded it there, and then you have some domain. So you add these things into your settings. py based on how your LDAP is set up, and someone in the organization should be able to supply you with that information. And then you create a class. It is a separate back-end

class. Fortunately, this is something that is a standardized get user, authenticate, create user. There's a bunch of functions that you write for a back-end And people have already written gists and and libraries that will encapsulate this. The idea here is that you should be able to take this backend and turn it on when you want it and for your own development purposes turn it off when you don't want it. So from our own local development, we don't want to authenticate against LDAP. We're just doing our own development work. We want to authenticate a mot against the model backend So you should just be able to flip that in your settings. py file so that when you're ready to deploy to your test or your production environment, you'd flip this on. So that covers active directory integration. Should be something that isn't scary anymore.

It should be something that, okay, I write it once and I'm done with it. The second thing is passing security audits. And this is obviously not just a technical problem. There's not like just a few lines of code that you can write. It's not like you can do pip install security audit. Well, that's actually a really good idea for a library. But there's a bunch of things that you need to know and and explain to a security team. So one of the things is as developers, we have our heads down in the code and we're thinking so much about where what problems are we solving, implementing new features, getting the system up and running. That we're speaking a completely different language than the security team. One of the things that I found so amazing was I expected the security team to be just as technical as I was as a software developer. But in general, in large organizations, they're not.

They actually sometimes don't even have a software development background. We're speaking completely different languages. And when you go into a meeting and you start talking about, well, here's how we're implementing things from a security perspective, and you literally hand them piles of code. on printed out on a piece of paper, they're like, these guys don't know what they're talking about. They're not speaking the same language as us. So the first thing to do, if there's a security team, is Find out how technical are they and generally try to find the most technical person or the person who's the most interested in the software development aspects and then work with them to figure out what standards are they using. One of the most common standards for web applications is called the Open Web Application Security Project. They have a top 10 list. of security vulnerabilities in web applications. And this is probably something that is the input to a checklist.

They're trying to check off boxes. The security team knows. There is no application that is completely secure. They want to make sure that the bases are covered and they generally take those bases, the boxes they're trying to check. based on some on some standard pro a standard open security protocol like this or security checklist like this. And it's looking for certain things that as developers we should know at least a little something about the first thing would be injection, particularly SQL injection. So coming into the meeting knowing what SQL injection is and understanding exactly how Django prevents that with the ORM. And highlighting if we're using SQL, if we drop down into the SQL and bypass the Django ORM, we call that out in our own in our own information that we hand out, hand to the security team

The second thing would be things like cross-site scripting and then also cross-site request forgery. These are things that Django has pages on, and the great part is about these. Is that each one of these top ten items, we don't have to reinvent the wheel as Django developers. There's a great talk by Jacob Kaplan Moss that talks about Every single one of these security security vulnerabilities, particularly in the context of Django and how Django hand Django handles them So you can watch that talk and read a blog post which I'll give to you which maps these security vulnerabilities versus what Django does. And obviously you're going to have to adapt it to the your own code that you've written. But what that means is that a lot of the heavy lifting has been done by people who've solved these problems before. So that is in general how I approach security audits.

Again, remember, the security team is often not nearly as technical as we are, and therefore we want to be speaking their language so that we don't end up in an adversarial relationship with them. Another thing is transferring legacy data. This happens constantly. We're building new systems in an ongoing ecosystem. The first situation which can often be the cleanest is we're just building a new application on a legacy database. There's already stuff out there and we we don't have to worry about another application touching that database. It is our database that we're working with It has existing data, it has existing relationships. We just need to know how to map our Django application to that. Fortunately, we do not have to do this manually. There's something called InspectDB. So once we've configured our Django app to read from a database, we can do InspectDB and we can pipe that into a or redirect that into a models.

py file and then it will generate the schema for us. Now, we may not know exactly what's in that models. py file, so we could look through it and we could read through everything, which certainly we're going to have to do at some point and figure out the structure of the database. But I'm a visual person and so there's another thing that we can use which is we can do a pip install Django extensions and then we can run a manage. py graph models with some output and it will generate getting off of our our whiteboard for a second, it will generate a f a file, a picture, that looks like this that actually shows us the structure of the database. And this is fantastic for getting up to speed on what is already out there. We don't have to hand hand figure out okay what table connects to what other table what are the foreign key relationships we can literally visualize this

and generate our models. py file now obviously that's the beginning of the journey we'll need to migrate tables we need to figure out if that structure is appropriate for our application, but it certainly is a starting point that is that we can get to very quickly and start understanding what that legacy database contains for our application. Now this the situation is not always that simple Sometimes there is an existing application and the requirement is, well, we just need to get some some of the data out of this existing legacy database. So there can be a temptation because Django, I believe it was as of 1. 3 or 1. 4, you can use multiple databases. So you may be tempted to say, well, we have our own database. We're also going to have our legacy database. We'll just use two databases for our Django application.

However, I recommend against doing this, if at all possible. In enterprises, weird stuff happens. You don't always have control over the situation. Sometimes there's some sort of political battle that is preventing you from the optimal scenario, and you may have to directly connect to the database. You can do this, but what I generally recommend is having some sort of wrapper around that existing application. If there's already a team that is developing software for that other application. Have them generate an interface, some sort of API that allows you to pull whatever data you need out of it on a regular basis. The r the danger if you do what I've X'd out here is that other systems make other teams and systems may come along and say, oh, you've got access to that data. Can you write me an API?

But you don't actually have control over that database. Someone else is touching that database. And so what ends up happening is you become this dependency that you never intended to be. So it's better to push it down onto the ex the application that really does have control over that database. Now, there's one of my other favorite applications. At my first job, they were called end-user computing applications. And they looked something like this. They were an Excel spreadsheet. Uh the great part was that the company I was working for, uh very large company, they had so many, literally hundreds of these things, and they had a lot of errors in them. There's no unit testing of Excel spreadsheets. And what ended up happening was they had all these cascading errors and they had to restate financials to the tune of hundreds of millions of dollars

because they kept having errors propagate throughout all their financial systems that were basically this We're talking about major financial institution in the country. Enterprises are very strange places. Now, fortunately, some environments recognize this is not a sustainable situation. So they'll ask you as a developer to come in and build a system that replaces a spreadsheet. But the caveat is, yeah, we want all the data that's in this spreadsheet. to like be in the system on day one so we have a hard cutover. So when we log into that system, it has all the data that we're expecting in our Excel spreadsheet. There are some tools that will help us do this. XLRD and XLWT, and I'll tell you exactly how you should use them. The first one is for reading Excel spreadsheets, so you can read Excel spreadsheets directly

But I recommend, strongly, strongly recommend that you have a lot of data validation and that you agree beforehand. You say what is the input that we should be taking in, and we will reject anything that does not match that criteria. Column B should have only numbers. And if there are any letters We reject that and we will spit that back out as an Excel spreadsheet with XLWT. So we'll give you back an Excel spreadsheet of all the stuff that does not pass data validation. We're not going to you know continue to re-ingest this information, but we will suck in as much as we can and then your team who handles the business side of things you can figure out what you need to correct. So you push

the data correction onto the team that has to hand figure out well why did we put you know a number in a sal or why do we put a why do we put a a bunch of letters in a salary column. I mean this this is literally the type of errors that happen in an Excel spreadsheet. There's just no validation. So you have validation on your side when you suck in the data and then you just say you have exception scenarios where you just pump out an Excel spreadsheet says You figure this out and we're happy to take it in when you've fixed the problems. That is by far the best way and probably one of the only ways to make this a sustainable situation where you can cut over on day one. Alright, so another thing is securing packages and installations. There's really two things I want to talk about here. Number one is I've gotten on client site and I've done pip install Django

and the thing fails. Well, often we're going through a proxy server and they're funneling all traffic through that proxy server. Now fortunately Pip has a has a setting. It it will uh it has a setting where you can specify a proxy server, but also as of later versions, it will it will respect this environment variable. So you can export the HTTPS on HTTPS underscore proxy environment variable and pip will respect that. It will say, okay, I'm going to go through this proxy server, and then it will begin installing your packages. Just make sure that you set this every time I've I've sometimes set it before and then I forget to set it in my environment variables and then my pip will fail and I'm like, wait, what's the proxy server again? So just make sure that if you're going to set this, just set it somewhere where you know what the proxy server is before you use pip every time.

Otherwise it can fail. Another way is this is our standard setup. So we're used to as developers, you know, we got to PyPy or PyPI rather and we download the packages and we have them installed in our virtual env and we're good to go. But this can be very risky for enterprises. They say we don't have control over this PyPI environment. We don't really care that it's a you know a a community resource or anything like that. What they care about is like they want to control that. They want to run their security audit. a process and they want to run their scans on a central repository. And that is understandable. There are malicious things that can happen to central repositories that you do not own. And so what you really want is something more like this. This is all self-contained in the enterprise. And you can certainly do this.

In fact, there are it actually does not take that long to set up a PyPI environment of your own. The only thing is you're gonna have to up you're gonna have to upload and establish all of the packages that you need there. And then make sure the developers are not installing from the central Pi PI that all developers in the enterprise are know that this is a centralized repository in the enterprise itself. So the only thing about this is if you've got one team that's working on a Django project, uh and they have to be the ones who are are uploading all the packages and making sure everything is scanned, it can be a lot of overhead. So I would fight back against this unless you have multiple teams that are willing to share the burden of hosting your own PyPI server But it is certainly doable. Alright, and then the final thing is this like what are these dynamically typed languages? As I said in the beginning, it's always amazing to me when CIOs are like

You know, I don't know about this whole Python thing, you know, it's dynamically typed. I'm like, you haven't programmed a lot of code in like 20 years. Like, what does it matter to you? Like, it but it's so funny. So what I used to do is I would always say, like, look If Python is good enough for Google, it's good enough for this big company. And the CIO would be like, but we're not Google. And I found that there was a lot of pushback. And so the way to mitigate this, the way to mitigate this with an executive team or people that are pushing back against dynamically typed programming languages. is to actually point out all the all the places that they're already using it. Look, we've been using Python in all these places for the past ten years. And guess what? The enterprise has not blown up yet.

So if you start going around to different use you take a look at all of the tools that your enterprise is using, that there a lot of these are actually have Python that you're using for you 're actually using Python with them already. So for example WebSphere uses Python to set up its configuration. You can write Python scripts to auto-generate or to do the WebSphere configuration. So, okay, there's one use case of Python already being in the environment in that environment. If you're using Ansible or using Salt Stack, or even if you're using Chef or Puppet, those are dynamically typed like they're built on top of dynamically typed languages extraction and transformation, taking data out of one system and putting it into another, you point out all this all the situations in which Python and dynamically typed languages, all of them, are already being used in the enterprise.

And that can be a powerful thing for someone who is just trying to say, well, we don't want to introduce too much change into the enterprise right now. You say you're already using it, and therefore it's obviously safe for consumption because it's already been here for 10 years. And after that, then you talk about something like here's the respected peer organizations that are building systems with Python and Django. And there's a bonus, if you can, if you know that there are certain organizations they bring up. And they're like, you know, these are the leading organizations in our industry, not necessarily like Google or Facebook or something like that. But if they're in government space or in the financial space and you know that this these group of executives always talk about this company is doing great things You can find out what those other companies are are doing. And the majority of the time, like Bank of America is doing a ton of work with Python.

They don't really talk about it too much, but it is something that is out there. So if this is If you're in the financial industry and executives are pushing back on, well, we don't want to use dynamically typed languages or Django, and they're but they're always talking about how great Bank of America runs some of their systems or how you know they're they're able to respond quickly with their IT systems. You say, by the way, like they're they're using this already, and that may be one of the reasons why they're outflanking us here. And so that puts a little bit of pressure from the standpoint of like social proof. You're basically saying, look, you don't want to be, there's there's a few different risks there. Like the in the first category, you're mitigating the risk of like we we don't use this already. So you're taking that consideration off the table. Then you're what you're doing here is you're mitigating like look your peers are using this and they're outrunning you.

And that is a fear that many executives have. They're worried about being overrun by other companies And so you can point out, like, look, these other organizations are already using dynamically typed languages. And then finally, I do generally have a laundry list of the leading tech companies, and I talk about Here's exactly what these tech companies are using Python for and these companies are doing really great things with small development teams and if that's the direction of your organization then that's probably something you should be thinking about as well. You basically want to start crossing down the list of all the rebuttals that they can have around dynamically typed languages. So we covered a few things here. First and foremost, integrating Active Directory. It's not something that should be too scary. There's some great resources out there with and using Python LDAP, this is not something that should take you two weeks.

You can generally write this stuff in about a day and be done with it and never think about it again and reuse it on whatever project you go to. Passing security audits talk in their language instead of just talking in software development language. It can take a little while to get used to that, but you remove the adversarial relationship between the development team and the security team Ingesting legacy data, there's a few s few scenarios here. If you're doing a hard cutover, then you can do inspect DB, you can do graph , graph the data model and figure out how it works. Uh and if you're talking to an existing system, I recommend using an API as opposed to using a set and just connecting directly to the database because that can be an unsustainable situation. And then also with Excel spreadsheets, make sure you do the data validation and spit out the errors and let them correct the errors as opposed to trying to correct the errors in your own system.

Securing dependency installations, PIP will respect the HTTPS proxy variable, but you can also set up your own PyPI if that's something that your enterprise is is large enough to maintain. And then finally correcting. Correcting misconceptions around dynamic typing. There's a approach that tends to work very well. Obviously every culture is different. All the pushback from executives can be different, but there is a structured approach that will allow you to Just say, you know, this is something that is not too risky. Um this is something that allows leading institutions to be able to use Python and use it well Alright, so I told you I'd give you guys the resources. I know not everyone has uh total cell phone service in here, but if you send a text message, I'll leave this up for just a minute.

If you send a text message to this number, it's 503 4763056, it will respond back with a text message. This is totally anonymous. Um if you could give me a score of 1 to 10, uh one being the lowest. and 10 being the highest, would you recommend this talk to a friend or a colleague? This allows me to determine, is this a talk that is something that is valuable to the audience? Or if this is something that maybe I need to uh work significantly on and maybe change out the topic so that it can be more appropriate based on the the talk title. A lot of times we don't necessarily get feedback. This allows me to get feedback based on the audience's reaction. So if you send a text message The benefit to you guys is you get the link that has all the resources to the blog post, has a link to the presentation, a link to the presentation

source code, everything you need that I've talked about throughout this talk. So again, 503-476-3056. And this will be up for a while. So if you don't have cell phone service in here, just send it when you go outside and you'll get the link back. Okay, uh one more little quick plug. Um I'm the author of fullstackpython. com. If you're looking for resources around Python, this is like my passion project for 2014. I have put a lot of time and effort into it. I actually just crossed 90,000 readers this year and super stoked that this really helps people. This came out of me just writing emails to a lot of developers and saying, here's some great resources. I put it out on the internet for everyone in the community to be able to consume. My name is Matt Mackay. I'm a developer evangelist with Twilio.

Here's my contact information. Thank you all very much for having me and I will answer questions outside afterwards. Thank you.