Managing technical debt in (Django) Projects by Chris Chang

Speaker 1: Hi, I'm Chris Chain. Uh you can find me on GitHub at CRC Check. My Twitter handle is C C R C C You can also find me on IMDb, I'm Chris Chain2. Uh I'm also looking for more Pinterest friends. I am I am cPanel on Pinterest. I currently work at Tabbed Out, but a lot of the knowledge that I'm going to be sharing is Things I've learned from uh my time at the Texas Tribune too. Uh you may have seen a lot of other Texas Tribune people around. Oh, I see some right there. So everyone always starts off

Speaker 1: talks like this with dictionary definitions, Wikipedia. It says technical debt can be thought of as work that needs to be done before a particular job can be considered complete or proper. And then something Gene Kim says is left unchecked, technical debt will ensure that the only work that gets done is unplanned work. Uh another thing Jean Gene Kim says about technical debt is that it's where you feel the next time you want to make a change. And for me personally, technical debt is every line of code. And just in case you were curious I checked and Urban Dictionary does not have a d definition for

Speaker 1: technical dip. But I also think it's important to just know about what technical it feels feels like. And it's when you fix one bug only to create another bug. And it's when you can't even deliver features anymore because you don't know what repercussions your change will have. And it can get so bad where you can't even install security patches because you don't know what will break. And it's when you're using a rapid development framework and you can't rapidly develop. Uh it's when you hear a lot of it works for me around the office every time you're trying to replicate something. And the great thing about Django

Speaker 1: is it does let you build things very quickly, but it can also get you into this technical debt trap. And the main thing that uh I learned is that uh you don't want to let Django become the scapegoat for your Team's inability to develop new features. And the the last thing about technical debt is it's It's like when everything is on fire all the time. And and this is this is your life. So like there's no magic trip tips for how to clean and technical debt.

Speaker 1: So anyways, uh here's some tips. I've kind of organized them from things you can do on your own to things that you can do amongst your team, and finally things that require going outside your team and to the non-technical people in your organization. So uh first thing is Django testing. You should you should do it. There's like three other sessions at DjangoCon about testing. Uh Don't let your dreams be dreams. Yesterday you said you would write tests today. So just do it. Make your dreams come true. Just uh just do it

Speaker 1: So when should you write a test? Because a lot of times a lot of people will just there's a lot of uh I found that that there's a lot of reluctance to writing tests in the first place. So maybe if you can ease people in, that would help a lot. And something that I was taught was if you're making an assumption, that's a good sign that you should be writing a test to check that assumption. And I can I can see justifying not writing a lot of tests up front, like doing TDD Uh a lot of people might consider that premature optimization to be writing a lot of tests up front. But if you're doing that, you should do at least be doing regression testing.

Speaker 1: Uh wise man once said, uh fool me once, shame on you. Fool me, you can't get fooled again. And you will get you will sleep better when you have good test coverage. It's a fact. So I want to give an example about testing your assumption. Let's say I had a function where I just wanted to turn in uh a zip plus nine or a zip, a US postal zip. I just wanted the first five. So that's pretty easy. I mean I just gonna it's a string, I'll just take the first five. But if you look at these examples I've set up And I've written this as in the form of a doc test, which I personally don't use, but it fits nice on the slide. You start to see at the at the bottom

Speaker 1: my assumption that taking the first five digits falls apart. And if you have tests, then it'll you'll be able to catch that. And if you don't have a single test Here's one that someone posted recently, and that's if you just use the client to load your homepage. I don't have the facts to back it up, but you'll get like 80% test coverage just from this one test. And the The thing that I really like is is if you like the feature, you better write a test for it

Speaker 1: So coverage is another great tool for just helping you discover where you should be where you're missing tests. And a lot of organizations will strive to get 100% coverage. But it's more important that your tests are quality. Just writing a test to get, for example, just checking the uh the Unicode method of a model may or may not actually be useful. You'll have to find that on your own. So some signs that your your tests are crappy. One is like can other people read and extend your tests? Will they take your tests and use it as a template for their own tests and be happy with it? Are the tests getting maintained whenever the code is getting maintained

Speaker 1: or are people getting frustrated because they have to may also maintain the tests? And the tests described the problem in a way where you could throw away the code, and people could use the test to rebuild the code And it's not only the tests, but the same could be said about your documentation and maybe even your README. Could you throw away all your code and could someone use the README to reconstruct your entire application? Another thing that's uh a big source of technical debt is bit rot, also known as software entropy. And I think that if you're not actively touching a piece of code, it's getting worse just lying around. It's also a sign for me, it's a smell as an organization if you don't have the manpower to actively maintain all of your code.

Speaker 1: And if you're interested in In this kind of learning more about this, there's a series of management books called the uh improvement kata, also known as the Toyota Kata. Every time I researched that though, all I got was a bunch of marketing seminars that were trying to sell me something. But maybe you'll have better luck. So if you imagine your co the code of your organization as as a globe, are there areas where there's like Here be dragons, where people just don't know what that code is and they don't want to touch it. No one knows what it is That's why it's important that

Speaker 1: uh everyone at the organization knows as much of the code as possible. And one tool that will help Developers be more a lot more productive in this is continuous integration. Uh one quote that I made up based off of this other quote is any task you do at least once a month. should take less time than it takes to get a cup of coffee. There's a post mouse pointer. Oh, it's been there the whole time. There's this guide here that is a really great intro to a lot of CI tools. Some that I particularly like are Jenkins, if you're doing stuff yourself. Travis CI is really great if you're doing uh open source stuff. The great thing about Travis CI is

Speaker 1: you can even do like post-GIS testing. I didn't know you could do that until last year, and I think it's great that a free tool will let you write tests that require postages. Talks is another great tool. It'll help you peek ahead run your test suite against different versions of Python and Django just to know will the next version of Django break my tool or not? Uh here's an SKCD I really like that just basically tells you whether or not it's worth to automate a task. And here's the here's the opposite XKCD saying that you should never automate a task, I guess.

Speaker 1: So code quality is also a really basic thing. There's a difference between a senior developer and a junior developer. Practice makes perfect. Always iterate on your code. A great thing to help with this is just working on open source projects. Keep your code simple. Code that requires less context to read is more understandable. So By keeping things simple, I don't mean keep your feature set as dumbed down as possible. You can do fancy stuff in your code as long as it's a pattern that's reused across your code base. And that the people you're working with are also familiar with it.

Speaker 1: For example, if you're in a Django project and you're using pandas to do some filtering. That may not be simple, but if everyone on your team is also a data scientist, that may be the most obvious tool for everyone on the team Like avoid snowflake s special snowflake patterns. Uh I always choose readability over performance. Let the tools catch up to coding style, not the other way around. Like it's not like Python is being directly executed by your computer anyways. And the great thing is Python has something built in called import this.

Speaker 1: And if you've never run it, this is what it looks like. Uh except I I was just wondering if anyone else constantly misspells import like I do. Uh I cut it off, but import this will just give you the Xenopython by Tin Peters, and I actually have it printed out, put it by my desk. And if you're looking at a piece of code and you're wondering what you should be doing with it, it's good to have this in the back of your mind too. Here's an example of two ways to do the same thing. Let's say I've got a iterable of objects and I want to get the Instagram app. Attribute out.

Speaker 1: I really love functional style, so I personally love using map, but every time I do it uh in code review My teammates say I should just use a list comprehension. So I guess now I do a list comprehension. But I guess just look at this in like right now and just see which one looks more readable to you. And there's so much discussion around code quality. Here are some like keywords you can look up. There's like whether or not comments actually help or hurt code readability. There's uh something I really like called cyclomatic complexity. The great thing is that is it you can quantify readability, and if you throw this into something like a Jenkins job, you can actually track how readable your code is over time.

Speaker 1: There's a whole Pep 8 discussion, and uh PyFlex is kind of an extension of that. And so I mentioned code review. Code review is probably like the single most practical way that you as a team can get better. A lot of people say though, like people take too much time to review. And pe people just want to like deliver. And some people might say like I'm the boss or I'm the big boss. I don't need code review or like we need to push this topics out, ASAP. But We all know that coding under pressure is is the best. And it's always, always perfect. Let's say you're you're a team of one person.

Speaker 1: Can you still do code review? You you can, I guess. Uh what I do a lot is uh I'll I'll sleep on a change. Uh just opening a pull request puts all the code in front of you, it gives you a new app a new uh View of everything, you'll discover patterns that were like, oh, why did I name it one way up here and change the name down here? That's things you can't see on a commit-by-commit level, but you'll be able to catch that on a pull request. And if you just want to practice, you can practice with others on an open source project. And uh this quote I really like. It's uh as iron sharpens iron, so does one person sharpens another.

Speaker 1: And that really applies when I I think with to code code reviews. And to do a code review, uh you do a pull request usually. And the general rule about pull requests is well a lot of people would say the best practice for pull requests is one pull request is only one change. And I am saying if you want to reduce technical debt, you should bend this rule. There's something I learned called the Boy Scout rule, and I thought it was do a good turn daily, and I wasn't sure what that had to do with pull requests. And then I thought it was like be prepared and I wasn't sure how that worked either. But it turns out that the Boy Scout rule is Based off of the quote from

Speaker 1: uh Robert Baden Powell, leave the world a little better than you found it. And the idea is if you're in a pull request and you're going in and you see typos Just go ahead and fix them. If you see like little mistakes here and there, go ahead and fix them while you're in the code, while they're right there in front of you. And the cutoff for this though, I would say, is if you start having to mess with tests while you're doing this, that's a sign that this little change you're making is gonna be is gonna actually hurt the readability of the of the diff enough of the pull request. Where you should leave it alone, open a ticket, revisit it later. And code review is also a lot more than just reviewing code.

Speaker 1: When you're reviewing code, you should also be making sure that what the code is doing is repeatable. I guess for example of that is I wrote this utility last week. And I was tried to run it and it wasn't working. So I went back to the pull request to edit it and discovered that I was using the utility in the completely wrong way, but I I had it documented in the pull request what the right way to use it was. And also along with the code is documentation. For example, if you're adding a new feature to your CMS. There's users for that CMS. You have to make sure that they know that this change is coming and they know you have to educate them about

Speaker 1: The changes that are upcoming. I think it just said that. And then there's also just is this code maintainable? Does the feature you're adding justify the technical debt that you're adding? Uh for example you may decide that to get this in the template you use a template tag. But then during code review, you may decide, wait a second, you can just use this jQuery one liner do to do the same thing. And That's exam an example of doing like two things, one with s significantly more technical debt than the other. And if you're really interested in code review, I think Raymond Hedinger 's Beyond Pepe

Speaker 1: talk is really great. It is the most hyped talk that I've ever heard, I think. And I think it justifies it. And going back to how if something is maintainable or not, there's something that I liked doing on my pull requests at the Texas Tribune, and that's I would add a technical debt note. And the basis of that Is on fiscal bills, they would have something called a fiscal note. And I'll read what the legislative budget board of Texas says about that and just substitute in your head. the technical pull request version. A fiscal note is a written estimate of the cost, savings, revenue gain, or revenue loss that may result

Speaker 1: from an implementation of requirements in a bill or joint resolution. It serves as a tool to help legislators better understand how a bill might impact the state budget as a whole, individual agencies, and in some instances local governments. So you can see the idea is that when you're building code, you have to make sure that you can deliver it over the long term. And probably the best way to actually really accelerate the code review process is is pair programming. It's like a code review, but instead of having to wait for someone to comment and

Speaker 1: Then you're like right next to them and it only takes a few seconds. My preferred tools for doing this are TMX and Vim, but you can also just stand Next to each other, we're not stand or sit. And just be on the same, you can even be on the same machine. In this case The human is a driver and the cat is the navigator. And there's also a variation of this that I've learned about. I've never tried it, but I'm curious about it. It's called ping-pong pair programming. And it's where one person writes a test, then the other person implements the tests, and then they swap back and forth. And the difference one difference is that with ping pong pair

Speaker 1: programming, it's actually an async process. You don't actually have to be there next to each other. So you can do it uh I mean you won't have the as tight as a a feedback loop as actually being next to each other. But you can get close to true pro paragra programming without without that. And it's also very important that you just get along with your teammates. When you're hiring, you should Make sure that you only hire people that you get along with. Uh I know back at the Tribune we assumed that once people came to us, they were technically proficient

Speaker 1: and at that point We would just hire them based off of whether or not we would get along with them Hylium. And I would say that the the worst coworker ever is is probably past you. So now going outside the organization. Uh I've mentioned this a few times already, but any any action you can take to get a faster feedback loop over every action and every change. Is something that you're gonna try and do, wanna do. You should learn how to say no. Uh, you should learn how to embrace failure as an organization.

Speaker 1: You should what I like to do is try to make myself obsolete. I think it should always work like you're training your replacement. And the idea behind that is that if you're automating away all the routine stuff you have to do, it frees you up to do the more creative things. And can you go on vacation or a conference without bringing your laptop? And as an organization, you should admit that technical debt is a problem. That way you can start treating technical debt as a ticket to be worked on instead of something that just swept under the carpet.

Speaker 1: If you treat paying off technical debt as a feature, then you can actually act upon it. One thing that'll help is actually writing out your your technical debt items as actionable small tasks Something like upgrade Django to version change this old view to this new pattern that we started using. And something I like doing is I I really like just writing a list of technical debt And I use fire for things that are gonna blow up. I use poop for things that might be publicly embarrassing. And then I use this table flip guy and I change the size of the table based off of how frustrating it will be to code the solution.

Speaker 1: And so there's some things about Agile that I think even whether you're using Agile or not, there's things you can take away from it. It's the main thing is it's not just about sprints and tasks, but it's also what I really like about Agile is you're always improving the process that you work. And I think a lot of agile is actually just common sense wrapped in marketing jargon. So Django, Django is great. It lets you write things with less technical debt, I think. The testing framework in Django is the best I've ever seen anywhere else. I've every time I use another framework, I wish I was using Django

Speaker 1: just for the testing. It has batteries included. There's less boilerplate. To do a lot of the same things and it's got a lot, it's got a gr great community. If there's a feature you want, someone else has probably already worked on it. But I do have some tips some tips for working with Django. One is don't customize the Django admin Think of the admin as a developer interface, not a CMS. If it seems like it's a lot of technical debt to write your own CMS. It's based off of my experience, it's actually not because when you're writing an extension to the Django admin, what you're doing is you're tightly capp coupling your admin

Speaker 1: to something that keeps you from upgrading Django in the future. Because guess what? The Django admin changes with every version too. And a side benefit is if you write your own CMS, you'll often end up with a better use user experience for your customers. And if you want to use a custom admin element, guess what? There's there's hundreds of them. This is just a screenshot of the Django packages packages page for admin extensions. The other thing you can do is clean as you cook. When you're in your code, just delete it. That's what source control is for. If you think that there's a feature that you might actually want, don't comment it out with a block comment.

Speaker 1: Wrap it in an if statement. Because if it's something that you actually may want to turn on and off, you should treat it like a feature switch and not like a code block. You could even extend that and use something like a Django Waffle or an A-B testing framework to actually turn it into a feature switch. And the main thing is if you find something I was gonna put this. If you see something, say something. If you see something terrible, make a ticket for it so it's not forgotten. Another thing with uh Django is naming themes is also really important. There's actually a talk about this uh on Wednesday.

Speaker 1: Are the names you're picking gruppable? Uh for example, if you're using S for a variable name. And you want to find and replace all instances of S, you're gonna have a bad time. Use unique app and model names. And the ri main reason for that is you like the Django will let you have your own namespaces and build your own apps and model names, but there's a few there's enough places where they all come together where it's worth it just to try and make everything unique. And the other thing is make your uh variables in your test read as if they're English. I guess and for example, uh here's a

Speaker 1: contrived testing example. Let's say I have uh uh article and I want a qu a manager that only gives me the active ones So I've got all these required fields, so when I make an article, I have to add all these fields Uh but here's an example using Factory Boy, where now my test setup only has what's relevant to the test. If I'm only testing active articles, here I've explicitly said Give me an article that's active. And then as opposed to like assert A is in this query set, now it's assert this article is in this

Speaker 1: query set, which reads more like English. And the thing about unique names is I really love Shell Plus, which is part of Django extensions. A great thing it does is from one line, you can start Uh messing with objects immediately. And if you don't have unique names using Shell Plus, you lose that ability of Shell Plus unless you start adding more technical debt. In the form of shims around Shell Plus. And another tip I would have to say is avoid the Django test client. Don't use it for unit tests, but do use it for integration tests. Because when you use the the test

Speaker 1: client, you're not only hitting your view, you're also hitting the like the URL router, the request middleware All the parts of the review, the response will wear, the context processor, and uh who knows what else? I would say avoid model inheritance. There's uh not to plug Django extensions again, but Django extensions does come with some ex examples of what should be you what what an abstract base class or a mix-in should do. There's like a timestamp date field, there's a title slug field, things like that that are very routine.

Speaker 1: You can definitely do that as a mix-in. But you shouldn't Read the feature that Django has model mixins and then jump to try and use it everywhere. For one thing, it'll make it very hard to trace what field comes from where you're i and where you're jumping around. You'll just get lost very quickly. Another thing is this goes back to kind of touching everything. Something I like to do is every time I have a pull request, I just like checking to see what's out of date. And I like bumping And your requirement that it can. Of course, this only works when you have test coverage.

Speaker 1: And the other thing you can do is you can you can trade, you can purposely get rid of technical debt and purposely give yourself technical debt. For depending on what you're optimizing for. Something that I think freed up the work at the Texas Tribune a lot is something I call Tucker Bell feature. And there were a lot of sections of the site that were frozen and The there was a lot of content. But the problem is even though this was old content, whenever we added new content, it would change all the old content as a as a side effect. So the idea was, is there a way to freeze this content in a way so that we can continue developing without having to worry about what changed?

Speaker 1: And you should you should ac ask me or anyone else about how that works. And the great thing is after we did that, we we deleted so many templates, so many CSS files. We could delete So many models after that. It was amazing. And when you're developing the future, you should decide up front what you're optimizing for. Is it more important? That you deliver the MP VP as quickly as possible or is it more important that this be something that that it can sustain for for a while? Like if you're a startup, getting the first to market is a real thing that that is probably more important than building something maintainable.

Speaker 1: Uh I'm surprised how many people haven't really seen this. This this works for j contractors, but it also works for code. You should decide. Good feast good, fast, or cheap? Pick two. You can't pick three. You can also pick one, but You what are you doing? And so Like uh I like to write write crappy code sometimes. My general rule is if everything can fit in one screen of code I can make it as crappy as I want. In fact, if this is throwaway code, I introduce errors on purpose just to make it look just to make it obvious.

Speaker 1: Because there's nothing worse than finding a piece of code That was hastily written, but also has unit tests and documentation and then realizing that the author had no idea what they were doing and was just making stuff up. So in preparation for this, there's a There's a lot of material about this that uh I found I really enjoyed. Uh these are actually videos, not reading. Uh I've got links to these links to these and if you just Google these words, they'll also take you to the right place.

Speaker 1: Uh here's some reading lists that are here's actual reading. Uh I actually read a real book in preparation for this called The Phoenix Project. So you'll have to find that in the bookstore. Uh look look for this book when you're there. And if you can't find that book, uh look for this book. So there's a lot of other similar tangential discussions discussions about technical debt. Each one of these could be their own talk and our talks too. There's like microservices versus monoliths. There's the single responsibility principle. Uh one that I really like is knowing about the not invented here

Speaker 1: principle or syndrome. Being aware of feature creep, you don't want your app to become this huge giant mutant. And I put insert product development buzzword here. But really like just if you're really caring about this stuff, you should look into like go outside technical talks and just look at like business strategy and product development. So like this is my homework for you. These are the these are all the important things is I think you should you should get frustrated without w while you're at work, but get frustrated in a c constructive way I also think you should be as lazy as possible. Automate away as many routine themes as possible.

Speaker 1: Try to always get a faster feedback loop in any process. Always be touching all your code. Uh treat fixing technical debt as a feature. Clean as you code. Foster a generally good coding culture at your office. Remember that everything must be repeatable and plan ahead for the whole life cycle of the product. And thanks. Uh keep making Django sausages. There's a link to view my talk, and for some reason you can also run my talk as a Docker container.

Speaker 2: Okay, um question. So when you're coding along and you're looking at this code and it's all building up, and inevitably every programmer in the world looks at and goes I'm gonna have to change that later. I just know I'm gonna have to change that later, but I need to get this out. How do you decide? Because it inevitably some technical debt will build up because of schedule, because of lack of information at that moment So but you could in theory stop and take the time to do it the perfect way. So and either way could be reasonable. So how do you make that call?

Speaker 1: I'd say it's a call by by by management almost And what I like to do is I like to letter my code with with to-dos. And I have like keywords I use. One thing that Uh I really like is with you can have a Jenkins job actually that just indexes all the to-dos in your code and you can kind of track that over time too.

Speaker 2: Thank you. Questions? I have a question. And just a note, c um statements that begin, I have a comment or not a question.

Speaker 3: Do you have any experience with a wide variety of skill sets where some people are far ahead with newer technology or far ahead with more of the legacy things you're working with? And keeping people on the same page when moving forward.

Speaker 1: Yeah, having as much overlap as possible is something you should strive to do. Pair programming is the best way to achieve that. But it's uh Everyone's different. So that's not a real answer, but you should try to do code review and pair programming.

Speaker 3: How do you keep the code reviews from teaching when they turn into classes?

Speaker 1: They can, and I kinda like that. Uh but they can also turn into bike shedding sessions too, which is why you should keep the pull requests as small as possible

Speaker 4: Um where does where does bit rot come from if the code itself isn't actually changing? I hadn't heard that term before, so.

Speaker 1: Well the I think the requirements is a good example. If you're if you can't uh if you freeze your your requirements and you never upgraded them, your app will always work. But eventually you'll want to make a change and you'll discover that it's not backwards compatible. There's there's one of my links actually talks about uh the organizational debt. Wait, the way we look at Tickle is wrong. One of the points he makes is that if code is left alone, it's fine as long as it works. And there's there's there's truth in that too.

Speaker 1: If you've got code that sits around and works, I would say just if so you have to change it, be prepared to throw the whole thing away though.

Speaker 5: Hi, thanks for the talk. Um, a really important topic, especially for um large projects. Um I wanted to ask you about Django Waffle and uh A B testing framework. How do you keep something like that from kind of exploding all the branches in your code and like how do you manage all that complexity it creates?

Speaker 1: I don't have that much experience with waffle, but in my own mental planning for that, I always thought Uh if you can track what switches you're actually using and make it a sp a task to go through and turn dead switches to static code. Uh if you if you make that a feature that you have to code and you plan for, then I would say that's how I would fe deal with that.

Speaker 6: Hey, thanks for the awesome talk. It was great. Um I was wondering, do you have any tips for how to incentivize good code policy in within a team? Because I find like the stick method is definitely much less effective than the carrot method of like getting developers to buy into this kind of stuff.

Speaker 1: Yeah my thought about this is to use bots. Basically, if you have a uh GitHub has a new feature where you can't merge uh pull requests unless it meets criteria now. Some people just may not listen. That's why also like the higher the whole hiring thing is also part of it too. Uh it's kind of it's kind of terrible, but sometimes you just need to change your change your team. But uh also I people may have a hard time listening to uh human, but if you have a bot that says the maca the simple matic complexity of your code is too high Reduce it or the code coverage is not there, add it.

Speaker 1: I want to say that people will look more likely to listen to a bot than another human.

Speaker 2: And thank you very much.