Lint All the Things! with Luke Lee

Speaker 1: Great. Thanks everybody for coming. Sorry for the delay. So um we'll get through the boring first. boring part first, but uh we're about halfway through the conference and I think it's went really well so let's just have like a round of applause for all the organizers And now that I know you're all awake enough to clap, we can uh continue on. Um so like I said, my name is Luke. Um I work at Octopus Energy and Kraken Technologies as a Kraken engineer. We'll talk a little bit about more what kraken means and that sort of stuff as we go on. Um also I'm a yoga teacher. You might ask yourself since you're very awake now like why the hell would he mention that right now? No relevance at all. It took me a long time so I like to brag about it when I can. If you're also interested in yoga, that would be cool to talk about.

Speaker 1: But that's obviously not why we're all here for a Django conference. So the talk title of the talk is Lint All the Things. So first off, what is a Linter? This is like a definition off of Wikipedia. I find it varying levels of success, lots of words, kind of confusing, not super helpful. Um flag errors, bugs, stylistic errors, and suspicious constructs. Sounds really fancy if I wanted to like sound way smarter than I am. But the way I think of it is like three different things. So first of all, I think of a linter as like a critic. Something that can complain about your code and kind of like coach you into doing better job. It can also be a teacher, so a lot of the times the way we use linters at work Is they can teach you a better way to do something.

Speaker 1: Instead of just complaining about it, it can say like, hey, by the way, did you mean to do something else? And then finally another thing is like being a caretaker. So sometimes you like to just you know go into your kitchen and make a big mess and then have something else like clean it up. Maybe you have a Roomba or something like that. Think of it like this, so maybe linters sometimes can not only like complain about something or teach you something better, but maybe they can actually clean it up for you. So, okay, great. Why do we have linters then? First and foremost, it's because we're all seemingly we're all humans and we do make a lot of mistakes. Coding is pretty difficult, it's very intricate, we can mess stuff up all the time. Um also it like helps us codify best practices. Like on our team, we have a really, really large team, so and a really large code base, and we have a bunch of people that have been on the team for a lot longer than other people, and it's really good to

Speaker 1: like code those like what I call like war stories or something like that. So codify some of those best practices. Another thing that linters can do really good job at is consistency. I always tell people on my team if I can open up a file and I know who wrote that line of or that file, then like we're in a bad position because that means it's going to be difficult for someone else to potentially work on that file. So if we use these linters, we can get some consistency. Also, these linters are since they're also just code, um, they can complain about things. And it turns out it's actually a lot easier to get criticism from a computer than it is from a person. It just is easier to have something in a pull request automatically come up and complain as opposed to maybe like someone coming to your desk and tapping you on the shoulder and saying like, hey, that code kind of sucked. Not great. And finally, all of these things kind of contribute to the fact that you can have faster code reviews.

Speaker 1: So like the canonical example about that is like if you're in like a code review and I'm not going to have to like complain about way you're doing white spacing or do you have a blank line here, do you not have a blank line and stuff like that? That takes up a lot of time and cycles and it can be really like demoralizing for like a junior engineer who's just trying to get their code merged. So again all these things contribute to faster code reviews. So You might say, okay, those are really great. That sounds interesting, but really like why do we really have linters? And for me, personal example, is because of this. It's meant to look a little scary because it is pretty scary to work on sometimes. So at Octopus Energy, we develop a platform called Kraken that is trying to deliver green energy to the world. And what Kraken actually is, is it's like a giant Django monolithic application. We support millions and millions of customers and we deploy it to like nine different countries every time we make a change.

Speaker 1: We have about 350 developers and inside of Kraken we have about two million lines of Python code only that we have written. That's not counting our dependencies, that's not counting any JavaScript stuff that's thrown in there. just two million lines of like raw insane like Django monoliths Python code. So like I mentioned before that can be pretty like intimidating as a junior engineer to come in and like work on a project that big. It's by I've been doing software for a long time, but it's by far the biggest project that I've ever worked on as well. So really a lot of the reason that we do all of these crazy linting things and have all this like guardrails in place is to one like make it much easier for our junior engineers but also to keep the rest of us accountable as well. But working on a system like this, you can sometimes feel um a little bit of every one of these and working on it. If we if you're really passionate about the conversation between like a monolith

Speaker 1: versus microservices or something, I really like to talk about that. So definitely come back and talk about that. to me afterwards. But if you're working on like sometimes when you're especially when you're first starting on Kraken, you can feel kind of like this. Sometimes it beats you up, sometimes you feel like you're on fire, and sometimes you just feel like you have a cold. But before we go on, I want to also mention this whole linting idea is not just a Python thing. If you're in the JavaScript community or in the C community or somewhere else, this linting thing is not like a unique Python thing. That being said, this is like a Django Python-based conference, so all I'm going to talk about are actually Python tools, but just keep in mind that it's not something that is just Python. So the way I like to do talks is like throw a whole whole bunch of stuff at you so that you can go and look at it later. I don't really think that we can like learn a whole lot in like 25 minutes.

Speaker 1: So um feel free to come and talk to me afterwards, but I'm going to throw a whole bunch of ideas and say, like, you know, show you how we do a bunch of different things and we'll go from there. First, um a thing we use a lot is called black. It's kind of like a linter. I throw it in here because I I think it's really cool. Black sells itself as what's called an uncompromising code formatter. I know in the Python community we're all used to like white space and we, you know, maybe you came to Python and you hated that at the beginning, and if you're like me, now that you like embrace it But we still have our own stylistic differences as the way we format code, even though like we are in a white space space language. So the easiest way to describe what black does is to actually just see what it does. So the details of this code aren't super important, but let's just say that I was planning my week and my weekends and you know

Speaker 1: you can see all this code is kind of scrushed together. There's a bunch of white space between like the word Saturday and Sunday. It's just kind of hard to read. I feel like it's Everything is just condensed. I always tell people on my team that whitespace is free, we should like use more of it. So let's like open things up a little bit, break things up into little logical chunks so that you can more easily read the code. So if you are like me, you really like um single quotes. So I always type single quotes. That's just the way my hands work. I'm not gonna hit that extra shift key. Drag So I'm gonna always type SQL quotes, but again, for consistency reasons, maybe we want to always use double quotes or we want to always have blank lines of different places. So if I throw black at this code, what I end up with is something that's like this. You may like quibble about the way the style of this looks, um, but in general I think it's much, much better than the previous version.

Speaker 1: The way I always one thing I always tell people about black is like normally before black existed or some of these other like code formatters. You'd start a new project and the 15 engineers would all get into a room and for like a day you'd just complain in a fight about like white space and stylistic changes. And what I what I think about black is that everybody ends up just being a little bit unhappy with it. But we're all in it together, right? So like I don't really like the double quotes. Some people really love them. There's like different things about the black style that I don't like. But the most important thing is that everything is consistent and a lot of those decisions are just taken away from you so you can solve the real problem you're there for. So I think that's pretty awesome. I can write code kind of the way I want to do it, the way my fingers work. I throw it into black. Black reformats it. So how do we use black? A couple of different like options. Black is really well known for not having a lot of configuration options.

Speaker 1: But over time it has gotten a few more This one really still kind of bothers me, like limiting the line length to 99. I'm very much like an old Unix guy and I want like 79 characters, but you know, I'm working through it. Also, we exclude different things because black can be pretty intensive, especially on a project the size of Kraken. Actually, um inside I think after Django 4, or maybe there's a certain uh point release in Django 4, we won't have to do this. But we don't run black on our migrations code because we have lots and lots of data database migrations and we don't really want to like run and reformat all those migrations. I think with newer versions of Django, um it's already gonna run the migrations through black, so we won't have this problem anymore. So a couple of configuration options, but then how do we use it from like the command line? We use the dash dash check option, which just black doesn't change anything. All it does is tell you, like, hey, uh, that's not quite black style.

Speaker 1: I would have changed it The couple of reasons that we do that is we do all these things in a pre-commit hook, which we'll talk about later, but we don't want black line changing the code while you're in the middle of a commit. It just gets kind of messy and the history can be weird. So we run it in a way to say like, hey, if Black would have changed your code, then you can't merge it. Another cool thing that a lot of people don't really talk about with Black and maybe they don't even know it exists is this dash-save option, which I think is the default. But since black is reformatting your code, it's kind of a to me it's kind of crazy that I'm gonna spend all this time like writing this code and I'm gonna throw it to something else and it's gonna do something to it. Like that kind of bothers me. But Black has this very cool option called dash dash safe where it when it reformats the code, it actually looks at the AST that's generated by the before and after and verifies that the code is semantically exactly the same. the same. Lots and lots of code formatters probably don't do that.

Speaker 1: You can turn this off if you want black to run a lot faster because it does a lot less stuff. But in general, I think it's like a great option to have enabled. So I can write code however I want, I throw it to black, it formats the stuff. Um another thing that's really interesting is i sort. Um I think i sort kind of has this cute little tagline. It's like i sort your imports so you don't have to. Again, similar to black, it's useful to see what it does. That would be way more explanatory than me telling telling you the details. So if you work in an application that's pretty big or general in the Python community, a lot of times you're gonna have a lot of imports at the topic. your file. That's definitely a problem in Kraken. We have to import a lot of different things. We have a very modular design. So you can see here it might be kind of difficult when you open up a file and you need to figure out if you need to import something.

Speaker 1: It's pretty easy to get like duplicate imports. or something because you can't sort through this very easily with your eye. Everything is bunched again together. So like again, white space is free. We can definitely have more of it. So if I throw this code into iSort, what is it going to look like when it comes back out? You'll notice the text is a little bit bigger, that we got a little bit of white space here, and things are actually broken up into logical chunks. And this will be every file in our project would be this way. So it's easier to find imports. So what iSort is actually doing is it's um sorting them alphabetically and then breaking them up into chunks. So like your standard library stuff is at the top, then you have some like third-party stuff, and then you have like all of your local stuff. Um and like I don't know that this makes me unreasonably happy. Like this is how I feel whenever I throw stuff to icewort and it just like makes everything really nice.

Speaker 1: Some configuration options again since we start a little late. I'm gonna speed through a little bit of that. Again, the way we use iSort is a check-only option for the same reasons we use dash check with black, but they're written by different developers, so of course they have different command line names. So that's i sort flake 8. Doesn't have as cute a tagline, but it's still very cool. It's your tool for style guide enforcement. So what does that actually mean? I'm an older like C developer, so I used to like throw everything in a compiler, it would complain to me. I'd fix them when the code compiled. I felt pretty good about it. We don't have that luxury necessarily in Python. So a lot of times you won't see an error until you actually execute that line of code. So like if you were really to look into the uh the code here, you would notice that there's a typo in this get to do list function. Because it's the variable is called to do and I'm passing in to dos.

Speaker 1: Very easy thing to miss, and you're only going to notice it on like the uh week, like if you run it during the week, if you run it during the weekend, the code's gonna work. This can be super frustrating if you're working on an application and you're deploying to nine different countries and you don't want to like take everybody in the world down. So you can throw this at Flake, Flake 8, give it a file name, and then it will complain. So that's stuff that's like comes built into Flake 8. Pretty useful stuff. But what's really expire uh like very, very cool about Flake 8 is that there's like a lot more stuff. You can fully customize your own rules, and we do this quite a lot with uh Flake 8 and with Python. So again, yes, this is very awesome. So what does this kind of look like? Or what are some examples of that? I don't know if you've ever written a test and you write something like the code at the top. You say like you're using mock library at a unit test, and you say assert mock.

Speaker 1: called once. Um it's a pretty bad test because that line is gonna always be true. Like that's never gonna fail. It's basically like if you've ever written a test and forgot to write assert statements in there, which I have definitely done, pretty useless test, right? You get great code code coverage, but you don't test that anything works. The correct way to do this is actually to call the assert called once method on a mock. So this is such a common thing to do. How can we just make sure that no one ever gets to check in code the other way? A very simplified version of this like custom flake gate rule that we have. We have all these incorrect method names, so like don't call, called, called once, called once with. What Flake8 does is it breaks all of your code down to lines and the AST so that you can do a whole lot of like interesting manipulation and verify like what's happening under the hood. Or in this particular simplified case, we're just looking at lines of code that Flake8

Speaker 1: is going to pass us. And we can say like, hey, did you call the called once method? You probably meant to call assert called once. And then Flake 8 will fail the file and we'll never be able to check that code in. So pretty useful. Um other kraken examples that we have are like our unit tests are meant to not use the database. A true unit test really shouldn't depend on anything. So we have some linters to complain about that. Again, like I said before, if you have a comparison in a test with the assert , we are pretty pedantic about the way we name fields because we have like hundreds and hundreds, maybe even thousands of models. When we have date time fields in the database, we name them with like underscore at and from and to very like specific reasons so that if you come across a model you've never seen before, you kind of know what you're looking at. And then some other things like making sure that all our task files are in like a task.

Speaker 1: py and stuff like that. Recently we started using something called fixit, which comes from Meta. It turns out Flake 8 is really powerful, but like anytime you deal with an AST, it's pretty difficult. And sometimes it can just be very, very difficult to like build something that targets the right syntax tree that you want to deal with. Fix it is a declarative way of doing that. So you can kind of say like this is what good code looks like, this is what bad code looks like, and like figure that out for me. So I don't have to write a lot of this complicated AST code. It will also actually apply the fix to you for your code as well, which is pretty neat. We're going to skip over my myPy pretty quickly, but it's a static type checker, so it's trying to get me back to that like C code where I can like run some code and it won't compile if it doesn't. Unfortunately the way Python works, my Pi

Speaker 1: is More like a static type checker in that it doesn't run at runtime, it runs before. So if you put a whole bunch of types in your code and you never run MyPy on it, it's pretty useless to do. Just some like various options that we use in MyPy so that we can complain and that sort of thing. Another very cool thing that we have, it's actually developed by someone in the Kraken community and it's open so uh it's open sourced, is called Import Linter. So a lot of people think that when you're in a monolithic application you have like no organization, it's just like a big ball of spaghetti. Almost all the time that can be true, but in our case it's really not. We have this really interesting way of like breaking everything up into modules and this layered architecture. So all of our dependencies always flow up downwards. So we don't like import back up in the chain. So it keeps this giant monolith like manageable. And what with the import

Speaker 1: linter we can do is we can complain about different things and we can say This layer should never import this other layer and will fail the CI if you want. So that way like junior developers should know like, okay, well, I actually structured the code in the incorrect way. So again, it's all about like trying to be able to be like get people up to speed with the architecture as fast as possible and have as little friction. So that's a bunch of linking tools. How do we like enforce it? One way is doing it locally with Git hooks. This is built into GitHub or built into Git. They're easy to create if you know bash. And there's like a various one. So like a we tend to use a pre-commit hook a lot. So right before you're going to commit locally, we run a lot of these checks so that your commit will actually fail locally. You don't have to wait on it for CI. One thing worth mentioning is when I say pre-commit, I mean the pre-commit as far as the git hook, not the project called pre-commit.

Speaker 1: Um we do have pre-commit in our project, but it's only in like another uh trial basis we're kind of like slowly introducing it. And a lot of people always ask like why don't you just use pre-commit hook? And really the only good answer to that is that we did a lot of this stuff in Kraken before pre-commit was really well established and like had a lot of that stuff built in. So over time we will definitely move to this and we're going to get rid of some of the things that I'm going to show you next. So how do we actually lint these things? It's like a two-step process. So first we need to find the files to lint and then we actually lint them. On a project the size of Kraken, we have thousands and thousands of files. We don't want to lint every one on a pre-commit hook, because then you get really like out of your flow. You type some stuff, you go to commit something, and then you wait ten minutes. Not a great workflow. So what we do is like a little bit of like get like simplified git magic to say like, you know, get me uh do the diff of what's about to be committed, do the cache

Speaker 1: and the name status so I get like an A and an M around and I know like which files were added, which files were modified. Interesting thing is it doesn't make any sense to lint a file that was about to be deleted. That's why you see A and M there and there is no other D or R, I think it's it in Git. So all this little uh bit does is like give you something to lint. So it gives you a list of files. We can store that into a bash variable, throw it to Flake 8, we can throw it to black, and we can throw it to iSort like I talked about earlier. You might notice that I mentioned uh MyPy and import Linter and fix it. We didn't run those MyPy is really advanced and it has to do a lot of stuff. Import linter also has to do a lot of stuff. Both of these are just too slow at the size that Kraken is to run as a pre-commit hook. So that is like a downside of the way we do things. We can't really lint those in the pre-commit hook because again it would take minutes and minutes. minutes.

Speaker 1: Other downsides for GitHub, one, you have to install them. All that usually means is like making a sim link. Not a big deal, but there is like a little bit of a barrier. There's always this dash dash no verify option in a git commit command. So technically even though you're telling everybody to run the commit hook, there's an escape hatch for them to get out of. of it. And finally, um you have to know bash which is not necessarily the most user-friendly language. Another reason to use pre-commit, the actual pre-commit project, is you can write them in different languages. Negative thing or like how do we like do that locally, but then ultimately we want to run every time we deploy Kraken all the time, we want to be running all these things. The bonus of doing things on CI is because it's guaranteed to run, like our build is going to always run this stuff when it gets pushed in or when it gets merged. And two, it's always in a predictable environment.

Speaker 1: So you know the whole like classic like it works on my machine kind of thing doesn't really happen when you're on CI. So that way everybody's running against the same version of Python, even if you haven't updated your dependencies locally or something like that. So you can be testing everything on the build in a predictable way. Um yep, I think uh that is about it. I know we were rushing a little bit for time, so thanks everybody. Um also if you are interested and taking one of these home with you. We are going to do a drawing at the booth in 15 minutes or something. And the the question is like how many times we've merged a pull request to Kraken since we started. If you want a little bit more details or something about that, we can talk about that. in a few minutes. But yeah, thanks everybody for coming. If you're interested in linting or like have other ideas for how we can do things better, or you're just interested in in what it's like to work for a green energy company, definitely come by and talk to me.

Speaker 2: Thank you, Luke. Uh we have a few minutes left for questions.

Speaker 3: Okay, thank you for the overview of uh various lifting options. Have you ever had a situation where you're unable to commit your code because of conflicting requirements between multiple code checking tools?

Speaker 1: Yeah, it's a good question. So since we do turn on a lot of these different things, um it is possible that like some of them conflict. It's kind of like a configuration problem. Like sometimes you just have to pick and choose like, okay, I'm gonna turn this on in iSort or black or something like that. Like I personally haven't seen that, but people on the team have run across that and it's usually because we just haven't like thought of all the configuration options. The the funny time like asking about when have you ever just not been able to commit your code? Again there's that dash dash no verify which really you should never use, but every now and then it's there for a reason. Um the thing that I end up running across is even though I wrote C for a very long time I was used to like type checking code. Um it's been a very long time since I've been writing only Python, so now I add types to all of my code because we're using MyPy. And oftentimes I'm just fighting with MyPy to like

Speaker 1: Get it to stop complaining at what I'm doing. Um so that's a very like interesting situation where you just keep fighting like banging your head against the wall because my pie is like, oh this typing is wrong and you're like, no, I promise I know what I'm doing, but So that like that one actually I run into all the time. I don't know if anybody else runs into that sort of stuff.

Speaker 2: Any more questions? I think there are no more questions. Thank you, Luke. Please give another advice.