How to spend less time writing tests

Speaker 1: And we're back. You thought you'd get away with one talk about tests, but no no no. This is far too an important subject. And um Uh I think uh mm maybe sometimes we've we want to spend less time uh writing tests. Is is that what this talk might be about?

Speaker 2: Yeah, great great showing up here this morning and seeing so many familiar faces from from last year as well as so many new faces. It's it's great to be back in Beautiful Copenhagen, I'm always so I mean I mentioned this last year, but like the the bikes are just incredible. You just want to like join in on the fun Uh so yeah, thanks so much for the brief introduction. I'll skip a little bit through this. Um my name is Will uh on the colo team as well as Lily here in the front. Uh so if you see Another person wearing this t-shirt, that's us. Um I've used Django for about uh nine, almost ten years, uh mostly building SimplePole, which is a Slack app. If you use Slack at work, you might have come across it, you know, you can do polls and surveys kind of within your Slack. And Colo

Speaker 2: actually started out as an internal tool for SimplePoll. So and SimplePoll also funds the development, funds Lily and I working on Colo. SimplePol. It's about six of us, you know, millions of users, so that actually makes money. Colo does not yet make any money, it's just completely free. Hopefully someday. And then yeah, at some point I worked at GitHub and I'd love to talk to people about Rails stuff and what we can learn from it. But you know, a lot to talk about. Alright, so that's me, and I have a confession to make. I don't really enjoy writing tests. But, and this is a big but, I do really like the tests that already exist. So what do we do with this? So I tests are great because um they let us deploy and ship with confidence. You know that everything that um

Speaker 2: You know everything that uh you've already built continues working. Obviously we had a great talk from Joseph uh about tests and how to make better test style guides, and it sounds like that's really resonating. So I always find myself building a feature and then being like, oh, now I have to write the test. And then I go and do it because I know that it's a good thing to do and future me will be thankful. But I wish I wish I didn't have to. Uh and I was curious, like is this just me who who thinks like this? Like is it just me who doesn't okay, I see someone raising their hand. Like you also don't like writing tests? Great. Nice. You like copy papers? Nice. Uh so uh I feel like sometimes you can go to like a tech conference and you hear uh talk about like test-driven development.

Speaker 2: And it c you can sometimes get the impression that you know everyone but you does software development in this perfect way where all the tests are written first, you know everything up front. And then it just like works out perfectly. And you write the test first. Maybe you don't even write any implementation because you don't need to. The tests are just, you know, that's all you need. So And that's just never the reality I found myself in. Uh so I actually started doing a little bit of research earlier in the year, doing some interviews with about fifteen Django developers. Some of you are in the audience. Shout out. So um if you participated, thank you very much. Uh and I asked you know questions like, do you enjoy writing tests? Uh how much time do you spend writing tests? And um I'll go through some of the results from that now to kind of set us up. One thing I'll say up front, just to cover it briefly,

Speaker 2: there's a bunch of academic literature about this as well, but the median figure seems to be that around 50% of the time we spend actually in our jobs is actually spent in or around tech. So it's a significant number. If it's not 50% exactly, you know, maybe it's 40%, but it's it's like certainly double digits. Um can be I I've actually heard as high as 80% of your whole day is spent on writing tests. So we spend a lot of time on writing tests. All right, so here are some of the responses I got uh when I was talking to uh 15 Django developers about do you enjoy writing tests? So a couple of different um takes. Some people seem to think, ooh, truly most people hate writing tests. Don't think that's quite true. One I found particularly insightful actually is on the left in the middle. And this person actually does enjoy liking tests

Speaker 2: or uh writing tests, uh, or at least often enjoys it, but says it gets annoying when you have to mock a lot of things. So I think that's an interesting insight which we'll come back to later. Actually testing is fun. Maybe setting up a test is less fun. So we'll come back to that in a second. And then I also asked this question and get out your phones because we're gonna do some live polling. in a second. I actually asked at DjangoCon Europe earlier in the year about do you enjoy writing tests? And these were the responses. So we have a couple of polar, like some hate it, some love it, and then a few in the middle. Uh and I would actually love to ask you the same question right now. So if you get out your phones and go to slido. com and put in those digits, uh those seven digits, then you can cast your own votes. And we won't spend too much time on this, so uh

Speaker 2: Get it in quickly. And then we'll come back to slido later actually. So leave leave that tab open. All right, eight responses. Do like another twenty seconds or so. It's just so fun watching this move. At the beginning a hundred percent loved writing tests. All right, that's that's 38 responses. That's pretty great. So it feels like mostly a similar picture actually.

Speaker 2: Maybe there's a few more haters this time. I don't know. Uh so that's that's cool. I mean so yeah, um I always think it's so much fun to watch these things. But yeah, so um almost over 40% of you enjoyed a little bit. Um a good third says you know it's just part of the job. And I think that's actually been a very common theme I've heard is like you know being part of being good software developer is making sure our stuff works. And part of making sure our stuff works is writing tests. So um It's almost like asking, like, do you enjoy writing tests? Like, do you enjoy being a good software developer? Those are kind of interlinked. So that's what that question is kind of getting at a little bit. 16% hate it, 9% love it. Super interesting. So given that, you know, maybe it's not the most fun part of the job, even though 9% of you love it, what actually is the value of tests?

Speaker 2: And again, a couple of responses from the folks I've interviewed here. Um, I think the most common theme with these is is this idea I mentioned earlier that, like When you're building a new thing, you're making sure there's no unintended consequences, that you're not breaking what's already there. So uh It helps you essentially maintain the status quo in terms of your functionality. You can move forward without introducing new bugs. And yeah, ship ship faster, ship with confidence. So that's kind of the the the value of tests. So uh this is what we're gonna talk through in this in this c uh in this talk. Uh six things. We've actually already talked about the first two. So uh we're doing decently well here. Um next up I'm gonna talk about recording traces with colo

Speaker 2: uh and and what that actually means. Then we're gonna get to the really fun, meaty bit of like inverting a trace to generate an integration test. And then more live demos as well as some some thoughts on kind of like what's next. Alright, so recording traces with colo. First up, like what is a trace? Some of you have may have come across tracing in like a production monitoring context. And the idea is kind of similar, although when I talk about a trace, I actually talk about something that's quite significant. So I have a little uh bit of demo code here and I appreciate this might be relatively hard to read but um so we have you know uh very straightforward code on the left. Or at least the the first function is straightforward. The second one gets a bit more interesting. But we have like a very basic add and then multiply that just is implemented using the add function.

Speaker 2: So it's a little, you know Obviously you could just do two star three and that would multiply. But I want to show you kind of a little bit of nesting and how that's represented in the trace. And then if you already use Colo, actually, quick question. I know a lot of you have seen the talk last year, so I won't ask if you've heard of Colo, but hands up if you've used Colo. Okay, nice. Awesome. That's I think maybe twenty-five percent. Um and you you I think the best way to use colour, or a very common way to use it, is using the middleware, but you can actually also just do color run and then any script. So that's what we're doing here. And then what's come out on the right side is a very simplified version of a colo trace. So I want to walk through that with you really quickly. And I've actually got this up, which is the same thing, but we have the the colour annotations on the on the left here.

Speaker 2: So maybe it's a little bit simpler to see. But the point basically is that um a trace consists of multiple frames and it's essentially call and call and returns for specific functions. So we run the script, the first thing that happens is the call to the multiply function. That's on line 10 here. Makes sense. Then the next thing that happens is the call to the add function. That's here. Then we have the return of the add function. So that's the first return. So you can see like essentially the trace captures everything that happens in the program. Or, you know, not everything, but like every call and return, as well as a couple of other things. So it's a little bit simplified, but um that's essentially what a trace is. All these different events that have some information about what happened in our program in chronological order.

Speaker 2: So same uh same overview of a trace, again a little bit simplified, but now using a Django view. So very simple function-based view, list to-dos. We're gonna have a little to-do demo app in a second. Imagine this lives at just like root. Um and then all it does is it uh yeah, gets all the to-dos in the database and renders like um renders them. basically. And then you can see actually on the right we have the same colour trace again, but now we have a few more bits of information. So at the very top we actually have information that there was a Django request. So in this case, the colour middleware captured the entire request. So that includes the Django request, which went to root. Then we have the call to the list to do 's function as expected. But then

Speaker 2: super interestingly we actually have information about the SQL queries as well. So the first thing that happens in list to-dos is uh that all of these to-dos get fetched. And that happens via the SQL query. So Encolo actually captures not just the SQL query that was made and sent to the database, but also what the database returned. So in this case we have two rows, ID 50 uh and then the title and like whether the to-do was completed or not. And we can use that information both to kind of Explain to a user, hey, here's what actually happens in the request, but then also we can use it later to generate this integration test, which we'll get to Then we have the return event from the list to dos, and finally we have the Django response. So in this case, uh this was status code 200 and some HTML was rendered.

Speaker 2: So this is what a trace looks like that was captured using the colo middleware. Alright, so let's go and record some traces. So I have VS Code open here on the right. I'm gonna make this a bunch bigger. That's a bit too much. And I'm gonna make this bigger here. And could anyone at the back tell me if you can read this? Both okay. Uh yeah, that's yes. Yeah, better. Better? Okay. If only the UI wasn't so big. I mean this seems ridiculous, but let's try it. So uh we have a very basic to-do demo app here on the left, um, and then we have uh VS Code open on the right. And uh I've already recorded a bunch of traces here, but like let's record some more.

Speaker 2: The the code itself is uh you might have to take my word for it, but it's like relatively simple, right? So we have some kind of decorator here, which is J used just as a demo, you know, if if evil is in the headers, then don't doesn't work. Um so y this might be like a user authentication decorator in a real app. application. We have the same code that lists the to-dos. We have a function-based view to add a to-do. And then actually, and this is something I think all to-do apps should have. Like if you have a big scary to do, like give a talk at Django Day Copenhagen, there should totally be a button to like break down the to-do for you. So uh Prepare presentation slides, practice the talk. And this is generated using using OpenAI and ChatGPT and stuff.

Speaker 2: Um so I think this this is just like a fun example, but like I think all to-do apps should have this. So that's what this last view does, which is um which yeah makes an API call to uh uh Or th this is the view rather here. So we we take this existing to-do, we actually edit it a little bit, then we we send it to OpenAI to break it down, and then we just um we just like re-render the pages. Essentially. So those are um uh those are our functions. So the first thing I actually want to show you is uh just Listing existing to dos. So just by the middleware working, um, we can now list the to-dos and then colo uh And if you know Colo, this will be very familiar to you. It gives you uh like an overview of everything that happened in that request.

Speaker 2: And usually if you have a slightly bigger screen, actually let me just make this big so that it's something is showing up. So uh it gives you this visualization of everything that happened in the request. Um this one was viewing a page. If I go to this one that we did a second ago, for um breaking down a to-do. There's a couple more things going on. We can actually also show you the outbound API call that happened to the OpenAI API. And we can even show you like all of the content in it. So we can actually see exactly like what uh Chat GPT returned here in case you're developing like this can be quite handy. So this is like a brief overview of Colo and like how um like what a trace uh looks like in kind of a visualized format if you're just

Speaker 2: um kind of doing local development or looking for essentially like an always on debug always on debugger to help you understand what existing code is doing. Obviously this is all pretty simple, pretty straightforward to understand. So this stuff is a little bit more useful when you're working with larger or messier code bases and there's nothing uh to do with test generation here yet. So this is just to give you an overview of what it traces and what Colo does. So uh Now that we've kind of established that, let's talk about how we might invert a trace to generate an integration test. So we've already got these traces stored. We've interacted with our application locally and we have these like big JSON files that capture everything that happened in each um each request. And I uh obviously Joseph

Speaker 2: already talked about this a little bit earlier, which is great setup because we don't really have to rehash too much what an integration test is. Um and I actually last minute added the arrange act assert here after hearing it from you, Joseph, because I think it's a great way to think about it. Um so when I talk about integration tests, I mean we pass it through the Django test client, basically. We pass it through the whole stack. We're not opening a browser or clicking on things, but equally we're not testing a single unit. We're going through the whole thing. Uh in this case we actually don't have any arranging, which is extremely rare. I think you could tell from that earlier response I read out that like um Often the arranging is like a really painful hard part of putting together an integration test. But we do have the acting and we do have the asserting. And this is relevant because

Speaker 2: With what we've built with colo test generation, we actually mostly want to help with the arranging. We found that that's like where a lot of pain exists, like getting the database into the right shape, getting your factories and fixtures if you use them into the right place before you can even do the acting, before you can even test and assert what you want to do. There's a lot of um uh setup required at the beginning. So um, you know, recapping of what we said earlier, integration tests give us confidence that like everything still works the way it used to. That's why I like integration tests. integration tests, that's why we use them mostly for simple poll, which is very user interaction heavy, lots of payloads that get sent around between Simplepoll and Slack. So uh how do you go about inverting them? Well, if you remember from earlier, we had um these two frames captured.

Speaker 2: So Django request and Django response. We know that the Django request went to root, and then we know that the Django response did 200. So I mean this might seem really, really basic, but yes, like you can just turn that part of the trace into like a Django test client interaction where you send a test client request to root and then you like assert on the status code. So this is like the most basic version for just getting something working. But like it's interesting because just doing this already gives you like a bunch of coverage, right? Even if you don't assert anything else, now you're actually um You have some you you have some coverage for your view. So this is already, I would very much argue, better than having no tests at all. Because if you break the root view and it suddenly, you know, has an exception, now you have a test that that makes sure it

Speaker 2: It always gives two hundred. So even though this seems like so basic and stupidly simple, it's already like quite valuable And now here's where it gets really interesting. So on the left here we have just a um a snippet of a um the SQL query I was showing you earlier. So a query which is also part of the trace. And this is a select, and we know that it selected these two rows. And um I believe this was for rendering the page. So if we want to actually render the page that lists the to-dos and have like a realistic overview, we need to seed the database with some data. Uh we either need to do that ourselves by creating to-dos manually, or we can just generate it from this existing trace and create these two values uh

Speaker 2: automatically. So based on what you see on the left there, Curlo can automatically generate these two lines of like um uh yeah like row database setup so that you can then um render the home page with some to-dos present. So I think this is like one of the m uh yeah suddenly it starts getting a lot more interesting. Like you don't have to come up with titles anymore or you know, not at least at this stage, you don't have to remember like uh w what's the um model actually called, you can just do that. And interestingly, uh we can do a similar thing with update queries. So with update, we know um what fields were changed and which fields remained the same. And if they were changed If the update query actually modified

Speaker 2: a specific column, then it's pretty reasonable that we want to expect that it changed in our test. So same thing for updates, we can turn that query into like an actual assert. And then so these are some basic examples, but in Kohler we actually have a ton of these rules and a ton of different logic. So we support a bunch of ways to make it like quite nice. We have support for defaults for Factory Boy Factories for HTTPrity, which is an HTTP mocking library. For um time freezing libraries, and then also we have a bunch of hooks for uh writing like custom test code or or custom templates. So let's go and generate some tests. So I'm gonna go ahead and close this. Um I am gonna go into this test file and then on the same trace we recorded earlier, I'm gonna just hit generate test, and it puts a whole test in my file.

Speaker 2: And then I can run the test and it passes, right? So and this might seem like okay, you know, this is not the most readable test yet. Uh and I think that's fair, but it's like at least a starting point, right? So now you have something that works and you can start editing it, you can start improving it, you can start using these custom hooks, which I'll show you in a second, to like cut customize it to your company's style guide. Um but like at least it works. So and this was actually the slightly easier test uh to show some more of the power. Let me actually generate the test for um the breakdown, which includes an outbound API call. So you can see there's a bit more a bit more in here. Make this smaller. So you can see like you know, we're freezing the time, uh, we're blocking outbound requests, we're making some um

Speaker 2: some seed data, and then we actually mock the entire HTT request that we send to OpenAI because we don't want our integration test to actually interact with the OpenAI API. Then we uh put the same data that was sent in that post uh form request in the browser in here as well, which is also uh extracted from the trace. We know that it should it should um redirect. It's like the save redirect pattern. And then, I mean, how cool is this, right? So we created the to-do earlier. up here. We know from the trace that this to do was updated. If you remember we added this bit to it, which you know For demo purposes. Um Colo knows that it needs to refresh this and then assert on the updated title.

Speaker 2: And then there's a few more things to ensure that these objects exist. Uh so if I run this again. It passes, right? So like this is like pretty cool, uh in my humble opinion. Uh cool. So uh and now you might say, well, this is like a pretty basic, you know, somewhat straightforward demo app. Um like it obviously it works in this, but does generating these traces actually work for like a larger code base? And this is where I'm gonna go back to simple poll, which wow, this is completely unreadable. So simple poll um is you know a pretty large Django code base. It takes like several seconds to boot up, which is annoying. Um and here I have a trace of someone voting on a poll. So uh with this, I'm gonna open the visualization and yeah, this doesn't look too nice, but you can just tell it's like a pretty fat trick.

Speaker 2: Like there's a lot of things going on. I can jump to them and like open the code and those of you who've seen Cola will have seen this before, although with a more readable editor. Uh and uh anyway, like the point of this is just that there's a lot going on, someone's voting on a poll. There's a very core interaction in the app, so obviously there's a lot of things happening. But like let's try and generate a test from this. And here's our test, like tons and tons of setup. Obviously, this is like not very readable. Um there's a ton of just like very basic data which is going to be the same in in a lot of tests. But we do get a test starting point made, and this should also pass. I might need to do something special to not run all tests at once.

Speaker 2: So this will just run that one test file. And yeah, like it passes, right? So it's possible even for like much larger, much more heavy views. And um and uh so since this is a real application, you know we're using it for colo, we actually have uh made use of a bunch of these hooks. So I mentioned colo has support for um uh factory boy. So we use a factory boy and we have a ton of factories. So you can actually in your colo config set up like all the factories and tell colo about them. And you can even set up these custom processors, which essentially just have even more power over shaping the generated test. So if I delete all of this and then I click generate test again. It will uh essentially build the uh you know

Speaker 2: integration is from the same um from the same trace that we did earlier, but now there's a bunch of custom stuff. So we inherit from our existing custom test case, we have our own setup, we use time machine instead of freeze gun. Uh we use the factories uh and so that we don't need to pass as much data into the into those. Uh and then um Yeah, we have cut we use custom existing functions that make all of this look a bit nicer. There's still a lot going on in this test because this trace is very core and there's still a lot of room for improvement here. But this should also pass. And it did. Nice. Cool. Love when a live demo actually works. So yeah, but so the point of this is you know um obviously larger code bases, it's much more difficult to get this right, and kudos

Speaker 2: honestly to Lily for getting this all working is like a lot of code. Uh and you can customize it as well because it's a larger code base. How am I doing for time? A few minutes, okay. Okay, okay. Um so the answer the uh to the talk uh title, how to spend less time writing tests? Well, maybe consider auto-generating some parts of your integration tests. Or you know, if you have like no tests at all at the moment, then maybe you can actually generate quite a few of your integration tests. And that can get your coverage up to like a nice level. And then you can start, you know, if you have the time, uh Tweak it and ensure practice of writing tests manually in your company, adopt the style guide, all the all the great stuff. Yeah, the best way to get started is a docs.

Speaker 2: curlo. app. In the past, uh there have been some uh so has been some feedback from people that they'd love to use colo, but like it's you know only in VS Code With the test generation, you actually don't need to even use VS Code. You can just use the middleware, use the command line thing, you can do color trace list to get that ID, and then you can just generate the test with just the ID. No VS Code required, but obviously. It can still be handy to view the trace. What's next? I actually have another Slido for you. So if you um If you still have your phones up, which I think many of you might not, um we have a ton of different ideas for like where else to take colo

Speaker 2: and like what else to do with it. We there's a bunch of ideas for beefing up the test generation. Like could we generate tests even more automatically, even without you having to click around? Can we do support for um different uh you know having multiple requests in a single test case. So a few questions for you that might help help guide the direction of Colo in the future. But um yeah, obviously we're both here at this conference because we'd love to talk to you to all of you about like what you think about the all of this approach, like if this is interesting to you. I think we're particularly um interested in talking to you if you work with multiple relatively messy code bases. and you just don't have enough time to like make them less messy or more um

Speaker 2: more tested because I think we can really help you in that case. So yeah, I think I'm just gonna leave this up uh and while the while we do the questions. But yeah, thank you all. Great to be here.

Speaker 1: Thank you so much. Nice way to ask questions too.

Speaker 2: Yeah. I ask the questions.

Speaker 1: Um yeah, we have time for questions. We have uh a couple of minutes. Let's do it.

Speaker 3: Thanks, Will. Great talk as usual. Uh not the first uh talk I'm uh listening to. Uh Um so my question is uh I saw w one test you are mocking the request being done to OpenAI. So if I have a view that sends an email and it doesn't talk to an API but uses uh the Django provided module. Uh is that uh mocked somehow?

Speaker 2: Yeah, yeah, this is a great question. And I think this actually came up um I think last time we we went to a conference. Uh I think so I think the short answer is that we um don't have anything for for this now, but we're interested in Building some kind of like generic mocking API essentially so that um like Colo will currently record that that happens and that the outbound thing happened like in local uh development So the question is like how could we then build like a nice or like generate a mock for you essentially, right? And I think for that we are st um we haven't like prioritized this yet. Uh and I'm not yet sure how the API should work. But my um current thinking is that you should be able to

Speaker 2: somehow specify that you want a certain thing to be mocked. So that could either be like a uh email thing or it could be like you calling out into like another complex bit of the code which you want as part of the integration test. as in you want that to be part of it, but you actually don't want it to pass through all of that logic. It would be cool to give like a generic interface where you can kind of like mock anything. And Somehow that somehow is why we haven't done it because we're not sure yet how is uh somehow tell Kurlo generate the mock for that. Uh email is a great example which I hadn't thought about and this context. But that's how we we we want to build a generic mocking thing basically. Maybe we can talk about how it could work.

Speaker 4: Uh yeah, so interesting application. Uh I had a question. Uh do you make do you have some logic to make distinction between behavior uh that is externally observable and the implementation details so that uh whenever somebody re uh refactors the code it doesn't break because of you tie to very specific implementation details.

Speaker 2: Yeah. Yeah. I guess the boundary for that would be at whatever kind of the integration test is. And I guess in some ways that's up to engineers. So um Yeah, like I think maybe you know as an example, like if you uh if you add a new column, right, uh to a new um uh to uh to a new table, then uh maybe some actually I'm not even sure these tests would fail because I think that would be fine. Um but say something benign happens And some of one of these tests does start failing. I think the idea could be either you uh re-generate it and then like look at the difference, or you just see, because they are still like readable tests, right? They're not like some background thing. Um you could just

Speaker 2: add to it like yourself. Uh but yeah, I guess broadly the point of integration tests is that you can refactor things and it still works and it still passes. So I guess the the line can get a little bit blurry, but l largely I would say it's like down to how like you decide which views to write tests for or which of the generated code you decide to keep, what to extend.

Speaker 1: I think there was another question or there was a hand up

Speaker 5: Um it's really cool. I I think it's an awesome project. Um I really liked how uh when it noticed that something was setting up. s that it it could then do a search based off that. Um would it also uh can it also detect when there were was some filter and there's some data that was generated that doesn't match that filter so that it also shows that you you didn't get just everything.

Speaker 2: Right, right. Yeah, yeah. So um like a this is like a query set Django filter. Exactly. Yeah, yeah. Yeah, so I think we have a couple of things around this. Um so one thing we try and do when we uh make the assertions is we um we have a couple of heuristics basically so we would filter by like unique fields So that they would then be like uh you know we would fetch the right row to be asserted on, if that makes sense. Um but that might not have been your question.

Speaker 5: No, no, I mean more that um Let's say it was just uh you filtered a query set and this this is in your view.

Speaker 2: Yeah.

Speaker 5: Um but in the preparation normally when you would be setting up this test, you would probably also create something that is not supposed to be there and you want to show that it's not there.

Speaker 2: Ah, I see, I see. So you would have like an assert before you act to show that it's not there?

Speaker 5: No, but

Speaker 2: No, I'm sorry. Wait, Lily, do you understand? Do you want to answer? Okay. Uh I think it's something that we can't really do because the the data that shouldn't be there isn't there in the trace. So it's something you would have to add manually. Right, yeah, makes also sense. Nice, and maybe we can chat about that more. Anything else? Otherwise, we have some some results. So it sounds like a good two-thirds of 40 of you wish you probably had more test coverage. Half of you worked in a project that had almost no tests. That's what I'd like to hear because hopefully we can make some tests for you. Awesome. And then uh a couple more

Speaker 1: What if you only have tests but no implementation?

Speaker 2: Yeah, then uh then you are the best software developer in the world. That's a joke.

Speaker 1: Right. Um let's give Will a big big hand. Thanks for uh sharing everyone.