Quality Assurance in Django - Testing what matters

Hello everyone. Hello, I 'll try to speak uh not so loud. Uh I'm really happy to be here. This is my first Django con as a speaker and um really uh I want to say thank you for the opportunity given. So I'm Radoslav Georgiev, Radoslav Georgiev in English, CEO of Hacksoft, end-to-end software development company. We are based in Bulgaria, which is the country where the big red arrow is pointing to, this is Eastern Europe. And A special swipe, twenty-second of September is actually the National Independence Day of Bulgaria, so a quick shout out to all Bulgarians listening or watching.

The goal of this talk is to be practical, pragmatic, and actually provide value for you. And if someone has something that's meaningful for him or her after the end of this talk, then I will be happy. So this is the final goal. And the context of this talk, we have experience with Django projects that were zero percent percent test covered to a hundred percent test covered, and also we are building a Currently quality assurance teams and we have some context about this topic. and something really important. Just like the beautiful city of Porto, this talk is going to have a lot of denivelation. So we're gonna go up, down, up, down. like really into context

and then zoom out and look at the bigger picture so we are in the right place for this So, quality assurance. This is an interesting topic. And first we are going to set the stage and give a point of view from like a development point of view We are developers, we are developing a new feature and we just want to we are ready and push it to production. Or we are changing an existing feature, we are also ready and we want to push it to production And usually if we just do the work, say that we are ready and push to production Everything is going to be alright, usually, most of the times, but some of the times

we are actually, as developers, the dog in this picture For the dock it's okay when production goes down because it has gone down many times if we know what to do, but looking outside It's not really okay. And the first for me definition of quality assurance is the process that helps us not be in this situation as often. But we'll see. So Before pushing to production, we need to take extra steps, which are make sure the new feature is working as expected, make sure the change that we've done is working as expected, but also Take into account something that sounds bad

and it's also bad called regression, which means this is the the word that we use for but this was working just last week Why it's not working right now. So we also need to make uh sure that we are not introducing regressions and accounting for regressions And of course we can do this manually. Once we are ready, we start clicking, using a CLI, calling KPIs, whatever the case is. And perhaps the first couple of times we can be pretty exhaustive and with confidence say, This is okay, let's go to production. But as we all know, doing this manually does not scale well and it also is very prone to uh human error and exhaustion. Like I'm not feeling very productive today, so let's just say it works and push to production

So we had the writing tests. Like this is this is the main uh argument for writing tests because doing it manually is not very good. And we're now here. This was We know that we have to write tests, but now the question is what tests do we write? We've been to a motivational talk. We are now hundred percent agreed that we need to write tests. We go home, we have a good night's sleep, we wake up, we open up our computer and let's write tests. You're wearing a t-shirt that says tests are best or best But the question is what to test? And since we are at DjangoCon, let's look a picture of a typical Django application, be it

website or just an API. It usually has some kind of view or API which works with form serializers serializers. It's really hard port. for for Bulgarians. Um we we might have business logic, we might have salary, we might have Redis, third party integrations, of course all REM. working with the database, communicating with the database, additional framework abstractions also we may build on top of Django. So We open up this diagram and we say, okay, let's write tests. And the question is, where? What to test? And as developers we are conditioned, at least this is my opinion, to think about unit tests.

When you're uh learning how to program or going to university or Studying computer science it's all about unit tests and unit tests are really great because it's input, output Everything's passing, everything's working. So as developers we look at this directed acyclic graph short for DAC, and say okay let's pick the leaf nodes and start testing there. And usually every project has some sort of utilities module This is a great place to start because it does not depend on many other things from our application. And we open up our utility module and it's uh computer

science one on one again. We have a pretty nice function. It's a random function, no need to look at it in details, just unique by. And we are okay. How to test this? How to approach this with tests. And remember, of course We are going to write unit tests using the unit test module in Python. And this is very important. We are in Django context We've started with utilities and the very first test that we write is using the unit test module and I'll call this the pure unit test. And the test is whatever. Input output it passes, everything's great. But We are in a Django project, so we might end up writing s

a utility that looks like this. It's a get object, wraps around get getObject or 404. Catches the exception, returns none because it gives a cleaner interface. And now in order to test this, it's no longer just Input output. Again, it's input output, but we need more things. In order to test it, we might need to actually hit the database and actually have a model. And the interesting thing here is that the it it doesn't really matter what the model is. We really don't care about the model. And we write three test cases. The first one is The happy case, the second one is nothing's there, and the third one is

we still want to raise exception if we are trying to get by something that's not unique. because this may show us that we have some kind of problem in our data model. So far so good. Still a unit test. The difference here is that we are now using the test case from Django. And why we're using it? Because we need Django test framework to do some additional setup for the database We can no longer just rely on the pure unit test module. And so far Testing utilities, it's quite straightforward. It really feels good. I I love writing test for utilities because it's a quick fake feedback loop.

I'm documenting usage. I can then show this to my colleagues and say, look how I'm using this and uh take notes from the test. So it's really good. Back to the diagram. The other leaf note or RAM database. We're like, okay I know that I need to write tests. I know how to write tests. I just wrote some tests for utility, so let's just continue. Box by box, we'll be fine. Eventually. And we have a model and we write a test like this. I'm pretty sure everyone in this room has written a test that looks like this. I have a model and I'm basically testing that.

The Django RM works. It's like this test give me gives me the information that the RM is actually working, the framework that you're using, it's working, and that's good. And the thing is, most probably the Django itself has tests like this that make sure that the framework is working. It still feels good because we're doing work and we're getting green ticks, but not much else. And there's an asterisk here because If I am, for example, testing a more complex multi-database setup and I just need something that hits the database, I may write a test like this. to test the database setup, not the ERM models, but we are not achieving very much with tests like this.

E RM is working. Great. And if we follow this approach, follow this strategy, where we go to the boxes, we pick each of the boxes and write tests for it, depending on our context. we might end up in like a very dense wood and b and we might um be in a situation where everything is a unit because we are writing unit tests and we're not very sure once we get to the boxes that depend on other boxes how to approach it. We might start our mocking journey and one month late one month later come back and rethink our priorities. But the thing is

this can generate a lot of busy work. And this is busy work that's not achieving actually very much. Yes, we can have tests, we can write tests, but the end result is we don't know what we've achieved and most probably we haven't achieved much. And ideally, of course, you can say Rabo just test everything, 100% test coverage, everything is going to be good That's the ideal case, but it's very rare that we are in the ideal case. We have to make time, we have to have a strategy, have a plan. So we're not in the ideal case And this is where we want to perhaps start thinking more strategically about the test test the tests that we write.

And ask actually ask the important question. Because again Just going unit by unit, a lot of busy work, but if we don't know what we actually want to achieve with those tests It's just busy work. And we want to ask the question, what do we want to achieve? And it usually the answers are around this. We want to have more confidence in development meaning develop quicker, ship quicker, don't have the anxiety that we've changed something And something completely unrelated someplace else is going to break and we're going to end up in the situation with the dog in the room and everything's on fire and we're again this is fine.

So we we we want to have more confidence We want to catch nasty regressions. Again, this is part of the confidence. We want to reduce the thing that's that's a joke, like the joke uh term uh user-driven development, which is Write the code, push to production, give it to users, and they'll tell you what's not working. Sometimes we have to do it, but we want to reduce it because users want a working piece of software. We want to avoid busy work and that's that's the hard part because we are working, we are producing code, we are having pull requests with a lot of lines in addition, yet We're not very sure what we are contributing to. And of course we want to remain in Django context, and this is very important for this talk

because we are at DjangoCon and we can go one step further to end to end testing clan with Selenium or Whatever else you like. But we want to remain in Django context. Let's say we take care of the back end. That's it. We have no control over the front end. So we want to remain in Django context, and that's important. So How to think more structured strategically about this? Well, our system has cold pathways, like cold stack You you call you call a function and then suddenly ten other functions get gets called down the road. Users are hitting those code pathways. And usually in a specific state, in a specific code pad way, we can have an error.

And when this error occurs Things are not good. And if we are to approach this more strategically, we will start with tests that cover As many code pathways as possible with as little tests as possible. Sounds good. And if we come back to our diagram It's a Django web application and usually it has an entry point, either someone submitting a form or someone calling an API. And this thing afterwards calls everything else.

So this looks like a good candidate to start actually testing. And if we are to approach Tests that hit the API or the view, we can start with something like this. This is the test case from Django, because we need a database And we can start with basically writing the method. The method name doesn't matter really much, at least for me. You can call it XYZ. It's whatever. And then in a doc string, as you can see, we can describe basically a bunch of steps that the user is doing And this is usually usually called the user journey or a user flow. And this is a change

in the way we think about our tests. because up until now if we're writing unit tests we are thinking about code. What Two Tesla. But here we're s we we start thinking about users. And let's say we have a bunch of APIs that take care of authentication. We say with some kind of verification, let's say SMS. Whatever User starts verification, tries with wrong code, tries with correct code, obtains access token. We actually want to see that this access token is working. We log out the original access token no longer works. Like this is a journey. And what this journey is going to give me is some kind of confidence that if a user does this

I have replicated the actual code pathways. I'm hitting the same code pathways and I have assertions at the end. So we start taking in user journeys and user flows. And zooming in This is again an example implementation. A more exhaustive assertions can be made, but they were not fitting very well on slides. So that's why I'm just Asserting counts. And what we can do? We can use faker, generate data, send request, assert, internal state. Okay. Okay. The thing that we can achieve with this approach, and the thing that's very important and it's a little bit nuanced, is

we are hitting bolt framework code, which is Django. We are in a way testing that Django works, but we are also hitting our business logic or app logic or domain logic code and we are hitting them in an integrated way Because we were we can test framework code but only in isolation no use of it but if we test framework code with business logic code then we're covering we're basically replicating what the user is doing And a test like this, if we zoom out, can look something like this. I have just copy-pasted everything from my dog string, put it in this strangely looking context manager.

And fill with the actual test implementation. And if you're asking what this context manager is doing, it's basically uh wrapper for sometimes I miss having blocks in Python so I implement my own block that just wraps around the piece of code just to visually Isolated from the rest so you can achieve something like this. And again if we zoom in An example implementation. This is quite a lot of details are going on here. We are calling the database, constructing payloads. Sending requests, getting responses, we can make assertions based on the responses, and continuing forward to

Assert that this specific journey is behaving as we are expecting. And this is kind of integration tests, kind of behavior-driven development. I think we can call it both and is going to be correct, but we simulate the user journey while also having the internal state and representation And this is the powerful thing about those tests that take the API and test everything after it. And why? If we go to Quality Assurance Land, people there have something that they call test automation frameworks And they spend quite a lot of time implementing various test automation frameworks. And by doing this, we can actually have a test

automation frame. We can have Django being the test automation framework because If we come back here, we are calling endpoints as end users, but we also have direct access to the database. I can just make a query and assert some kind of internal state and say okay this is what needs to happen and this is actually good. And it's really powerful. And again, let's let's have a let's have a different test. We can have a test that given we want to test uh access token expiry, we say given a user New word here. User can access alt

requiring APIs. This should be cannot And so what we do? We override settings, we uh send requests, but the important thing here is the given a user. And this is something else that uh we found quite useful to have in those test cases because it reads very well and it gives you a user and the implementation of given a user is totally up to you Whatever your tastes are. You can replicate API calls. So any given is like executing another set of user journeys. You can use factories, you can just use RM

code, you can use whatever you like, and you can return whatever data structure you like, and it's just the concept of wrapping the tools and abstractions that you're going to use. So you're kind of decoupling from them and just using this given um This kind of notation. And later today there is a talk about factory boys, so make sure to uh come and listen to it. That's why I'm not going to cover factories at all And another thing that we can do with tests that cover as much ground as possible is we can cover even more ground because usually our apps have some kind of uh tasks integrated into them like salary

and we can say for my tests I will run salary in memory I will mock whatever's going outside. I don't want to call S3 or whatever. Send emails. And cover even more ground, basically replicating, going as close as possible to end-to-end tests, basically replicating what users are doing. And there's a really nice thing that was previously a package and now it's part of uh Django, which is capture on commit callbacks, because you most likely execute tasks in transaction callbacks, so Django is providing the tools for that.

If we are to show some kind of a pattern for those kind of tests, it's going to look like this given XYZ I want to replicate a user journey, use Django as my test automation framework, have access to the internal state, cover as much ground as possible. And you 'll say, okay, Rattle, so yeah, this looks good, but Those tests are slow as hell and if I have thousands of those tests the CR is going to take forever. And the truth is, yep, there are always trade-offs to be made.

Those types of tests can be slower and more heavier than just unit tests. And for example, if you give me a project with 0% test coverage that I need to support and maintain Most of my effort initially is going to go towards writing those tests because I can write those tests without even knowing what the project is. I just need the entry points and I need the user journeys. And then I can start fiddling with it, refactoring and doing things with with the project. But if you for example give me code like this, there is a clean method on a model or something else And if I start testing if this is behaving as expected

with the heavy API tests It doesn't make much sense. Then again, I will be slowed by the type of test that I'm picking. And the more focused unit test will do better here. So Yep, again uh this coming back to this diagram. And since we are talking uh about quality assurance, uh it's a mandatory to include the test pyramid And I'll talk about quality assurance, but since the test pyramid is quite boring, this is the test cake. That's why that has a cherry on some. So the test pyramid says most of your tests should be unit some integration, some end-to-end. We are excluding end-to-end since we are we want to remain in Django land.

And perhaps this is the most important slide for me for this talk. When I approach quality assurance as a developer in a Django project, I open up this diagram and try to figure out Where am I going to be? More towards unit tests, where is utilities and business logic layer that's decoupled from the database? Or more towards integration tests with views and APIs and business logic layer interacting with the database. And there is a gray area in between That can be covered with both pure unit test cases or Django unit test cases, depending on what we actually want to test.

An example here is uh if I want to test this validation, I don't need to store this model in the database. I can just use it in memory It's going to do the work. So this is extremely important and it actually answers the question what to test. And sadly, the answer is it it really depends on your context. And it's important to have a test plan. That's why there is a quality assurance Position, teams and it's like software engineering but for other parts of the process and it's really it's really important and this can help you navigate where to put your energy, where to put your focus, and what to test.

And it's all about Which test case you're using? This is a great marker. Can I use the test case? The pure one? Yes, great. I will go with it. Otherwise, I will go with the Django test case or something that inherits it. And of course, quality assurance is quite the broader topic. And testing is a part of it, but it's not the entire part of it And the thing is, quality assurance can also include developer experience, testing software architecture, the how you handle deployment. How you handle different parts of the process. You can assure a decent amount of quality on a project without

writing a single line of test. And this is really important to know and to think about. It's not only tests. We as developers we're wired quality assurance go write tests. But sometimes we can Increase our quality by doing other things. For example, we've developed for ourselves an internal Django style guide which if we follow is going to guarantee a good amount of quality because we know it's working and we have a pattern that that we've been using quite a lot. Another example for me, my Pi is a quality assurance tool It can help you catch certain errors by having type checker. Same with TypeScript

And of course, testing in Django is really fast topic. There are tools, libraries, best practices, and you really need to put in the effort and think about how you approach it. It's not just write tests for everything, but how you write it, what you test, what's your res what are your resources and how do you want to spend your time And as I mentioned, the goal of this talk was to be for you practical. So I hope I have kindled some kind of fire that you're going to follow up after this. And since I have one more minute, I'm just going to tweet a bunch of things that I found useful around the topic of quality assurance on Twitter. Yeah, that's it.

Thank you