Testing Modern Web Apps Like a Champion with Andrew Knight

Speaker 1: So Andrew, aka Pandy Knight, is a software quality champion who loves to help people build quality software. He's an avid supporter of open source software. He's a playwright ambassador, as well as the lead developer for BOA Constrictor, which is the. NET screenplay plattern. He is automation panda online if you want to find him. Pandy is a local here, and I got to meet him when we were putting together the proposal to bring DjangoCon to Durham, and I'm glad he's here to give us a deep dive on testing today. Please welcome Pandy Knight.

Speaker 2: Good morning everybody. How are we doing today? Great! Yes, so exciting. My name is Andy Knight and I am the automation panda. I do things with testing. It is truly an honor to be here today. I am local. I live in Kerry, and the fact that DjangoCon is in our backyard is just absolutely wonderful. So thank you all for coming to my hometown for this amazing event. It is really an honor for me to speak here today. I actually learned web development through Django. So Django has a very, very special place in my heart. In fact, if you see my my profile picture, I'm wearing the same shirt

Speaker 2: That profile picture was taken at DjangoCon 2019, where I was giving a tutorial on Selenia Web Driver for web testing. So it is it is quite special to be here on keynote stage. Four years later talking about more kinds of testing because while some things have stayed the same, many things have also changed. Today we will learn how to do modern web app testing like champions. My life's work essentially is to teach the art of software testing. There is a way to do software testing. And testing is an art as much as it is engineering. There are certain practices.

Speaker 2: It's not just about tools or frameworks, but a certain way in which we approach this problem. I always like to think that development and testing are two sides of the same coin. So if we are to build excellent web apps together as Djangonauts, then we must include testing in our practice. And today I hope to share a little bit of what I've learned in the 45 minutes I have on this stage to help y'all build excellent web apps too. So Djangonauts, are we ready to take this journey together? Yes! Awesome! I would like to introduce y'all to one of my best friends. This is Suki. She is my French bulldog. She is a gift from God.

Speaker 2: I love this dog so much. She just turned two years old. And Suki is the first dog that my wife and I have ever had. We've never had pets before. And how many people out there have a dog or a cat or some sort of pet? A lot of people, right? It's a lot of responsibility, isn't it? You gotta remember to like. feed the dog and and take the dog on a walk and clean up after it after it goes potty and get its vaccinations and take it to the vet and if it pukes its guts out you have to clean it up right there's a lot of things to remember. And so me and my wife being new puppy parents were like, crap, how do we remember all these things? So I developed a reminders app to help me remember all the things to do for my dog.

Speaker 2: I called it Bulldoggy, the Reminders app. It is available on GitHub here. You can use the QR code uh automation panda account bulldoggy reminders app note this is not a production level app I developed this as like a tutorial thing but for what it's worth it's pretty cool I've sometimes you you build something you're proud of and I'm proud of this look how cute this little logo is right Cool. So if we take a look at what the Bulldoggy Reminders app is, it's a web app. It's basically got two pages. You have a login page here where you can enter your username and password to log in with your account. And then the main page shows the reminders that you're trying to track. And so the left column will show the different reminder lists you have. So maybe you have a list for chores. Maybe you have a list for groceries. Maybe you have a list for projects.

Speaker 2: And you can dynamically add new lists and delete them and edit the names. And then on the right side, you have when you highlight a particular list, you have all the items in that to-do list. So if, for example, on your chores list, you might have things like buy groceries or mow the lawn. You can strike them out by clicking on them. You can add new things, edit text, delete all. The dynamic modern web app functionalities that you would expect. And every account has a particular set of stuff that's private to them. So if you log in as Andy and then you log in as Jessica, you'll get two different sets of things all privately. Maintained. Pretty cool, right? So how did I develop this little app? Um it's quote unquote simple in terms of

Speaker 2: A web application architecture because simple is better than complex, right? But there is complexity here. So uh the data I stored in a tiny DB database. Why? Because it was small and it was meant for tutorial stuff. I have pydentic models on top of that so that I can use those as like, you know, the the account or the the list or the item. The controller is handled with a web framework. You could use Django, you could use Flask, you could use Fast API, basically anything you would want. For rendering the views and all, uh I'm using uh templates and static files, so that just like you would if template, you pop in the stuff and boom, pretty web page. But the thing that makes it dynamically interactable is HTMX.

Speaker 2: How many people have heard of HTMX? Yes, oh everybody. Awesome. Yeah, Chris May gave a wonderful talk on it the other day. Uh HTMX is incredible because it'll it enables you to do more dynamic kinds of things as if it's a single page application. Um in my view, HTMX is a great way to democratize front-end web development for those of us who may not necessarily know or use JavaScript. So my app, that's how you're able to add and delete things. Um the application also has an API, so if you just want to do direct API calls to manage the re um the reminder data, you can do that too. Now, I'm not going to go into too much more detail about how this app is built. If you want to get a real deep dive into that, please scan this QR code.

Speaker 2: I gave a keynote address on this application at PyTexas. focusing on the fact that this is a full-stack Python web application. Everything is in Python. And as I said before, development and testing are two sides of the same coin. You could see that talk and this talk is very much bookends. That talk focuses on development. Today, here we're going to focus on the testing angle. So, how do we test this app? As we said, there are complexities here. The user would see it basically just as rendered in their browser, but we know there's many moving parts to look at. And um testing is challenging. That is no lie. So for the remainder of this talk, we're going to look at how we would go test

Speaker 2: a reasonable modern web app like the Bulldoggy Reminders app. So isn't testing hard? How many people think testing's hard? It's okay. I think testing's hard. Yes. It's not easy. There are lots of challenges we face in testing and lots of unexpected challenges, especially when we're doing black box style testing. Some of the things that I've hit in my time. Tests can be very slow to execute, whether you're doing it manually or you have a script that's running something like Selenium or Cypress or Playwright. It can take a long time to grind through all the different behaviors you're trying to cover. Tests can also be brittle. How many people have suffered flaky tests?

Speaker 2: Right? Oh my gosh, doesn't that stink? It's the worst. Cause you have to handle the the race conditions and the proper synchronization with the things that you're testing or otherwise it can just be like oh can't find the element because the page hasn't loaded. Die And then you're like, I gotta rerun this test. Oh my gosh, it's horrible. It's a waste of time. Makes testing hard as a practice. Uh test being flaky. Tests that don't make sense. How many times have you been assigned a test case to run and you read this procedure and you're like, what what does this mean? What does it maybe you didn't even get a test procedure, you're just like go test the thing and you're like, what? I have some user story for this, but how does that help me, right? From a business standpoint, tests don't make money. Right? There's right, right.

Speaker 2: I mean that there's a whole risk mitigation thing going on. You can justify this, but ultimately it's like Well, we we're a software development company and we push features and so we have to do all feature development because that's what brings in money and testing as an activity does not bring in money, therefore it gets perpetually Deprioritize, it's not important. Just keep pushing code, right? And so you're like, well, hold on, wait. We need to, if we're gonna have sustainable development, we need to make sure that we're building with quality and we're checking ourselves before we're wrecking ourselves. But that doesn't always translate through the bureaucracies and the leadership and all this kind of stuff. And so testing resources can ultimately be squeezed, if not just eliminated. And finally, in the context of being perhaps a programmer or developer or somebody who

Speaker 2: sees themselves as a builder. When you develop something and you when you build something, but then you go to start testing it, there is a mental shift that you have to do, right? There's a change in mental context. And that can be a bit difficult, especially if you're not used to that. And it can seem almost as a, it can feel like an interrupting force in your in your workflow. It doesn't need to be that way, but oftentimes it can feel that way. And so that that creates this this speed bump almost in a sense. that that testing is challenging and that you have to change the way that you look at something that you're already doing. Do these resonate with folks? Have we felt these? We've probably felt some others, but these are the ones that have particularly been painful for me.

Speaker 2: As a result of these kinds of things, um historically testing strategy for web apps has revolved around this thing called the testing period. Pyramid. Have people I'm sure many people have probably heard of the testing pyramid. It's a thing that's been around for longer than I've probably been in a professional environment. Um but the thought was that, well, okay, you have Yeah, you want to have a pyramid shape structure for the types of tests you do. You want a lot of unit tests. Unit tests are good. Why? Because they're white box, they're right next to the code, they're low level, they're very fast. They're touching your Python functions and methods or whatever you're doing. And so unit tests are good. And so just pile up on unit tests. Like you're at the buffet. You're like, don't get the rice. That's a lie. Just keep getting that general child's chicken man. Just pile it on, right? Unit tests. Unit tests are good.

Speaker 2: We've kind of indoctrinated ourselves in this thought. And then moving up the layer of the pyramid. Um component tests. The idea of okay, well maybe we have these UI widgets. Maybe we don't, but if you do, have a UI widget library when we can test these UI units as components, right? And this has become very popular within the web space. We're going to talk a little bit about later, but then it's like, yes, component tests are good. Because they're essentially like unit tests. We keep moving up the pyramid into API territory. And APIs, ooh, well APIs are still pretty good, right? You know, it's like you can make a request, get a response, verify 200 code. Um they take a little bit longer to run, and if there could be network issues in there, right?

Speaker 2: But overall we said, okay, well maybe we have Some API tests, maybe not as many as unit tests. Um they're they're questionably good. Like yeah, we should have some of those. And then we get to the very top of the pyramid, the UI tests, the most complex, the longest. the slowest, the most brittle, the most prone to flakiness, and we say, ah, UI testing. We want to do the bare minimum of that because they're risky. Right? They take the most time to develop, they're the most likely to break, they're the slowest to run. So we just we want to have very few of those. And it's we we have developed an entire testing strategy with this pyramid around presumptions that certain types of tests are good and bad. And I'm here to tell you today that's bull crap. Right, bull city, bull crap, bulldog. Ha ha ha, the bull jokes. No, it it's

Speaker 2: it's truly bullcrap because every type of tests Adds value in unique special ways. We need every kind of test. And to try to label some as good and bad is is not healthy for us as we approach the quality of our applications. I mean I could argue that UI tests are the most valuable ones because they're the ones that are testing your application as a user would. Right? And so it's it it is mitigating a different kind of risk at that level than say a unit test would. So when I talk about modern testing, I want to break away from this thought of the pyramid and break away from these preconceptions we have about certain tests being hard or bad. And there are three points I want us to learn about modern testing.

Speaker 2: First of all, I want us to focus more on building fast feedback loops than building certain types or counts of tests. Ultimately, what we're trying to do with testing is we're trying to build fast feedback loops into our development process. Right? We don't really care what kinds of tests you have. We want to know that if we push a change and we accidentally broke something. That something is there to kind of tap us on the shoulder and say, hey, you broke this, and this is what happened. Here's a view of it, and this is exactly where it was, so that we can go fix that. Ultimately, that's the whole purpose of testing. We don't do testing for testing's sake. We do testing to find answers and reveal truth. And so these what we want to do is really build fast feedback loops. with things

Speaker 2: tools like CI with tools like automated testing versus just trying to check a box and say we ran this, right? Or to say, oh well we we developed a hundred new tests this sprint. Right? Nobody cares about that. What I care about is protection. What I care about is safety. What I care about is if I want to move fast that I don't break things. Or if I do break things, when I do break things, I can catch it. A second point of modern testing that I want us to focus on is that we want to make test development as fast and painless as possible. Why? Because the business people were right. We don't make money off of testing. Testing is a risk-mitigating activity. And so we we want to spend as few resources and a little time into building those fast feedback loops as possible that can give us a confidence to say we're good.

Speaker 2: And so we want to be able to develop our tests. And when I say develop tests, I'm focusing mostly on an automated sense. We want to make that fast and painless as possible. We want to make that mental context shift from coding the application to coding the test against the application seamless. And third, together with that, we want to choose test tooling that naturally complements our dev workflows to keep us moving seamlessly so that development and testing truly are two sides of the same coin. Does this make sense to everybody? We're still awake. Yes, okay, I see heads nodding. Ah, Drew's like, I don't know. Gosh. Okay, good, good, good. Good save. Alright. And so while testing can have challenges, our approach to testing doesn't necessarily need to be challenging itself.

Speaker 2: One pattern that I have found for writing good tests is the arrange, act, assert pattern. All functional tests, whether they are unit tests, component tests, API tests, or UI tests, can fit this pattern. You may have seen it in other forms. In a BDD sense of given when then with Gherkin. I believe there was a talk on that the other day. Same thing. Arrange the things in the system, act on your target behaviors, and assert your expected outcomes. This pattern helps you write individual atomic tests that focus on singular behaviors one at a time. You can write all kinds of tests like this. And so as we go through the rest of this talk, and we're going to go through different kinds of testing, you're going to see this pattern appear over and over again. And if you would like, I have postcards with this pattern on them that you can come see me after the keynote

Speaker 2: so that you really won't forget this pattern. You can take it home with you. Alright, so we're not afraid of testing anymore. We recognize it's challenging, but it's not impossible. Can't we just use PyTest or unit test for our testing? Like why are we turning this into like a whole keynote address? This doesn't seem like a huge deal. I'm sure most of you have probably touched PyTest or Unitest or a similar kind of thing, maybe even in another language like JUnit, NUnit. I can just code prattle off on these frameworks, right? Well, I love PyTest. If you know me, I am a PyTest stan. I think PyTest is not just the best test framework in Python. I think PyTest is the best test framework in any language. I stand by that, and you all know me. I have done scary things in other languages with testing.

Speaker 2: Py test is by far my favorite test frame because of the way it does things. I better stop here because I got tangent way too far on that. But come come talk with me afterwards. I I love PyTest. But PyTest by itself cannot test web apps. Unit test by itself cannot test web apps because what PyTest and Unitest and these other core test frameworks do is they simply provide the structure for which for you to write test cases. Right? They they say for example with PyTests, what you do is you create a test underscore something module in a tests directory, and then you create a function prefixed with the word test, like test my bulldoggy app. And then what you can do from the command line is you can say python-mpy test, and it finds all these test functions and test modules and executes them, captures any problems, and then pops out a little report at the end.

Speaker 2: You still need to add the magic sauce inside each test case to test your apps. So while PyTest is wonderful and awesome, it alone is not the answer. We need more than just PyTest. It's Py test and other kinds of things. Okay, that's cool. Um well this is DjangoCon, and so we're talking in a Django context here. What about Django's testing support, right? Because Django has this awesome test client that you can do stuff with and there's like database management it handles, and that's true. Django's testing support is probably some of the best I've seen in any Python web framework. It's pretty pretty fantastic. You can use PyTest or Unit Test with it. But I want you to keep in mind that it's basically going to be limited to white box style testing, meaning the kind of testing where you're directly accessing your code and calling classes and methods, or sorry,

Speaker 2: methods and functions, excuse me. And so you can test Django models and views with the test client. And you can also test non-Django code as well, right? Because it's just all part of the same PyTest or unit test suite that you're developing. And like I mentioned before, one of the really nice things is you can actually set up a test database so you don't not hit your production database when you run your tests. But the main the the the big limitation, like I said, is this going to be primarily white box. You're not going to really test the app like a user. You're kind of going in the backdoor and monkeying around with things, which is totally cool, right? That's that lower level of that pyramid that we saw. But it's not going to bring it up on a browser and be like clicking and poking around with it like that. So it uh also like if I'm sure you know the the

Speaker 2: the Django admin the the managed PY if you want to run tests, Python managed py test. Gathers them all up. So here would be an example of a unit test that we would write. Now this is not a this one does not use the Django test client, but it could be run as part of a Django suite here. And what this is testing is uh Token serialization for authentication. So you can see kind of here we see this arrange act assert pattern. You know, we're serializing the token and checking things on it as like the arrange we're trying to set up. We're deserializing the token as the act. And then we're asserting that we got the username back from the deserialized version of that token. Right? Arrange act assert. White box unit test. We're directly calling the serialized and deserialized functions here.

Speaker 2: We're using some of the models that we have. All this good stuff. Does this make sense? Great. Okay. So example unit test or white box test. So what about black box testing, right? If if white box doesn't test it like a user is just kind of testing code, how do we do this black box testing? Meaning I bring up my application in a browser and I login with clicks and scrapes and stuff or maybe I call the API with a REST client. How do we do that kind of testing? Back to our pyramid scheme again. I use this because it's very common and most people can uh can understand the layers here. There is a separation in the white box with unit testing from the black box of UI, API, and arguably component testing.

Speaker 2: This is where we're kind of drawing the line here, just to kind of give context of what we mean by these terms. White box meaning you can look in and see it. Black box meaning is like a black shroud is over it. You can't see the code. You can only interact with it like a product. There are many tools out there to do black box testing. When we're looking at the different layers, UI, API, and components, um You'll notice a trend here. There are traditionally you could use things for like Selenium for your UI testing, requests for your API testing, and Storybook for your component testing. Other tools like Cypress and Playwright can actually do all of these kinds of things.

Speaker 2: They tend to be more all-in-one versus piecemeal. And so we're going to look at each of these tools and see how we can do these different kinds of testing with them. There are also codeless tools available. Typically, codeless tools are proprietary, they're built by a vendor, and you need to pay some sort of license. Maybe there's a free tier or not. I'm gonna focus more on these open source tools. Why? Because we're in the Python community and that's what we do. So, how do we do component testing? Well, like I said before, component testing would be instead of having these all these giant pages of stuff on them You can build libraries of UI units. Maybe there's a calendar widget, maybe there's a login form widget, you know, all those kinds of things.

Speaker 2: The nice thing about creating a UI library is that you can have a consistent look and feel throughout your application. Now, not every single application has one of these, but if you do, it's very wise to do kind of do component testing because you can test just this one widget in isolation One of the best tools to do this today is uh Storybook. Have people heard of Storybook? Maybe some? Yeah. It's not the only one that's out there, but it's It's a pretty nice tool for building your components and then checking them right away. It is manual checking. You kind of see it render and look at it, well, it looks pretty, or oh no, I didn't mean for my button to be red. Those kinds of things But I found it to be helpful. And also I used to work for a company called Apple Tools. They had this cool thing where they could

Speaker 2: plug into your storybook library and do visual testing on it automatically. which was really, really nice. So that you could make sure with that, like if you made any tweaks to your components, that things would still be appearing the same as you would expect. So what about API testing? Things get a little more interesting here. API testing is very, very important because the APIs are kind of like the underpinnings of the application. That's kind of like the entry point, the exit point from the back end. Large applications can have hosts of APIs. It could be that you have a more uh traditional architecture where you've got like your your view layer and then you've got your backend layer and then your persistence layer. Or it could be that you're dealing with a a microservice mesh going on where there's just web services just out in an ecosystem and they're all kind of spewing stuff to each other

Speaker 2: It's very important to test APIs because like we said, they are um that that's where you can hit a lot of the um Permutation, different kinds of permutations of data, right? Uh API testing is a lot faster than UI testing because calling a um An API endpoint usually takes a few microseconds if you're on a good network connection, whereas trying to navigate through a web app can take multiple seconds. So we have orders of magnitude of different speed and execution. So here would be an example of a basic API test for a successful login for the Bulldoggy Reminders app. And here I'm writing it using playwright. Historically, I've typically just used um

Speaker 2: uh py test plus requests For doing straight API testing, and you can do that. That's totally valid. But in a project where I have to do both API and UI testing together, it's nicer to have fewer tools in your to to manage in your toolbox. And so Playwright can do API testing as well. And I think underneath it just uses requests anyway. So but here's what a basic API test looks like. So I have my PyTest function. Again, PyTest, you write your test as functions, not as classes. Test successful API login. I'm passing in certain values, and these come in as what we would call fixtures Fixtures are basically functions that do setup stuff and then return a value that gets injected. And so I have separately written fixtures for Bulldoggy API, which is an API request context.

Speaker 2: That's going to be what I used to call my API. I have a user model, that's the user who's logged in, and I have a base URL, which is passed in, and that's going to be where I point my API calls. It's good to not hard code your base API in your tests. So here we go, uh making the request bulldoggyapi. post for the login resource path. My form that I'm passing in for my data username comes from the user, password also comes from the user. So you kind of have your re your arrange and act in this one step together, and that's okay. You post it out, you get a response object. From this response object, anytime we do API testing, we want to make sure of certain things about this response. It's one thing to just call and be like, yep, didn't get an exception.

Speaker 2: We didn't blow up. We're good We want to go a little bit deeper. So right off the bat, we want to make sure our response is okay, meaning it got a 200-ish value, right? Because this should be a successful login, not a failed login We want to check our response URL that it sent back, that it's giving me the base URL plus the reminders page, because that's where it should be redirected after login is successful. Also when I'm doing login, I should be getting some sort of cookie back for authentication so that I can save the state of like yes I am logged in And so what we can do is we can go to the API context, fetch that storage state, and get the first cookie out of it. And we can check things about that cookie. It should have a particular name. to be what we expect. In this case, this application uses the cookie named reminders underscore session.

Speaker 2: And we're going to assert that the value is not null. Now I could go a little bit deeper and be like, well what is the actual value of this cookie Right? But I don't I don't need to do that. Why don't I need to do that? What did we test before? Remember that unit test we had? Did the serialization and deserialization of the authentication? Well that authentication token is what got passed in here in this cookie. So I've already verified that the value of that authentication token is going to be correct. Here, I just need to make sure that it got there. Does that make sense? We can use our different types of tests together to make sure that we have full coverage without necessarily needing redundant coverage. Cool stuff, right?

Speaker 2: Yeah, cool stuff. Awesome. So does API testing seem reasonable? Makes sense? We've probably done some of this before. API testing doesn't need to be too over the top. It's it's a fairly doable thing. And in fact, I mean there are there are certain companies where they have entire organizations dedicated to just testing APIs. Go figure. Now there is a another flavor of API testing as well, and that's called contract testing. So it's not a new thing, but it's not something that's necessarily practiced as much As we said before, the API testing, it can be fast, it can be more stable than a UI test, but you still have the issue of the network. Right? Because anything that goes over a network could get lost or it could get interrupted or it could slow down.

Speaker 2: And that that can introduce um longer times for your feedback cycles, as well as just random failures along the way as you're trying to do your testing. And so one way you can try to mitigate that is with a strategy called contract testing. The way contract testing works, it's almost like take an API test and make it a unit test. What? How? The thought is that with a contract Whenever you have an API call, there's a consumer and a provider. The provider is the one hosting the API, the server, the service, whatever you want to call it, and the consumer is the one calling that. And so with this, there is this under there's this implicit or understood contract between provider and consumer. Consumer has to call the API in a very particular way.

Speaker 2: It's got to have the right headers, it's got to have the right format, all the right stuff. And the provider, if if the consumer has provided that, provides a response again in a very particular format with certain headers, there's a certain structure to the body or whatnot. And so that forms the contract in this request-response dance. So what contract testing does is to say, hey, well, if we can verify that both sides, consumer and provider, are meeting The expectations of that contract, as long as we can make sure that they can shake hands like that, then maybe that's good enough for our API testing. And that's in a nutshell what contract testing is. There are frameworks out there like PACT. That's probably the most most popular one.

Speaker 2: And you can do things like you can test the consumer in isolation to meet the contract. You can test the provider in isolation to meet the contract. You can generate these contracts and send them out both ways. And then you can even rip out the contract and actually run it like real. So um I don't want to go too too much more detail about that, but if that seems like something that could be really valuable for you, definitely check it out. I can recommend PACT. A contract testing is also typically more for like if you have a a microservice mesh versus just like one little web service you're you're poking out there. Because in a giant microservice architecture, you've got many little tiny services around all interacting with each other. And so it can be very hard to do full on API testing for every single one

Speaker 2: when really you just want to make sure that they can all just kind of talk to each other. Alright, so that's the EPI testing. What about UI testing? And specifically, what about Selenium? How many people have heard of Selenium? Yay! How many people have used Selenium? I'm so sorry. No, it's okay. It's okay. Bad joke. I've used Selenium too. I've used Selenium for way too many years. Ouch. Selenium is a good tool. It's a good tool. So, what is Selenium Web Driver? Selenium Web Driver is the classic and still today the most popular browser automation tool. The way it works is it manipulates the browser via the Web Driver Protocol, which is a W3C

Speaker 2: standard, meaning all browsers are supposed to adopt and conform to the Web Driver Protocol standards. So you should be able to use this protocol to interact with any browser. Selenium uses open is it is an open source project. So you can look at his code, yay, open source, right? Yay! Yes, yes, clap. This is the moment to clap. Yes, open source, yay! Good. But not only is Selenium open source in that you can go read the code and contribute to the code and become a committer, but it is also open standards and open governance. And what does that mean? Selenium is open standards in that the web driver protocol is W3C standard. This was a a standard that the team worked very hard to get in place, and it is something that

Speaker 2: will r be sticky for a very long time in our in our tech scene, right? Because if all browsers have agreed to adopt this thing according to the W3C, that's gonna be something that sticks around for a while. So you can trust that web driver is going to work. Not only open standards, but it's also open governance. Um there is no single company that controls the Selenium project There are a group of companies that come around and maintain, support, put money towards companies like Sauce Labs, Lambda Test, Browser Stack. My former employer, Apple Tools, was a major sponsor of Selenium. But that they they have an open governance in that you can you can go to their their leadership team meeting notes and see what's going on, right? It's a transparent open door.

Speaker 2: They're welcoming more people in. Selenium project isn't just gonna like die one day because some company said this isn't making enough money, right? There's that kind of protection to it. It's very community oriented and very much in the open source community. Selenium WebDriver also works with, like I said, all the big browsers: Google Chrome, Mozilla Firefox, Microsoft Edge, Apple Safari , Internet Explorer. I'm so sorry if you still have to use Internet Explorer. Oh gosh. I have pain there. But also Selenium not only supports many browsers, but also supports many languages. There are bindings for the WebDriver protocol for Python, Java, JavaScript, C sharp, and Ruby. So we can use Selenium and Python. Yay, it's awesome.

Speaker 2: Excuse me. So the way that the web driver works is that you have your automated process. And for most of us, that's going to be a a test case. like a pie test test case or something, but it doesn't necessarily need to be a test because Selenium is just a browser automation tool. But let's for our context say we have a test case Test my Bulldoggy app. Remember we what goes in the PyTest thing? The Selenium calls go in the PyTest thing. Boom. And so the bindings in Python are going to be making calls, making Selenium web driver calls. That's a pip install selenium, and then you make the calls there. Then what that does is it doesn't directly go to your browser, it goes to a proxy server on your machine called the WebDriver Executable. So every browser needs its own special executable.

Speaker 2: So if you're using Google Chrome, you're going to have to have Chrome Driver installed on that system path. If you're using Firefox, you have to have the Gecko driver installed on your path. And so from the automated process, from your PyTest test case, when you call Selenium and say like you know, browser init or whatever. Um it goes to look on the system path to be like, okay, let me wake up the Chrome driver or the gecko driver or whatever driver you're using. And it's going to the script is going to communicate with that as a proxy. Then the Web Driver executable communicates with a real live browser. So the Web Driver executable then wakes up Chrome and then starts sending those Web Driver protocol commands back and forth. So when you click in your script, it goes to the proxy and then hits the browser and you see it go click. Magic, right? I remember first time I showed one of the developers on a team

Speaker 2: previous team I worked on how Selenium worked and it did it. She's like, oh my gosh, it's just going on its own. I don't need to touch the mouse. This is magic. I'm like, yep, automation, it's great. So here's what setup looks like. Again, within a PyTest context, I'm going what I recommend people do is create a fixture, which is just a setup and cleanup function. I'm gonna usually I call mine something like browser or driver. And inside, to set it up, I say selenium. web driver. whatever the browser I want. Usually I do Chrome because it has the highest market share and it's the fastest. And this will, like I said, go to the proxy and then wake up an instance of the Chrome browser. You can add an implicit weight here if you want to help slow things or to help you with the erase conditions.

Speaker 2: Normally I recommend explicit weights, as we'll see in a little bit. I believe I have some of that on the next slide. You can also set other options like if you want this to be excuse me, headless if you need particular window size, full max, whatever, you can set Chrome options. But ultimately we are going to return and by return I mean yield. the instance of that browser so that when I call this fixture in my PyTest test function, it receives the instance of that web driver object. And the reason why we yield is that means this becomes a generator, it's not just a regular function. So that the PyTest will call this fixture before the test runs to get up to here in yield. And then once the test function is complete, it will come back into this fixture and do everything below the yield as a cleanup

Speaker 2: operation. Pretty cool, right? And so we always want to quit our browser at the end of testing. You know, the beginning of the test is open the browser, the end of the test is close the browser. If you don't quit your browser, you're gonna have zombies, and then your Jenkins machine is gonna topple over because it's using too many resources. I have learned this the hard way. Don't make that mistake. Yes, dude in the front is chuckling. Yes, somebody gets it. Alright, so now that we have the um the browser setup and cleanup handled, I can write UI-based test cases like this. So I create my test function, test successful login, I pop in that fixture, that's the browser. So before the test, it's going to set up my Chrome driver. That's wonderful. And that way it's that fixture becomes reusable because I can reuse that for any test.

Speaker 2: And here we're going to do a we're going to do a UI-based login scenario. So here again we're going to see the arrange act assert pattern, but we're just going to see it phrased as given when then. Given the login page is displayed, browser. get pop in your URL. Again, you probably don't want to hard code this. You probably want to pass that as an input, but for the sake of the example here to keep it simple, this is what we're doing. When the user logs into the app with valid credentials. Now we're going to see the very typical kind of syntax we see with Selenium interactions. Browser. findElement with some sort of locator. Here I'm using a CSS selector. You can use XPath, ID, anything. But browser, you need to locate the element on the page, find that element, and then

Speaker 2: send keys for your username Find element, password, send keys, your password. Find element, expath, the login button, click. This is how we do interactions. Finally, then the reminders page is displayed. We have arranged the page, set it up, we have acted by doing a login, and now we need to assert that. The reminders page actually did get loaded, meaning we did get to log in successfully. And so here I'm I've created a web driver wait object. with a default timeout of 10 seconds. This means that it will, the calls I'm going to say here with wait until will actively keep checking for an element to appear. So that if it didn't appear right away, because we know what

Speaker 2: web browsers can take a few seconds, it's not going to just carp out and die. That it'll be like, okay, well, let me keep waiting until this thing becomes true. And if it doesn't become true after the 10 second timeout, then and only then will we throw an exception and say we didn't get there. This kind of synchronization with Selenium, you need to handle yourself. Right, so here I'm making sure that the logo appeared and the title appeared. And here I'm even getting the a particular element from that page, the title, and getting the text and making sure the text is bulldoggy. So does this test scenario make sense? I mean it's a little bit like the the code is a little bit janky here because it's long, right? I have long lines, but still it's a range act assert If we wanted to make this better, we would probably use something like a page object model or a screenplay pattern.

Speaker 2: I don't want to go too too much in depth with that here because we have so much to cover to breadth. But if you want to learn more about how can you make this better, come chat with me afterwards. So Selenium is great, it's awesome, it's works, it's been around forever, it's got all the open stuff that we love. What are the problems and shortcomings of Selenium? Well, as many of you probably know, uh there is no automatic weighting, meaning if your tests are susceptible to flakiness You either have to explicitly handle waiting everywhere or do an implicit wait, which is not a recommended practice, or you just YOLO and hope that your app doesn't time out. And unfortunately, that's what most teams do that I've seen. That's why it is recommended to use a layer on top of Selenium, like page objects or screenplay, to help mitigate that. Another issue that people have had is slow performance.

Speaker 2: The web driver protocol is not the most expedient. But not only that, but every test you need to open up a browser and then you need to close a browser, which means not only is it the Chrome driver, but it's the actual Chrome instance or whatever browser instance instances you're using. And that adds a time hit to every single test, right? That gets back to what we're saying with UI tests are slow. That was one of the reasons historically why UI tests were slow, because of this constant setup tear down setup teardown. And also uh to keep in mind, Selenium is just a browser automation engine. It is not a full test framework. You need to build your own framework around Selenium. And Selenium does not support API or component testing. Those other two vital parts of our testing strategy.

Speaker 2: Alright, so if Selenium has some pain points, well what are other other tools out there? What about Cyprus? Who's heard of Cyprus? Who's used Cyprus? Not as many, okay. You know Cyprus had their first conference this past Tuesday. Virtual, very cool. Um Cyprus is fairly new, but I guess it's not so new anymore. I first found out about Cyprus in early 2018, and at that point it'd been around for about a year or so, maybe two years. Cyprus was the darling test framework for front-end web developers for a long time. Why? Because it brought a full rich modern web testing framework Not just a browser engine, but a full framework, with very rich developer experience.

Speaker 2: And I'm sure those of you who have used Cypress know exactly what I'm talking about. It's smooth, it's pretty, it's nice. The way that Cypress works is it manipulates the browser via in-browser JavaScript. So like I was saying before, development testing, two sides of the same coin, Cyprus fully embraced this. Right? The web application process that's running, your test process is running in that same loop with it. So it's like right there next to each other. Cyprus is an open source project, but it is maintained by the Cyprus company, Cyprus. io. So it is basically company-backed. The core framework is free. There is a paid like a web platform dashboard thing that you can do with it as well.

Speaker 2: Cypress works with Chrome, Firefox, Edge, and Electron. It also works with WebKit, not full Safari, but it does work with WebKit, which is the project underneath Safari. Does anybody know how it has its WebKit support? It took it from Playwright. True story. Open source for the win. Right. It was it was pretty funny when that happened. I was like, oh, I see what you did there, right. Um Cypress can also test APIs and components. In fact now when you you open up your Cypress uh browser um window the first time it'll ask you do you want to do end-to-end testing or component testing? However, it does have bindings for JavaScript only, right? Um

Speaker 2: that that is inherent in the design of this. If it's going to be operating inside the browser, sending the JavaScript, you're you're going to be JavaScript only. So here's what a Cypress test looks like, and I'm sorry that I'm showing JavaScript at a Python-based conference. Please forgive me Um I can get away with it because it's Django, right? It's web based. The Cypress tests, they follow the describe it should pattern that is very common in the JavaScript world. I don't know who came up with this. It's kind of weird to me, but whatever. I'll go with it. So what gets me is that some people in the JavaScript world say this is BDD and I'm like, no, it's not. Not the BDD I know. But anyway, we'll give it to them. It's okay. I'm not I'm not trying to try to throw shade or anything. I'm just like, it's interesting, it's different. But you basically you would set up your test, you

Speaker 2: say, describe this kind of like a block of tests. Bulldoggy, the reminder is that boom. What should it do? It should log in with proper credentials. And so here we have that same arrange act assert given when then tests for login that we had before. Code's a little bit different. CY dot visit, cy get type, cy contains click. Right, these but again these are getting the these are locators, getting elements and then performing interactions on them. So if you're if you know Selenium, you can probably get around with Cyprus. Um I will say the there are some aspects of the syntax that I'm still kind of scratching my head about. I have to go look it up in manuals when I use it, but I haven't gotten stuck yet, if that makes sense. But then we also have built-in assertions with Cypress. So cy. get, a particular thing, should have text, and these are chai assertions.

Speaker 2: Bulldoggy. Cypress get this logo, it should exist. Uh Cyprus contains the logout button. That logout button should exist. What you might notice between the Selenium code and this code is this one's a bit more concise. Right? And that's by design. Cyprus has a very, very nice syntax. Also, there are no explicit weights here because Cyprus handles waiting implicitly. And it does it right, which is very, very nice because that's one less thing that you have to worry about when you're writing your tests. It helps you focus more on the testing and less on the automation mechanics. And when you go to run your tests, if you're running it in UI mode, Cypress will pop up this Electron app window and it will run the tests, as you can see in the sidebar here, step by step, and you will see the test running in real time on the right. Again, that nice

Speaker 2: like eye candy, that that developer and tester experience is really something that Cypress is focused on. And really crushed it. And that's why Cyprus started taking such a large market share within the uh web UI testing space. I mean it I I've seen the data to back that up. It's pretty incredible how quickly Cyprus rose. But Cypress, as we know, is not perfect. First of all, it is JavaScript only, so it is not ideal for Django knots like us. And I mean you could say, well, uh JavaScript is the the language of the modern web. And I'm like, well Not everybody needs to be shoehorned into JavaScript to be able to build cool things, right? You know, like I just because I know Python or I know Go or I know

Speaker 2: I don't know, whatever, OCamel, doesn't mean that I shouldn't have access to be able to build cool web apps, right? Do I r are you really gonna have some data scientists trying to share their their stuff? with their their peers and tell them, oh well if you want to make a web app to share that stuff, you have to learn React. It's like, well, I just spent all this time learning data science in Python. And I have a Jupyter notebook and there's this cool Django thing, like why do I need to switch thing , right? So there's there's that tension there. Um Also, like I said before, because it is so but because its design is to be in the browser, it is Cypress is essentially trapped in the browser. Right. Um for example, you can only run one tab at a time with Cypress. That is a limitation of that tool. Well what if your web page opens up another tab?

Speaker 2: Well you have to do some hacky things to make it work. It's like, uh that's not So there are those kind of those rough points with Cyprus as well. Never fear, there's one more framework we can look at, and that is playwright. Yes, playwright. Who loves playwright? I love play yes, yes, playwright is awesome. I love playwright. I'll be biased. Playwright is a relatively new test framework from Microsoft. It comes from the same division that get brought you Visual Studio Code and TypeScript. Like Cypress, it is another modern web test framework that gives all the bells and whistles and all the wonderful user experience. The way it works though is it's not doing WebDriver protocol or in-browser JavaScript. It's going in the back door with debug protocols.

Speaker 2: And that makes it super duper fast. If you benchmark playwright against any other browser automation tool out there, playwright just blows it away. Like like no competition. It is fast. It is like Jimmy Johns. It is freaky fast. Like I said, this is a Microsoft project. It is a spiritual descendant of Puppeteer, if you've heard or used of Puppeteer. Uh but whereas Puppeteer was more just browser automation, Playwright is truly focused on testing and test automation. It works with browser projects for Chromium, Firefox, and WebKit, and I'll talk a little bit more about that in a moment. Like Cypress, it can also test APIs and components. But the nice thing, unlike Cypress and like Selenium, it has bindings in multiple languages.

Speaker 2: So while the primary binding is TypeScript and JavaScript, you can use Playwright in Python as well as in Java and C. And so I've done a lot with Playwright and Python and I love it. I think it's great. So how does the browser work here? Um not only is is is Playwright super fast in that it uses debug protocols, but it really optimizes how it uses browsers and how it does setup and cleanup. So by default, Playwright does not use your full browser. It doesn't use Google Chrome or Apple Safari or Mozilla Firefox. It uses the browser projects underneath So sh it targets Chromium. It targets WebKit. And that gets you like 99 -98% of the way there for what you need. But it also means that when you go to run your browsers, it has a smaller footprint on your system.

Speaker 2: Playwright also will manage and install these browser projects for you so you don't have to mess around with these web driver executables or anything. Another really cool thing that Playwright does is that it doesn't start and stop a browser instance for every single test case Instead, it uses one browser instance through the entirety of your test run, across all tests, and instead it peels what it calls a browser context out of that instance. So one browser instance, but for the entirety, but one browser context per test. What a browser context does is basically like an incognito session for your test And so it will have its own session storage and all that kind of stuff, but it won't have access to any of the others. So it's almost like mini containers or something going on inside your browser.

Speaker 2: And contexts are very fast to create and destroy. So you don't have this multi-second hit every time you start or stop a test. Within each context, you can also have multiple pages. And the page object in Playwright is the object through which you interact with the page. And so you could most times you only ever need one browser context and one page for a test. But if you have a kind of test that where you click a button and it opens a page in a new tab or new window, Playwright can manage that as well. And you can have both of those pages happening simultaneously. So, this is what a playwright test looks like. Uh again, the code is going to be very similar. This is the same login test we've seen before. This time I don't need to set up browser, user, or anything like that. I mean a user I could, but like I don't have to s I don't have to make my own browser fixture like I did with Selenium.

Speaker 2: Here, uh there is a fixture called page and this is given to you by the uh playwright uh plugin for PyTest. So it's just you know pip install that, but then you get this fixture for free and all you have to do is declare page and you get it. So the calls are going to be very similar. Again, we have locator, fill, expect, all your assertions. Again, all the waiting is built in. So pain points with playwright. One limitation could be that you know you're not testing your full browsers, you are only testing browser engines. Maybe that's an issue for compliance if you've got a company with very strict rules. There might even be small test gaps in there, like there's some session thing that goes on. I don't know. I haven't hit anything like that.

Speaker 2: And if it would be very, very fringe, maybe that's acceptable risk to take. Another shortcoming is that play rate is the newest one. So even though it is gaining in popularity and market share rapidly, it still doesn't have that footprint that Selenium and Cypress have. And features historically like they were evolving pretty rapidly. We're seeing more stabilization now So if you want a quick comparison of these tools, here, capture a quick snapshot of this table. I've outlined the differences to compare and contrast. This is one of the money shots. Take a moment here. I know we're we're kind of out of time, so I'm going to wrap up rather quickly here. Three, two, one. Cool. So, uh

Speaker 2: what about performance and load testing? Yes, they are important. Uh functional testing verifies that a behavior works correctly according to assertions. Performance testing verifies a behavior works well according to metrics. And so it's a different style of testing. And load testing. It often gets lumped in with performance testing, but it is a distinct thing. The idea of load testing is how do the functionalities and performance behave when we crank up the scale of load on the system. Some tools you can use for these. Performance testing. You can look at a tool like Lighthouse. I mean you can even do very bare bones basic things like just open up your Chrome Inspect tab and look at your networks, go. Right. Uh for load testing, there's a very cool Python-based tool called Locust.

Speaker 2: Has anybody used that? Yeah. Now you can just crank things up. And it's I like that one because it keeps things rather simple in terms of how you crank up that load. How should we run these tests? Like we said before, build fast feedback loops. You can build a whole pipeline that goes through building your code, doing static analysis, unit tests, component tests. Deploy to a test environment, API tests, and UI tests. I would recommend running your performance and load testing separately, but ultimately remember what you're trying to do is get that fast feedback. If you are not running your tests continuously or in CI or at some regular cadence, your tests are worthless straight up. Yay! Cool. One tool I really like to use to that to keep things um

Speaker 2: I don't want to use the word simple, but keep things a little more um basic, manageable for small projects, GitHub Actions. You can build all of this into GitHub Actions. For example, here is a GitHub action I've written. What it does is it uses a Ubuntu branch. Anytime I make a change to my main branch, I check out Python or I set up Python. I install my my bulldoggy dependencies. I install the playwright dependencies. If you're doing this for Reels, you probably want to pre-build an image, but just roll with this here. You start the Bulldoggy app, you wait for that to get up, and then you execute your Python tests. Like boom, like amazing. Right? And this could include, you know, playwright tests as well as API tests as well as all that. So, how do you learn more? Test Automation University. Has anybody taken courses on TAU? Anybody

Speaker 2: know about this Oh my gosh, nobody knows about TAU? This is the best place to learn about testing and automation online. Free courses, video based with quizzes. Anywhere from like half an hour to a couple hours long. I've taught courses on TAU. Everything is free. It's and they have courses on everything. I'm proud to say I've developed many of the courses in the Python track. I also have TAU t-shirts to give away if you want one. Yes, come see me afterwards. I love TAU. When I was at Apple Tools, I was director of TAU. It's a phenomenal program. Definitely check it out. So, I know we're over time. Let's wrap up. As I've said many times, development and testing are two sides of the same coin. It hits me especially hard because While many people think of me as the testing guy, I've actually never considered myself to be a tester.

Speaker 2: I've always considered myself, first and foremost, to be a software engineer. who is building problem, who 's building solutions to different problems. And so my specialty has been building solutions to testing problems. And that's why I'm in this space and that's why I care about what I do. But ultimately, if there's anything you take away today is that know that Django Nuts, you can do testing. And not only can you do testing, you should do testing. Because if we are to build high-quality web apps, we need to make sure that they're working well. So thank you very, very much again for your time here today. It's been a truly an honor to deliver this keynote here in Durham at DjangoCon. Second money shot slide. If you want to see everything we covered today in a nutshell, bam, there you go. Again, my name is Andy Knight. I'm the automation panda.

Speaker 2: You can follow me on Twitter at AutomationPanda. I blog at automationpanda. com. If you like the Bulldoggy app, hit me up on GitHub with some stars. Thank you so, so much. It's been wonderful.