BDD To The Bone: Acceptance Testing with Behave and Selenium with Pat Viafore

Thank you, Drew. Uh I'd like to start off by talking about requirements. Spoo. We don't want to talk about requirements at a programming conference. You know, us programmers, we're little black boxes and our output. is beautiful code and systems. But our input are requirements. Every one of you has requirements when you work on code. And if you think you don't, well you do, it's just they're not written down and they're in everyone's head and they're all different. But we all have requirements. That's not to say that requirements don't have their own problems. The first problem I want to talk about with requirements is communication. Tell me if this sounds familiar. Your some business person talks to the customer and the customer says, uh, something's wrong. I need I need help on something. Can you do this for me? And your business people go, Yeah, you know, we have a team. We're listening to your your

requests. And then they wait a day and they go, hey, should we have written anything down what the customer said? Yeah, probably. Oh, you know, just write a Jira. It's fine And then the manager gets it, says, oh, what's this JIRA? Oh, the customer needs this requirement. Oh, great, great, great. Is it important? Yeah, it's super important. Can I put it in front of everything else? No, don't do that. Put it at the bottom of your backlog. You'll get to it eventually Then a second customer comes around and says, oh, I also have this requirement. And the manager goes, oh man, two customers? I'm gonna put in the middle of the backlog now. And sooner or later you get to work on it. And the developers look at this little ticket and it says, implement the thing. I don't know what that means. When you play requirements, it's like a game of telephone, that game we play as children where we sitting around in a circle, whisper a phrase at each other and see how it gets distorted around the circle and we all laugh at the end.

Well, in a business setting, no one's laughing, but it still happens. Think how many people touch the requirements as they go from customer or end user to the developer. There's a problem with communication. I also can't stand non-testable requirements. Some people call these non-functional requirements. I like calling them operational requirements. Things like your site should be fast. Your site should scale on Black Friday. That's not really testable. Like, yeah, okay. How fast do you want it? How many users are we talking about? It it's not good guidance to the people who have to work on it. And if it's weeks or months, you know, since that requirement has been generated. Some of that context gets lost. Lastly is traceability. You see this more in a formal requirements setting where someone will say, hey, we have this requirement, the site should be able to do X.

And someone says, oh yeah, yeah, the site does X. Great. Where's the test that shows me that? Uh I don't know if we even have a test for that. And it starts becoming like, how can you even say that your system does it or doesn't do it? If you don't know where the test is, where the code is, that's what I mean by traceability. So, how do we start addressing this? Well, I'd like to introduce you to something called Gherkin. Has anyone heard of Gherkin before? Hands up. A good bit of you, great. So what Gherkin is, is it is a language we use to express our requirements in. This is what Gherkin looks like. This is for a hypothetical link shortener, something like bit. ly. I'm gonna call it Patly because I love attaching my name to dumb little projects. Uh and what happens here is I'm describing what I want the system to do So I have a scenario, a login

user can create a shortened link. Given a logged in user, when the user creates a shortened link and the user opens the shortened link, then that link redirects to the original link. This is Gherkin, and it really all boils down to this kind of format. So a feature is some grouping of use cases. A scenario is a specific use case. Given some precondition, when an action occurs, then I expect some post-condition to be satisfied. If you've ever done unit testing and you're familiar with the AAA pattern, arrange act assert, this looks very familiar So matter of fact, this looks very familiar to anyone who's ever written a test before. Who has written a test before? Thank you for everyone who puts your hands up. What if I could make my requirements

more testable? What if I could make my requirements tests? And this breeds the idea executable specifications. Now when I say executable, I don't mean we're killing our specifications as much as we may want to, but I want to mean specifications, requirements that I can run. So I want to be able to take this, this natural language uh requirement language called Garkin, and I want to be able to run it. And we're at a Python conference, a Django conference, maybe there's a Python tool that does this So what are my options? Well, I asked a British super spy from cinema if there were any options. Do you know what he said? Give you a clue. This is who I asked. He said, oh, just use behave. And he used that exact intonation and deflection because he knew it was a professional text setting with no other meaning behind it.

But yeah, Behave is a Python library that can map Gherkin to runnable code. And that's pretty nice. So let's take a look at what that might look like. So this is behave code. You'll notice I have some functions here, login or create short and link. I've decorated these functions with behave decorators. Given a logged in user. When Behave sees the phrase a logged in user and the Gherkin requirements, it will run this function. So now I'm starting to be able to map functions to certain clauses and it's quite expressive. I can put regular expression matching in here. I can parameterize these just like you can on like PyTest. I can make tabular examples just like uh PyTest parameterization as well. And what's really nice, so the what how Behave works is we're mapping a function to a clause,

a given, a when, or a then. Behave calls them steps. And there's a context that gets passed from step to step. You'll notice in my when clause, when the user creates a short link, I'm actually saving some things off in the context. Cot context. old link equals some old link. com, and I'm saving the value of a new link. In later steps, say when I want to check that some post condition has happened. Here's my then. Then that link redirects to an original link. Well, I'm asserting about things on the context. I'm using just plain old naked asserts, just like PyTest does. I don't need any special functions. If you write But just straight up asserts, behavioral work with it. So what happens when I run this? This is an example of some output. So the very first line will

say I am creating a test database. I'm going to ignore that for now. That's something Django specific I'm doing and I'll talk about that later. But I have a feature, and this is like my test suite. My my feature. And for each scenario, it will show me all the steps that it ran and whether they passed or failed. Green means passed. So um for those who uh may have trouble distinguishing green reds Um everything where it starts given when and and then is green at the moment, which indicates the past. And there's a little output at the bottom to tell you as well. If something fails. This line right here is red and that will say, hey, I have a failure. This assertion failed. The current URL was not set to the old link. This was that assert that I showed you just earlier. So I now have a way of saying pass or fail based on natural language. Pretty cool. I didn't even have to introduce an LLM to get that.

So I mentioned some problems I have with requirements, and I want to revisit them now that we kind of have an idea of what this sort of technology can give us. Communication. If I were to write all my requirements as Gherkin early on, I start building a shared vocabulary. All those little steps. Those are things that I can hand off to my business analysts, my end users, my support team. They can write their own tests without writing code. There's been a lot of other sort of technologies that have come along that have kind of promised that. Anyone here has used a robot framework? One person. So I've I've used robot. Robot does support Gherkin, which I love, but Robot had a system where you could try to do pro uh

write tests without programming, but in a Python file. And when I tried to uh I saw that used at one of an organization I worked at. And what ended up happening is they started mixing Python with natural language with Python with natural language, and it was so confusing. We like separation of concerns. Let's put the test cases, the test steps into something that can't be polluted by code. I know we all can read code, but let's face it, reading code can be hard sometimes. It's way easier to read natural language. Use this shared vocabulary to write better tests and to enable other people who might not want to touch Python to write better tests as well. All they have to do is just mix and match whatever steps that you've provided for them. You also get reporting uh capabilities.

So this behave outputs JUnit is one of its options. I just grab JUnit to HTML off PyPI and I create an HTML report If I had to hand this to a PM and say, hey, here's all the things our systems do, they love that. Way better than handing them a set of PyTests. tests. Like this is about as clean as I can get a pie test test for one of those test cases. I still don't want to hand 1,000 of these to a business person saying, hey, yeah, this is what our system does. And let's talk a little bit closer to home. Think about the junior engineers that you're onboarding or a new hire who's walking into a large code base Maybe that's you. Do you want to be going in and say, hey, what does our system do and try to understand the communal knowledge from every single person on your team and still get an incomplete picture? Or do you want a nice set of natural language requirements that aren't just written after the fact?

They're written in a clear, explainable language. I know which one I'd like. This also starts driving the notion of acceptance testing. So acceptance testing is a form of typically end-to-end testing. It doesn't always have to be, but there's a there's a crucial difference in the testing that we are more familiar with. So developer testing, unit testing, component testing, integration testing, stress testing, those are all verification. Are we building the thing right? If I implement something, I want to make sure you know it doesn't crash, doesn't throw an exception, doesn't leak resources, does the right calculations. That's why I write unit tests. That's why I write component tests to make sure that, yes. my code is doing what I as a programmer intended. Acceptance testing is a different kind of beast.

It's validation. Are we building the right thing? This is meant to drive conversations with end users. And again, that's more than just your customers. That's your support team, ops, anyone who might have a vested interest in your application or system. Are we building it right? Just a few weeks ago, I was working on a feature. Um, and we used a word hybrid to denote a certain workflow And the team was saying, yep, we need you to add this workflow when we're in hybrid mode. We went and did that and said, it works, it's fine, gave it back to them, and they said, nope, still not working. Something's broken. Okay, is it environmental? Is it some sort of like one-off issue? We spent another week just trying to figure out what was wrong before we finally traced our way all the way back up to the person with the original requirement.

When they said the word hybrid, they meant something different. Then what our team meant. We were building the wrong thing and we were not happy about it. We lucked out 95% of what we did was still applicable. But there's many other times in my career where we've said, oh. We didn't understand the requirements well enough. We built the wrong thing. And it doesn't matter how much coverage you have. It doesn't matter how perfect your tests are. It doesn't matter how perfect your code is. If you build the wrong thing, none of that matters. A quick note, who's seen the testing triangle before? Yeah, but you see this in like every book that ever introduces testing, including mine. We say unit tests at the bottom. Yeah, you should write lots of unit tests. Maybe a few integration tests, but UI tests, only write a few of those. And most people take this as a face value, but there's a dimension of this testing triangle that I think is different.

And I've seen this triangle in many different forms. I've even seen it inverted with the point at the bottom, and I don't know what was going on there. So To combat all these different triangles, I'm going to invent a new one. Because we know how standards go, right? Insert relevant XKCD. This is what I think every testing triangle is trying to say. You have tests that are high value to cost or have a high value to cost ratio, and you have tests that have a low value to cost ratio. You want to do the things that have a high value to cost. You want to do those fast and you want to do them often The things that aren't so valuable compared to their cost, don't do them as often. There can be unit tests at the top, there can be unit tests at the bottom. It depends all about value to cost ratio. I think acceptance tests can be anywhere in this triangle as well. If you have acceptance tests, even if it's an end-to-end test, most people will say, oh, that's so slow, don't run those often.

If their value still outweighs their cost, run them often because you're going to get more value than the cost you're doing on there. And I think using Gherkin and driving these conversations first and then using Behave to test these, is a great way of lowering the cost of acceptance tests. They have high value. They're testing things of how a customer would see it. Let's try to drive those costs down so that we can run them more often. So that's communicate. Um I mentioned another problem I had was non-testable requirements. So you know how it is, Monday morning comes in, your PM comes to you, like, hey, I have a new requirement for you. Creating a link should be fast. And what's the first question you ask? How fast, exactly. Let's start putting numbers behind it. Um there's a uh a talk by a few Google engineers, Titus Winners and Hiram

Wright, uh where they talk about the Beyoncé rule. If you like it so much, you should put a test on it. Not mine, it was theirs. So use Gherkin to put numbers to back up things. I want a link to be displayed within 200 milliseconds. You can start tracking that over time, but embed that in your requirements. Don't just say it should be fast. Oh, we should be able to scale to this many users. How many users? Put it in a test. You can use, I mean it's Python. We know Python can do so much for us, and we have so many tools, so many packages on PyPI Use your requirements to drive testability because your requirements are tests. On that note, traceability. I said, hey, I have a requirement. Where's the test that goes with it? My tests are my requirements

Sorry, I forgot advanced. But my tests are my requirements. I know I'll show the output again later. When I run my uh behave test, it tells me what line of code is getting executed for each test. We have executable specifications, specifications that can be executed or ran. And that leads me to BDD. BDD, behavior-driven development. Once you start getting into this kind of mindset, you start practicing BDD. And most people think I'm actually going to take a tangent to test-driven development. TDD, most people think the most important word of test-driven development is test. The most important word is driven. The tests drive your development. The tests aren't the output. The tests are d serving as a design methodology. TDD is a design methodology.

not a testing methodology. BDD is very much the same. You're talking about your requirements with your end users up front and you are making sure That you have the behaviors that you want to test covered and you're letting that drive your development. You're making sure that you can build the right thing, or at least mitigating risk that you are going to build the wrong thing. We all know that sometimes customers just don't know what they want. I can't solve all of that with what I'm talking about today, but we can get closer and we should strive to get closer So now I recognize this is a Django conference and I haven't said really a word about Django, so let's talk about how we weave this in with testing our Django projects. So I know that there was a talk about uh empathetic testing earlier today, and uh I really liked that talk, and I see this sort of requirement testing as something very, very similar.

It's creating empathy of how do we make sure for like the developers that come after us, what requirements are we testing? I know there's a talk tomorrow, a deep dive, about how we want to test Django, and that's going to be covering unit tests and stress tests. I want to focus specifically on the acceptance testing of this. So typically, again, end-to-end as a customer would see it. So what are our options? How am I going to test out Django? So, could I test the API? Yeah, I mean this is a great thing for integration testings. How confident are you that everything in the API is in your user interface? It's typically extra work, so you probably should test both. So testing the API won't give us get us everything we want. How about scraping HTML?

I can use Beautiful Soup. I saw someone mention PyQuery earlier today. Also not a bad idea, but the problem is when a user looks at your system, they see your product as a whole. They don't see you don't get a bug report that says, mm-hmm, your JavaScript's broken. Please go fix your JavaScript. No, they say your product's broken or your system's broken. So if we just look at the HTML, we're missing out on a wealth of things. all the JavaScript interactions, all the rendering, all the just various interactions a user has, session management, logging in, logging out, scraping HTML will only get us a part of that picture, a snapshot, if you will For the same reason, JavaScript unit tests. Yeah, I'm driving things through JavaScript, but let's be honest, we're at a Python conference. I don't want to write my tests in JavaScript. It's also just only gonna get you a part of the system.

If you want to say, oh, I gotta start introspecting databases or back-end services from my client, that gets messy real quick. It'd be really nice if we could test the browser. Like, mmm, you know, the browser already does the rendering. It has a JavaScript engine. It's talking to the back end with requests. So should we test the browser? Yes, let's test the through the browser. And for that, I'm gonna talk about selenium. There are many other tools around that do it too. I'm just gonna talk about selenium today. Just know that they all kind of do the same thing. If you learn one, they're it's transferable between them. So selenium, what does some selenium code look like? This is my environment. py for a behave. Environment. py lets you set up things like fixtures and py test where you can say, although, you know, this happens before every test scenario. This happens.

After every test scenario. And you'll see I have some Django specific things in here. That's just so that databases are set up and torn down every time I run tests But what I want to pull your attention to is this line right here, context. browser equals webdriver. chrome, about halfway down through the screen. I'm creating a Chrome window right now through Selenium. So Selenium is just gives you access to, you can use Chrome, you can use Firefox, Edge, Safari. There's all these little web driver-like programs. that you can install that Selenium will talk to to drive your web browser. And once you have that, you can do some really cool things. I in my steps some of those uh give and when then steps implementation. You might have saw seen that I had a login page. Here's how I'm logging in in a user. From my browser, I'm getting a URL

through a browser. I'm finding some elements in the DOM by an ID and I'm sending keys to those elements. And then I'm clicking on another element. Selenium does some really cool things. Like if the element is not visible, it will error. Not just it's not in the DOM. If it's not visible, if there's something else overlaying it, if there it's invisible, Selenium will error by default. So it shows you, hey, you tried to click on this, but if I were a user, I would not have been able to click on that. That's an error. Here's how I'm creating a uh a link in the little link shortener that I created. Finding some elements by IDs, clicking on them, sure. Um what the way I have this set up is when you add a new link, it just appends the old link and the generated link at the bottom of a list. So I'm using Selenium to find elements by XPath to find out, okay, let me get the last element that's added.

Let me make sure that it's correct. And that it actually returns the right thing. Now it's kind of tough to look at code and say what is it really doing? So I think a demo would be a lot better on this. So I have uh just a console window here. Um I'm just gonna run behave on the two uh f scenarios that I showed you, my hands are off the keyboard. You should see Chrome windows popping up being driven automatically. I'll do it again in case you missed it, because hey, it was fast. Also a benefit. I am logging in, I am checking my link shortener, I'm redirecting to uh the DjangoCon US page. That's my behave tests. So as the curtain falls on this talk, I want to just reiterate what just happened. We took natural language requirements. Yeah, they're in a little bit of a structured form, but again, they're English. They're not Python or you know

XML or anything like that. I took some English level requirements. I mapped them to Python functions. Those Python functions launched a browser and interacted with the browser all through the safety and like the comfort of Python. And I just ran tests. I ran executable specifications through all of this. So that was with Behave and Selenium As I close out, I just want to remind you, build the right thing. Don't waste time building the wrong thing. Uh, you know, what's there's some con um idiot some phrase. It's Don't let weeks of coding prevent you from hours of planning. Have though that conversation early. Talk with your business users, talk with your end users, talk with anyone who uses your system, and come up with those requirements in a nice, easy form.

That shows that we are going to build the right thing. Have the right conversation. Make sure you're talking to the right people. And make sure you're really driving down what it is you should be doing. Execute your specifications. Earlier I said, you know, I only meant run your specifications, but you you can kill your specifications too. But like a, you know Specifications are dead, long live specifications sort of thing. Think about them in a new way. Thank you very much. My name's Pat Viafor. I'm a principal software engineer at Cloud Software Group. I'm an owner of a consulting company If you want any Python advice, come hit me up. I'm the author of Robust Python, a book by O'Reilly. Most of this material came from uh chapter 22 in there. There's 23 other chapters that you can go read about if you're interested in this sort of things.

And I'm an organizer of Huntsville. pie. If you're ever in the North Alabama area, come uh get send us a message. We're always happy to see new friends and faces. Thank you very much. Enjoy the rest of your DjangoCon.