Performance Testing for Modern Apps by Dustin Whittle

Speaker 1: Alright, thanks everyone for joining. Again, this is Performance Testing for Modern Apps, and we're going to be talking about how some tools of the trade for testing performance on the server side, but also understanding client-side performance. So I have a ton of content. There's a bunch of notes at the bottom of my slides. I'll make them all available online. So if I go a little bit fast, know that there's plenty of notes for later So the reality is that top engineering organizations think of performance not as a nice to have, but as a critical feature of their product. And it's because they understand that it has a direct impact on their business's bottom line Most of the time, developers really don't think of this until uh you go to launch. And really I'm here to help change that. So you can find out a little bit about me. Uh follow me on Twitter at Dustin Whittle or DustinWittle. com. So why does performance matter? Microsoft found that Bing searches that were two

Speaker 1: seconds slower resulted in a 4. 3% drop in revenue per user. And when Mozilla shaved 2. 2 seconds off their landing page experience, Firefox downloads increased 15. 4 %. So they got 60 million more downloads just because the page was a bit faster. And making Barack Obama 's website 60% faster increased donation conversions by 14%. But the most impressive metric that I've come across is that by decreasing the end user latency of Amazon. com's retail operations by 100 milliseconds results in a 1% improvement in revenue. So whether it's Yahoo, ShopZoa, AOL, Amazon. com, all these engineering companies, all these uh engineering organizations understand that ultimately performance directly impacts the bottom line. So the question is how fast is fast enough? So 0. 1 seconds, it feels instantaneous.

Speaker 1: It feels like you're flipping a page in a book or 100 milliseconds. So you should really strive to keep your load times in this range. But one second allows you to think seamlessly, and after ten seconds you really start to lose the attention of your users. So there's been a bunch of performance studies to understand basically the attention span in applications and user experiences And what they realize is that performance really is key to a great user experience. I think everyone's probably had the experience where you go to check out in an e-commerce store and you click the checkout button and then it just waits for a long time and you really quickly start to lose faith I think the engineer in all of us knows to just wait it out or you're gonna get charged again. So again, how fast is fast enough Um 100 milliseconds is again instantaneous. It feels like flipping a page in a book. 100 milliseconds to 300 milliseconds, the delay is perceptible, so your users are going to start to notice.

Speaker 1: And after about one second, you really start to interrupt the user's flow So users expect a site to load in two seconds, and after three seconds, 40% will abandon your site. And this comes from a Nielsen uh performance survey that they did a long time ago. And the reality is that mobile applications are even worse This is really hard to do because modern applications are really complex. When you have a hundred microservices talking to each other and you're making calls to external providers, a shipping provider, payment processing provider, fraud detection provider, it's really hard to have Great performance. So with application complexity exploding, how do you manage this and how do you test for this? Um I think most companies treat performance and really uptime as a critical differentiator. And if you look at some of the major enterprise companies, they all treat uh uptime as a critical metric.

Speaker 1: And I think we've all encountered this. If you provide a service to others, it's enforced by an SLA. So, what's the goal here? The goal is treat performance as a feature, and we're going to talk about some tools of the trade for performance testing to do exactly that So the first thing I like to start with is how to understand your baseline performance. So just raw Python is going to be uh pretty quick pretty fast. When you start to add Django, you have a framework layer that's going to add a bit of overhead. And then you're going to have your application stack that's going to add a bit more of overhead. Really what you want to understand is what's your baseline performance on some hardware, on a specific set of hardware that you're going to run in production. What's the performance of a static asset being served not from Python? What's the performance of Python hello world? So very simple script. What's the overhead of your framework?

Speaker 1: And then what's your actual application Because oftentimes what you'll find is that uh the business transactions in your applications are have very different performance. The homepage is going to be highly cash, whereas the checkout process is going to talk to a bunch of third-party providers So it's going to be inherently slower. So you should understand the performance of each one of these transactions and really understand uh how that affects your users. So I like to do that by understanding what the static threshold is, what the hello world baseline is, and what the application benchmark is. So if you've ever installed Apache, you've probably access to Apache Bench. If not, you can app git install Apache 2 utils on most Linux platforms and it becomes available. Apache Bench is a very simple tool for benchmarking the performance of applications. This isn't specific to Django, this isn't specific to Python. You can use most of these tools across any different applic

Speaker 1: application platform. So Apache Bench is really crude and dead simple. So if you just want to get an idea of uh how fast a particular transaction is like your homepage. It's really easy to test a single concurrent user. So in this case, what you're seeing is Apache Bench Dash C for concurrency. So I want to test one user going as fast as possible for 10 seconds to AcmeDemoapp. com So we'll run Apache Bench dash CU concurrency of one for a time of ten seconds and dash k is benchmark mode. So I want no delay between the requests And what you'll start to get is a response that looks like this. So you'll get the at request per second, which is a use very useful metric, but more importantly is the latency. So what's the average response time per transaction? And the balance to figuring out how much capacity your infrastructure can support is understanding when you max out your requests per second and the latency starts to rise.

Speaker 1: So you can always serve more transactions just very slowly, but you don't want your users waiting 10 seconds just because there's a thousand of them showing up. So you really need to understand the balance of those. And with Apache Bench it's really easy to start to increase the concurrency. So in this case I want to start testing 10 users. So we'll just change the concurrency level here to 10 and test again for 10 seconds. And what you'll see is that we have more requests per second. In this case we have 65 requests per second. Uh but the time per request has gone up to an average of 151 milliseconds. Now, Apache Bench is great. It's uh quick and dirty, it makes it very easy to load test a server. Um, but I prefer Siege. So Siege has a very similar format. If you can app it install siege on most uh platforms or port and brew

Speaker 1: install siege, it's pretty straightforward. And it has a very similar format. So you can run siege-c, again, concurrency of 10 users um for a time of 10 seconds. Now what you'll notice is in these examples I'm load testing one endpoint. I'm only load testing the home page. But that's only so useful. And we'll get metrics that were very similar to Apache Bench. So for 10 concurrent users, we get about 65 transactions a second, and the uh average response time is about 30 milliseconds, I think. So you can keep increasing the concurrency uh until you max it out and really what until until you can start to see the latency start to skyrocket. And really what you want to understand is what's the maximum request per uh uh per second that the machine can get uh where

Speaker 1: before the average response time starts to increase. Now this is fine for a very simple application, but most of the time applications aren't one endpoint. It's many different endpoints. You have functionality to you have the homepage, you have login, log out, add to your cart, and checkouts, process and order, etc. So there's a quick tip to be able to crawl the entire application to discover all the URL endpoints because most of us inherit applications and we don't have the ability to just build them from scratch every time. So you often don't know all the functionality that exists So there's a tool called SProxy, which is a transparent HTTP proxy. Basically, what it allows you to do is make a request through a proxy and it will log the URL that you're requesting. So if you want to interact with an application and you want to crawl all the endpoints in the application, it's very easy to use S-Proxy and Wgit.

Speaker 1: uh to basically emulate a search engine spider. And the goal here is that we want to uh find all the URLs of the application. So it starts with S proxy-o, and all it's going to do is run an HTTP proxy on port 9001 and all the URLs that we access are going to be put into a URLs. txt file. The next thing we're going to do is use wgit. So Wgit has uh spider mode that allows you to emulate a search engine spider and it'll go to the homepage and recursively crawl all of the links So if you really want a quick and easy way to discover all the functionality of the application, you can run sproxy and wgit and discover all the public URLs in your application. And what you end up with, we'll sort the list so we have a unique list of URLs at the end, and you end up with something that looks like this. This is just a simple e-commerce application that I'm using, but there's an about

Speaker 1: page of view carts, change the currency, login, register for an account View by category, view by tag, etc. And what really what we want to understand is what is the how do we benchmark traffic across all the unique URLs so that we can understand the performance of each transaction? Because the processing in order is going to be a lot slower than going to the homepage, especially when the homepage is highly cached. So you can use Siege again, this time feeding in the list of the URLs text file, so that we can benchmark traffic across all the unique URLs. So we're going to run the same command that we ran earlier. So this case it's going to be siege dash v for verbose. txt file. And what we'll end up with is something pretty straightforward to the output that we had earlier, which is the average response time for each one of these transactions

Speaker 1: and the transaction rate. Now, this is only so useful because oftentimes you don't want to just go through a list of URLs, you actually want to perform a transaction. And in that case, let's talk about multi-mechognize. So I'm a big fan of multi-mechognize. It's an open source framework for performance and load testing. But what makes it really useful is it allows you to script transactions. So I can actually post these credentials to the login form. I'll maintain state by gra creating a cookie jar and maintaining the HTT session along the way so I can log in, then I can add something to the cart, and then I can check out. And I can actually manage uh the entire HTTP transaction or a set of transactions and then I can start to load test those as well. So it's very easy to install. It's Python tool. So you can simply pip install multi-mechannize. And then you'll have access to a command line library and you can just bootstrap a new project pretty straightforward.

Speaker 1: Now I presume everyone here is pretty familiar with Python, so this should be pretty straightforward. The idea here is we're going to import the HTTP request library So it's an HTTP client for Python. And then the only thing that multi-mechanized provides is a transaction class. And what you do inside of the transaction class, it doesn't matter. It's just going to run many of these transactions at a high level of concurrency. So I can do this for a single URL endpoint or I can run a series of URLs. So this is going to be the same example that we just ran earlier with Siege or Apache Bench, where I just want to load test uh AcmeDemoapp. com as a single endpoint. So here we'll import the request library, we'll define the method run, and then we'll simply make a git request to apme demoapp. com. But oftentimes you want to do much more than that. You actually want to script a user scenario.

Speaker 1: So I actually want to go log into acmeedentoapp. com, go to the cart page, and then start to script this out So here you can import multi-mechannize. Again, the only thing that matters is what you put inside of the run method. So in this case we have a transaction and we just want to run this transaction at high levels of concurrency So here we instantiate the multi-mechanized browser. This is what's going to allow us to emulate the HTTP state and maintain a cookie jar. We're going to disable the robot so it doesn't block public endpoints by uh using the robots exclude. And then where it has a custom timers API. So this is where it gets really useful because based on the custom timers, it will generate fancy graphs that will allow you to understand for each one of these transactions the average response time and the number of requests per second.

Speaker 1: So for each one of these we'll create a custom timer and we'll create a custom timer for the home page and a custom timer for the card functionality. And obviously you can script this for the specifics of your application, but it becomes quite easy to to script complex transactions So here we create a start time. We're going to use the cookie jar to open acme demoapp. com. We're going to read the response. We're going to record the latency. So the start time minus the end time gives us our latency. And then we're going to add that to the list of custom timers. And then we're going to do the same thing but go fetching the cart. We could very easily do a post, post the login, fetch a bunch of products by their tag, add those all those products into the cart, and then process an order. But that doesn't fit on a slide. So once you have the script, you can create as many of these scenarios as you want.

Speaker 1: So this is just one transaction. So I could have a home page transaction, I could have a buy uh buy stuff from a shopping cart transaction, file a support request. transaction, all these different user scenarios. And then once I want to run these scenarios, I can use the multi-mechognized command line to run these at different levels of concurrency. So here you'll simply run multi-mech run demo. So this file is called demo. py and it just has the transaction class inside of it And once you run that, uh it'll run to whatever level of concurrency you've configured and then it'll generate a list of output that looks like this. For each one of the custom timers. Uh you'll get the response time uh plotted out. So the average response time versus the request per second. Now by a show of hands, so this is pretty easy for uh any scripted transaction.

Speaker 1: But what you'll notice is that you're running all these transactions off of a single server. By a show of hands, how many of you uh have a real production application that's more than one machine? Right. How many of you live in the cloud? Okay. So we're going to be talking about my favorite open source project, bees with machine guns. Big fan of talking about this one. One because it comes with a notice that says it's a felony to do this against any other site except for your own, because effectively it's a distributed denial of service tack It's brought to you by the fine folks at the Chicago Tribune. Uh so what is bees with machine guns? It's a utility for arming mini bees to attack targets. We're creating micro EC2 instances and the Amazon Web Services to load test web applications.

Speaker 1: So when you live behind a load balancer and you have hundreds of serv hundreds of machines behind a load balancer, how do you generate enough traffic to actually test that? You're not going to do that off a single machine. You're not going to do that off your your MacBook. You're not even going to do that off an EC2 instance. You'll max out your network throughput before you can generate enough load. So the idea of Bees with Machine Guns is it allows you to run a distributed load test. And it's very easy to use. So again, Python. So simply pip install these with machine guns. No reason this can't be fun, folks. Uh and then it uses the AWS Bodo library. So if you're not familiar with Amazon Web Services, you can go to aws. amazon. com/slash free and get a free year's worth of

Speaker 1: uh computing power and then all you need to do is configure your access key in secret here and that'll give uh the Bodo library uh AWS credentials. So Bees with Machine Guns relies on the Bodo library to spin up EC2 instances Once you configure this, you're good to go. And you can pick whichever region you want. In this case, I'm running it off the West Coast. Now you can spin up two machines or you can spin up 200. It just depends on the level of concurrency that you want to generate. So in this case, uh it looks pretty fancy, but the only thing that really matters here is you call BsUP-S2. So I want to spin up two servers so I can run a distributed load test. There's some other stuff in here. Basically, I want to be in the default security group. I want to spin up these two servers in the US West 2B data center. I want to use this AMI instance ID and these SSH keys and username.

Speaker 1: But all of that other stuff is irrelevant. What you really need to know is B's up dash s two. So I want to spin up two servers to start load testing These people have a great sense of humor, so they're going to connect to the hive, attempt to call up two bees, they're going to wait for the bees to load their machine guns, and then they're going to be ready for attack. The swarm has assembled two bees. And then you can check how many machines are actually up and running and ready to be used. So you can call B 's report, and it'll tell you that you have uh two bees from the roster and the IP addresses. So it's we're going to start the load test with just a thousand requests. So in this case, we want to load test a single endpoint. So we'll call B 's attack and then the end, the number of requests. I want to fire a thousand requests with the concurrency level 50 at a time against this URL endpoint. So I want to uh basically throw 50 concurrent users until I hit a thousand

Speaker 1: requests at AcmeDemoapp. com So something very similar to what we did with Siege, just with a bit more concurrency. And then we'll get this response. So it's going to read two B's from the roster, connect to the hive, assemble Bs. Each of the two Bs will fire 500 rounds, 25 at a time. Singing the URL so it will be cash for attack. So whenever you use a framework like Django, whenever you use uh The mini web application stacks you should prime the cache because you're going to do a lot of things on the first request that you uh you want to when you're doing a load test you want to understand the real world performance. So you want to have one cache request that gets fired and bees with the machine guns will automatically do this. So it'll Create one UR one request to that URL to prime the cache and then it will start load testing. And then you'll see the requests per second at fifty uh fifty uh concurrent requests per second we get

Speaker 1: three hundred and six uh requests per second with an average time of 163 milliseconds. So here again the goal is that we want to increase the level of concurrency until we start to see the latency start to spike. And that will tell us what the this particular hardware running this application can support. But you can very easily increase this uh to ten thousand or a hundred thousand or five hundred thousand very easily. So in this case, we're just gonna crank up the concurrency to a thousand requests a second. And we'll call this again. So bees attack. This time I want to fire off 100,000 requests, 1,000 at a time, at acmedemoapp. com. So bees attack dash n 100,000, dash C a thousand. So concurrency of a thousand And this time we're going to do the same thing. And what we'll find is that the requests per second, now we support 502 requests per second, but the latencies start to spike.

Speaker 1: So now the requests are averaging out at 360 milliseconds. So that just means that yes, while we are not gonna fail over, the users are gonna wait longer. So you decide how much money you want to spend and how much the business can support on the threshold there. Uh but ultimately you want to be able to start to add hardware when the latency starts to spike, especially if this is going to be sustained normal workload. Again, they have a great sense of humor, so uh sometimes things don't always work out. Each of the two Bs will fire 550,000 rounds, 500 at a time. Once the offense is complete, uh actually we missed it. The target crushed to be offensive. So the server survived, basically. Uh and if things don't go so well, no bees completed the mission. Apparently your bees are peace-loving hippies. So the swarm is going to wait some new orders.

Speaker 1: And then uh the benefit of the cloud is that costs scale linearly with demand, so you only pay for what you use. Uh so you you can call Bs down and it will tear down the two instances you just spun up. So if you really want to easily run a production load test You can spin up a couple of machines for an hour or so and then tear them right down when you're finished. So Beesdown is simply gonna turn off the two instances that it spun up to create the load test. Alright, so we talked a little bit about Apache Bench and Siege when you want to load test a single endpoint. We talked about multi-mechannize when you want to script different endpoints. And we talked about bees with machine guns when you want to generate massive amounts of concurrency. But sometimes you want to do all of those things at the same time. So enter locus. io. So uh this is a relatively new uh open source framework for load testing.

Speaker 1: I'm a big fan. Uh it's again written in Python. Uh go to the website locus. io Pretty much all the slides are directly copied from there. So you can pimp install pip install locust. io because again it's a Python tool, and it gives you uh the benefit of both worlds. One, it allows you to run uh load tests from multiple servers, but it also allows you to have complex transactions And then even better is it has a bunch of great reporting and ships with a web application so that you can do this very easily in an automated fashion So the reason I talk about performance testing is this isn't a one-off. Most of the time the problem with performance testing is you only want to do it right before launch. How many of you know about healthcare. gov? So this is the only time I've seen a president go on live TV and apologize for a broken website because they didn't do any testing and they spent some obscene amount of money on it.

Speaker 1: They did some testing right at the very end to realize they were It's it wasn't gonna go well. And it didn't. And uh a lot of people uh got in trouble for that, right? So the the goal is don't just do this when it's launch time. This is something that should be part of the software development lifecycle It's a continuous process. Everything that you do affects the performance of your code. You should understand that. In the same way that you have a continuous integration set up and you write uh unit and functional test , You should write performance tests because you should understand when you're going to significantly impact the end user experience as part of every release process. So we'll get more into that later, but for now more about locust. io. So the idea behind locust. io is that you can very easily script a website task. So the same way that multi-mechannize has a transaction, you can define a set of tasks.

Speaker 1: In this case Yeah, we'll just go right into this. So you have a user behavior in this case. We just want to log in and fetch the profile. And on every request we need to log in. So on start, just like you have a setup and teardown. Uh you have an on start uh where we're automatically gonna log in and this is just gonna do an HTTP post request to the H uh the login endpoint. So if you see back here, uh it's just uh the call self. client. post and posts the username and password to the login endpoint at the start of every request. And then we want to run the task number two, which is to fetch the home page. And then finally the third task, which is to fetch your profile. So I want to log in, then I want to go to the homepage, and then I want to go to the profile. Now I we can get into some of the rest, but the idea here is that

Speaker 1: it also comes with a great UI Because for each one of these endpoints, you want to understand what's the average request per second and the average latency for each one of these transactions. By scripting them with locust. io and by giving you a nice UI to run these tests, they make it very easy to automate all this stuff. Very easy to script complex transactions and then generate high levels of concurrency as well. So check it out, locust. io. But the idea here is there are many tools. It's not just going to be a bunch of Python tools that I handpick for this talk, but there's Gatling work, uh Sung, there's a ton of different server-side tools. The idea is that you use what works well for you and your use case. But that's just the server side. The reality is what about the client side? Most people when they think about performance, they think about only the server side because that's the application stack.

Speaker 1: But what you don't realize is that in modern web applications, more latency comes from the client side than the server side. The 200 milliseconds that you're waiting for the response from the server is nothing compared to the two seconds that you're downloading JavaScript CSS resources, waiting for the page to execute JavaScript and paint the actual page to become usable. When you talk about uh the end user experience time, it's about users perceived experience. It doesn't matter to them whether it's on the server side or the client side. It's all about am I waiting in my browser? Can I use this right now? Can I click on that box to start searching? That's why Google's homepage loads very quickly. It's just a box. I just want to be able to type in the box as fast as possible. Right. So when you think about performance testing, don't just focus on the server side, also understand how the client side is impacted. So Google's invested a ton of money in performance engineering because it's really important for them because they understand that performance impacts their business

Speaker 1: And one of the great tools that they've released is PageSpeed Insights. So Google PageSpeed Insights allows you to analyze and optimize your websites using the PageSpeed rules. So what it will tell you is, hey, you should be using Far Futures Expires headers for your CSS and JavaScript. you should be using gzip compression for delivering your CSS and JavaScript. But even better, they've made it extremely easy to automate this piece with Nginx page speed and Apache page speed. So these are modules which will rewrite your responses in real time. uh to make them more optimized. So most of the time I don't recommend using Apache or Nginx PageSpeed. This is the cheap, lazy way to do it. Cheap as in engineering time and lazy as in you don't actually have to fix anything. You just install a module and it does all the hard work for you. You should really use automation and front-end automation to make this part of your development cycle so that when you package your application up to deploy to production, you automatically

Speaker 1: minify your CSS and JavaScript. So that you automatically set the Nginx configuration up to serve it compressed. So you can do that very easily with Nginx page speed, but you're not going to really fix the root of the problem unless you fix it in your software development stack. So it's not just uh a PageSpeed uh a module for your web server. They also have a great website so that you can go and get actionable advice on how to improve the performance. So you can go to developers. google. com slash page speed. and type in your URL and it'll give you tips on how to improve the performance. And this is the end user's perceived performance. This is what usually matters the most. So it's available as a web page and standalone API. It says available as a Chrome developer tool, so it's very easy as

Speaker 1: to make part of your development setup. And it's also available as a gulp package. So you can, if you're familiar with Node. js, you can npm install PSI, and this will allow you to automate this as part of your front-end workflow. So what that will look like once you npm install PSI, you have access to the PSI command line, and then you can simply run PSI and then the domain. And what this will do is run PageSpeed Insights using the PageSpeed API underneath And give you some formatted output. It will tell you your page speed score, so 73 is not too still. The amount of CSS and JavaScript that you're using, how many are cached, uncached, how much is compressed, uncompressed, etc. So this generally is a good way to integrate with your continuous integration setup. So they'll fire this off as just another task

Speaker 1: and then start to do some benchmarks across this. So if our page weight ever goes over 500 kilobytes then uh we should probably have something fail, like a test would fail. And then if you want to just integrate directly, of course they have a REST API that's available. So sometimes you want to actually understand what's going on in the JavaScript world. So Wbench is a Ruby tool that makes it really easy to understand the uh what's happening inside the browser. So it lets you understand the page load times. So this is a Ruby tool, so you can simply gem install Wbench. And Wbench uses effectively uh Chrome and WebDrivers, so that it'll run inside an actual browser instance. and then record uh the the page speed times. So in this case I'm just going to run it across the very wonderful dustinwittle.

Speaker 1: com. And you'll see how horribly it's built. So I rely on uh Google has a Google Fave icon service that they don't really publicly expose, but each one of those requests ends up in a redirect uh which means everything takes a really long time on my website to load. Uh and what you can see here is really the end user timings. Uh that yeah so the user timing APIs So the W3C has a set of timing APIs. One is the user timing API, so how much time have I spent doing DNS lookups? How much time am I spent doing SSL negotiation? How much time am I spent waiting for the browser to download CSS and JavaScript, waiting for the page to paint and for the page to become available. And then they also have a resources timing API. So if you've ever added a social widget on your website like the Twitter share button.

Speaker 1: or a LinkedIn share button, you'll realize they really start to impact performance. So that gives you performance timings for the individual JavaScript and CSS that you include on your site. And then they're just releasing a custom timing API so that you can do whatever measurements you want in JavaScript. And let me tell you all this so that I can tell you this, which is to automate your client-side performance testing with Grunt and Gulp. So uh this talk's gonna deviate a little bit, but use Bower for managing dependencies, use grunt or gulp for automation, front-end automate your front-end workflows. And Yo Man for bootstrapping a project. So if you're getting started with something new today, there's a bunch of YoMan generators for Django to make it really easy to do all the stuff that I'm talking about with best practices built in by default. Uh grunt or gulp doesn't really matter. Pick which your flavor uh which flavor works for you. Okay, so by a show of hands, how many people understand, how many of you would call yourselves uh

Speaker 1: professionals Yeah, hopefully all of you. And then how many of you understand that you have a problem when a user complains directly to you or via customer support? Which says you have no level of performance monitoring at all built in. You find out when users go and say, hey, I can't check out into your website. So really the reality is that most people don't uh measure performance. Um I work for a performance company, so I'm not here to pitch that. My goal here is to simply say that this is really impactful and uh affects your businesses more than you might know So you should be tracking performance and development and production. Even those companies, the few companies that do track performance, they only track performance and production And they use some variety of APM tools out there.

Speaker 1: But if you're going to do load testing, the whole goal of load testing is to generate useful information. You can only capture that useful information if you're monitoring. So if before you go and start doing the load testing, the client side performance testing, Instrument everything. Instrument your code, instrument your databases, your caches, cues, third-party services, and your infrastructure. Because oftentimes it's not the application stack that breaks. It's not the Django app that's the issue. It's uh logging errors with some error provider, making API calls to some payment provider, or the cash hit, miss uh or database lookup. So unless you're instrumenting that, once you generate all this load you're not going to get any meaningful insights out of that. So the whole goal is you should instrument your applications, your infrastructure, so that when you run a performance test, you get useful insights.

Speaker 1: So there's a ton of different open source tools. Pick your favorite stack, but uh the reality is that you should be doing some level of performance testing and some level of monitoring. I'm a big fan of Chef and Sensu, uh stats D Graphi and Grafana, so that you can have a cool dashboard that looks like this. I know I'm covering a lot of this stuff very quickly. But the idea here is that uh if you can instrument your applications and you can instrument all the API calls that you're making, all the database lookups, your cache uh key hit-miss ratio if you're talking to memcache or Redis. you can really start to understand where you need to focus on improving performance when you do run a load test. Running a load test to understand like capacity planning and whether you're going to fail on launch day is one thing, but making it a part another level of functional testing for your applications is really the end goal. And you can only do that if you actually have some level of instrumentation to get meaningful insights.

Speaker 1: And then the last part is, again, it's about the end user's perceived Performance. So you should track performance of your end users. Because if you have an application hosted in Amazon Web Services in the West Coast And you have a bunch of users in Australia, well a bunch of users uh on Telstra service in Australia might be having a very serious problem, and you're never going to be able to see that. Sorry to be specific. You're never going to be able to see that unless you actually monitor the end user's experience. There's a bunch of open source tools that make that possible. So there's the episodes framework from the one of the uh Steve Sauders, one of the guys behind Google PageSpeed. And what it essentially does is it captures the JavaScript timings from inside uh every one of your browser uh real users browsers So you'll see very quickly if users in a specific location or on a specific carrier are having performance problems

Speaker 1: because that wouldn't show up and on the Python stack. That's not going to be an exception, a century, or whatever error logging platform you use. It's just going to be a user who has a bad experience who complains to customer support because they think it's your fault. Even when it could be their carrier, it could be a particular browser. So check it out. And then webpagetest. org. So I'm a big fan of webpagetest. org for a couple of reasons. One, it allows you to very easily check performance across a different array of browsers. So you can check uh Chrome performance on the West Coast from specific locations over specific network conditions. So I want to check Chrome from New Zealand over 3G. How does my application load? And more importantly What does the rendering process look like? So one, uh

Speaker 1: webpagetest. org, you can just fill out a URL and tell where you want to run the test and from what browser. And then it will generate test results. So one of the things that I really like is this is a collaboration amongst a bunch of um technology companies, so it's available free. And it'll give you a both, you can use it for a QA purpose, so it'll give you a screenshot of the final rendered form, but it'll also give you a video of the rendering process. So oftentimes what you want is the to optimize the critical path so the uh the the key parts so the browser can paint the page as fast as possible. That's what matters most. And you'll very quickly be able to see oftentimes what you'll see is the a sudden flash and then the page gets painted and that can be a one-second delay. Or you can see immediately the page show up and then it gets progressively built beneath

Speaker 1: Uh it all depends on how you architect it. Oftentimes it's very hard to test those things. Webpagetest. org makes it very easy to test and QA. And like I said, if you've ever used the uh developer console in Chrome or Firefox developer tools, uh you've probably used to the waterfall view of uh loading your client-side resources. This will allow you to see where you're spending time. Like I said, I steal the Google uh Fave Icon service, which ends in a bunch of redirects, which kills my performance. And without this tool, I would have never realized that was happening. But it basically makes two requests for twenty different icons on my homepage. And my homepage is only twenty icons, so it's a bit of a waste. Okay, so I think we've covered a lot here. Uh the final thing we're going to talk about is sitespeed. io.

Speaker 1: So sitespeed. io is very much like Google PageSpeed. uh except for it's uh much more automated and much more comprehensive. So again you can npm install sitespeed. io And what this will allow you to do is analyze your website speed and performance. So the same way that page speed will give you actionable advice on how to improve the performance, like you should cache these uh CSS and JavaScript files or minify them There's a bunch of rules that SiteSpeed incorporates and will test for. So this is really great because when I talk about uh automating all of this, you want to integrate this in your Jenkins flow, your Travis CI workflow. And you want to be able to have some real metrics to use. Sight speed makes it very easy to capture each one of these as individual metrics and then fail based on that.

Speaker 1: So you should fail if uh the average end user experience time is over two seconds or you're including more than 500k in JavaScript or whatever the conditions make sense for your site. Uh it's a pretty comprehensive listing of tools, so it's not just CSS and JavaScript cached or uncached, gzipped or un uh uncompressed or uncompressed. It's also how many DNS lookups are you do using, SSL negotiation times. It's very comprehensive, so take a look at it And then again, you should be monitoring production applications. You know, there's I think a lot of you are familiar with probably New Relic. App Dynamics is a company that works with enterprise applications. There's um CompuWare is another one. There's Ruxit, there's a bunch of these uh performance management companies.

Speaker 1: The goal of all these companies is to effectively map out both the server-side performance and the client-side performance and allow you to understand exactly what's happening. I've talked about a bunch of open source tools that allow you to do various pieces of this. The goal of all these tools is to tell you where your traffic's coming from in real time And then being able to capture the individual metrics and then give you visibility into the client side. It doesn't really matter whether you roll open source or you buy a commercial package, but you should be doing some level of performance monitoring. And then oftentimes I've talked about a bunch of open source tools for writing all of these load tests, but oftentimes it's much easier to buy this than to build it yourself. So we'll just run through a couple of load testing services from the cloud. So I'm a big fan of Pipeka just because they have a global footprint and they make it very easy to just write a load test and say, hey, I want you to go to this URL and send 50,000 users and give me a report on what that looks like.

Speaker 1: Uh Blitz. io is probably one of my favorite in terms of just easy to get started, uh Blitz. io. It's paid and free, but they give you a decent free plan where you can just say, hey, I want to spend up 500 users, send them This URL and then plot the response time versus the uh level of concurrency. And then Blaze Meter, which works well if you have uh complex interactions or you want to test mobile devices. So sometimes uh I know it's you know in the vein of open source that uh we try to write all of our code ourselves, but it oftentimes it's a business problem that you're trying to solve. Uh and you should remember that. That's not always code isn't always the solution. I can't s uh emphasize this enough, but tests for failures. The reality is that most people test under ideal conditions, but at some point in life you realize that it's an imperfect world.

Speaker 1: and uh things will happen. So you should be prepared for them. So Netflix uh understands this very much, which is they've released a suite of tools called the Simeon Army, and one of those tools is Chaos Monkey. What Chaos Monkey does is it goes and creates chaos inside of your AWS setup So what happens if you lose your caching layer? How does your website respond if you drop your memcache instance or your Redis instances? What happens if uh PayPal's API that you rely on slows down 3x? Do you uh retry and exponentially back off? Do you cue things up and uh or do you block on them? You don't really discover those issues until you have a production outage. Oftentimes that that's when you learn the most when things are failing and it's the most painful to learn the lesson is like, hey, we really probably should have cued that up and not blocked on that, or we should retry that request if it fails, not just ignore it.

Speaker 1: Um you should really test for failures. There's a bunch of tools that make this very easy to do, but what happens if you run a load test and you kill off your memcache instances? Does it surge on the database? Can your database handle that, etc. So performance matters. Hopefully I've emphasized this enough. So treat performance as a feature. There's a bunch of tools and tips on how to improve front-end performance, but basically use the 14k rule so you have instant loading. So you want to make one HTTP rule uh one small request and get all everything you need to paint the page and then progressively build the page from there. There's a bunch of tools to do this, but the core of this is use task runners to build and deploy production code. So grunt and gulp Or there's a ton of tools for Django that make it just as easy to incorporate these best practices like minifying CSS and JavaScript

Speaker 1: and combining all this stuff. And then best practices for performance testing. Capacity plan and load test the server side. Optimize and performance test the client side. Again, you spend more time on the client side than you do on the server side, so don't just think because the server can respond to a request that your users are going to have a good experience. Understand your starting point. You can't understand where you're failing unless you know where you're starting from. So instrument everything. And don't just instrument and monitor production. Instrument and monitor your development and production environments because you you don't want to fix it when it's red, you want to fix it in the yellow, you want to fix it before it goes live. It's very easy to learn these lessons when the executive team's yelling at you because you're losing all this money and revenue per hour because everything's broken. You get a lot of resources, but that's also the most painful time to do it, so try to do it earlier.

Speaker 1: And then measure the difference of every change. It's not just upgrading from Django from 1. 7 to 1. 8 that's going to impact performance. or upgrading for your version of Postgres or Linux or moving from one data center to the next. Everything affects performance. And everybody cares about performance. They just care about a different perspective. Business people, they care about revenue. Um the operations team, they care about uptime. Engineers, they care about the end user experience. They you just need to carve off the perspective that's useful. So you should measure the difference of every change. And then automate performance testing in your build and development process. It should uh really be part of the software development process and something that's automated that happens every time. And then ultimately understand how failures impact performance. Now there's been a lot of developments recently. Uh HTTP2

Speaker 1: specifically, a lot of the best practices for the HTTP1 world are actually going to hurt you in the HTTP2 world. So sharding, concatenating all your CSS uh CSS and JavaScript, spriting a bunch of images into one file. These are all hacks around shortcomings in HTTP 1. Uh HTTP2 eliminates the need for most of these hacks, so just be conscious that the protocols are changing and that you also should adapt your development techniques accordingly. Uh Ilya Grigoric has a great book on high-performance browser networking that I'd highly recommend if you're interested. So again, integrate automated performance testing into continuous integration for both the server side and the client side. Understand the performance implications of every deployment and package upgrade, and monitor the end user experience from end-to-end in development and in production.

Speaker 1: And that's all I have for everyone. Any questions?

Speaker 2: Thank you very much for the talk. Um I had a question uh you about um whether you think um given the complexity of modeling i involved in creating a load test and the and an a an an enormous p possible investment of developer time. Whether there's any value in taking a feature ramp approach uh with regards to and I I noticed you didn't mention I don't know if you have a

Speaker 1: So most of the time what I try to do is uh take the basically replay production traffic. Don't model them out based on what you actually what you think your users are doing because your users will very quickly surprise you. Go and take a look at production access logs and uh find the most common user paths and then replay that traffic The feature branch, the fe like uh when I write a new feature, test out this feature, does work quite well, but what you're gonna miss is the complexity of the overall. So the most common use cases are things that you built originally. Um w anything net new that works well, especially as a way to s ha have sustainable performance testing. Um but I definitely think that is something that you should replay production traffic versus just what you think users are going to do.

Speaker 1: I think when you b go building uh sort of the feature branch approach or uh when you build individual features and you try to just test the net new functionality The users often use that in ways that you don't expect. So I think you rely on the source of truth, which is those access logs. So if you use some sort of load balancing solution, pull it off of that.

Speaker 2: Um I'll I'll ask a follow-up as long as no no one else has. So when I say feature ramp, I I meant something I I meant something very specific. So um releasing to say one percent of your of your users Oh okay yeah.

Speaker 1: Yep. Um yeah, so uh most companies well uh a lot of the smarter companies end up slowly releasing features through that. Uh that's a great way to actually test that out, especially in multivariate testing. You see that a lot with uh very substantial changes where you just want to slowly roll them out Either way I would write some sort of automated performance test, but you can very quickly learn by doing custom instrumentation around metrics that matter for those scenarios. So if you're going to introduce a new feature You can use like stats D to create some custom metrics around that and that's what I would that's generally how we release new features is when I want to release something new we'll track some uh important measurements for that specific feature. And then we'll slowly roll it out to the users. Yeah, that's a much better approach.

Speaker 1: Anyone else? Let's not all ask it once.

Speaker 3: All right. That's our time. Thank you very much, Justin.