Predict Lightning Strikes using Django and AWS with Calvin Hendryx Parker

Speaker 1: Cool. Well I am super excited to be here. It's one of my like favorite ways to kick off a talk. Talk about something as cool as predicting where lightning will strike. As Frank said, my name is Calvin Hendrix Parker. I'm co-founder and CTO of Six Feet Up. Hopefully you saw our impactful intro this morning, so I won't give the whole bio again, but if you didn't, you can catch it in the recordings afterward. What we want to talk about though is actually something one of our impactful projects actually has been around this specific one where we're helping Predict lightning strikes and we well, I didn't come up with the algorithm. I am nowhere near that fancy of a nerd uh when it comes to those kind of things. But The science nerds need us software developers to actually make these kinds of things become a reality.

Speaker 1: So I'll start off with kind of the the story of why part of why this is important. The uh the scenario up here on the screen actually comes from a real uh lightning strike incident that happened. Um I can't remember 2020. So in September 2020, for those of you who don't know, Tampa Bay in Florida is like the lightning capital of the United States of America. There's a couple other places on the face of the planet that rival it in terms of quantity of lightning, but not by men not by many other places. Like this is if you want to see lightning, go to Tampa. you're guaranteed to catch lightning at some point. The most lightning happens there, period. So uh a jet skier was actually out on Tampa Bay. Uh about 310.

Speaker 1: And you'll see here that we actually the red dots to the right are the potential lightning. Coming in at 310, you'll see the time of the lightning strike actually was three thirty-seven. So here we are, you know twenty-seven minutes ahead of the lightning strike occurring. The algorithm output three minutes later. So you can see the difference between 310 and 3 minutes later that those red dots moved right over the top of where that jet skier currently is And so we're at the the algorithm, I say we, I didn't make the algorithm, but I helped them make it do awesome things. But the algorithm actually predicted the lightning would be right on top of that jet skier who did get struck. uh 25 minutes later. So that was a that was a prediction 25 minutes ahead in in the future for exactly where lightning was going to

Speaker 1: strike. So the the kind of cool part of all this is like the 25 minute lead time and the accuracy of like 98 99. 8 % accuracy on predicting where lightning will actually be. The issue was when they came to us to actually do this project, it took 90 seconds to get that inference done. So obviously it's hard to do this in rapid succession or scale this out if it takes you 90 seconds to just do the inference alone, plus then the time to rebuild that model on an ongoing basis. Because you want to be continually rebuilding these kinds of models based on all the input data available to you. So this actually was an algorithm developed by a gentleman named Jason Deese.

Speaker 1: He's a research meteorologist for the National Weather Service. And he'd been working on this for years, but been running it exclusively in Jupyter Notebooks. So on local machines, it ran on as one of those things that runs great on my laptop, but good luck getting it running someplace else, kind of a problem. And then they basically wanted to see how they could scale this up and even make it a reality to then go take this to be a viable commercial product. What was interesting, uh, we got to talking and said this is a perfect use case for a lot of the serverless technology on Amazon. So really, this talk should almost be like how to scale detection of lightning with serverless on Amazon, but We obviously are using Django behind the scenes and lots of Python as far as as part of this whole process. The scope of the data is pretty large.

Speaker 1: Now when you think about the fact that there are um NOAA has Nextrad uh radars that are, there's I think you threw right here, 159 Next RAD stations in the uh United States. Each of those stations is sending sweep scans. You can see here like three to fifteen megabytes per sweep, depending on the activity that's going on in that area. And I think each scan. So every every five minutes you get these archives of all the sweeps that are going on. And that produces a tremendous amount of data. We're talking about around 500 gigabytes of next rata data alone to ingest and process. And then another 450 gigabytes of data of what's called wrap data.

Speaker 1: That's like a rapid forecast prediction data. These are all open data sources available to the general public. The next RAD one's actually hosted on S3 buckets, and I'll talk about that when we look at some of the architecture. But there's just a tremendous amount of data, and that doesn't include uh new data sources that this group is continually adding in. So as we as this scales out the amount of data that they're going to process on a Well, almost a minute by minute basis is becoming quite quite large. And so you can see we actually were able to get their uh refactor their algorithm, help them get the predictions down to under 500 milliseconds. which made this much more of a reality than the 30 to 90 seconds that was taking before to produce those kinds of kinds of results. But we did want to take the cloud native from the start.

Speaker 1: approach with all this. And one of the big reasons for that is going to be how can we accelerate getting more releases and more features into production quickly for this group. We wanted to focus on the developer experience as well. We knew we wouldn't be the only ones who are around helping them out. We want to make sure that they could actually onboard new developers into the mix. So we also wanted to keep the stack as simple and as easy for folks to understand in their heads as possible. So not trying to overcomplicate things. So I'm going to start the journey with the developer experience. We started with the ability for the developers to run almost the whole stack locally. Uh so we were using Lambdas and Django. So setting up and being able to do a Git pull from

Speaker 1: in this case they're using GitLab. You could pull, do a you know, Docker compose up and be developing, hopefully in the theory, be in minutes. And generally that's actually pretty true, and help them onboard new developers along the way as we were working with them. So it actually allows us to run all the basically the whole stack so it looks like it's real for the developer locally. The bottom bit there is everything else has been from this point in the presentation it was been um built out with infrastructure as code so using Terraform but we would do basically a AWS console clicks out to kind of sketch out maybe some ideas and then build the terraform that would actually make it all a reality. And that included everything from the developer experience into the deployed application experience. Uh in this case you'll see we've got uh

Speaker 1: Lambdas, we've got um ECS Fargate S3 buckets. the ECR image repositories, and actually the GitLab configurations are also done with infrastructure's code. So they could change things quickly and easily and make sure they could track those changes that they were making to any part of the developer experience or the release. uh experience. So that's why you're seeing um code artifact and all the bits here are all done with infrastructure's code. So key to this bit with a being able to use serverless on Amazon in addition to using, we're using Fargate for hosting of the Django bits. We're also using lambdas for some of the ingest when it came to the radar data and rebuilding the models, things like that.

Speaker 1: So we want to make sure we're actually keeping the the environment very consistent. So a developer didn't have to know, oh, when you're working on lambdas, it's gonna work like this, and you're gonna do these five things to make it all go. And when you go work on the Django, oh it's in a container and you're gonna do this. So we actually wanted to make sure that everything was containers all the way across the board. About a year and a half ago, Amazon finally announced full support for Docker images in Lambda. Which makes it just again super easy to kind of keep things uh similar across the board. And so if you want to use Docker images in AWS Lambda, uh you basically just have to make sure you inherit from their base image. So That first line here, which is the from line, pulls in the base image for any Python-based lambda you would want to make and deploy into Amazon. Everything else here is pretty straightforward.

Speaker 1: You know, you copy your app into there, call the handler, and that's basically the API for using containers with Lambda. Now the issue came into play where if you want to deploy an AWS image that has C dependencies in your your project. So a lot of the scientific libraries that they're using are going to have C dependencies. And so you need to be able to compile those for that specific target platform so you could deploy them into your ECR containers and deploy and actually have it all run. So the trick there is going to be to use a multi-stage build, which is generally a a good practice for keeping your images slim anyway. But in this case you have to because you want the base image not to actually be the Lambda image for Python, but actually be the Amazon Web Amazon Linux image, Amazon Linux 2 in this case. You install the matching version of Python that you're going to use in your follow-on

Speaker 1: steps to deploy into the uh Lambda And then in this case now you've got the C compiler available to you so you can actually build. So I've installed Python 3. 8 Devel on there so I can actually compile the crazy radar libraries and stuff like that are built with all the kind of C dependencies. So the follow-on now is you do a second stage of this build with the actual Lambda image And at the very bottom, uh you'll see we we install or copy from the build stage prior, where we're using Amazon Linux, the wheels directory. So if you didn't catch it here, we are actually building wheels. And then install basically just building the wheels, putting them into a folder, and then the next step we actually copy those wheels from that folder into the

Speaker 1: Amazon Lambda image so they can actually run. And that makes it all compatible. And makes it nice and slim. Oh actually, there is an efficiency I have not done here yet, which is that copy line from build into the app wheels does copy the wheels in there and then we install them. It'd probably be better to remove those at some point to make the image even slimmer. But the the size of the image actually wasn't the issue here. The issue was getting C compiled or C dependencies compiled in. So then the deployed experience, like what actually is on the front side of all this? The NextRad NOAA data, the big NOAA logo over there. Like I mentioned before, they're about every minute or so are throwing these new uh radar sweeps into a public S3 bucket, and you can subscribe to SNS notifications.

Speaker 1: This makes for a really clean implementation because I don't have to use Kafka or deal with like polling. Like it's just all nice and built into the Amazon platform to support this open data. So you get events whenever new radar sweeps are put into that bucket. Ultimately we went with the archive archive sweeps, which are six minutes of data at a time, because the that resolution and time was good enough and it was easier to deal with because if you're dealing with the sweeps. You get partial sweeps and you have to like kind of put piece them back together and it gets tricky. Whereas if you use the archives, they're full and complete sweeps and easier to to deal with. So We basically have a lambda running that image that then consumes those based on the SNS notifications. And then based on that, it will Uh we use Redis caches in this case.

Speaker 1: So Redis is actually kind of key here to a lot of this. And I'll talk about that here in a moment because it's actually a key point if you take anything away from this talk. is be really really careful with your Redis caches and the and the client classes because it can really throw a wrench networks if you don't have them lined up correctly. So we then dump data into Redis so that the other applications and other uh Lambdas and serverless components can all consume that data and use it as part of the deployment. And so we use Fargate for all the Django bits. So any uh inference and API, all that's written with Django Rest framework and just deployed and uses uh basically we just as we compute and cache the the builds of the statistical models. We then just deploy those with Redis, you know, publishing those into Redis to be consumed by the other API parts of this.

Speaker 1: So the part I was mentioning before If you are using Redis from Django out of the box, like you're using Django Redis, there is going to be a very Django-specific implementation of how it serializes data into Redis itself. You can override that with a custom client class for your Redis serialization, but it you you don't think of normally doing that. If you're ever going to use this data outside of Django, you're going to need to use some other alternate implementation there. Because they're just not compatible. They store data in differently optimized ways. It's not the standard kind of Redis key values that you would expect. There's going to be other bits kind of wrapped into that that you need to take advantage or take care of or take care to pay attention to. So that was kind of the important pit there about the uh Redis

Speaker 1: thing. And then oh another thing I forgot to mention in here. All along the way, all these pieces are basically using some custom client libraries that we built for the client. So for our client. The uh all the um prediction algorithms and things like that are actually built into a separate Python library that we can now run CI against and run tests against. And then deploy that into the Lambda images just like we deploy them into the Docker images for Django. So they all share a common library across the pieces. And that's where excuse me. That's where the um code artifacts and like the ECR repository come into play is that they all install from a common location that we publish privately into that VPC or into that uh Jen

Speaker 1: or Amazon account. Okay. Initiative if there's any questions. I've got five minutes. Oh plus questions, right. Gotcha. Okay. So we still have time for questions in addition. That's enough time for me to get through the rest of what I've got, which is good. Yeah, please don't hesitate to like ask. Frank can run you the microphone if you've got something you're burning to know about along the way here. Because the next part or last part of the talk is really going to be about what are the next steps. and where we ran into issues developing this. One of the tricky bits with developing a truly cloud native application and trying to do it all locally is it's really hard to simulate things like the SNS notifications coming in for things like the radar sweeps. uh it can be tricky to simulate some of those really

Speaker 1: really cloud specific functions. Excuse me. But uh there's some nice tooling around there. If you have ever looked at local stack. cloud, uh I found this because the people who do PyPy, the Python package archive, uh the warehouse group, they use that in their development environment as well to simulate their cloud. pieces and bits locally without having to be able to to go to the cloud. Uh the real issue here is the amount of time it takes to run things through a CI pipeline just to deploy them up to check them to see if it worked. And then be like, oh, back to drawing board. I mean it feels like the old days of compiling software. So who does that anymore, right? Oh wait, there are people who do that. Uh just us Python people are used to the kind of immediate uh instant gratification. that comes from just being able to run your code right away. So being able to run all those pieces front to back, simulate a NextRad

Speaker 1: la radar image landing. simulate all the data coming into your local laptop, at least in for uh like a minimum number of stations, allows you to run the whole thing front to back much, much easier. And so it really was the big speed up here is going to be in terms of developer velocity and getting more features out the door. Now another thing that's tricky when you're doing local development like this is going to be when you have to interact with uh private uh wheel or Python Python uh private Python packaged repositories. So there are much nicer ways of doing that now, but if you basically can pass in you know use pre-signed URLs or to the package Python package pip and package. URL environment variable.

Speaker 1: Definitely leverage environment variables for all these kinds of interactions. Hard coding you will end you in a very much a place of pain, which is not fun. So now based on what we've all built, the next bit is actually going to be taking this whole stack to the next level as far as Handling things like true streaming data. I mean right now we're basically picking up data as it comes off of a queue and and doing something with it, but actually being able to take in streaming data straight into various pieces of the architecture that may be receiving that. Doing forecast validation based off of other parameters outside of the weather model weather models. And this is some of the additional data that that team was talking about bringing into the mix. And then leveraging the same architecture for doing MLOps. So really this is the next stage for that team.

Speaker 1: is they really changed how they thought about deploying their application now they saw how we transformed the Jupyter Notebook into a fully productionized, you know, serverless uh deployment. They've actually look actively looking for how can they reuse that same pattern to do other bits like MLOps and do additional kinds of predictions like tornadoes or floods and things like that. So actually it's opened up the market for them quite a bit to do new kinds of um new kinds of bits of predictions. One thing here to note, uh that was a mistake, the kind of a path we went down early on in the project was some of these things are possible to do in a lambda, but shouldn't be done in a lambda. The Amazon has recently, over the last couple of years, opened up the capacity and the runtime, the memory, and capabilities of like Lambda

Speaker 1: functions, but they can be very, very expensive. They can absolutely do it, but you're going to pay for it. So sometimes it still makes sense to still spin up an EC2 instance to do certain kinds of work because it's going to be a lot cheaper than if you're trying to do a full serverless route. Or if you want to run your own like Kubernetes clusters or leverage EKS in that case. So lambdas aren't the uh end-all be-all solution for everything. So you just because everything looks like a nail doesn't mean it's a nail. Uh any questions? I think I've got we've still got a good five minutes uh for any kind of questions. Yeah, we're here

Speaker 2: Are are you are you not afraid of seeing a lot of AWS in your stack right now? I mean the the as as the time goes by it seems that Uh one end up being more as a AWS employee kind of thing with all of that.

Speaker 1: So you're you you s y your question seems to be like are am I afraid we're getting too much AWS lock-in on the platform? I mean there's certain decisions you have to make if you want to go fast and actually be able to get to market quickly. And so I think it's a fair trade-off that most of this was to get us up and running quickly and get something that could actually start getting to market. Most of the pieces in there could be reproduced in other platforms, but I don't feel like it would be at this point in that company's life cycle. prudent to do that. Uh you'd be spending a lot of money trying to accommodate like cross-platform or multi-cloud capabilities when I feel like you know I I'm a little bit biased on this one. I am an Amazon AWS hero. And I feel like Amazon is kind of like the 800-pound gorilla.

Speaker 1: And they're probably the most stable uh cloud out there when it comes to if they release a feature, they don't typically take it back. Uh as like unlike some other cloud providers who we will remain nameless

Speaker 3: I have a question that is from the very tail end of what you talked about having not done a ton of serverless functions and I feel like they're often touted as a way to save costs. So yeah what's the sort of heuristic for figuring out looking at like a use case and and sort of knowing like, uh this would be better in ECT or EC2 and like no but that would be, you know, this is a lambda function for sure sort of a

Speaker 1: Oh 100%. I mean there's a there's with with an interesting problem like this You get to see certain things can scale to zero and you can only pay for when they actually get invoked. A lot of these lambdas mainly run at the top of the hour when fresh data is actually available from the weather services. Some of these things may be streaming or the models may take you know multiple hours to potentially build. You have to balance that performance and time and and then obviously now that you're in a public cloud, you have to look at the what the cash impact or the money impact is of any of those kind of design decisions. All the tools and pieces are there to do, and you can do it five different ways, but there's going to be a little more of an optimal way to do it from a cash perspective, or if you're looking at like a performance perspective, you just have to make balance those trade-offs.

Speaker 1: But yeah, serverless is not the solution for everything. Certain things just don't fit that workload. Any other questions? Awesome. Well thank you very much. I've appreciated being here at DjangoCon.