Django for IOT: From Hackathon to Production by Anna Schneider

Speaker 1: Come on, no.

Speaker 2: Yeah, so I'm Anna. I'm proud to be speaking opposite another Anna. So if you don't want to hang out in the overflow room, you have two Annas, go see. So I was introduced to Django at the hackathon where my startup was born. So we're called Wattime, and we use Django to fight climate change. So we do that by enabling internet connected devices like smart thermostats or the electric vehicle charging stations that you see pictured here to turn on when energy is clean coming from the grid and turn off when it's coming from dirty fossil fuels. Another side project that I'm working on right now is the Unconscious Bias Project. And for that we're collecting evidence-based research resources to help you identify, reduce, and cope with the effects of unconscious bias in science and tech.

Speaker 2: But neither of those are what I'm actually talking about here today. So I'm talking about Django for the Internet of Things, from hackathon to production. Uh what? That was a lot of the words. Let's break this down a bit. So Django, we all know and love everyone's favorite web framework. The Internet of Things. So this is when something that you don't normally think of as a computer, like a thermostat or a car , it can receive and transmit data and even respond to controls all in real time. And when some people talk about the Internet of Things, they talk about writing the embedded code that runs on the device. But a lot of the devices that you can buy these days have uh APIs at the bench provides. So today I'm going to focus on the use case where you're writing code that interacts with that API.

Speaker 2: Hackathon. So hackathons are weekend events where you form a team, write a bunch of code, drink a bunch of coffee andor beer. These usually winners and prizes, and everyone has a great time. And as IoT has grown in general, it's become particularly popular for hackathon projects. But because hackathons are so short and compressed, they're just the weekends, they're kind of notorious for producing really bad code. So what if you want to put your code into connection with real users? So reliability matters in a different way for Internet of Things apps than for a lot of other things you might work with on. So say you have a Wi-Fi enabled lock on your house. What happens if the server goes down? Can you still get into your house? Can anyone get into your house?

Speaker 2: So you'll want to be a little bit more careful when you're coding for production. Eat your fruits and veggies while you're drinking your beer. And make sure that your code is extensible and testable and uses dependencies that are reliable so that when you ship it, you can trust it and go have fun. So today we're going to talk about some design patterns that have worked for us at Watttime while writing and deploying Django projects. That monitor and control Internet of Things devices using the vendor-provided APIs. Some things will streamline your work for your next hackathon, and other things will help you as you move into production. I mean as we go, I'll be using this emoji for hackathons, this one for production, and this one for anti-patterns to avoid.

Speaker 2: So there'll be a couple of those. Great. So imagine it's Friday night. You're at a hackathon and you have this awesome idea for an IoT project. Where do you start? So usually start with some models. So for comparison, let's think about the standard books app that you may have seen in the tutorial. We have an author model with a name and the hometown attributes, and we have a book model with title, a foreign key's author, and the year. That the book was published. An analogous IoT model would look pretty similar to start with. So instead of book, we have a device with a name and a location. And instead of author Uh sorry, other way around there. So instead of device we have an uh instead of author, we have a device with the name and the location. Instead of book, we have an observation

Speaker 2: with a value, a foreign key back to the device and the timestamp. So even though I said the words wrong, there's a lot that's similar, so what's different? The main change to the device model is to add the vendor provided I tags, something like a MAC address maybe You'll almost always need this to pass on to the vendor's API to uniquely identify what device you want to be talking to. So this might be analogous to the author's social security number or other national ID number. The bigger sort of mental change when you're writing these apps is thinking about uh the observations. So IIT is fundamentally about time series data In the way that most other Django starter projects aren't. And the number of observations that you'll be dealing with grows a lot faster than the content in the early stage of a typical content-driven app

Speaker 2: So at Watttime we tend to deal with medium-sized data since we pull uh new energy data from the power grid every five minutes. So it's not like huge data Project might enter big data territory much faster than you expect because many sensors have readings that pull data extremely quickly. I'm gonna even that my very first hackathon. Adding a database index to timestamp was the difference between our demo timing out and it actually running quickly and small. So thinking about the underlying structure of your database tables might be more important. And the last difference I'm going to talk about is you'll want to be storing different types of observations in a lot of of cases. So with books, it's a good bet that the title is going to be well represented by character data.

Speaker 2: But with IoT you'll probably want to be storing both numerical data like how much energy is being being used by the device and maybe Boolean data like whether the device is on or off, possibly also care data like display messages, or data with choices like the color of a light bulb. So you might want to use different models for those different data types. Great, so we have a basic data model on to views, right? Wrong. So instead of views, which are for interactions with people, our basic unit of interactivity is going to be tasks, which are for interactions that can be triggered without a human pressing a button. And to see why this is important, let's think about the normal vanilla MVC In this case, your app is only responsible for sending data to a user if they request it from you.

Speaker 2: But most third-party APIs, including almost every uh API I've used for an IoT project, they put the responsibility on your app to make the request to them whenever you want to push or pull data. And of course a big part of the point of the Internet of Things is machine-to-machine communication, doing what the user magically wants without making them ask for it. So it's crucial that all the important stuff that happens in your app can be initiated outside of a user-driven request-response cycle. And pretty much the entire rest of the talk today is going to be focusing on how to make that happen because it's pretty different to the normal views. So, what tasks do we need to be doing? The two basic kinds of functionality for an IoT application are going to be monitoring, so getting the state of the device, and control, changing the state of the device.

Speaker 2: So for each kind of observation you have, you might want a separate task function to get the data and to set it if the vendor lets you modify it. Some APIs don't let you modify as many attributes as you might want to, so check that before getting your heart set on specific applications. You'll also need a driver task. So let's call it the do something awesome task that's going to stitch all of this together. So okay, that's more or less readable. Cool. So for this particular demo application, we're gonna pass in a device. We're going to based on some external data decide whether to turn it on or off We will pass that isON

Speaker 2: message onto the device using the device's API. We will save that new status to the database. So that's what's happening here. It's kind of funny, after working on Internet of Things apps for a while, the part where you interact with the device itself has become less exciting to me. Because like that's the part that's kind of the same between most apps. Like if you're talking to a light bulb, that part's gonna be the same. But the part that I get excited about now is the decision criteria for why you're turning that light on or off, why you're actually taking the action, because that's the part that makes your app unique. Um yeah, there's the awesome part. So once you've decided what sort of awesome thing you actually want to do, uh sorry.

Speaker 2: There we go. Yeah. So writing the code is going to be as um just as simple or as hard as your idea. But there is one gotcha that we're going to talk about quickly. So we're building up to running these hasks asynchronously. So in a production situation, you can't necessarily count on the state of the database being the same. when the task runs as when you called it. So here, since we're passing in the device object as the parameter, what happened with the database may be different by the time we get here. So a better convention that I like is passing in the primary key to the device. And then you can make the query to get exactly the right object that you want from the database right when you need it.

Speaker 2: Also, if you use a convention of returning primary keys to new objects instead of something like error codes, then you can chain different tasks together really easily. Great. So we have our models, we have our tasks. Let's put it together into apps. So in a hackathon situation, just toss it all into one app. No one really cares. But if you want to organize it a little bit better, I like to split up my apps this way. So devices has the device model. Observations might have attribute, status, anything. else. Interactions is where the tasks go. And it'll have the wrapper around the vendor client API in its own app.

Speaker 2: So this might seem like a lot of structure for now, but each app will grow over time as you start adding views for adding and removing devices so the user can manage what they have. If you want to start adding fancy D3 dashboards, you can put those views in the observations app. You're probably going to want to log the interactions that you're making with a device, so that can go in the interactions app And if you want to start having multiple kinds of devices that you're supporting, then you can swap out the vendor APIs pretty easily. Cool. So you have models, you have tasks, there are in apps. Let's deploy it But how are we going to deploy these tasks? Uh running a web server isn't actually going to help us here. What we really need is something that acts like, if you're familiar with the Unix concept of cron or cron

Speaker 2: tabs, where we want something like that, but that can run in a distributed cloud environment. So we want to be able to run our tasks anytime we want at frequent and deterministic times. and outside of the request-response cycle. So and this is the situation, this is the part of the code where hackathon and production differ most. So I'm going to talk about two different ways to run these tasks, both of which you can deploy to Heroku or a lot of other cloud environments. So let's do the hackathon way first. I'm using Heroku has an add-on called scheduler, and we'll use that to run management commands. So you're already familiar with the Django management commands like make migrations or run server probably. And you can write your own management commands too.

Speaker 2: There's boilerplate in the Django docs, and you essentially just copy that, and that's a great way to get started. In this case, we'll just have a simple management command that wraps the task that we already have. And then to get it running on Heroku Scheduler, there's this free add-on with a cute little web interface to set up to run your tasks. So it's pretty easy to set up, which is good. The downsides to using this, which make it only suitable for hackathon situations, is that it has very limited frequencies that it lets you on the tasks So you can do daily, hourly, or every 10 minutes. But if you want to do every five minutes, you're at a lock. And it's also, they only operate as a best effort service. So I have noticed it skipped some tasks if you let it run long enough

Speaker 2: and then are sad when you don't have that one data point. But it's awesome for hackathons. If you need to go into production and do it for real, you'll want both more flexibility and more reliability than these free add-ons can give you. So as the fits production, you gotta eat your veggies. So we're gonna use a package called celery. So what is Celery? Celery is a distributed message queuing system for asynchronous stuff. So most tutorials for Celery focus on how it's really good for long-running event-driven background tasks, like maybe sending the user a sign-up email after they signed up for your site. That's something that doesn't come up as much in tutorials, but that we're going to be using it for today

Speaker 2: is scheduling periodic tasks. So in a generic app , you may want to run some daily analytics like Clinton was just using for a aggregating all those uh data for the police calls. Uh but here we're going to be using it for doing our cron in the cloud solution. The main reason I don't recommend Celery for hackathons is that it complicates your architecture a lot. So do a brief overview of everything that you need to have set up to run a Celery app. So we have our web servers. Those can trigger the event-driven tasks. We also run a scheduler server that can trigger the periodic tasks. Both of those put messages on

Speaker 2: a message broker transport queue. That's what celery calls it. But it's just a first in, first out queue. where wherever tasks get put on, they can then get uh consumed by worker servers that actually do the work of running uh that task function. So maybe they'll go out and have external APIs, maybe they'll put things in the database, they'll do whatever you need them to do. And if there's a return data from that function, it'll go into something called the result store. There's a bunch of different options for backing services for both the queue and the result store. Redis is a popular one because it's really good for both of them. Rabbit A and QP Yeah, AMQP services are also good for the queue, but not for the results

Speaker 2: door. So to make it more concrete, For our IoT application, we're going to have every few minutes our scheduler is going to put a do something awesome message into the queue. Our worker will pull it off. It will Decide whatever you want to do, maybe turn on our light bulb. It will send the API request. to our vendor to say to an outloite bold, the vendor will return, yes, it's on now. Good job. The worker will then store that new status in the database and return the primary key to the response. So once you've got all that architecture configured, actually changing our plane of Python tasks into

Speaker 2: Celery tasks is really easy. You just add a stackerator. And then to configure when the tasks are going to run. There is this dictionary syntax. And the syntax for defining the periodicity itself is pretty similar to the Unix cron tab. So you can have really flexible intervals anywhere between the once a minute and once a year. including things like day of the week or like only at noon on Tuesdays or anything like that. And after all my talk about passing primary keys around, you're probably tempted to do a query here and pass the results as arguments into the task, right? Unfortunately, this doesn't work very well

Speaker 2: because this dictionary is only evaluated once, like anything else in your settings file, for instance. So you can only use static arguments here if that's an option for you. If that's not an option, one good sort of workaround is to have a wrapper task that spawns all the daughter tasks. you need. And so your database call only happens here. Cool. So to that, we have a final uh production ready project. So we have the same apps as before at the bottom. The schedule file has been added to the interaction. And then there's a bunch of other boilerplate to configure all of the salary architecture that's in the

Speaker 2: bold. So there's some pretty important subtleties math that I don't have time to get into today, but uh the docs are good. And If you didn't memorize all that, there's also put together a cookie cutter template. Github. com slash ASCHN slash cookie cutterjango IOT So this has all the boilerplate configured for you. It has all of the data models configured for you. All you have to do is write your actual interactions. with the vendor API and whatever you do something awesome is uh this should uh make things a lot faster for you to deploy either with Selloue or with the management commands. And it's all configured to run on Heroku So hopefully this will be helpful for your next hackathon.

Speaker 2: So what have we learned? IoT is fundamentally about time series data in a way that most other Django starter projects aren't. So that'll mean bigger data and that will mean you have to think a little bit differently about which of your models are going to be big and which of them are going to be small. IoT is also fundamentally about asynchronous processes instead of the standard MBC request. response cycle. So think about putting the actually important stuff not in views, but in somewhere that can be accessed through tasks or something else. To run those tasks, an easy but really rigid way that's going to hack down situations is Heroby Scheduler. And the flexible but more complex way is using salary.

Speaker 2: And I keep talking about Heroku just because that's really common for hackathons, but celery of course can run anywhere. If you want to get started using some of this better and faster, then Cookie Cutter Django IoT is ready for your perusal. And just to get a little more philosophical for a sec, I'm getting back to the keynote this morning. IoT is typically thought of in very privileged situations. It makes life easier for people who already have it really easy. But a lot of time we think that the Internet of Things can be more powerful than that. So we like to use IoT for environmental applications, but I encourage you to think about ways in your life where

Speaker 2: having devices respond to what's happening around them could be much more broadly impactful Uh thanks. Uh I think I have a few minutes for questions.

Speaker 3: Um so uh I tried to have two. Um the first one was the time series data and the primary key? Were you saying make the time the primary key and why? Well like I didn't you kind of went over that pretty quick. I didn't

Speaker 2: Sorry, um so I haven't actually tried doing it that way. Um I just use the standard Django primary key.

Speaker 3: Oh and then then the other one was why use the celery um like what benefit did you find using the celery

Speaker 2: So if you're running on one of these platform as a service situations, you can't access cron.

Speaker 4: Oh I I guess he might have answered that question. I understand that Celery has the advantage of the uh of uh queuing for asynchronous tasks. So you have the consumer-producer model rather than an invocation. But why Heroku versus Kron? Is that also, is that also Because you may not have access to cron on the d on the devices.

Speaker 2: Um I'm not sure I quite understand.

Speaker 4: Well in other words, well just setting up a job in a cron tab and invoking invoking invoking a uh like a a con a uh Django command rather than having Heroku uh invoke the command.

Speaker 2: Yeah so if you're running in an environment where you have access to con and you know that you're going to only be running on that machine then cron can do the job for you. But in some sort of a more distributed situation where you can't rely on having access to those. The

Speaker 5: question about running celery in production and and how you what you're doing I guess to monitor the gas and making sure that they're running and running successfully and all of the smoke circles and

Speaker 2: Yeah, I didn't have a lot of success with celery flower, flower, however you say it. Um I wanted to like it, couldn't get a lot out of it. Um I actually don't use Redis for my production situations I use uh cloud amqp um and other uh amqp services uh and I find the the monitoring on that is pretty good but again not quite as much detail I want. So lately I've been looking into using Labrato and some of those sort of logging things. One nice thing that I just found out about Labravo is that you can uh You can send alerts based on when a line show up in your log file. So either based on the content or if it just like if your queue stops recording for 10 minutes, then you can have Labrato trigger an alert based on that.

Speaker 2: So I'm my internet is setting that up as we speak today, so I'm pretty excited to see how that works for us.

Speaker 6: Is there a reason I I don't think you use the task or periodic task operators from solar? Is there a lot or like

Speaker 2: sorry what?

Speaker 6: Um did you use the password period task? Those are really helpful.

Speaker 2: Yeah, so uh the the task decorator here?

Speaker 6: Yeah, there's a periodic task it writer where you can just feed it the proncav object and is there a reason not to use it?

Speaker 2: Um I haven't because I don't like to have the tasks only be available periodically. Um so if you do it this way you can call it from multiple places uh in different ways.

Speaker 4: What's the biggest challenge you've s had to solve with doing an IoT project? Um

Speaker 2: I guess uh one application that we're working on right now is controlling smart thermostats. Uh so we have uh data coming in about whether the power grid in Chicago uh how much uh carbon pollution There isn't that electricity. So the API that I'm working with for that partner is a SOAP API. It 's been incredibly annoying to deal with. So that like little line where it's like status is actually like a a huge, really annoying uh method there. So So picking your parties carefully is important.

Speaker 4: Never mind another question. I noticed early on in your slides you you had a A set status and then a status set create. And I'm wondering what's the purpose of duplicating that? In other words, I'm thinking about the EchoB thermostat where Everything you do, it just queries everything again from scratch. And I wonder if if that model avo avoids desynchronization of the state of the device versus the the the our the database's view of the state of the device rather than

Speaker 2: getting those out of sync can be a problem. Um for some devices they only let you uh access maybe like ten days of history historical data and so having your own record of it can be really useful too, even if that does create the additional challenge of trying to keep them up to date.

Speaker 4: just depend on asking re-asking the state if that doesn't take too long rather than depending on the state in the database being

Speaker 2: Yeah so if you are able to ask the device for its own historical data that is likely to be more app So if that's an option, do it. Last question, Peter.

Speaker 5: Did you have to implement the option to prevent all instances of the task from trying to mutate some device states at the same time?

Speaker 2: Um that is something that celery has some helper functions for um that is complicated. Yeah