A Management Layer for Scalable, Multitenant Django with Addison Hardy and James Ray

Speaker 1: Good to go. All right. Thank you everyone for coming out to the talk. We're here representing uh JPL. It's the Jet Propulsion Lab. We're part of NASA. And my name is Addison Hardy. And presenting with me today is James Ray. In the audience, we've got two team members, Scott and Lewis, and then a couple team members back at the lab who couldn't make it, Stephanie and Alicia. They were all a big part of this project. I'm gonna hand off to James to talk about uh what JPL is.

Speaker 2: Yeah, so you might ask yourself, you know, what is JPL or the Jet Propulsion Laboratory? And um, you know, while the lab creates robots that go into space We focused on turning the data they get back into cool websites for the public. You might know JPL as the people that hit an asteroid off its course with a dark mission or landed a helicopter on Mars. Um

Speaker 1: I can hold this for you.

Speaker 2: So like I said, um JPL is the place that puts rovers and helicopters on Mars. JPL's main focus has been creating robots to go where humans cannot yet go. Our center doesn't focus too much on human spaceflight. We assist other centers like JSC, Johnson Space Center with hardware, software, satellites. communications arrays like the deep space network. DSN is used by entities around the world to monitor and track things that go into space. This is our campus. We're in Pasadena, California, like two hours up from here. It's where some of us live. Scott Hills from Rochester, New York. We're a very hybrid team. We proved this by recreating our main website a couple years ago with the team from Torchbox, which is in the UK, and a kind of a multi-time zone team

Speaker 2: right at the beginning of the pandemic. Uh so okay, that's cool, but you know, what what does JPL do with Django? We don't put it on our flight hardware, no? Um

Speaker 1: not yet.

Speaker 2: Not yet, but heavily back on Earth. You know, we use that we use Django as a platform to disperse the data that our flight hardware gets You know, we want to enable the scientific community to do all kinds of things, whether it's a climate scientist figuring out um you know, climate change uh trends or whether it's someone just being very inspired by space travel uh or Things about the orbit that are just very, I don't know, uh intrinsic to us as human beings. So, you know, you probably saw the Mars 2020 um All the artwork, all the things, we also um produce that stuff as well. Sorry, no worries.

Speaker 2: And we produce all kinds of collateral, including web experiences , internal websites, business apps. Kind of uh we're the maintainers of a lot of legacy code. I know you've heard that word before, and that kind of leads us into the reason why we made Launchbox in the first place. We run a lot of internal and external websites and web apps. A lot of those were created by many different teams, many different languages, architectures, frameworks. all hosted separately, running different dependencies. Dependency held to the nth degree. You know, uh JPL's been around since the Almost since the Cold War. So we

Speaker 2: sometimes have software that's literally that old. We wanted to create a standard platform. For running web applications internally at JPL where we could enable developers to not have to think about uh configuring SSO the right way NASA wants you configure SSO or you know, renewing your SSL cert and not taking that m government website down and you looking bad. So that's basically what led us to create LaunchBox. And led us to create the management dashboard that we can see everything at a at a glance. And I'll kick it off to Addison to talk about implementation.

Speaker 1: Cool. So The platform that James mentioned, we set out to build it about a year and a half ago. And we wanted to create sort of a standard CMS platform that could be used by different teams in the lab to create content websites. Um we decided to architect it with Django and the Wagtail uh CMS uh package. And one of our big architectural goals was to make the platform multi-tenant. So we wanted to have one sort of container cluster where all of the platform sites host names would be pointed at a single load balancer and all of the containers in the cluster could handle requests for any

Speaker 1: of the sites. So the architecture we came up with containerized using Docker. And for that original platform, we were baking two things into all of our container images. the the Django Wagtail application stack, and then this additional component that we called the management layer. And the management layer was something that we started building about halfway through the project. And we built it to help us manage the multi-tenancy within the cluster to keep all of the site resources separate from each other. And also to take care of like repetitive DevOps tasks. And it also provides a dashboard and API where you can see

Speaker 1: an overview of the cluster state and manage the sites within it. So um platform ran on AWS ECS as an auto-scaling cluster. And the the management layer was was the key component that that that enabled the platform. So what did the management layer provide? It handled all of the container networking. So it would automatically generate Nginx and UWISGI configuration files for each site. Anytime you created a site or you added a hostname to a site or changed anything about the cluster state. And it would manage things like SSL certificates for host names.

Speaker 1: take care of issuing and renewing those. At JPL we have like an internal cert authority. So that was important. for for the internal platform. It also handled monitoring, so it would automatically monitor the sites, you know, status for each site and statistics about requests going to each site. And resources. It would handle database creation for each site, things like Django migrations with each deployment. Um it would go through and migrate each of the site databases and keep them up to date with our Django models. And then synchronization. So anytime the cluster state changed, if you went into the dashboard or made an API call, to add a host name or add a site.

Speaker 1: All the containers would be notified via a Postgres notifications channel. And that would trigger them all to regenerate the configuration for that site. And it provided a dashboard, so an interface for creating and managing sites, managing host names, monitoring the platform status. So this gave uh our team a lot of benefits. We could view the you know sort of the platform state, a single interface. easily monitor the platform health and it reduced operational overhead. So routine ops tasks were handled automatically. Things like like normally at JPL, if you spin up a new site, you've got to uh configure it a certain way for security rules

Speaker 1: Um you need to get an internal SSL cert, configure it for internal SSO. There was a lot of repetitive tasks that were associated with all those different sites we were running before this platform. And so by moving all the sites to the platform. it really started to reduce the the operational overhead for our team. So what did we learn from building that that initial platform? It was a good fit for a lot of different sites that we were running before we built the platform. We were able to move quite a few of them into it. But it wasn't a perfect fit for all of those sites. Even after building the platform, we were still hosting some sites separately. um mostly purely static sites

Speaker 1: and applications that had a lot of custom functionality that it didn't make sense to bake into this this standard uh CMS platform. And and all of those separate sites continue to have that high operational overhead. The management layer that we built. had you know provided a lot of advantages, but it was designed to run that one application type, the the Django Wagtail stack, that that particular code base. So we started thinking, could we extend this management layer and the dashboard and API to work with multiple different site types? So we started working on a new version of the platform and we gave it a new name, calling it Launchbox.

Speaker 1: So Launchbox is a scalable platform for running multi-tenant web services. And you can run more than one web service at a time within the platform. Services are based on a repository URL, a branch name, and an environment name. And sort of the goals that we had for building this new version of the platform were to allow developers to focus on building applications. And let LaunchBox handle the infrastructure, deployment pipeline, and resource allocation. And we wanted to extend the resource allocation to more than just database management. We wanted it to manage resources like the cache and also storage backends

Speaker 1: like S3. So in in Launchbox, the containers only contain the management layer. And service builds are distributed to each container during deployments. So if you trigger a deployment on a service, You're deploying a specific commit SHA , a specific commit. And the container cluster has a build system. That is distributed across the cluster. So all the containers in the cluster can participate in builds and handle different aspects of the builds. And then when those builds are completed, they get distributed to all of the containers. We wanted to allow extensibility through plugins. This was really important because we wanted to open source this new code base.

Speaker 1: And we couldn't keep the GPL specific bits in the codebase. So we created a plugin system. We also want to have first-class support for Django apps because that's the primary stack that our team's using. So we built a helper package for Django. You can add it to any Django project. And it makes it really easy to configure that Django project to work with Launchbox resources. I'll talk more about that later in the presentation. Architecture-wise , Yeah, so in the old platform, sites were always backed by the same service type. It was that Jingo Wagtail application. The new platform. You choose from the the available service list when you're creating a site.

Speaker 1: And services can tell LaunchBox how they want to be run with this config file called launch. yaml. So that limitation of the previous platform where it could only run the one code base, it's because the management layer was pretty tied to how that codebase was set up. We switched to this this new architecture where the service code base, you put this launch. yaml file on the root of a repo that you want to run And it tells Launchbox about the environment variables your service needs, the resources and routes and things like that. So I want to give a quick demo of what the launchbox uh dashboard looks like.

Speaker 1: Alright, so when you first sign in, you land on this dashboard interface, you're going to land on the sites page. And if it was like a new environment you had just set up, the first thing you'd want to do is go create a service. We're going to work on adding like a first-time user experience, but Go to the services tab and add a service and again it's based on a repo URL, a branch name, and then an environment name. And whatever environment name you provide is going to be passed to the service via the environment environment variable. And then once you've got a service, you can go in and view it. It'll pull in all the recent commits.

Speaker 1: on that repo and branch. And then you can click a deploy button over here and deploy a commit. And then on the deploys tab you can see we've got a new pending deployment. And you can view that deployment and watch it go through the different deployment steps. So this is the the service part of the deployment. You can see it's cloning the repo, parsing the launch. yaml file. It sets up a Python virtual environment, installs Python requirements. And then once the deployment finishes, it automatically gets pushed to all the sites that use that service type.

Speaker 1: And over in the sites interface, that's where you can manage things like host names. You can go to the resources tab and see the resources associated with that site. And I'll explain more about this later when we talk about the launch. yaml file. But these resources, database, cache, and storage, are coming from a section in that launch. yaml file that tells LaunchBox. Hey, this service type needs Postgres database, a Redis cache, and a S3 bucket. So I'm going to hand back off to James. This is the current dashboard that we're currently using. But James has been working on a brand new dashboard that we're hoping to release in about a week. We open sourced the repo, we open source launchbox about 30 minutes ago.

Speaker 1: So that your first time last time. First last one?

Speaker 3: Was it all already lost?

Speaker 1: So we've just been using uh we've been using that original platform we built internally, and we're getting ready to uh in the next couple months switch that platform over to run on this new code base. And more than likely, other development teams within GAPL will create their own launchbox environments to run their applications.

Speaker 2: Yeah, so the new dashboard, we're trying to simplify things, make a kind of a growing event feed UI, less small buttons, bigger click points. More descriptive areas for like you know at a glance data, really power user stuff. You know, you can see your list of deploys, quickly see which one failed. Dig in Y go create a new site pick which service you wanted to run on. A lot of the functionality hasn't changed, just kind of putting new gusto on it. We have a full uh API spec as well that lives within the dashboard. We're definitely looking for contributors.

Speaker 2: So feedback, contributions. the community to grow around this. I think of it like um open sourcing cPanel or something. So it's really exciting.

Speaker 1: Yeah. And uh the other thing about the new dashboard, the the current one is uh it's it's sort of uh static web um It just uses like static uh vanilla. js and HTML. The new dashboard is based on Nux3 and Vue and hopefully uh will allow us to iterate on it faster. Um we'll switch back to this one for the presentation. Yeah. Well, that's a few months.

Speaker 1: Cool. So the the key aspect of the platform is really this launch. yaml file. That's the most important thing from the platform's perspective because it tells the platform how to run your site or your application And it's also the most important thing from the end developer perspective because this is how you tell the platform how your application works and the resources and environment variables that it needs. So the watch. yaml file uh it's got five sections. Uh we're still working on the last one workers, but um the first four, uh there's an environment section So you can provide base environment variables and then environment specific variables.

Speaker 1: So an example of that might be in base you have debug true and then you have an entry for production with debug false. There's a resources section. There's a phases section. It's got two parts, service phase and site phase. Those run during deployments. So the service phase is going to run once. That's where you do things like install Python requirements. The site phase is where you do things like run database migrations. The site phase is going to run for each site that uses that service. And then routes is where you define network routes. So to sort of explain all that a little bit more, here's an example of the environment section. So I've got that whole debug true-false thing going on, and then just an example variable

Speaker 1: And everything that you define here is going to get passed to each of your sites that are running. So if we created a service that had dislaunch. yaml file and we set that environment variable to production. then it would be debug false and the example var volume one would get passed in. The resources section, so each environment has a dedicated Postgres server, Redis cache. and an S3 API compatible storage provider. Locally that's achieved with a Docker compose file. It spins up a PostgreSQL Redis and Minio which would provides that S3 API compatibility. The resources section looks like this. So you can have as many resources as you want.

Speaker 1: You can have multiple Postgres databases. But the most important part is the name you give the resource. So this one's named Database One. And that's how it's going to be referenced in your application. And over on the right, you can see we've got four sites that are based on this service And so this launch. yaml file is going to yield four different Postgres databases that are named with the site ID underscore and then the resource name. And all that stuff gets created during a deployment if it doesn't exist yet. For the phases, In the service phase, we're installing Python requirements. And in the site phase, we're migrating the database, updating the index.

Speaker 1: The way to think about it is that the service phase is things that you know are You only want to do once and aren't specific to any site's individual data. And then the site phase. Each time the site page runs for each of the sites, it gets passed extra environment variables that are unique to that site So each of the resources generate a set of environment variables. Like for this one, database one, it would generate five environment variables that get passed to each site. And they follow a set pattern. So it's like LB underscore the resource name, database one

Speaker 1: underscore, and then the uh parameter name. So it would be things like the database name, username, password, host name, that kind of stuff. And it's all documented in the documentation. The route section is where you define the routes. We support two right now, WSGI routes and static. So this example of a WSGI route, you're just providing the path to your WSGI file and the variable name in that file. And then a static route, you just point at a specific path, and it can be either a folder or a specific file. And then workers, we're still working on this, but it's going to allow you to run basically cron-like jobs.

Speaker 1: So the deployment system that I was referencing before has these two main parts. The first part is the service part of the deployment. And all of these parts get created as sort of pending jobs in a queue. So when you start a deployment, it's going to add a pending service deployment job. And then once that's finished, it's going to add jobs for each of that service's sites And any of those jobs can be handled by any containers in the cluster that are not busy at the moment running another job. And it's architected in a way to be eventually consistent. So each container, when you do anything like add a host name or

Speaker 1: deploy, Each container will is guaranteed to be eventually consistent with that new configuration. And that happens pretty quickly. The second part is for the sites. So we generate environment variables for each site, create the site resources. The way that works is if you've changed your resources section in your launch. yaml, if there's any new resources, or it's like the first time you're deploying. it's gonna detect that those databases don't exist yet and create them. So yeah, each container participates in the deployment process. Processes architected to achieve eventual consistency.

Speaker 1: Work is shared across the cluster And the architecture is designed to prevent timing issues. A good example is what if a site was created during a deployment? So a deployment starts and If we immediately added a bunch of jobs to the queue, like the service deployment job, and then jobs for each site, that would be based on the current sites that exist. If someone were to create a site while that deployment was running, there wouldn't be a job for that site. And you could add one, but The architecture that we landed on is basically a loop that runs on each container. It runs as a separate process. And it does four steps in order. So it looks for completed service deployments that the container doesn't have yet, completed site deployments,

Speaker 1: pending service deployments, and then pending site deployments. And it does that loop in order. And if anything fails, it restarts from the beginning. And The the effect of that architecture is that you know no matter what order you do different actions in the cluster is guaranteed to be eventually consistent. Or at least that's that's the goal. Networking rise, each container is running Nginx and UWSKI. We generate that network configuration during container startup, during deployments, anytime settings change. A little more info on that. So for each site, we parse the routes from launch.

Speaker 1: yaml. We generate an Inginx site file for the site server block for each host name and then within each server block we have a location block for each route and then we generate a USGI Vassel file for each USGI route, each WSGI route. And we're running Yawisky in what they call emperor mode. So um, and then the Yewisky workers are called vassals. Um And each one of them has a unique config file that points at a particular deployment, particular set of environment variables. And uh yeah, the the deployment is handled by that loop. But other types of changes like adding a hostname

Speaker 1: are shared using Postgres notifications. We send them in JSON format And we use different channel names to represent sort of different data types like sites, services. And there's an example of those. The last thing I want to talk about is the Connect API. So One of the key things that we were trying to avoid with this new platform was not having to do sort of SSO or LDAP or the equivalent integrations in every single application code base So we wanted Launchbox to provide a centralized integration and then allow services to easily hook into those integrations.

Speaker 1: So you can build plugins for all providers, identity providers, and actually CERP providers as well. And then to connect with those services can use this internal Connect API. So it's only available within the container. And each service, really each site, gets passed a custom LB Connect API endpoint that's just for it. And it can query that endpoint to find out like its current resource information, but also to talk to any of those API integrations. And that means that it's really simple to like add SSO or LDAP integration to your services without having to have a lot of duplicated code.

Speaker 1: And then we built this Django LaunchBox plugin which talks to that Connect API. And um Does those API calls for you. So it can return database, cache , configuration information in the Django settings format. So with just one line you can set up your Django database, Django Cache. And yeah, it provides these helper classes. So resources is for configuring databases and caches and storage. And then two more classes to talk to identity or auth providers. It also optionally provides routes that LaunchBox can use to query

Speaker 1: Django user data So if you want Launchbox to be able to display user data within the central dashboard, you can enable these routes. And they work with any Django user model. And Launchbox can fetch you know, all of your users create update users from the dashboard. So

Speaker 2: what's next? Um, you know We open sourced it. Where do we go from here? I think we, you know, we want the workers continue working on that, get cron jobs running, get it to where you can basically run your everyday application in there and maybe replace that with some system you're running today. We want to work Smarter, not harder. You know, like really what our goals on this thing is to not have all this code that we all know It's just it's out there in multiple places for no reason. It all it all runs the same functions. So that's like don't repeat yourself is kind of number one principle. We want to adopt the Django ORM for the management layer itself.

Speaker 2: That's definite conscious decision. We want to move into that. Allow multiple Git remote providers. Beef up the docs and along with that examples and tutorials. And we're we're looking to build a community. I mean we're we open sourced it for for it not to just kind of get dusty and We all forget about it because while we we have our goals inside of JPL, our management also has goals inside of JPL. So we're trying to align all that together and show that we can also contribute back to the community. And uh the new UI. It's been really fun working on that and getting feedback and really making something that you look at it first time and you know it's oh that's

Speaker 2: that's the launch box thing running. And yeah, watch us contribute. Hopefully we'll be contributing on this project for a long time.

Speaker 1: I want to share one more aspect of the demo that I forgot. It's kind of the most important part, which is these sites actually running. So uh yeah. Um so we've got two sample services here uh and one site based on each of those services. So the first one is the bakery demo. So this is the Wagtail Bakery demo. And it shows off a bunch of different Wagtail features. And Integrating LaunchBox with this repo involved very little changes. Basically took the Wagtail Bakery code, added a launch. yaml file, and and then updated the the Django settings to use that Django Launchbox package to configure the database cache and S3

Speaker 1: storage. Um let's see. The other one is a Launchbox demo service. So this is one that's in our documentation. It deploys really quickly and it kind of shows off what Launchbox can do in terms of the routes. So the home page is served by this pages route, just serves a static folder. The API route is actually a bottle API. Launchbox will work with anything Python that's WSGI compatible. So there's a sample API here and this is bottle. And then this one shows just a single static file.

Speaker 1: So static JSON file. And For things like JSON for all these different file types, Launchbox determines the right or tries to determine the best content type header to set for the different file types. And uh yeah, so I just want to make sure I showed that.

Speaker 2: Nothing else. We we have a few minutes for questions. Um Anybody else anything?

Speaker 4: Curious what open source license that you guys chose to use

Speaker 1: MIT.

Speaker 2: How do you guys handle secrets management?

Speaker 1: It's a good question. So obviously you can't put the secrets in the launch. yaml file. We're planning to add a new tab to the um New tab to both the services and the site sort of detail views that allows you to add secrets. And those will get passed into your sites. One big benefit that we found in in sort of experimenting with this is it it's really cut down on the number of secrets we actually have. since all of the database uh

Speaker 1: configuration information, for example. Um is being handled by launchbacks and generated automatically. You know, we don't have to specify environment variables for things like the database host or username and password. But yeah, absolutely we're going to be adding a secrets section.

Speaker 5: Have a uh meta question. How do you launch LaunchBox?

Speaker 1: It's a good question. Um so Uh if you coin down the repo, which uh let me go back to the slide. The repo is right here So on the left, that's the main launchbox repo, and on the right is the Django Launchbox plugin. Clone that down, and if you check out the documentation, it walks you through setting it up. Mainly it's just building the Docker containers, building the Docker container and then running it with Docker Compose. And that'll bring up the Launchbox container as well as a Postgres, Redis, and MinIO container. And then you can there's a make

Speaker 1: file and you can type like make dashboard and that'll bring up the dashboard for the first time

Speaker 2: There are some getting started guides um and more being polished, I think so.

Speaker 6: Yeah, looks very promising. Thanks. Uh I hope I didn't miss it in the presentation, but is it possible to use the system to manage multi-region deployments?

Speaker 1: So right now we have mostly been running LaunchBox locally during this development process. but it's currently set up to be deployed to a ECS cluster on AWS in one particular region. Um we're planning to open source our Terraform config for doing that. and hopefully in the next week. But um I definitely think that's something that we could work on adding support for over time. But a as of right now it's it's

Speaker 2: single

Speaker 1: designed to run in a single although I guess if um yeah if they were all pointed at the same database then it would work Um it's a good question.

Speaker 2: Think about the underhand.

Speaker 7: Any more questions? Um actually I will always think one in if the what Wait for other people to ask questions. Um the context of which I've normally heard multi-tenancy used is in the idea of you've sort of described here, yeah, my bakery and your cheese shop as two different sites that need to share uh SSO resources and and And things like that. I've normally heard multi-tenancy used in the context of uh my bakery and your bakery as two white boxed versions of the same underlying application. Is that a use case that you support at all? Or

Speaker 1: yes. So um it really depends on how you architect your your service applications, but Sort of launchbox ideally would like your services to support multi-tenancy. And we've tried to divide design like the launch. yaml file and sort of the the way that services integrate with Launchbox to support the idea that you can create multiple sites based on a single service. And just as an example of of that.

Speaker 2: Hopefully it's a little open-ended on that way too, because you could You know, my cheese shop could be a fork of my bakery even. And those are two different services now.

Speaker 1: So this this bakery one site is based on the bakery demo service and It's a branch that we modified a little bit. We added a launch. yaml, updated the Django settings. That's really all the changes we made. And so I've created this Bakery One site on that Bakery Demo service. And if you go to the resources tab for the site, all of these values are site specific. So if you look at like the database name, It's got the site ID, underscore, resource name in it. Same for the cache key prefix and the storage bucket name And if you create another site based on bakery demo, if I call this one bakery too

Speaker 1: , create this one The resources that are getting passed in are specific to Bakery 2. So as long as your code is um you know configured to talk to the launch the resources that you specify in your launch. yaml and you can either do that through the the Django Launchbox plugin or you could just use these environment variables listed here directly then any command you run is like in that site phase of the launch. yaml is going to be specific to that site. It'll be talking to that site's unique database.

Speaker 2: And it works really well for intranet sites where you have tons of sites that basically need need to run the same code base. But they don't want shared resources. They don't want a shared media library or even a shared database at that point.

Speaker 7: Any more questions? All right, in which case, thank you very much, James.

Speaker 1: Thank you. Thanks everyone.

Speaker 7: And a small token of appreciation from the organizers. I mean , I'm gonna find a bell fellow.