Security strategies for multi tenant applications - Raphael Michel

Hello and welcome to this Saturday morning talk about security strategies for multi-tenant applications. My name is Raphael, I'm a software developer from Heidelberg, Germany. previous series of JengaCon Europe and if you're watching this you're watching a recorded version of me and the real-time version of me is with you in this chat and can answer questions as we go. I will also be available at the end of my talk for good for live video questions Before we go into the security strategies, I want to make sure that we're all on the same page what a multi-tenant application is, so we're quickly going to define it and give some examples So, multitenancy is commonly referred to as the concept of a single instance of a software serving multiple entities.

Those entities can be individual users or entire enterprises or other less organized groups of people, but the idea is that those entities use the software in a pretty separate way. So an application that is not a multi-tenant application is a social network like Facebook or Mesodan or Twitter where everybody operates on the same databases where everybody can see and interact with everybody else on the platform. An example for a multi-tenant application would for example be Slack, where you are inside your own workspace together with others from your organization. and you can interact with each other but you usually there's a footnote you usually don't

uh interact with people of other organizations who might also use the same product Basically, multi-tendency covers most software as a service businesses out there, at least most of the ones targeted to business users. In our case, we're developing Pretix, which is the ticket shop application that you used to sign up for this virtual conference. It's an open source event ticket shop. And one installation can power many events organized by many different people. So it fits the definition of multi-tenancy that we just gave. There are now multiple ways to implement this multi-tenancy

inside our database layer. The simplest one would probably be to separate clients on a database level. We could just have a separate database per client. This gives us a really clean separation between our different clients and we can interact with them separately and make sure no data leaks between them. However, it requires us to open up multiple separate connections to our database because with most interbase systems it's not easy or at least not uh what quick to switch between databases in running connection. And it's also very hard to do in Django because Django um has a static concept of database connections and it's not really easy to define a list of databases that we have for multiple clients.

on the go with Django. It's also pretty bad for performance probably because not having ket not being able to reuse the same database connections means we can't really use connection pooling so we're opening up a lot of database connection so that that's not really nice. The next approach that is actually in use unlike database level separation which I haven't seen with Django there are probably people who do it but it's very uncommon is schema level separation To understand schema level separation, you need to know that on PostgreSQL QL there are multiple levels Of hierarchy in your database. There's the PostgresQL cluster, which is your server or combination of multiple servers. There is your

database, and you can have multiple databases in one cluster. And inside that database You can have different schemas and those schemas contain tables and indices and views and so on. So usually you only had one schema. That is called public and all your tables live in the schema called public. But you can have multiple schemas within one database So unlike multiple databases, they are still the same database. You can connect with them at once and you can also query from them. in the same query. But I'm like, but they are separated in a way that you can have different tables for different schemas. So they kind of work like namespaces for tables.

So using schemas and using one schema for each client allows you to have everything in one database but have everything clearly separated by client. It also possibly allows you to scale up easily because once you grow too large to keep everything on one database server, you can just move the different schemas of different of different clients to different servers. And it has a very little overall impact on your code base because you just need to make sure that you're selecting the correct schema when you're querying your database. However, it works only in PostgreSQL, there's no concept of schemas, for example, in MySQL SQL, and it's very hard to do cross-tenant queries

For example, I used Slack earlier as an example for multi-tenal deputation. So a couple of years back, this was 100% true because inside Slack you could only interact with people from your own organization. However, in recent years, Slack has introduced a feature called shared channels where you can share a Slack channel between two different companies. And so even if you don't have it now, it's very likely that one day you will come across a use case where you will need to run queries efficiently across multiple tenants. And that again is very hard to do with schema level separation. So for those two reasons, schema level separation can be a very nice and clean approach, but wasn't an option for us. In pretex, we want to be able to run

On MySQL and SQLite as well, and we want to be able to do cross-tenant queries. If those restrictions don't apply to you, then schema level separation might be a good way. and there are multiple different uh in different implementations for Django out there including Django PEG schemas, Django tenant schemas, and Django tenants. So with those out of the way, the only way that really let that is really left to us is row level separation. And by row level separation I mean that with every row, with every model instance in our database We keep track of which talent it belongs to. For example, if we have a client model that represents our talents, our customers, and then we have a product model, like in our ticket shop

case where we have Different products per client, then we would need a foreign key from the product model to the client model, so we'd like to know which client a specific product belongs to. And then whenever we need a list of products, for example, to share to the user, we just query all products filtered. by the client that is currently active either by looking what users logged in or by checking the the domain that we were on or something else in our request. And cross-level row level separation is very explicit. You explicitly see in your code base what belongs to which

element. Where do we have separation? Where do we not have separation? And it's very readable. It's very flexible. You can easily switch to doing queries across multiple tenants over all of the tenants and it works on every database but it's supported by Jagger. However, there is a downside as well. It's very easy to get wrong. And to prove that, let's play a short game, spot the bug. The first one's pretty easy. Say given the models we used before, we're looking through our code base and we're seeing a query like this. Selects products that are currently available.

And with a trained eye you quickly see there's something missing. We're missing the filter for the current client Say uh we should edit in or we will have a data leak. So that's still pretty easy to do, but how about this Django gives us with very powerful things to do things really really quickly in this way as a list view. It's a complete definition of a view that renders all our products. We just need to hook it up to an URL, write a template, and it'll give us a list of products. There's no query set in there, but it will still be a problem because it will render us the products of all our clients. So whenever we use a generic view, we will need To supply a custom query set method and make sure that we're passing in the correct filters.

The same is true normally for generic views It's also, for example, a problem for model forms. For example, if we had a model order that has a foreign key tube product, which in turn has a foreign key tube. to a client and then we generate a form for our admin backend for our event organizer to modify an order. And we say, Django, please give us a model form of the model order with a field product. And it looks all fine. There's no query sets or anything in there. But it will give us a select box, a very nice model Choice box that I believe Carlton will be talking a lot more on later today. And it will list as the products of all clients in the system. And

the way around this is again pretty straightforward, but a little bit cumbersome and requires CAT override the underscore underscore init method and set a custom query set for that field. And those things are not only hard to spot. In the code base, they're also hard to spot during testing because on your local development machine, for a feature you just developed, how likely is it that you actually have data in your database for multiple different clients? So you won't even notice that it will return you too many results. And I've talked about generic views and model forms, but the same is true of Model serializers and Django REST frameworks, filter sets and Django filter, and so on.

With very thorough code review, And some experience it is possible to prevent this kind of error 99% of the time. And when it doesn't, when you don't catch it, things can go bad. Really, really bad. A few examples from our history as a company and as a software project. In 2017 we encountered just that. We had two incidents of data leaks due to a list view or a model form without a custom query set. It wasn't that bad, it was not really It it was no personal uh personally identifiable data involved. Um it was caught early on, so it wasn't that bad, but

It it was a real incident of this. And then two years later, just last year, we nearly leaked all of our customer data due to a missing filter clause in a query. The only thing that prevented a disaster from happening is that we tested the feature again shortly after it rolled out to production. And we noticed there was a really really large file being exported instead of the expected really really small file and we immediately shut everything down and we were able to see in our log files that we were the first one to test it and nobody had gotten any data that they shouldn't have Gotten. But it was really, really close to an absolute privacy and security disaster. If you want to read more about the specific incident, it's all in our blog, as with any security incident that we have.

So we needed to find a way to make sure this never ever happens again And if you want to make sure that something never ever happens, the best way to go about this is with a defense in depth approach. Defense in depth, it sounds very professional, it's a very simple term, it just says that you have multiple security measures in place which are redundant. So even if one of the security measures fails or gets broken, the other security measures are still in place and you can fall back on them and the system is still secure. So in our case, we settled on three layers of protection. The first layer of protection was to lock it down, or better, to keep locking it down.

We would still, and to this day, we're still writing all those field statements I've talked about, but we're still We're still subclass we're still overwriting those things on every list view and every model form and every serializing and so on to make sure that it's very explicitly in our code base which client are we talking about, which objects do we want to fetch from the database. I'm not a fan of magic. There are packages out there like Django Multitanner which will do this for you magically. Ye activate tenant and they will you'll they they provide you with a model based class that injects into your models and model managers and automatically only returns the correct statements which can be convenient

but I would say um Might get really dangerous if you get used to the fact that everything is done for you and then you're in a in a weird place down below in the state where it doesn't do everything for you and then you forget the query gun So we're still training ourselves to put in all those finter filter statements ensure with code review that they're in there. Because we believe explicit is better than implicit and we'd rather write all those statements and go through the code review one more time than miss one of them in a place where we can't miss it. See, this is what we've done all the time. It's still in there as our first layer of protection, but it's nothing new.

The first thing that we added was our second layer. We wanted to have a fallback So if we ever were to forgot one of those statements again, it wouldn't have catastrophic consequences. And the outcome of this is Django scopes. Django Scopes is a package that allows you to annotate your Django models with information about how they are related to your tenants. For example, revisiting the model structure from earlier, if you have a client model and a product model, you would use a custom model manager for your default manager objects um that is a scoped manager from Django Scopes and you you tell it two things. First of all you tell it that your tenants are called clients

You could have Django scopes would allow you to have multi-dimensional clients, but that's not a topic for today. And The second thing that you tell it is how it finds the client if it has a product object, so you say the you tell it which field is the foreign key to the talent model. This could also be a foreign key across multiple hops using the common underscore underscore machine notation that they can use in Django. And then once this scope manager is activated, if you try to fetch all products from the database It will just blow up and it will show you an exception, scope error, a scope on the imagined client needs to be active for the square.

We don't know Which client you want, so we're better s we're better off just returning nothing and throwing an exception. If you do want The data for a specific client, you need to activate a scope. You need to use a context manager called scope and tell the context manager which client is the currently active client. And while that context manager is active, um Django scopes will hook into all your queries and make sure that you're filtering for that client. For example, but but as I said, we will still kind of write the filter statements in there. If you do forget to write the filter statement, Django Scopes will edit for you automatically.

That's something we don't rely on, but it was easier to make it implicitly filter. Then to throw an exception. We might change this at some point to just throwing an exception if the filter statement is not in there, but we wanted to prevent false positives uh from from parsing the the SQLite so if the library isn't sure whether the filter statement in is in there it just adds an in there as a protective measure. Of course, you can also run queries across multiple tenants. For example, you can use the context manager to activate multiple clients at the same time. If you have a piece of code where you're pretty sure that you either want to access all the tenants or that you

um that you uh verified it very very thoroughly and there's a performance impact on generoscopes, which rarely happens. might happen, then you can disable it either also through a content manager or through um a function decorator. Scopes disabled and Uh within that that function everything behave like if jungle scopes wasn't installed at all So this all sounds very inconvenient. Adding all those context managers all over your code base sounds like a lot of work and cluttering your codebase. Well the thing is, it doesn't It's very easy to integrate because in in most projects you will know which clients

data you're gonna access based on the URL being accessed or the user being logged in on CRM So what we did is we implemented a middleware, we call it scoping middleware, that figures out which client this request belongs to and that activates the scope for the whole runtime of the Django view. Now, if we look at the things that we identified as the more invisible problems before, for example, like a model form. Just declaring a model form like we did before with saying I want to have a form for the model and it has a fore key on Will give you a scope error. You can no longer declare this way and will it will find all those problems in your code base

immediately after you install chain of scopes on trying to run your project the first time And the way around it is to add in a custom field class, what we call the safe model choice field, and there's also a safe model multiple choice field. And then safe model shares field does two things. It prevents Django scopes from throwing an exception and it limits the default query set of that field. to an empty query set so we will return no data at all and you now need to go in, overwrite the init function and set your query set the way you like it. Another really good example of where this becomes relevant is the Tango shell.

We all don't like working on the Django shell in production, but we all know it happens. And also there it's really easy to forget a filter statement and access data for the wrong client or delete data for the wrong client So you can also use Django scopes there. And in our project we have a shell scoped command. That allows you to get a Django shell where the scope is restricted to a specific client and you won't be able to access data from the clients. This is currently not part of Django scopes, but Something we took over from the pre-talks project, and there's a blog post here that describes how to integrate this shell scoped parameter into your own project So how is this working in the real world?

Integrating Django scopes into Pretex, our main project, with at the time 55,000 lines of production Django code was around three hours of work, so not a lot at all. It was a little bit more tedious to integrate into our test suite , but the the full time from Django Scope being a library ready to use and Django Scope being fully integrated into pretext and running in production or still on the order of two or three days. So it's really easy if you have a multi-tenant project to add in Django scopes just as a safety layer so you know See you in there whenever you write a model form or serialize that but uh uses a too broad query set, it will just fail and whenever you forget the folder

statement, it will automatically be added. But I wouldn't call a two-layer approach defensive depth already. So there's of course third layer. Third layer is to verify that you're actually not leaking data. And there are multiple ideas on how to do this. For example, you could log queries to your database, maybe not all of them, but most databases have an option to, for example, log 2% or 3% of the queries to text file and then you could parse them back and check if there's actually a filter for a client in there and if it's not you could trigger an alert you have an engineer check what's going on. Or you could have something on on one of the levels either in the database or in your Django

code or somewhere else that looks at result sizes and whenever you get a result from a database big that has a lot more records than the largest of your clients has, then there's probably something going wrong. Or you get to have honeypot clients in your database containing specific data, and whenever that data passes out of your database into your application, you can trigger an alert because that should usually not happen. So these things rely a little bit on security by obscurity, by the attacker not knowing how it works. Security by obscurity on its own is a concept that should absolutely be avoided. But security by obscurity as part of a this defense in depth strategy can be really, really valuable. And unfortunately, this also means that I'm not gonna tell you in too much detail on

how we implemented these because if an attacker would know in very much detail they could probably structure their attack in a way that doesn't trigger. those um it doesn't trigger those detections but I just wanted to to give you the idea that um in addition to joint scopes there's even more you can do And you can get creative there and do something that allows you to notice when somebody tries, uh, when somebody found a security vulnerability in the application and tries to actually pull the data out So this brings me to the end of my talk. As I said, I'm going to be around live for questions. If you have questions later on, you can reach me by email or on Twitter.

You can also find more out and more about our project on pretext. eu or the pretext GitHub page. And yeah, I'd love to hear from you about your ideas on how to secure multi-talent data. Thank you very much.