Zero-Migration Encryption: Building Drop-in Encrypted Field in Django

This video features Vjeran Grozdanic at DjangoCon Europe 2026 in Athens, Greece.

Zero-Migration Encryption: Building Drop-in Encrypted Field in Django
0:23:40
Published October 26, 2026
60 views

TALK: Zero-Migration Encryption: Building Drop-in Encrypted Field in Django by Vjeran Grozdanic

Django doesn't come with a solution to encrypt database data out-of-the-box. While community projects exist, they often require complex migration processes or double-writing to new columns. At Sentry, for our high-traffic distributed system, we needed a solution that would abstract encryption logic without worrying about encryption keys on a case-by-case basis or performing risky migrations on millions of existing entries. We built EncryptedFields, a custom Field base class that provides drop-in, no database migration, replacements for CharField, JSONField, and TextField. Our approach allows for a "hybrid" state: the field handles both old plain-text formats and the new encrypted format on-the-fly. This eliminated the need for complex data migrations or secondary columns, making the transition safe and seamless for our engineers.

More information: https://2026.djangocon.eu/speakers/SYTSVG/

Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope

#django #djangocon #djangoconeurope #djangoconeurope2026

Transcript

3,815 words · auto-generated · en Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:02

Speaker 1: So our next session this morning is from Viran and Viran works at Century, one of my favorite companies, and we have a talk on zero migration encryption. Round of applause, please, thank you

0:22

Speaker 2: Yeah, zero migration encryption or how century encrypts millions of database rows without downtime migrations or double writes. So who am I? My name is Vieran. I work at Century and I tried to be a professional basketball player, but now I have a different career. Uh centuries for those who don't know application performance monitoring software. We do errors, tracing, logs, metrics. But what's important is that it's a large Django backend. And it serves thousands of requests per second, just the product, not the ingestion and everything else. And uh that's a really huge scale, and this is very important for the next problem. And that was a task that I was given. It was supposed to be a simple task, just add a new table to store the third-party API credentials for a new partnership that we were about to do.

1:16

Speaker 2: And at first it seems like a really simple task. It's just like a new Django model, you add a API key field, and that's it. Forget about it. But then the question arises, should this be encrypted? Because we are dealing with sensitive data here, and probably it's smart to also keep it encrypted at the database So the answer is of course, yes, we should encrypt these things always. Uh so I was still fairly new to the code base and I just wanted to check for the existing patterns. How do we deal with encryption? uh at century. And after some time spending uh and researching, realized that each team at Century has its own way of encrypting the data.

2:01

Speaker 2: We have over ten different environment variables, each team added their own way. Uh all over the code base they are encrypting and decrypting things as they fetch it and some of the stuff weren't even properly encrypted. For disclaimer we were encrypting stuff at rest in the database disks were encrypted but in the database itself they weren't And now I was faced with the dilemma. Should I just continue with the existing patterns? Uh just add a new variable, environment variable, have my own way of handling things at code, or maybe solve this a bit better. So after talking with the manager, I found some time to solve this holistically. And to recap the current state We had ten

2:46

Speaker 2: different places that uh ten different tables and ten different decryption keys that needed some kind of way of Being merged together and abstracted. A lot of places across the whole code base where people were doing encryption and decryption, and it was really hard to chase it because it was multiple levels deep of abstraction. And also there was no easy way of even telling if the data is encrypted just by looking at the models. You would have to find all the usage of that model in the code base to figure it out. So the solution is always some kind of abstraction. We needed to make this simple for others to use it So if it's complicated people will not use it. So the the first requirement is always make it as simple as possible to use. People don't need to know about

3:33

Speaker 2: what even encryption algorithms are behind or they don't need to deal with the encryption keys. So to start it, this is a probably a well-defined problem. We weren't the first one hitting this in 2025. There had to be someone else who already thought through this and there has to be a library to to solve this problem And of course, there are plenty of libraries out there. Uh I will not name all of them, uh I will not even name a single one of them, but basically the gist of each of them is uh They do everything for you. You just kinda set up the keys and you replace your field with the encrypted version of that field. And this seems very simple.

4:18

Speaker 2: That would perfectly solve my problem. But that's not really the case. Because there are still 10 different database models and combined over billions of rows. So there needs to be a way to migrate all of this. Uh we c like the new table that I was doing was easy. It's a fresh start, but the existing data is the problem. So just a sum to lay some foundations here, uh Sentry needs to be highly available. And what that means is we cannot really Easily do database migrations. So we need to do zero downtime migrations. This is a diagram of

5:05

Speaker 2: very simplified distributed system. Let's say we have a two uh instances of our app and the database and they all talk schema version one. But then when uh version two comes and we need to do a database migration, how do you do it safely? The easiest way and how I do it for most of my side uh projects is basically just stop all the active connections, roll the change in the database, then roll it in your application and call it a day. But we cannot afford this at century because this needs to be always on, always available. So the way you usually do it, so this is very simplified way of doing it. Because just the this topic could be a talk of its own. But you first roll the schema changes to the database and then gradually to each of your apps, instances of the app.

5:58

Speaker 2: And uh just to give a simple example, uh there are rules how to do this safely. You cannot really do whatever you want So when you add a new column, uh a new column must be null nullable and or it has to have a safe default. Why? Because in Postgres If the column is nullable or has a default, you don't need a full table rewrite, you can just do a metadata change. So when inserting a new row where you can have a state, Your database speaks schema version two, but your app is still talking about version one, you would have a safe uh way of inserting or retrieving data. And another thing is uh we can never change the existing column type. So if you change the column type, most of the times in Postgres

6:46

Speaker 2: this is this will require a full table rewrite. And Postgres is very, very lock happy and it will take the lock on the whole table and you will prevent the all the reads and writes on that table until that full table rewrite during the migration is finished So now knowing this, how do we safely uh do a rollout, encryption rollout on our existing tables? Uh well there is a process. So this is let's say this is our already existing model, it has some data in the table. And Basically, the way to do it would be to add a new column. We already know we can do it in a safe way, safe way if it's anallable. So this is just a simplified code, but this column should be probably innullable.

7:31

Speaker 2: We had a new column. We start double writing data to both old and the new column, but this is a problem now because now we have to find all the places in the code base that either write or read. From here, and we need to adjust this, all these places in the code base. After we are sure that we are double writing at all the places, we need to backfill all the data So now our new column is empty for all the old existing rows. So now we need to write a huge migration to Just backfill it and encrypt the data. Once you've done all of this, then you can start using only the new column in the code base. And once you're really sure that nowhere else, nobody else is writing to the old column, we can safely Drop

8:16

Speaker 2: the old column. Now we need to do it ten plus times. And with the rise of the agents that might be possible, but we also have to make sure to make no mistakes while doing this because every mistake is very costly and might result in loss of data or production incident So now it's time to take a step back. All this process, while doable, would require a lot of work. And it's super easy to mess up something. So we need to extend our requirements. Very important requirement is, besides all of the already mentioned, that migration needs to be as easy as possible to do And now

9:01

Speaker 2: another uh sidestep is actually look how the existing implementations of this kind of way, like existing libraries, do things And do it then do it they do it by uh modifying some things in the Django-based field. Django is super modul modular and easy to extend, and there's a very nice abstract class that represents a database column called field. And the important thing is that it maps Python types to database and vice versa. So it's very flexible to extend it. There are two amazing methods that we could use, and those are getPrep value and from db value. So in getPrep value that's the method that's definitely gonna get called before you save the data to the database

9:47

Speaker 2: and when you read from the database. your your method from db value is called. So this means that we can probably hook into those values. And in the life cycle of writing to the database or reading from the database, we could do our stuff. We could implement our logic. And this would actually take us from easy migration to no migration at all because we could just start doing the Start handling the existing values as well. So the first thing to migrate is car field. And it's already a R field under the hood in Postgres. So need no need to do any schema migration And instead of saving a plain text value, we can just save the encrypted value. And when the if

10:32

Speaker 2: when we fetch the value from the database, we can just decrypt it and serve it in the process as the plain text value Everything is abstracted away, works perfectly. So in the get prep value here, we are going to encrypt our stuff, and from db value, we are going to decrypt this. That seems like a problem solved now. Well, when we think a bit more about it, how to even tell apart the existing values and the new ones? So this is our database after we rolled out our change. And this is just a Fernet base 64 encoding. But new value and the old value, yeah. When we look at it, yes, for us it maybe makes sense, but there's nothing that really gives it away That the new value is encrypted and the old value is just a plain text.

11:19

Speaker 2: So do we just always decrypt and catch the exception? We could do this, but there has a probably a performance hit. So since we own the whole protocol, we could just change the format and the way we write to the database. So why not add a magic prefix to kind of like mark like a magic bytes that Linux using? And we can just add a prefix to always say like this is the field that we actually modified and this is our format. So that's what we did. We added a c a prefix and always now and Retrieving it, we can just check the prefix and we know if the value should be decrypted or should we just let through as a existing old value. So now the problem is really solved, right?

12:05

Speaker 2: Well, there's still thing that we haven't handled key rotation. And encryption keys have these tendencies to get leaked or just rotating them is a good security practice. So We need to support that use case also. And in encryption there's always not always, but usually you have one main encryption key. And you have multiple decryption keys, because you can be in a state where you just rolled out a new encryption key, but you still have all your all of your old data in the database encrypted with the old ones. So now you need to handle multiple keys in your system. Old data will be decrypted with the old key and the new data will be decrypted with the new key. And this is how our value currently looks like the first row, that's what in the database.

12:51

Speaker 2: And let's say that we have a three different decryption keys. Which one do we use to decrypt the value? Do we just go through all of them? Uh we could, but then again there's a performance hit and if you rotate your keys often enough after some months or years This is going to add up and you're going to do a lot of decryption before you hit the right key. But we already said that we own the format, so why not extend it further? And we could just encode the information about the encryption key in the format itself. So now in the database, this is how it's going to look like. We are going to have our magic prefix And we are going to have our key information. And now

13:36

Speaker 2: the problem really is solved, right? There's nothing else to worry about. Well, there's still something else. What about JSON field? Because we mentioned in the beginning also there were a couple of JSON V and Varchar fields. And with JSON it's not really so easy to just do this kind of thing because encrypted value is a string or base64 encoded bytes or whatever. But we need JSON B under the hood and we need to somehow make this a JSON. So we can just make it a JSON We still own the

14:22

Speaker 2: format. So now uh our JSON field becomes encrypted JSON field, and we are just the way we did it before We can just save it in a JSON format. We are going to add an underscore value or what some key that will mark this as our encrypted format. And then the Postgres is going to look like a JSON, like it is. So the whole process now is you get a JSON, you have to first stringify it, you have to encrypt it, you get the well-known format that we already defined, and you have to create a JSON out of it And then you save it to the database. Now the problem really is solved. And this time for real. To recap how it all works, uh

15:09

Speaker 2: in the application code, when you set that secret value on your model, we are going to use two of the already offered methods in Django, get prep value and from db value. And it's gonna go to our encrypted field middleware or some layer, and that layer consists of format parser that knows how to read the format that's in the database. It will know exactly which method and was it even encrypted or is it a plain text the old value? It will know which key to use to decrypt the data and it will know which part really to just was a string to that needs to be decrypted. It will have encrypted reg encrypted registry because you could have multiple encryptors, multiple algorithms that deal with this. And it's also going to have a key store. So I haven't talked about the keys, how we do it.

15:56

Speaker 2: But what we did is just we mount different files and each file name is a key ID and the value in the file is a key itself. And to talk about the performance, we have been running this in production for six months now. And this might not seem like a lot, but we do like tens of encryption operations per second and hundreds of decryption operations per second. It's a very read-heavy uh workload. And the performance overhead is, well, at least in our case, not a concern because it's less than a millisecond of the whole overhead.

16:47

Speaker 1: Thank you very much. Do you want to take questions?

16:50

Speaker 2: Yeah.

16:51

Speaker 1: And do we have any questions right here in the middle?

16:56

Speaker 3: Thanks a lot. One of the things you said in bit uh before was that you couldn't need uh you would need to go through the entire code base to make sure how the field was being used How did you make sure that it's not being queried on because you can't do that anymore when it's the data is encrypted?

17:16

Speaker 2: Yeah, so that was like it's not fully dropping replacement, obviously. You cannot do filtering on that anymore. But that was like a trade-off we were willing to make because we we knew that nobody should filter on one of the secret values like API keys. Those val those rows were always fetched by the ID

17:38

Speaker 4: Uh hey um the JSON B um trick that you did like encrypt it and put it in JSON B but like the field in the database is You can store a string in it anyway. So what's the value of like turning it in like encrypting the entire stringifying the JSON? uh and then encrypting it and then putting it in a different JSON. Like what what's the benefit of that? Or is it even needed?

18:08

Speaker 2: So Uh okay I I'm not I'm not hundred percent sure if I can store the string in the JSON B column. If that's the case, I wasn't aware of it.

18:18

Speaker 4: Yeah. Uh I just solved a bug where we had a default value that was not a list but a string that contained the list and it stored it Great and when you pull it out you just get a string. So that's why I know that I just solved it yesterday, so but yeah, it doesn't need to be

18:36

Speaker 2: Okay. Yeah, I was under impression that I'm not hundred percent sure in this, so I was under impression that I had to create a JSON. Could be a mistake on my end. Yeah, just question.

18:50

Speaker 5: Thank you for your your talk. So for what I understood you moved the encryption and decryption phase on the Python course. Have you tried to think to uh move this uh loads on the database itself maybe using some j generated field.

19:07

Speaker 2: Yeah, yeah. Yeah we were thinking about it but it's very hard to do the key rotation in the database And it's requires a lot of ops uh in help here, but this is kind of self-served developers can do it themselves. So yeah, thank you

19:25

Speaker 1: Do we have any more questions? Um

19:32

Speaker 6: By storing the entire JSON value as an encrypted like single string, you're losing all of the structure that was probably the underlying reason for using a JSON field in the first place. I presume that the only thing worth encrypting are the values and not the keys. So wouldn't it make sense to only encrypt the values and just have like multiple like ang blah blah blah blah blah things in your JSON field instead of having one flat value.

20:03

Speaker 2: So you would have to somehow specify exactly which key in the JSON you would want to encrypt. And that already makes it a bit more complicated. Uh we first try to do this, but then the problem is that developers need to read a lot of documentation how to actually specify these things. And we were lucky enough that for us most of these JSONs were containing just some tokens and some metadata that were never queried. So we could just say like let's simplify this, hide away all of the complexity and just tell them like it's encrypted, you cannot query it, that's it. That bet was a trade-off we were willing me to make.

20:38

Speaker 7: Hello, you said that Each team had its own encryption method there. So um was were they all basically equivalent uh or did you I Actually teams have different requirements that made them use different methods.

20:57

Speaker 2: No, they were they were mostly equivalent. I think somebody first started doing Fernet and everybody else just followed the pattern with their own key. There's one more here in the first row.

21:16

Speaker 3: You said at some point you had full control about the format of what the encrypted data looks like. How did you make sure that there isn't any data in there that's

21:31

Speaker 2: Yeah, so the the real prefix isn't really and and we added like a very long underscore century encrypted value prefix. To make sure that nobody will by accident. It's still a risk. If somebody really makes it, we'll try to decrypt the data. But it's going to fail, and we still have like a handling of the failures to just then return as it plain text. But we added a very long substring hoping that nobody will match it.

21:57

Speaker 4: Yeah, thanks for the talk. My question is at the start, you said you had some unencrypted values in the database at the start. And maybe I missed it, but did you actually encrypt those values at runtime or did you run a specific migration just for those values?

22:12

Speaker 2: No, we had to run a specific migration. We did a separate job that ran through Probably the whole night through billions of rows and just resave them, fetch them and save them, and now they're going to be saved.

22:32

Speaker 1: Bear with me. I'm almost there.

22:37

Speaker 8: Thank you. Um Do you also store the encryption algorithm in the key somehow, or does each key have its own algorithm linked to it? And if you want to change the algorithm you have to change the key that's used?

22:50

Speaker 2: This prefix, this ang, is already also including the algorithm that we are using. So we modify the whole prefix to know how to know exactly which encryption decryption method to use.

23:04

Speaker 1: And do we have any more questions? Just here? Um

23:11

Speaker 2: is this open source? Yeah, whole century is open source. Yeah, could you add the link

23:16

Speaker 5: to the uh

23:17

Speaker 2: Yes, I'll send uh I'll send it in Zulub

23:22

Speaker 1: Okay, looking around any more? No? I think you had an incredible amount of questions, which means it was an incredible talk. Thank you very much Thank you, applause.

Conference

DjangoCon Europe

April 15 - 17, 2026

Athens, Greece

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos from DjangoCon Europe