Beyond the Basics of Migrations with Charlotte Mays

Speaker 1: Thank you very much. So this QR code should take you to my slides. If you want to look at them on your own device, it should also be attached. That link should also be attached to the talk in louds form. I haven't checked that, but I have been told it's there So when we get started with Django, one of the first things that we all have to do is manage. py make migrations, manage. py migrate. We just saw that in the last talk if you were in the room. And so we're all familiar with the fact that there's this this this migration thing that's happening. But there's a lot more to it than that. And For a lot of applications, you may never have to really get your fingers into it, but then every now and then there's so much power in these, it's very useful to know what's going on under the covers there.

Speaker 1: So at the core of it, you know, despite the fact that this migration might look intimidating, especially to somebody who's, you know, this is one of their first exposures to coding and that sort of thing. At the core of it, it's literally just this one migration class that's going to hold two main important things. It's going to hold your dependencies that t tells Django what your migration builds on, what the last migrations were, and your operations that tells it what to go and do to the database to make the database line up with what your Django app has defined. And this is what that's going to look like. It's just going to be your import from Django. db and then creating that class. And dependencies and operations are just simple lists, just like a lot of other Python code.

Speaker 1: That you will have seen. If you look at some of the migrations that Django has created in your own apps, these are some of the common operations that you'll see in that operations list You'll see create model, delete model, and rename model for exactly what that sounds like when you create a new model, remove a model, or change the name of it. Similarly, one layer deeper, add field, remove field, rename field when you are changing the fields that are on your existing models. One layer even deeper than that, we get to alter field, and this is used to change those attributes of the field. So it's an existing field. you're not changing the name of it, but you're changing maybe its nullability, you're changing its max link, all of those things that we define when we make our field definition.

Speaker 1: If you change those, those will get changed in alter field And then a little bit less common, if you're using indices, defining indices directly on your models, you'll see add index and remove index when you make tweaks to that. So let's take a look at what this actually looks like in practice. So I'm going to use a sample model here. Let's imagine I have an application talking about conference talks. What was on my mind when I was creating this? Can't imagine. So I've got a model talk here, and it's got three fields on it. It's got title, speaker, and conference. So we can store that information about these conference talks And these are all just simple char fields, nothing unusual. So this is what the migration looks like. Aside from trimming to fit more on the screen, this is exactly what Django

Speaker 1: generates, absolutely no tweaks So you'll see this initial equals true at the top that just tells Jenga that this is the first migration we've done on this app. The dependencies list being empty also accomplishes the same thing. But when Django auto generates, it does the it explicitly sets initial equals true. And then we've got our operations list, which only has one item in it because all we did was create one model. If you created four models, it'll have all four create models in that one operations list as just items in a list. And you can see it translates exactly to what we have, except what's that? We've got four fields. Well that's part of that Django under the hood magic, where if you don't specify a primary key, Django will give you one so that you have that in your database.

Speaker 1: This is yeah if you're used to referencing your models by you know you've got that dot id or dot pk automatically without having to define it. This is where that gets defined is in your migration that Django auto generated This line is cut off, but if you want to see all of the parameters that are in it, just go open up one of your standard migrations that Django's auto generated when you created a model. But you'll see the title, speaker, and conference are exactly what we set up. That looks exactly like the way we set them up: models. char field and our max length that we set. And that's exactly what these migrations are doing. This is what I mean when I say that these are not as intimidating as you might think to come in and start trying to make changes.

Speaker 1: Now I will caution you Keep your models and your migrations in sync. This is not an invitation to go and I'm just gonna change it in the migration. That's gonna cause you problems. So let's make some changes so we can see what a migration looks like if we make some changes. So I'm going to change the name of my speaker field to be presenter instead, and I'm going to change the max length of my conference field because I gave it eight, and that's not nearly long enough for most conference names. So that migration now is going to have a dependency. We didn't have a dependency in the last one. But this one is going to depend on our first migration. So it tells us it's in the app talks. And it tells us the name of the migration that this one should come after.

Speaker 1: And so when you have these Django migrations, and this is something, this is one of the things that I've seen is sometimes people's first tweak that they ever have to do to a migration If you and somebody else on your team are working on the same Django app at the same time and they push up their pull request that has a migration on it, you might have to change your dependency. To depend on theirs instead of depending on the one before theirs in order to make all your migrations work. Sometimes that's the first time anybody has ever had to go in and manually do anything to a migration. But anyway, so that that migration or that dependency is just pointing at the first migration that we did. And then the operations here, when we make our changes, we've got that rename field.

Speaker 1: And it again is very transparent. The model name is talk, the old name of the field is speaker, and the new name of the field is presenter. And then the alter field, the one layer deeper than the rename field, because we changed an attribute. So this is just telling us the model name is talk, the name of the field is conference, And then the new field definition is models. char field with a max length of 64. So you can see how this just exactly mirrors what we've done in our models And even though the code is automatically generated for you and isn't immediately explained, isn't thoroughly commented, it's all very transparent when you start actually looking at what it's doing. So when might we want to have to edit some auto-generated mutations, uh

Speaker 1: migrations, sorry. Um So as I mentioned before, you know, sometimes you might edit the dependency. It can be helpful to make some tweaks to the defaults occasionally. One of the big ones that you may have to go in and edit is to ensure that a rename is used when you're renaming a field or a model. Here I've got a clip of the the prompt that Django gave me when I generated that second migration It asked me was talk. speaker renamed to talk. presenter? And I had to tell it yes. And so that way it used a rename field instead of removing a speaker field and adding a presenter field. Part of the reason I point this out is if you don't get that prompt when you make a migration and you've renamed something, you probably want to check and make sure what it did.

Speaker 1: Because if you change too much at once Django may not be able to recognize that this was supposed to be the same thing, just with a lot of changes. And what you don't want is you don't want to see that your old version of it got deleted with all of your data in it and a brand new one got created with no data in it. So if you're renaming something, you always want to make sure that that's being used. If you get this prompt, you know it's using the one you want. But if you don't get this prompt, you may want to go check your migration and make sure. And then the other time that we'll often be editing auto-generated migrations is to ensure data integrity. So if we are making a change where we want to make sure that our data shifts

Speaker 1: We might, for example, have some information that's an adjacent field, and we want to give it pride of place as its own field. So we might add an operation to our migration so that we can pull the information out of that Django JSON field and put it into the new field that we're creating. And that's something that Django isn't going to know how to do automatically, so we have to do that ourselves. So, that gets into some of the same stuff we do when we're building migrations from scratch. I use the term from scratch loosely here because Django is still giving us a scaffold. But it's going to give us an empty migration. So we'll often do this for data migrations. If you need to make

Speaker 1: extensive shifts to your uh data if you need to clean up something, etc. You might make a data migration which is only going to touch your data and not your models. Or occasionally you'll have to build a migration from scratch if you have a complex database situation. I'm not going to get into that in any depth because if you're in that situation you know you have a complex database situation We're talking about things like you're building Django app on top of a big late legacy database, or you have multiple databases that your Django app is talking to Those sometimes you might have to go in and build your migrations yourself. But if you're in that situation, you know you're in that situation, you can go and dig down in that. But that is sometimes when these will come up So how we're going to do that?

Speaker 1: We're just going to give Django that make migrations command again, except if we just tell it make migrations, it's going to tell us no change is detected. So we have to give it this dash dash empty flag. And that dash dash empty flag is going to tell it, I know that there's no changes to do, but I want you to give me a migration file anyway so I can make my changes. And this is what that looks like when you have it create that. So you've got again your import line, your migration is set up. It automatically populates your dependencies because it knows, okay, this is going to need to depend on the last migration that's in this app. If you're depending on another app as well, you'll need to add that as a dependency But if you are just building on what you have, then

Speaker 1: your dependencies are all set and ready to go. And then the operations is just an empty list, because we gave it dash dash empty. We have an empty list of operations. So what are we going to put in there? Let me give you a hypothetical so that we can walk through an example. Let's imagine a data cleanup scenario where I did a data import into my talk model. And unfortunately, the way the import happened, it was supposed to have the title and the presenter as separate things, but it had Title of Talk by Presenter of Talk all as one string that it dumped at a title and it didn't put anything in presenter. I need to fix that. So my fix is going to be that I'm going to clip the presenter name out and I'm going to store it in the correct field I'm going to give caveats that I'm doing this very quick and dirty.

Speaker 1: I'm doing it to illustrate the migrations, not to illustrate how to manipulate data. So there's a lot of stuff I'm not putting safeguards in that I would normally put into place. So you you know take that grain of salt. I'm not trying to illustrate good data manipulation. All right, so this is what your basic structure of a migration function is going to be. It's just a basic Python function. We're going to give it a logical name. We're going to give it these two parameters, apps and schema editor. That's what Django is going to pass it when we call our function in our operations. And very important here is how we import our model. Instead of importing our model from models. py like you would in almost any other Django context.

Speaker 1: We're going to use this apps. get model and give it the app name and the model name that we want to get. There's a couple reasons for this. The biggest one is that when migrations are run, they are run in sequence. And there are going to be presumably migrations after this one that are going to change the model in other ways. And our migration that we're creating should get run in the context that it is in this sequence. And this git model will get the version of the model that matches where we are in the migration sequence And so you very much want to do this. I want to provide the caveat that, or not caveat, but the the warning that If you were to do your regular import the way you would importing from models, it might work, but you will probably get unexpected behavior down the road.

Speaker 1: Do not do that. Use your git model. You are not going to have access to a lot of your custom functions if you've defined custom functions, custom save, that sort of thing. So you do need to take that into account and account for that functionality in your migration directly But it is important that you use the model through git model so that you don't end up with issues down the road because you're using the wrong version of your model. All right, so all of those caveats in place, here's what my function looks like for my fix. All I'm doing is iterating over all of the talks in the database. If it's missing the presenter field, then I figure it's one of these oopses. I split my talk title using that by

Speaker 1: text that was in between the title and the presenter name And then dump those components back into the fields where they belong. Very important, I'm calling my talk. save. It's not going to do that automatically for you, just like anywhere else in Django, you have to call your talk. save Also, just a callback to when I said the custom functions are not there. Your custom save isn't happening here either. So if there are safeguards you have in your custom save function, you need to make sure you account for those here. All right, so I've defined my function for my data migration. So now all I have to do is dump that into my operations. I'm going to use migrations. runPython There's a uh sister to this that is run SQL if you're uh running SQL directly.

Speaker 1: Um And so we're just putting our function title in there. We're not calling it, we're just passing our function. And then what's that migrations. runpython. noop. So I haven't talked yet about reversing migrations, except very tangentially. Normally, we're always running migrations forward. We're starting from wherever we are, and we're running whatever new migrations we have. On occasion, you may need to back up if there's been something that's gone wrong on your database, that sort of thing. You may have to back up. You may be familiar with Using manage. py migrate, your app name zero, which reverses all migrations for an app, sets you back at zero, lets you start your migrations again from scratch.

Speaker 1: You may also just back up to a certain point where you pass it the number for the migration you want to back up to. So you do that manage. py migrate your app in this case 0002. And what it does is it actually Reverses each migration in sequence going back through to get back to a certain point. So we use that no-op when there's no changes needed. So in this case I'm doing a data cleanup and so I don't need to re-dirty my data if I'm going backwards. I can leave my data in its clean state. So in general, you don't need any sort of reverse operation on data cleanups, but it is much more useful when you have data structure changes.

Speaker 1: So for example, if you have that JSON field and you're pulling something out and you're making it its own field. You probably want to put that back into the JSON field if you're reversing so that you don't lose your data. So in that case, you just make another function exactly like the one we already wrote, just doing the same thing in reverse, and you'd put that function name. But We're using the no-op because there's no reverse operation that we need. It's important to note that a lack of that defined reverse operation would make a migration unreversible So you would not be able to back up your migrations past that point. You'd have to manually reset things. If you didn't have a reverse operation on there. So that's what those two parameters are.

Speaker 1: The first one is our function that we expect to run. The second one may never be run if you never have to back up your migrations. But if you don't need anything to be done in the reverse, you can throw that no op in there. Otherwise you need to write a reverse function and you would put that as your second parameter. Alright, so a few best practices when you start getting your fingers into migrations and starting to mess with things. Don't reinvent the wheel. If the auto migr auto-generated migrations work, just use them. Because that is going to be the smoothest I definitely encourage looking at them and seeing what they're doing so that you can understand what Django is doing under the covers for you. But don't try to change things just because you can change things

Speaker 1: Do however rename your custom migration files. So when you do that dash dash empty, it's going to give you your migration number at the beginning and then auto and then some gibberish. And that is not very helpful later if somebody's trying to diagnose a problem or they're trying to figure out, okay, what migrations do I have, so on and so forth. When Django does its own migrations, when you have it generate for we made some changes to the model, it's going to put a name on there that tries to tell you something about what this migration is supposed to be doing For your custom migrations, it's good to go ahead and change that migration title, but you do need to do it when you create the migration because otherwise the next time you create a migration, it's going to depend on the auto-generated title, so don't go back and do it for old migrations.

Speaker 1: But when you have just created a custom migration, rename that so that it has a logical name. And again, always use that reverse parameter when using runPython and RunSQL. The no op is there if you don't need it to do anything in the reverse, but use that parameter so that it is reversible. Do make sure that your coworkers know if you have customized a migration. I know we're all human, and I certainly can admit to having seen Oh, they made this change to the model and there's a migration here for the same app. That makes sense. Django created that. Skim And you know, we fully expect that's what people are gonna do. Realistically, would it be great if they did a thorough review of the migration? Sure. Are we going to no But if you have made changes to the migration, make sure your coworkers know who's who are doing the review

Speaker 1: so that they can actually review that code and make sure that it looks right. Don't use migrations for repeat actions. So I gave a data cleanup example here. In that scenario, I'm assuming that I have fixed the problem that would have created that import issue. If it were something where I'm going to repeatedly be importing from this service that I don't have control over and it's just always going to come in and need this change made. I wouldn't use a migration for that because I'd have to write a new migration every single time we need to do that cleanup operation. Instead, I would use something like a management command. So that I can have that be repeatable and be able to be run every time we do that import, that sort of thing. So if you're gonna need to repeat the action, migration is probably not the place to put your code. Migrations are generally gonna be one and done.

Speaker 1: With them being one and done, you don't necessarily always have to worry so much about efficiency because this is not something that's gonna be run all the time. It doesn't matter if it takes a lot of resources the one time it ever gets run. But caveat, efficiency matters on large tables, especially if you have CICD that might time out if your migration takes too long to run. So, you want to make sure that when you're dealing with large tables, when you're dealing with big data sets, when you're dealing in scenarios where you might have a timeout issue, That you take more care with your efficiency, make sure you're not doing too many database operations in your function. And that's all I have. So I can open it up for questions.

Speaker 1: You can also reach me on Mastodon here. And again, there's that barcode if you want to access the slides directly.

Speaker 2: You've added the um No op option in you know in place of the reverse parameter. Um so what would you do if you wanted to actually reverse something?

Speaker 1: Yeah, if you so the question is that no op parameter is if you don't want to do a reverse action, but what would you do if you did want to reverse it? You would just write another function exactly like This function that we have here that makes our that does our data manipulation. Whatever is the reverse of your data manipulation, you have a second function. That, you know, in this case, if I were going to reverse this, I would probably call it reverse removal of presenter from title or something along those lines. I'm not super worried about short titles here because again it's going to stay in the migration, it's never going to be referenced anywhere else. So I prefer clarity to shortness. But so I would have another function that does that, and then the code in that would do whatever that reverse action is. And then you would just In your operations, instead of the no

Speaker 1: up line, you would just list that function title the same way that we listed the first function title in the first parameter Excellent question.

Speaker 2: All right, thank you.

Speaker 1: Okay. Well, I will be available in the hallway for the rest of the conference. So if you come up with a question later, feel free to come find me. Thank you very much.