Only reliable Data: Protecting Database Integrity with Eva Nanyonga

Speaker 1: All right, thank you. Uh my name is Eva, like I've been introduced. And today I'll be talking about how to keep our data reliable using Django. Um we'll talk about uh uh uh atomicity, which is basically uh at the base of uh having reliable data and uh how we can perform um atomic transactions in Django And then we will also see how to report failures in these transactions to help us both debug and also make sure that our transactions

Speaker 1: um are reliable. Um Postgres, uh one of the D B engines, uh defines Atomicity as the property of a transaction that either all its operations complete in a single unit or nando. In addition, if a system failure occurs during the execution of a transaction, no partial results are visible after recovery. And this is one of the acid properties of a database. Basically, the database to to keep the database reliable, we need to make sure that we apply atomicity to it.

Speaker 1: And some some would even argue that atomicity even helps all the other properties in uh the the the acid. acronym uh because I mean to uh uh keep it consistent you really uh need it to uh perform all all the transactions either all at once or you know fail if if uh um uh uh there's uh an error, an execution error that has occurred. So I think atomicity is really important, but some of sometimes we as uh developers overlook it Maybe database engineers, you find that database engineers

Speaker 1: put it at the heart of their uh operations because uh they want to have data that is not corrupted, data that can be reported on Uh so we also need to make sure that we make it easy for the uh database engineers So how do we perform um atomic transactions in Django? I I used an example of uh a um some APIs. Um And uh what you see there is uh uh the user, uh creation of an account, a top-up and uh a debit.

Speaker 1: I'll probably use uh the account creation to uh demonstrate atomicity, but uh That doesn't mean that you can't apply automicity to all the other APIs on the list there. So it can cover pretty much all things that are listed there. So when I run my API um for creating a transaction, as you saw In uh the first screenshot, uh I ran an API using Postman and uh I have a model that describes uh

Speaker 1: the accountants, the account name, the number, the balance, the status uh of that account, and uh who created that account. And so After I run the creation of that API, you see the database shows that we have um a record in there. I first ran uh I first checked the database uh before I ran the um creation uh and then I I I checked the database after so you can see the difference. So right now I'm gonna override uh

Speaker 1: I used uh rest framework um Django Rest framework to create this API. And right now that is me trying to uh over over overwrite uh the perform create um A function within Django Rest framework so that I can do something else with my API. For example, I want to send out an email after I've created an account to show someone that they have an account with our our company. So I go ahead and um uh uh create the account and then uh try to send an email.

Speaker 1: What happens when I uh try to do that? So what happens is I get an error, but I'm not sure what the error is really about. I mean I read uh down down below it reads There's an attribute error, but it's not very clear. But what do you see here? There was a record that was made to the database. Another record is there as uh uh stella. This is this is a second uh uh creation that I'm doing So a record has been made, but I have an error.

Speaker 1: So it means the email was not sent, but We have another user. But so it it so happens that maybe an another user won't know that, you know, uh an account was actually created And they'll try again. So what happens is I introduce some error reporting And uh because I already know, uh uh I I create a new account uh uh called uh with another name, as you can see. But you can see that

Speaker 1: before that I tried again and it created another record with the same name. And the same account. This is this is not good data. It means there are two records that are the same. And uh I mean what what happens if For example, you're going to put money on that account. Which one are you going to choose? It it becomes a big problem for anyone that's using the system. Especially the data guys, when they're trying to find how much money you have, this is going to be a big problem.

Speaker 1: So the new error that I have is that the code is invalid and this is because I introduced some error reporting. Well, it is a good error, but I'm not too sure yet. It doesn't have everything. So I introduce even better error reporting. I uh Try to find out that uh try to find out which type of error it is right down there by uh uh wrapping the error in a type. So what happens then? What happens is I'll get a uh a a much better error, uh a connection refused error.

Speaker 1: And this means that uh my um my email was not configured well. So what do I do? I I try to um uh configure my email and everything is, you know Going on well. So what what happens then? So as you can see Um right here I there was an error. I tried uh to Um I tried to create an account and well it wasn't created but we still had

Speaker 1: uh if you if you look at uh If you look at uh the errors that are introduced and uh uh the atomic the atomic transaction that are introduced uh with atomic here. Under the uh account creation, you can see after the perform creation, I use uh the the with transaction atomic as Django does it to Introduced atomicity to this function. This means the function, when the function fails, a record will not go to the database.

Speaker 1: So then when that error is reported, I get a new uh uh database That shows me, if you see the database right now, we jump from five to eleven. It shows all the number of times that I tried. There was no record then But after successfully configuring my email client, my SMTP email client, I managed to get a new record in. And I can comfortably say that we have demonstrated how atomic transactions work If you uh

Speaker 1: want to maybe uh perform more atomic transactions You can use decorators. Django gives you another way to perform atomic transactions using decorators. um it would have uh an art and just transaction the atomic just as uh just as it is with uh With transaction. atomic, you just decorate your function right there with the at uh decorator at transaction. atomic So we go then to uh some of the most common use cases of uh

Speaker 1: uh atomic transactions. I would argue that almost all uh systems require atomic uh transactions, but they are most crucial in financial transactions. uh for example banking you want to um uh make payments or in accounting systems These uh these transactions, it's very important that you record uh, for example, you're uh transferring money from one account to another. What if something fails within the middle? You need to find a way to roll back everything as to

Speaker 1: have the integrity of the database intact. Otherwise you will have issues with one bank showing a balance that doesn't exist in another And uh we have uh e-commerce systems. Uh uh it it's used in uh Order order management, order uh when you're making orders on your e-commerce systems, this is uh one other use case that's really important. We can see uh Transactions being used in telecommunication systems, online ticketing, supply chain management systems, and

Speaker 1: so on and so forth. So I would say that atomic transactions or atomicity of databases is uh really, really important. And because Django uh uh automatically commits uh uh a a transaction when it started uh we we do not we we we do not know uh if when one of them fails if uh we are going to get reliable data. So we need to make sure that the atom the uh transaction is actually atomic What I mean here is that when uh when you run an SQL command, say of create of uh

Speaker 1: uh update uh or add uh To the database, it automatically saves that transaction. Django automatically saves that transaction. So if there were say two transactions of maybe um uh using someone's you know financial wallet system and then uh updating a balance after that It means if the there is a failure on updating the balance , there is going to be a problem where you have more money in the account uh than than the the person is supposed to have. So uh atomicity is really important.

Speaker 1: Um uh uh Yeah, I think that's pretty much it about my talk today. I don't know if there are any questions or

Speaker 2: Now we have time for questions. Just uh raise your hand. I'm going to uh get your the microphone to you

Speaker 3: Thank you, Eva. If you go back to the previous slide, you've got a list of common use cases. And They all seem to be about um transactions that kind of involve money at one level or goods at one level or another.

Speaker 1: Right.

Speaker 3: This also applied to cases where things are moving in a system, for example, I don't know, medications in a drugs delivery system, in a in a uh medical equipment or in um uh where it's m robotics for example

Speaker 1: Yeah, absolutely. Yeah, that's true. It it it uh applies to pretty much everything. Um for example, an electronic medical record You want someone to get a prescription, but you haven't uh one of the uh transactions hasn't uh recorded So you need to roll back. If if there was uh if the prescription did not go through, you need to roll back uh the entire transaction. Uh the patient, you try to uh get a patient to get a a particular subsc prescription rather So you managed to create a patient, for example, but the prescription was not attached.

Speaker 1: You need to roll back the entire thing. Because then you will have you will think that you sold something that you didn't. And you you might think that this patient got this medication, but well they did, they could physically get it. uh but you don't know when you check their records for example to to see maybe if their uh if an allergy was registered on this patient, you may not find the record of that particular prescription medication. So I think it's pretty important.

Speaker 4: Hi, um do you have any um practices or recommendations around uh retrying transactions? Like in your uh in your example, there was just like a misconfiguration in the SMTP server, but what if there's sort of a Some kind of a state causing a failure or just in a you know, there's no not necessarily like a a really obvious bug in in the code or configuration like that, but it's just in running in production something happens and then you've got you've got a I don't know, maybe you've got 150 transactions that fail, then you don't have a that you don't have like a mismatching data in your database, but maybe you've got you have to do something about that situation.

Speaker 1: You mean like uh run the transaction again, for example?

Speaker 4: Yeah, I mean can you just kind of speak to different options ways that you might think about those kind of kinds of problems

Speaker 1: I I haven't thought about uh that but a solution I could think of right now uh is uh Maybe having a script in case something like that uh happens where there's a failure to run uh the transaction again. I mean if it's just Obviously, you'll notice if it happens again , maybe the server went down. Um you you notice that the script maybe is overrunning. I don't know if that makes sense. Thank you.

Speaker 2: Any other questions?