Contributing to Django or how I learned to stop worrying and just try to fix an ORM Bug

Good morning and welcome to my talk, Contributing to Django, or How I Learned to Stop Worrying and Just Try to Fix an ORM bug. I wanted to find out uh how many people are attending their first DjangoCon. Oh that's so awesome. Oh goodness, that's great. Thank you so much for all uh coming, and I'm sure that you'll have as great a time here as I did at my first DjangoCon, which was last year. As previously stated, my name is Ryan Cheeley. I've been using Python for about seven years now and Django for about five. I work for a medical services organization in Southern California called Desert Oasis Healthcare. Part of my duties include managing a team of web developers, and one of our newer projects is an internal Django application.

But today I'm going to talk about the fantastic experience I had in trying to fix a bug in the Django ORM. But first, a side quest. This is a picture of the Trailcrest trailhead at uh Mount Whitney. When I was 15 years old, my Boy Scout troop did a hiking trip to Mount Whitney. Uh for those of you that don't know Mount Whitney, it is the highest peak in the contiguous United States with an elevation of 14,500 feet, 505 feet. So there I was at the Mount Whitney Trailcrest, 13,600 feet above sea level. For our metric friends, that's 4,159 meters. We'd spent seven days hiking to get there, and I was only three miles from the summit of Mount Whitney.

But I wasn't sure I could go any further. You see, my hiking companions told me I didn't look so good. Honestly, I didn't feel so good either. Did I make it to the summit? Well, I'll say that trying to get to DjangoCon US at times felt an awful lot like trying to get to the summit of Mount Whitney. You see, I've been trying to come to Jane DjangoCon since 2018, but I had a work conference that conflicted the same week and so I was unable to attend. In 2019, the same Work Conference, the same week as Django Khan, made it impossible for me to go two years in a row. In 2020, I was really, really determined to go. But we kind of locked down the world and none of us got to do anything that year.

In 2021, I was able to attend the online version and I had a really great time. So much so it encouraged me to do whatever I could. to make it to in person in 2022. And I finally did. And it was amazing. It really was. This is a shot of all of the participants at DjangoCon US last year in San Diego. And yes, that is actually me circled there. See? There I am. And there I am without my mask. Uh I had slightly longer hair back then. I didn't like going to get my hair cut in the middle of the pandemic. And there may have been a little less gray in my beard because I'm a year older now. I was very excited about attending the sprints

and working on something ORM related. My strategy for finding a ticket was to look for something old that seemed straightforward-ish, and then just kind of go for it. I'd spent some time during the conference trying to find something that met that criteria, and by the day of the sprints, I'd been able to narrow it down to a single ticket. So the criteria. Was it old? It was open in January of 2009, making it almost 13 years at the time of the sprints. Was it straightforward-ish? It was related to SQLite, which is a bit easier for me anyway to get set up as a database. And with that, I settled on ticket 10070. Which was titled Named Parameters Not Working on Raw SQL Queries with SQLite, first

reported by Matthias 30. And so I went into track, I assigned the ticket to myself, and I added the comment. I'm at DjangoCon US and looking at this ticket. I was fortunate enough to have Simon Charette at the Sprints last year. Uh for those of you that maybe don't know who Simon is, he's a Django contributor that spent the last 10 years working on the Django RM and migration framework. So I spent some time reviewing the ticket, talked with Simon a bit about it, and it seemed as though it wasn't really a problem anymore. So I added a comment to the ticket. In looking at the tests in raw

underscore query slash tests, I see a few tests that already exist. These are already available and appear to be testing the very thing that the ticket is asking for. And with that, I mark the ticket is done. Later that day, a comment was added to the issue thanking me for my work on the ticket from the original reporter, Matya Surti. I was so excited that I let the world know about it on Twitter, now known as X. Here we see a screenshot of my tweet which says, I closed a ticket while at DjangoCon US 2022 sprints with Simon Scherette. I'd spent the rest of my time with the Sprints working on other non-ORM Django

related tickets. I did a quick documentation update and in other Django adjacent packages. Other Django adjacent projects, I did an update to Django packages to add a VMware button to the homepage. All in all, I had an amazing time at the conference and at the sprints last year. and met some truly amazing people, as I'm sure you will this year. And when it was all over, I'd headed home, having been able to close a bug in the ORM. And I felt really, really good about this. This is a screen capture of a scene from James Cameron's movie Titanic, which was released in December of 1997. Where Leonardo DiCaprio's character, Jack Dawson, says, I'm king of the world.

And that's kind of how I felt. A true story, I've never seen this movie. I might be the only one. Very not. Awesome. But then another comment by Scheiberger, who's a member of the Django security team. Sorry, this is still broken on SQLite. The additions and fixes mentioned are mostly for Oracle and other back ends. The SQLite backend at the time I write this still has supports param style pi format equals false and still borks with the syntax error if you try to replicate the session in the ticket description. But what is support's Pyram

style pieformat? Well, it's a flag that indicates whether or not support for Pyformat style is available Uh this indicate that indicates if the backend SQLite Postgres Oracle supports the Pi format style, which looks like this. There's a lot of stuff up there. And for SQLite, this was set to false. Oh no. I was faced with two potential choices. Choice one: I could ignore the bug. It had been a bug for years. As you'll recall, it was open in January of 2009, and it didn't seem to have caused any major issues

Choice two. I could try to fix it. Try to work on it on my own. But I also had a lot of feelings about the discovery of the bug still existing. I felt embarrassed. I declared it fixed on Twitter I had people at the sprints last year ask me how I did it I felt disappointed. I I felt like I let people down. I felt like I let myself down And fear. Imposterum syndrome started to sink in.

That voice in the back of my head asking. Can you really work on this on your own? Do you really have any idea what you're doing? But then I remembered a talk that Carlton Gibson gave at DjangoCon US 2018 titled, Your Web Framework Needs You. It's an amazing talk. If you haven't seen it, you totally should. If you've already seen it, you should watch it again. It's great. Here are the highlights that I got out of it. The vast majority of tickets are no harder than the work you do to solve problems every single day using Django.

Yes, they need time. Yes, they need thought. And yes, they need potentially a little bit of your love. But if you get stuck while trying to solve it, you can ask for help. The review process is challenging. But everyone gets the same process. It's not personal. Where I work, we try to always make code reviews about the code and never about the person who wrote the code. We all want the code to be the best that it can be, and these code reviews, while challenging, will make the code better

and you'll more likely than not learn something I found this to be true as I worked on this ticket. It was never about me, but always about making the code better. You can do it. You are qualified. If you spent time working on an issue, you are the world expert for that ticket. And to drive home that point, I'm going to talk about a comment on another ticket. In June of 2021, Sarah Boyce, who's now a member of the triage and review team. Posted a comment on ticket 32359. I was wondering if this is being worked on or if I could try and pick it up.

Happy to work on it in a group as well, to which former Fellow Carlton Gibson replied, Hi Sarah, you're very welcome to pick it up. I'd suggest an initial time-boxed investigation to work out what the state of play is. At that point, you'll be the world expert on the issue, so you'll either have an idea or you'll want input. If you open a PR at that point, even just adding comments in the second case, To explain what's going on, it's easier easier for others to get up to speed and provide input. And so with that, I went back to working on the ticket again. And I did what most of us probably do when we're assigned a ticket for work or an open source project or a personal project.

And in all honesty, I should have done a better job when I was at the sprints last year. Write down what you learned. Replicate the bug. Read some docs, which can include doc strings. Write some code. Test the code. But obviously before we can write down what we've learned, we're gonna have to go learn some stuff. So First, I set up the connection to point to a Postgres database. Here we see the standard connection using the Postgres Django database engine. And taking code from the original ticket, I put it into the REPL, and when we run this, it runs without error, as expected. Now, we swap out the Postgres

connection and put in a connection to a SQLite database. Here again, we see the standard connection using the SQL Lite Django database engine and try to reproduce the bug. Again, taking the code from the original ticket and putting into the REPL, we can stack trace, which is what the bug told us should happen. Now, there have been a comment posted by Baptiste Mespilon in November of 2019. Their comment was I don't know how recent it is, but SQLite does support named parameters, albeit with a different syntax. Both of these work on my machine, and Baptiste gave two commands that work. Baptiste then asks the question, do we want to try and make the syntax consistent across all backends, or would it be enough to document the syntax for each one?

When I applied that workaround in the code that borked to use Shy 's term above, it does run as intended. So here's the the updated code, and it worked Okay, great. So we've replicated the bug. We're gonna write down some stuff that we've learned. Next, we're gonna read some docs, which again can include doc strings. Now, the second to the last line of stack trace says Django slash DB slash backends slash SQLite3 slash base. py in line 357 execute. So that seems like a good enough place to start as any. And when I look at that line in that file, I see that there is a class called SQL like cursor wrapper, which has three methods: execute, execute many, and convert

query. Here we see that both execute and execute many call upon convert query. So my next thought was let's go take a look at that. And here we can see the entire method for convert query is just a single line that uses format underscore qmark underscore regex. But format qmark regex is equal to underscore lazy, underscore re underscore compile, which is a method that lazily compiles regular expressions. And in looking at that helper method, we see that it has that it compiles a regex with flags

and has two parameters: regex. The regular expression to be evaluated, and flags, which will modify the behavior of the regular expression compilation. Some examples of these flags could be re. i, which will ignore case sensitivity, And re dot s, which makes the dot character match any character. In this particular use case, we don't need to worry about the flags. I'm just including it for completeness here and to be a part of writing down what I've learned. Okay, so what have we learned? Well, the issue is with the execute method, but that seems to be driven by x by convert query, which will also likely impact the execute menu method as well. Okay. Replicated the bug.

We've read some docs, writing down what we've learned. Next we're gonna write some code. Now, I've got some ideas for a fixed at this point, and we know from previous slides. Then this select statement will fail while this one will succeed. So one solution could be to somehow transform the first statement. into the second statement, accounting for all the ways that someone could put a pr in a parameter set. So my idea was to use a regular expression. And so we want to go from this to this, which means that we want to replace closed

parent s with an empty string. percent open paren with a semicolon to get here. And this transformation can be done with the regular expression seen on the screen here. I posted my idea to the ticket in track. And then got some feedback from Schieberger. As the paraphrased saying goes, which was originally attributed to Jamie Zawinski, I had a problem, so I used regular expressions, and now I have two problems. Shy said, this looks quite fragile. Uh consider for example, for whatever reason

A query which includes close parent s in a string constant. A more robust approach is to create a naming dictionary from all the parameter names and use that to format the original text And then he gave an example, where param names can be a collection, can be collected either by parsing the query or from the parameters given. Now, if this would have been all that Shy had put in the ticket, I can tell you I would have been a little discouraged because I didn't really know what he was trying to tell me there But Shy also left this cut in the comment. Take a look at the Oracle backend. I wrote that piece too long ago to remember which it takes, but at the time I knew what I was doing there. And that

felt good. How many times have you written code, forgot that you wrote the code, went back to look at it and go, this is really smart. I wonder who wrote this? Yeah, me either. But Shy did. There was some similar feedback from Simon Charrette. Saying, I agree with Shy that we should avoid using regex here. Simon stated the ticket, the most straightforward solution I can think of. I have haven't looked at the Oracle implementation in details, would be to simply implement Dunder get item. This will ensure the backend reports a meaningful low-level message regarding underscore missing underscore parameter in case of a parameter missing. Instead of a key error during the Django translation phase, which should make for a hopefully better developer experience in debugging.

Okay. So the code for the the original code for the convert query method is given by the code on the screen. And again, it's returning format qmark regex, which again is a lazily compiled regular expression. So we've got this feedback, and now I wanted to incorporate it into my solution. Recall the hint given by Shy regort regarding using the Oracle as a starting point using a naming dictionary. And so I applied that idea to, I applied that feedback to my use case, taking Shy 's hint and making some slight transformations, as we see here. And I implemented it in the execute method.

Here, the if statement checks to make sure that the params pass through is a dictionary-like object. So we check the power amps to ensure that keys exists using has adder. I use the same idea here for execute many. Again, checking for a dictionary like object and using a list comprehension. to get information from param underscore list, which is potentially a generator. Okay, so we've replicated the bug, read some docs , wrote some code. This whole entire time we've been writing down what we've been learning. Next, I need to test the code.

When going through something like this, we want to make sure that the current tests pass and then write any new tests if they're needed. Okay. So as I worked through the test, through testing the issue, I found a few things. There is a features file, Django. db. backends. sqlite3. features which has a database feature class with supports Pyram style pi format value set to false. This was also something that Shy called out in his initial comment. I see a class called raw query tests that has two methods, test underscore piformat underscore pyrams, and test underscore query underscore representation. But in looking at test pyformat params, I see this decorator.

And since supports param stylepyformat equals false for SQLite, this will cause the test to be skipped So, in order for the tests to check what is happening, it looks like I'm gonna have to add my proposed code and then change the flag for supports prom style pie format from false to true. Based on the tests discovered above, it doesn't look like new tests are needed. Okay Before changing the code, let's see the outcome of running the two tests above. And we can use This command right here, which uses the Django test runner and then the K flag to specify which tests we want to run.

And when we run it, uh they they pass as you would expect. Now, since support param star pi supports param style pi format equals false, The test test pieformat params should be skipped, while the test test query representation should pass. And as expected, here we see the test results. One test is skipped, denoted with an S, and one test is passed denoted with a dot. Okay When working in a situation like this, the strategy I use is something I'm calling a possible state testing matrix. I'm not sure where I got this idea or term from, but I'm pretty sure I didn't come up with it on my own. So what are our possible states? Well, there's the original state, that is, where the flag is false

with the original code. Here we should have one test that passes and one test that skips A broken state, that is, when the flag is set to true with the original code, where one test should pass and one test should fail. This is why the bug exists in the first place. A check state where the flag is set to false and the code is updated. Here is in the original state, we should have one test that passes and one test that's skips. And finally, the desired state where the flag is set to true with the updated code. Here we should have two testing passes, two to test the pass, and we can celebrate. Now, while I'm only focusing on these two tests during development, we also need to make sure that the entire test suite

passes. So after I got to a point where those two tests had passed, I did run the entire test suite and it did continue to pass. Okay. Now so I've replicated the bug, I've read some docs, written some code, I've tested the code. And I've mentioned a lot of times here writing down what I learned as I learned it. And this was one of the most important parts of this entire process. And I'm sure that we all do this, right? As we're learning new things, we write them down. I had never done this before, not until working on this issue. I was inspired by a talk given by Simon Willison and Django Khan US 2022 titled

Massively Increase Your Productivity on Personal Projects with Comprehensive Documentation and Automated Tests. Try saying that three times fast. In that talk, Simon included an idea called Public Notes. This is another talk I highly recommend watching, and if you've already seen it, watch it again. This was the first time I had used this idea, but I've used it a more a few more times since then as I try to learn about a thing. I've even started using it at my day job. But what what are public notes? Essentially, you use an issue In my case, a GitHub issue is a spot to work out a problem

to work through a problem. And you leave yourself tips, hints, breadcrumbs, whatever you want to call them. basically doing what you would do in the scientific method. In my notes, there's a lot of back and forths. A feeling of going two steps backward and one step forward. But even in looking at it now, I can see my thought process for how the bug ended up getting fixed And honestly, were it not for these public notes, I'm not sure I'd be able to give this talk. Here's a short video of my public notes on GitHub

I have 40 comments as a conversation to myself back and forth from October 30th of 2022 to November 8th, 2020. That's just 10 days. Okay, we spent quite a bit of time talking about the process, but what was the fix, Ryan? How did you fix it? Recall the feedback given by Shy about using the use of a naming dictionary and that I adapted for my use case. and put into the execute method as we see here and the executeMany method while leaving convert underscore query unchanged.

When I pushed up my code to GitHub, there was a comment added by Nick Pope. There is duplication which should be pushed down into convert query. Converting percent open parent value closed parent s to semicolon value in this way will also convert % % S to percent S, which will then be incorrectly converted to question mark in convert underscore query. Converting param list into a list in execute many will also saddle us with poor performance. We may have been passed a generator, and the current approach will force all of the parameters to be materialized.

The approach needed requires that we can peek at the first item in the generator without consuming the lot I'll provide a PR to your branch so that you can see what I mean. And truth or word, Nick submitted a PR against my code where the duplicated code was pushed down into Convert Query. And updates were made to the execute method and the executeMENY method. Now this is great, but sometimes seeing a full diff is a little bit better. So for those of you that might want to see the full diff, here we can see what it looked like for the convert query method.

And execute. And execute many of the events. As a bonus to all of this, the SQL Lite docs in the Python standard library got updated. Here's the documentation pre-change with a note. The SQLite module supports both QMARC and numeric DB API parameter styles, because that is what the underlying SQLite library supports. After the change to the document, documentation seen made by Nick, we see named was added. The SQLite module supports both QMARC, numeric, and named. All because of the work that had happened before

in the the pull request that Nick had put onto my code. Since that update, other updates have been made to the standard library documentation and is now less verbose, but the named parameter style is still a part of the note. From this, I hope it's clear that this fix wasn't just implemented or made by me. I had a lot of help in getting the code to the right spot. Schaiberger, Simon Charette, Nick Pope, and someone I haven't talked about yet, but I will later, Marias Feliciak, one of the fellows. So Schieberger. He gave feedback that the approach had some fragility, that regex should be avoided, with Simon Charrette agreeing, and gave a place to get started for investigation using Oracle as a starting point.

Simon Charette. He gave a great keynote at DjangoCon US last year on the state of the ORM. And at the sprints last year gave an awesome introduction to the ORM in general. And he reminded me to update the docs inside of Django after the change had been made. And Nick. As we saw above, the PR that Nick did against my code helped improve it dramatically. It removed code duplication and pushed down that duplicative code into the convert query method. And Marish. He simplified some of the comments in the code and ultimately was the one that merged the code into Django.

I hope I've been able to convey what a positive experience this was for me. I also hope I've been able to convey how much I learned. This process taught me more about SQLite, specifically, that it does support Pi format style. The Django ORM, specifically its structure in the code and how to better reason about it. And Python, generators in particular. Up to this point, I really didn't understand generators, and I still probably don't 100%, but I'm closer now because of the work I did on this ticket. I started using public notes to learn about a thing, to write down what I learned. Public notes since then have included upgrading the OS on a digital ocean server.

Installing Python 311 on a Raspberry Pi, and figuring out how SSH keys work and how I should implement them. All things that are thanks to my experience of using public notes. Now this was a great experience, and you might think, because I'm telling you about it, that I've made tons more contributions since then. But sometimes life gets in the way. I've not had an opportunity to contribute to Django since then , but I have contributed to another open source project, most notably Django Packages Where I've helped to make some documentation documentation improvements for developer onboarding and done some code reviews. I've also been able to implement Django at my employer, Desert Oasis Healthcare.

where we use the admin to manage what we call administrative tables that could only be updating using using SQL Server Management Studio previously. So that probably tells you that we are using Microsoft SQL Server on a couple of different servers for a couple of different databases for that particular project. And the lessons. The ORM can seem big and scary. The code for Django can seem big and scary. But remember, the Django ORM is Python In fact, all of the code for Django

is Python. So when you're ready to go look for a ticket to work on, whether it's ORM related or not, remember that there is an amazing community here that wants to see you succeed. And at the end of the day, the code that you're going to look at, the code that you're going to read, and the code that you're going to write is Python. So just keep that in mind when you're ready to dive in to one of the currently 1012, that was a couple of weeks ago, open tickets. And because this is Django, there is great documentation on how to triage tickets and to work through them.

Okay, back to Mount Whitney. As I said before, I was only three miles from the summit, but I just wasn't feeling it. My fellow hikers didn't do anything to discourage me from continuing, but they didn't really encourage me either. I didn't make it to the summit that day, and it's been a regret ever since. I've not attempted a hike quite that big since then either. But why am I telling the story? What does it have to do with fixing a bug in the ORM? Well, when the bug wasn't resolved as I thought it was, I had the option to ignore it.

I had the option to give up like I did at that day at the trail crest on Mount Whitney. I could have not gone to the summit of fixing the bug. But the community helped me to work through it. The solution that was ultimately submitted and merged wasn't just mine. It was through the hard work of several people. My experience, and I hope this is true for everyone else, is that this community wants you to succeed and they want you to learn. If this community had been with me there that day, I'm 100% sure that I would have made it to the summit.

Because of the encouragement that is provided and the determination to want you to succeed. So remember Your framework needs you, and there's a great community to support you and wants to see you succeed and learn. And this year, there will be both development sprints and contribution sprints If you're looking to get involved, this is a great opportunity to do so. And I highly encourage you all to go. And I really hope to see you all there. Thank you very much.