The Design and Development of Choices in Django 3.0 - Shai Berger

Hello and uh welcome to my talk. Choices were one of the highlighted features of the Django 3. 0 release And I want to tell you about how the feature came to be and what issues we ran into in its design and implementation. I hope we can learn some things from this story. My name is Shy Belgian. I'm a Django security team member. I used to be a Django core developer for a few years. I've been using Python for more than 20 years and I've been working as a programmer for 30. Here in Israel, I work at a small company called Kaplan

Open Source Consulting. That's the orange logo. And I volunteer at the Israeli Free and Open Source Software Association, Hamako. Just to make sure we are all on the same page, let's see what we're discussing. Choices as a feature of fields have been there for a very long time And you specify them with a sequence of pairs, where each pair sets the value and the label or presentation for a possible choice for the field. If you also wanted constants for the values, you needed to define them separately, leading to code like you see on the screen now. Many people thought

that this was violating DIY and generally looked ugly. In Django 3. 0, we added choices types, which allow you to put the constant names, values, and labels all in one place by defining a kind of enum. When you use such a type, you also get an option to have the labels generated from your constant names if you don't need translation. And some other features of the Enam class from the standard library. So, how did we get here? As I said, a lot of people

for a very long time were not happy with the way you had to define choices in Django. So they built their own classes. Sorry. One important example was Tom Forbes' Django Choice Object, which was published in 2013. and was not using the standard library in num classes because it came before them The history of the feature in Django begins with a ticket opened in March of 2017, asking for enums to be used to specify choices on fields Django 1. 11 had not yet been released at this point, but was already feature-frozen, so the suggestion targeted Django

2. 0. As you may recall, 2. 0 was the first release to be Python 3 only, and so it was suitable for a feature relying on a part of the Python 3 standard library. The suggestion itself, however, was a little naive, asking for enums to be used directly and for labels to be always generated from the enum member names. It was pointed out immediately that this left no room for specifying translatable labels, which for Django is a strict requirement. So the ticket was closed once weeks over this issue.

Five months after it was closed, Tom Forbes from three slides ago showed up and posted a message on the ticket suggesting an API for adding translatable labels. This API was taken from Django Choice Objects, but Tom did not link to his implementation, and the ticket was already closed, so basically nobody noticed. In December of 2018, I was working on my first major Python 3-only Django project. This project had choices which also needed to be translated, and I joined the ranks of people who looked at the issue and said,

maybe we could use enums for this. I looked around, found the closed ticket, saw Tom's suggestion, and inspired by it, I found a way to implement it with Inams. I presented a proof of concept on the Django developers mailing list right on New Year's Eve saying basically we can restore him. We have the technology. The reaction was mostly positive, but there was also a major objection, which we'll discuss later as a design issue. The discussion went on for about two weeks and then it died out.

I was busy, I guess others were busy too, and that's the way it stayed until DjangoCon Europe. Which was in April. When I went to JengaCon in Copenhagen, I already planned to work on Enam 's for choices in the sprint. I'd talk to people there about it, I managed to get the ticket reopened, and indeed, I spent the sprints fleshing out the proof of concept to an initial pull request. At this point, we still saw this as mostly about enams, and we still saw it as a minor feature.

With the PR available, others got involved. The PR got reviews. I am not naming all the people who participated, and I hope they'll forgive me For about a month I was responding to comments properly and adding fixes, but then I got very busy again. And at some point I just said, I need help with this. If anyone wants to take over, please do. At Marius Feliciak's suggestion, Nick Polk took over development of the feature in July 2019 Nick did lots of more work, handling issues, polishing the code, and improving the documentation.

The way it looks now is more Nick's doing than mine. At some point, the class name was changed to choices, with subclasses, integer choices and text choices for the most commonly used value types. The change was made in order to de-emphasize the connection to Inam, because at that point we realized that Inams were a tool and not so essential in the definition of the feature. In September, Choices were declared one of the highlights of the coming release and merged into Django's Master Branch on September 4th. After one more bump in the road, a bug that was found in the beta and will be discussed later, the feature was released as part of Django 3.

0 on December 2nd, 2019. Yeah. So, in the process of getting this feature accepted into Django and developed, we ran into several design issues, which I would like to share with you now. The first objection raised against the use of enams for choices was that they don't fit the bill. If you define an enum and give its members values, then the enum members don't compare equal to these values.

For a type that is supposed to provide value to fill fields, that is a very bad property. But as it turns out, if you use enuns in a slightly different way, adding a base type, then the equality issue is resolved and with it a small set of related issues. The objection was overcome Enams can surprise you and when people usually think of them, they don't think of using base types. Among other reasons, because the standard library documentation recommends against it. But this recommendation has an exception

for the case where you need to support interoperability with other systems, which is exactly the Django use case. We also had to modify some other details of the behavior of Venams to make them fit Django well. But all in all, it was best to rely on the familiarity and good design of the standard library. Nick Pop summarized it very well in a comment on one of the pull requests, saying, we are providing a choices type that is enam-like and enam-backed. but has to make its own rule. Another issue we had to deal with was the empty choice, that is, specifying the label for nothing being chosen.

With the traditional list of pairs, this was done by attaching a label to the value NUN, but with an enum with a base type, we couldn't do the obvious analog Because the members of such enums must have values of the base type. We considered several options. One was to designate a specific value to mark empty. That is, say you're making an integer choices. Pick some numbers that isn't one of the real choices, zero or 2020, or any other number you don't like, and use it to set the label for the empty choice. Another was to create a special base

type where none is a valid value and use that instead of the real base type. We found more palatable options when we remembered that we have some control over the values that go into the enam. At the point where we take care of the labels, we could check if the user specified some value as none and then pick its label and not pass it into the none. And then we figured that instead of checking the value, we could check the name. If we set a special name for the empty choice, Then the user would not need to write none and we'd save them the trouble of finding a name for a constant that would never be used.

So that's what we did The last and most involved major design issue was about Dunder Str. It was initially raised in the context of text choices, which are themselves string values. Take, for example, the year in school choices type we've seen earlier Shortened here to YIS on the right side to fit in the slide. If we take one of its members, say senior, and cast it to a string, what should we get? The default implementation from Enam is clearly unsuitable. The option

one option, sorry, suggested was to return the label since Dunder Str is supposed to return a human readable presentation of the object. Another option is to note that text choices inherits string. So its members are already strings. So we must return the string we started with, which is the value. A variation on this is to return str of self-value for all choices types. When I realized that both above arguments were valid I made a suggestion based on my favorite tenet of the Zen of Python. In the face of ambiguity, refuse the temptation to guess.

Raise an exception and let the developer resolve it explicitly on a case-by-case basis. The option not to provide text choices and let users run into the issues themselves was raised but never considered seriously. The debate was left unresolved and the default Enam implementation was left in place until a bug was raised during the beta. By a user who put a value from a choice into a field and then had that rendered in a template. This made us realize That choices are not objects to be used in their own right, but just a way to pick values to put into fields. And once those fields are in, I'm sorry, and once those values are in fields

They should behave as normally as possible, especially for common functionalities such as Dunder Str. So we chose the SDR of the value for all choices types. Now let's talk about the implementation. We are going to see some advanced Python here And delve a little into things usually referred to as dark magic. I hope I can make it clear enough, and if not, I hope to at least get you curious about these topics. The main challenge of the implementation is to make a class based on Enam's Enam

such that a user who defines subclasses can add labels in their definitions. We need the labels to not interfere with the definition of the enum, but still be stored and made accessible later. So we want to subclass inamzinam, but what we really want to modify is the subclass creation process To enable the creation of classes which are subclasses of our class and have special properties. For this, what we really need to inherit is the metaclass In case you're not familiar with metaclasses, a proper introduction is out of scope for this talk.

But in a word, classes in Python are themselves objects, instances of other classes. And these latter classes are called metaclasses. And just like classes can control the initialization and behavior of their instances, metaclasses can control the creation and behavior of classes. Enams are defined with their own very special, very intricate metaclass called Enam meta. We inherit this metaclass mainly to override its Dandanu. And when we inherit choices from Enam, the most important change we make is to use our own metaclass.

So let's look at the details of this Dundee. Dunderneu in Metaclasses takes four arguments, of which the most interesting is the last one, called Class Dict. It holds a mapping of names to all the objects defined in the class body, methods, class attributes, and whatnot When our metaclass inherits in a meta, we get a special class dictionary with additional properties defined by our parent metaclass. One of them is the list of member names. When we say member here, we mean one of the values of the Inam. When you define an enum subclass, you can define enum

members, but you can also define methods and other things which are not members. The Inan Meta class takes care to produce a list of just the member names. We go over this list of names and for each one of them we pick up the value and we look at it. We check if it looks like something that is a value with an added label. That is, if it's a sequence. with more than one element and the last element look looks like a label. It's either a string or a promise, which is what you get if the user puts in a translatable string. If this is the case, we take the label of the value

and leave in it only the preceding parts. Otherwise, we leave the value alone and generate a label from the key, which is the member name. The label goes into our list and the value without label goes back into the class list. When we have gone through all the members this way, we take the cleaned app class dict and pass it up to the Enam metaclass to generate the new subclass. This is how we accomplish hiding the labels from the parent while collecting them for our own use. But we are not done with the Dandonim yet. We still need to make the labels accessible.

So we have the list of labels ordered according to the order of Enam members. And from enam, we also get an inner dictionary, value to member map, in the same order. We have everything we need to pair the labels with the members There were two ways suggested to make the connection. One was to go over the members and labels and set each label as an attribute on the corresponding member. The other was to build a dictionary mapping values to labels and give members the property to access it. The map and property option was implemented first, and to be frank, the simpler

attributes option never got the consideration it deserves as far as I can tell It was suggested on the pull request at a time when I was no longer responding properly to comments, but before Nick took over, and there is no discussion of it. That said, I still believe that the option chosen was the right option, because choices are generic. And if a base type already has a label attribute, there will be less breakage this way. Although a mapping property sounds complex, the implementation is very concise.

In this line, zip treats both its arguments as iterables. So from the value to member dictionary, it gets the keys, which are the inarm member values As we noted, the dictionary and the labels list have matching orders, so this creates the right list of pairs to pass into dict. Then, at an early version, Nick suggested this line. I'm going to pause for a few seconds to give you a chance to figure it out for yourselves before I explain it

Okay , So this is one of the most impressive single lines of Python I have seen in years. If this was chess, it would be the revealing move of a deciding combination. If this was drama, it would be the point where three separate plot lines converge into one. Let's take it apart. Let's first look at this side. We normally see property used as a decorator, but decorators are just callables called on a function or a class to modify them in some way.

One may just call it on some function to turn it into a property. Now what do we do with the property? Remember, we are still in DunderNew of the metaclass. By setting it as an attribute of the newly created class, we are creating an instance property for the class. But what is this property? The property decorator is usually applied to a function or member function taking just one argument, self. And this looks nothing like that. But it is a function taking one argument. And when used as a property, the argument passed will be the enum

member itself, which for the purpose of dictionary get Is equivalent to its value. And so the lookup is made. As impressive as this line might be, it is a little obscure And in the final version, it was changed to this one, which is less of a wonder, but a little clearer, and makes the code more maintainable. So that is all I have to tell you about this story so far.

But it doesn't have to be the end. There are still things left to do. There are still things you can do with lists of tuples, but cannot do with choices classes. And they're waiting for someone to solve them. There are features of other implementations which haven't found their way into Django. So there may be room for more people to get involved. And that is really all. This is how you can reach me on my personal or work emails or on Twitter or where I do open source And as you may or may not be aware, today is the Jewish New Year's Day.

So a happy 5781 to you all. And may it be better than last year. And uh thank you and bye-bye.