I never Meta model I didn't like... by Russell Keith-Magee

Speaker 1: Well, good afternoon, Cleveland. As I said, my name is Russell Keith McGee. My day job is as CTO and co-founder of TradesCloud. We're an Australian software as a service for tradespeople, plumbers, electricians, carpenters, people like that. But the reason that I'm here today is Django. I've been a core team member for the last 10 years. I'm president of the DSFs for since 2011. I've served on the technical review board for the 1. 7, 1. 8 releases, and I'm also on the security team. But one of the other roles that I've assumed over the years is as a mentor in the Google Summer of Code. For those who aren't familiar with the program, Google Summer of Code is effectively an internship for college students. Open source projects apply to have students. The students then apply to be a mentored in that particular project. Google very kindly picks up the tab of paying them a stipend

Speaker 1: in exchange for 12 weeks of contribution to an open source project. The only catch, it needs to be open source and it needs to be done as solo work mentored by an expert from that project. Now this year, this year is actually one of the first years in about six that I haven't been actively mentoring a student. But over the years, the Google Summer of Code has resulted in a number of very large features that have been added to Django. The fact we've got multiple database support, the ORM aggregation APIs, the systems check framework in 1. 7, and in 1. 8, a formalized meta model. Okay, so what is a meta model and what is metaprogramming? Well, in short, metaprogramming is when you're writing code that can reason about the code that is running It's especially useful when you're writing generic frameworks because instead of encoding specific situations, you're encoding how to respond to a generic situation.

Speaker 1: An easy warm-up example. Metaprogramming can be found inside Python itself. Let's say we're going to do some graphics. Okay, so we're going to need a point class to represent uh the points on our graphics uh graphics canvas. We're gonna do both uh 2D and 3D representation, so we're gonna use some subclassing. Okay, nothing too challenging here. We've got some initialization methods, we take some arguments, we instantiate some some objects We want to be able to output those points in some way. So you write some output methods. You write output point for a two-dimensional point and output point for a three-dimensional point. Okay, nothing too challenging there. And when you run them, you get exactly what you'd expect. But what you really want is rather than having a separate method for each type of point, you really want to have a single method that will take an object, a point of any type, and output it appropriately.

Speaker 1: And that's easy enough to write as well. You know, if you're coming from a language like uh C that doesn't have metaprogramming tools, or Java, which has metaprogramming tools that doesn't encourage you to use them that much. you might be inclined to do something like this. You do a class check. Okay, we've got one method called output point. We're going to print the things that we know are common, the x and the y coordinate. And then if this point that we've been given is an instance of a 3D point, then print the z coordinate as well. Okay, and that does work. There's nothing wrong with that. But a more Pythonic approach is to use metaprogramming. What you do is you can every you can check for a class, if you check for a class uh that you're only checking for 3D points, that will work, but if you check for properties of the instance, you can adapt to any type as long as it has the right attributes. And so

Speaker 1: Python provides a tool, provides a built-in method called Hazatra. That is, what you're checking is rather than saying is this object of this class, you're saying if this object has the attribute Z, then print it. But what if there's even more attributes? We're going to start modeling Doctor Who now, so we need to include time into our XYZ coordinate. So that means we've got to do another Hazatra check, right? Well, no, because again we can use metaprogramming here. Every Python object has a special attribute named dunder dict that contains all the attributes of that instance. So Second line there, it says p dot dict, that's all the attributes of that instance. We sort those and we iterate over them. For each of them, we're going to print out that attribute, the name of the attribute, and get the value of that attribute off the object P.

Speaker 1: Now the thing that's interesting is that that definition will work with any object, not just points You can point any Python object at that method, and it will as long as the uh the attribute can be output as a string, that will successfully output a representation of that object. Now, that is admittedly a very much contrived example. You probably wouldn't actually build it that way. You'd actually use the the the wrapper mem method or string method on the actual class itself. But It's enough to demonstrate the point. By leveraging metaprogramming, you can do some very powerful things with very little code. And in Python at least, it's fairly easy to understand code. Okay, so what does this metaprogramming mean when we move to Django? Well, it means being able to do some the same sort of thing we just did with Python objects, but do it with database models.

Speaker 1: One obvious place this is really useful is model forms. I have a Django model. It has a bunch of fields. I want to be able to display a form to edit an instance of that model. Now, I could define a form instance for each of my models and then add the fields that I want to edit for those on those models, or I can use better programming. Using metaprogramming I can iterate over all the fields on the model and for each of those fields add an input that's appropriate for that form. Where's that particularly helpful? In Django's admin. Django's admin allows you to just register a model and you get an entire admin interface for that model automatically. How is that possible? Because Django's admin uses metaprogramming to introspect the models that form up a Django project and then generate forms for those objects.

Speaker 1: Now, of course, you can get a much better, much more customized interface if you spend a whole bunch of time configuring your admin views. But as a first pass administration interface, the metaprogramming approach lets you hit some very big runs very, very easily. Django's admin also requires you to register all the models you want to display on the admin. Now, strictly you could even avoid this step if you wanted to. Django's App Cache is a form of metaprogramming which allows you to introspect the applications that exist in a project And then the models that make up that app. So if you were so inclined, it would be possible to write a completely zero configuration admin application for Django. Now, although the Meta API was just one of the banner features for 1. 8, that doesn't mean it's a new feature, it's just a new stable feature. Django has had a meta API since the very beginning.

Speaker 1: There were some very big changes in the early years, but by the time Django 1. 0 was released, the API had pretty much settled down. What did that API look like? Well, if you had a model called My Model It had an attribute called underscore meta. And that object had a bunch of methods on it for introspecting the fields on that model. And there was get field and get field by name and get fields with model and get concrete fields with model and get all related many to many objects with model. Looking at that API, you can probably see why we weren't too keen on making it stable. It worked, but it wasn't pretty. Now what you can't see from just that description are the warts. Some of those methods include fields from the parent. Some don't. Can you tell which ones? As a result, the Meta API was an

Speaker 1: unofficially stable API. It was an API that formerly wasn't stable, and officially we reserved the right to change it. But in practice, we knew that enough people would be relying on it that we didn't we wouldn't change it unless there was a really good reason. And the best good reason there is to formalize it so that we're actually publishing it and making it official stable API. Last year, as part of the Google Summer of Code, Daniel Pirathen took the job of clearing out all that craft that had accumulated in the API so that we could formally document and publish Django 's Meta API. This is Daniel graduating from university last year. As you can tell, he's a very serious, very somber, quietly spoken individual. But what was the result of Daniel's work? Well, a new stable meta API. And it looks a little something like this. The meta

Speaker 1: object is unchanged. You still say here's my model, my model dot underscore meta. But now there's just two methods on there Get field to return a field with a specific name and get fields to return a list of all fields on that note on that model. You can optionally include the fields from parent models and hidden fields if you're trying to learn something a little bit about your inheritance tree So the uh the uh the uh uh parent fields are the ones coming from your s your superclasses, and hidden fields are fields that back another field's functionality. For example, the underscore ID field on a foreign key, or fields that have been explicitly hidden, like a foreign key with a related name starting with plus. What you get on top of those two calls is a is a from those two calls is either one or zero to many field objects, which have attributes that then tell you the properties of that field.

Speaker 1: So is it in a relationship with another model? Uh what is the cardinality of that relationship? What uh uh what is the other model that it is related to? Was it automatically generated or is it explicitly defined in a model file? Does it have a direct manifestation as a database column? So is it concrete? Internally, this meant a whole bunch of code churn. So we could replace get all related many-to-many objects with model with a much much simplified call. And that much simplified call is effectively just a list comprehension over all fields, filtering out the ones that don't have the field properties we want, or the ones that do have the field properties we want. It also meant we gained some new functionality. Previously, generic keys weren't represented in the meta model. They couldn't be, because they were in contrib and we'd have to make a special case of that particular class to get them into cause

Speaker 1: meta -representation. Now we've removed the need for that special case. Any third-party field, including contrib including things in Contrib, like all the GIS fields and all the generic key fields. Any third-party field can get the same capabilities as a core Django field and be included in that meta metadata representation. Okay, so why is this a useful thing? Well To be completely honest, for most day-to-day builder blog use cases, it probably isn't. But it is extremely useful when you start looking at larger projects, in particular anything that starts approaching framework-level stuff. A stable meta API means you can now write functions that take a model rather than a model instance as an argument. This means you can write tools that produce high-level functionality that respond to the characteristics of a model.

Speaker 1: and generate functionality based on those characteristics. So here's a practical example where a Meta API can be very, very helpful. My commercial day-to-day Django application allows tradespeople, plumbers, electricians, to keep track of the work that they have to complete. Part of that means keeping track of customers. So I've got a customer model. Users can create new customers and if you raise a piece of work, it's linked to that customer. When you issue an invoice, it's linked to the customer. When you record a payment, it's linked to the customer. So if you ever want to see the full history of a customer with your with your company, you can easily retrieve all the work, all the invoices, all the payments that relate to that customer. However, this data is being entered by humans. So sometimes you end up with two records for Mr. Smith. What you need is a way to be able to merge two records together.

Speaker 1: Now this is easy enough to do. All you need to do is nominate which is the record you're going to retain and which is going to be the uh be deleted, and then you update all the foreign key and many-to-many references to the duplicate instance to point at the original instance instead Hey, that's easy. So what does this look like? Okay, well we've got some models. We've got a customer model with a name field and a bunch of other data. We have a work order model that has a reference, a foreign key to the work order and the order ID and a bunch of other interesting details. We also have an invoice and a payment model, both also owned by that uh by that by that customer. And so we can easily define a merge operation for those customers. An existing customer has could have work orders, it could have invoices, it could have payments, so given a duplicate customer We need to find all those related objects, update the foreign key references, and then delete the duplicate.

Speaker 1: Magic. So now you can merge customers to your heart's content. Okay, so time goes by, you decide to add some new functionality to your system, say the ability to raise quotes. So you define a quote model and some views, you roll out that code, and then someone merges a customer record. and all the quotes related to the dead instance disappear. What happened? Well, the merge mechanism doesn't account for quotes. You updated all the work orders, you updated all the invoices and payments, but the quotes weren't updated. So when you deleted the duplicate customer, there were still quotes related to that customer. So all the duplicate customers' quotes were also deleted. Oops. So how do you fix that? Well the first obvious solution is to update your merge mechanism, add the extra line you need for quotes, and that will work fine.

Speaker 1: Until you then add the appointments model to keep track of each appointment you have with your customer and so on and so on. What you really need is something that will adapt to any new model as you add it. And how do you do that? Your metaprogram. Rather than encoding the models that need to be updated, you use the meta model to discover all the related models and then update them. That way when a new model comes along with a foreign key to customer, it will automatically be included in that merging operation. So what we do here, we iterate through over all the fields If the field is a one-to-many, so it means it's on the remote end of a foreign key relation, and it's auto-created, so it's the far side, the the underset side. Get the accessor name, get the attribute, the uh get the attribute of that object, update the attribute to uh the

Speaker 1: customer to be the original, and off you go. Interestingly though though this doesn't depend on customer. You can actually make it a completely arbitrary merge operation with just one change, abstracting out the explicit reference to customer, if you notice just at the end there, customer equals original. Replace that with a metaprogramming lookup of the name of the field you're being related through, all of a sudden you've got a completely generic merging two models into one. That will work on any Django model. Now, caveat here, this is only working with foreign keys. You also need to reproduce this logic for many to many's and one-to-many's and so on, but the principle is much the same for those types of fields. Alright, so that's that's a simple practical example of somewhere you might use Meta in your own code. But the real killer app for the Meta

Speaker 1: API, the reason that I got interested in the project in the first place, is the potential for exposing new data stores into Django. For almost as long as Django has existed, people have been asking the question: how can I use X with Django? Where X is some non-relational data store. MongoDB, Google Ack Engine, React, Cassandra, CouchDB, whatever this year's favorite flavor happens to be At one time, the Django Core team harbored the dream that the ORM was completely data store agnostic. It was an object relational model, which is why you don't do joins, you have a filter. You don't group by, you aggregate. Now that dream didn't quite pan out. The year after Alex Gaynor did the multi-database project for the Google Summer of Code, he did a second project looking at building a non-relational backend for Django's database API. And it did work, but there were a bunch of problems.

Speaker 1: So the idea of a database backend for NoSQL was kind of shelved. But the question didn't go away. So, okay, what's the actual use case here? Well many parts of Django don't care what data store you're actually using. Views, URL routing, the caching layer. Just go ahead, use use MongoDB, use whatever. As long as you're just accessing data, you might as well use the native API provided by your data store. There's no reason to force the square peg of MongoDB into the round hole of a query set. But there are two big pieces of Django that are dependent upon the data store, model forms and the admin. So I put it to you that the question can I use MongoDB with Django Really means can I get a Django form for my MongoDB model and can I query and edit my MongoDB data in Django's admin? And if that's the bar you want to set, it's a whole lot easier and a lot more plausible to clear that bar.

Speaker 1: First question, can I get a Django form for my MongoDB model? Is right in the sweet spot of Django's Meta API. I have an object, what attributes does that object have? What is an appropriate form representation for each of those attributes? These are questions that aren't dependent upon having a relational data store. They can be asked of any model. Even a basic Python object could answer those questions for you. The second question, can I query and edit my MongoDB data in Django's admin? Only requires a little bit more information than that. You need to be able to know what models exist in a data store and you need to know how they're organized. You need to be able to do some basic CRUD operations on it. These query primitives are a much smaller API than the entire ORM. Now, okay, for the sake of historical consistency, let's call those operations filter, get, and save. If you can provide a duck that quacks the right way, Django's

Speaker 1: forms and admin don't actually care that it's not actually a Django model under the hood. It just needs to adhere to that basic API contract. Okay, so you won't be able to take your arbitrary Django application from PyPI and push it onto an arbitrary Django data store. But let's be honest, that was folly and it was never going to work in practice. But what you will be able to do is get the benefits of much of Django's tooling while using a non-relational data store The best part of all this is because it's public-supported API now, you can do all these experiments without actually making modifications to core. To prove the point, let's look at a case study. Django Mailer is a proof of concept that Daniel put together during his summer of code to demonstrate that the Meta API actually worked. What does it do? It uses Google's Gmail REST API to expose your Gmail mailbox in Django's admin.

Speaker 1: Now, as a quick yes As a quicker setup, it's not the same Django mailer that James Talbot released as part of Pinax, naming things as hard y'all. The code is available on uh on Daniel's uh GitHub repository. The other caveat is that this example does depend on a couple of small changes to Django itself, things that we are in patches that we're looking at getting into core , essentially breaking the last of the hard-coded exceptions in Django that say this is a model. So you can't just use this out of the box, but it shows what can be done with relatively little effort. How little? It's about 500 lines of code. Now, I haven't got a time to tear down 500 lines of code in the remaining 10 minutes I've got here, but I can give you a whirlwind tour of what you need to do. Now the following code is heavily edited for clarity. It will not even begin to work as as described, but it will sort of give you a flavor of

Speaker 1: where we're headed. Okay. So we start with the manager. You say you want authors. objects. all. Objects is the manager. We need a generic one that will point out Gmail and store your Gmail credentials when you get there. Okay. Then we subclass that base manager to provide a mechanism for issuing queries on specific object types. We need one to look at message threads and we need one to look at specific message messages because they're going to hit on different API entry points in the Gmail API. Okay, they're going to return query sets. What about those query sets? Again, we have a base class, a base query that sort of abstracts the idea of calling a REST API, and a base query set to generate those queries. Now you'll notice here that query set is a subclass of list. Because all it's really doing is representing a list of results. You could use your own container if it's useful, but you can just use Django's list, just use a base

Speaker 1: Python list if you need to. And again, we subclass that base class for specific data types. Get thread by ID is the actual A Gmail API call there. So when you call get on a thread query set, you're invoking getThreadbyID. And we do a similar thing for message query set, but when we do, filter is a method that returns a cloned version of itself. So you can start to chain those filter operations. And every time you clone it, every time you filter, you modify it to add additional query parameters. Next, we've got the meta model. Here we subclass Django's own options model, the class that implements underscore meta, because that gives us a bunch of functionality for free, and all we really need to implement is get field and get fields, and the rest all works. Now, we pull it all together. We define a base Gmail model

Speaker 1: that retains state, and we define a Python meta class that will control the process of object construction. Then we subclass, set a meta class for each of our our models and a default manager, and we're done. Now you can say thread. objects. get and message. objects. filter and retrieve data from your Gmail instance, assuming you've got your uh your credentials set up correctly in your settings file. All that's left then is to register those models in the admin. You have a message inline, look works on messages. You have a thread admin, it uses a bunch of inlines and displays the number of messages in each thread. Register that with the admin. Do exactly the same registration process as you would for any other Django model. And you don't. So That's a lot to absorb in a short amount of time. And it might it is does seem probably a little bit overwhelming.

Speaker 1: Keep in mind the whole thing is 500 lines of code, and I'm rushing through it to kind of show what is actually possible. But it is just 500 lines of code. And what you'll get what you end up with is reproducing a limited version of the API that should be familiar to you as Django users, models, managers, query sets, queries, and so on. So here's a call to action. Who's coming to the sprints? Show of hands. There we go, goodly number. Alright. Who's looking for something to work on? Yeah, a goodly number of those. Alright. Have I got the project for you? How about one of these? Have you got expertise in a non-relational data store? MongoDB, Google Ack Engine, uh Cassandra, React, CouchDB, or anything else that happens to be out there. Why not build a Django MetaDuck for that Jang for that data store?

Speaker 1: You'll be able to see your data store in Django's admin, generate forms for that object, off you run. What about more traditional data stores? You could wrap an LDAP store, you could wrap your local email inbox, your file system, even a pool of AWS resources with a Django MetaDuck and browse, even potentially do some light management of your AWS resources. Of in in Django's admin. Okay, what if you don't like Django's RM? Well why not replace it? There's no reason you couldn't make a Django MetaDuck interface for SQL using SQL Alchemy. So you'd be able to take a SQL Alchemy model as a Django resource, expose it into uh into Django's admin for most of your business logic. You're using the SQL Alchemy API, because that's the one you prefer. But for convenience and prototyping, you can just drop that model into Django's admin or into a Django form.

Speaker 1: Now, if you do choose to take on one of those projects, you are going to hit some rough edges. So I will warn you. As I said, Django's uh Daniel's Django mailer depends on a couple of patches that haven't been merged into trunk. I can't guarantee that we've identified every single one of the places in Forms and Admin that you might trip up. There's a couple of places where we probably have made implicit assumptions about the fact that a relational database underpins this application. But if you do find a bug, it's a bug we're interested in fixing. And the more people we have contributing these Django Metaducts, the more confidence we'll have that we've broken the RRM dependency in model forms and admin. So, I hope I've sparked your interest about the possibilities of the meta interface. We probably have got time for a couple of questions, or you can grab me over the next couple of days or online. Thank you very much.

Speaker 1: I can hear crickets. Here they come. Here they come.

Speaker 2: How's the documentation for this?

Speaker 1: Um well pretty good on the basis that there's not much to document. So there's literally two methods and a bunch of attributes. There is actually even some really good documentation for so you were using the wrong API that wasn't formal. Well, okay. that uh here's how you transform one to the other. So uh yeah there is documentation there and it's pretty solid. The documentation of how do you write the Django MetaDuck? Uh no But like I said, the only thing you need to be doing is essentially appearing to be a meta model. So if you implement getField and GetFields and a couple others, you're basically there.

Speaker 3: So do you think there is room in the meta -abstraction for some theories that wouldn't make sense in relational databases do in things like Mongo or other stores? Like extending the distraction to be a little bit more flexible around the edge.

Speaker 1: Yeah, absolutely. I don't know that's it's sort of where that uh is it a one-to-one, is it a one-to-many, is it a many to many, is it a Yeah, a blob in some meaningful way. Whatever that I don't know what that abstraction is, but yeah, I I think that there's no reason that we can't have that flag in Django, even if Django itself doesn't use it. But also on top of that you can you could you could set it up in such a way uh if you know you've got your MongoDB data type that needs to be able to filter on a particular flag, that flag is not going to be on for any other data type. So you only need to have it in your data store. You can say getAtra defaulting to false And any model that doesn't have it will will show that up anyway. So

Speaker 3: okay.

Speaker 4: I don't think this is a a new uh problem that I'm about to ask. New to what you've just described. But um I'm just looking at the the Django documentation and there's this page on model meta options and then this This other page on the model underscore meta API which is talking about the class options. And I'm really confused what the relationship is between these two things and is there a possible naming

Speaker 1: Yeah, okay, so some of that is a little historical. The n the name of the class options is never surfaced to the end user. Uh it's an internal name and underscore measure is the only one that is there. If there's something that we need to clean up, it's probably the documentation referring to options. because it isn't something you should ever really be instantiating. So yeah.

Speaker 4: But there's also options under the model meta options, which talks about options. Abstract.

Speaker 1: Uh oh okay, I see where you're going. Um

Speaker 4: I think there's actually two naming conflicts or potential naming conflicts for pe people who are not intimately familiar with how all this n is you know yeah working underneath.

Speaker 1: Um yes, I agree. Yes there is probably something that can be cleaned up there, yes.

Speaker 4: I'll talk with you later and see if we maybe there's some suggestions about

Speaker 1: Which one we call this out or take out of the documentation. Yeah. So I guess the the word options says he I can't think of anywhere that's actually surfaced as public API. It's something that is used because that's the name of the class. Not because it's actually uh a name you need to use. So it's a candidate to be deleted anywhere it's being used effectively. So Thank you. All right, well I think that's that's pretty much time as well, so thank you very much.