Wagtail as a Platform - Matt Westcott

Speaker 1: Thanks very much. Yeah. So hello, I'm Matt. I'm one of the core developers of Wagtail and one of the original developers. If you've spent any time around GitHub or Stack Overflow, you might also know me as Gasman. And yeah, it's great to be uh here in Cleveland for my first foreign trip in a very long time. So now um Okay. Focus. So um yeah, when I was uh young I uh had uh this book uh from the Mr. Men series, Mr. Clever, all about um this uh The man who's the cleverest person in the world, quite the cleverest person ever. And you know how as a small child your mind is constantly forging all these links between concepts to make sense of the world?

Speaker 1: And I have a feeling that as a small child, when I first heard the name Cleveland, I might have unconsciously made the link between Cleveland, Ohio and Cleverland, where Mr. Clever lives. And in Cleverland, clever trees manage to grow apples and oranges at the same time. In Cleverland, clever flowers get up and go for a walk. Clever worms drive around in cars all day, and clever elephants play tennis. So if it seems that I have unrealistic expectations of this place, uh I apologize in advance. So uh, but um anyway, my talk today is about the uh cleverness that we uh build into our Wagtail projects.

Speaker 1: We've adopted the term Wagtail as a platform for the idea of going beyond Wagtail's standard models and apps, pages, images, snippets, documents. And using the building blocks of the Wagtail admin to build your own apps. In the early days of creating Wagtail, we didn't really Think of it as a platform. Django was the platform. And if anything, there was a conscious effort not to create another platform inside of it. At the uh last Wagtail space, I two years ago, I spoke a bit about the uh inner platform effect and the tendency of software developers to build something that grows to badly replicate the platform it's written in.

Speaker 1: And this is an easy platform, uh easy trap to fall into, especially if you come from a background as I did of writing web apps to serve customers. And now you're suddenly switching to building a product to serve developers. The natural inclination is to treat them as customers of the web app. and give them an environment to do their developing in. If you think of something like WordPress, which is a PHP-based application. But when you're building a WordPress WordPress site, you're generally not going to be writing PHP code. You're going to be inside the WordPress environment. And the disadvantage of that, of course, is that the things you can create are limited to the things that we as developers of the platform

Speaker 1: have anticipated and got around to implementing. So to avoid that, we chose to leverage Django as a tried and a tried and tested platform wherever possible. You're defining Django models, writing Django template. The role of Wagtail is more of an extension to Django. So we provide a bunch of models like page and image. and the admin backend to maintain them. And if you want any more than that, you build it in Django. But despite our best efforts to not build a platform A platform of a different kind kind of emerges accidentally because in order to keep the code base of Ragtail maintainable, we built it in a sort of pluggable way

Speaker 1: The more peripheral bits of Wagtail like images and documents exist as their own apps that push their functionality into the Wagtail core. The core code of Wagtail Doesn't know anything about images and documents. There is no code in there saying if you encounter an image field, render it this way And in principle, those apps, the images, documents, snippets have no special status, and anyone could build a third-party app with a similar feature set. And I say in principle because I think this was more a guideline for our code quality and keeping concerns separate rather than something we really expected to be a common use case.

Speaker 1: But our first inkling that this did actually have relevance to the real world, that this was something that people really wanted to do, was probably meeting Prycelt Foundation in South Africa. to uh 2016 quite early in the uh lifetime of uh Wagtail and uh Prike Help Foundation's uh mission is to get health and social information out there in the lowest friction way possible, which often meant working with feature phones with text-based menu-driven interfaces It's very much on under the banner of content management, but a far cry from our existing notions of managing a collection of web pages. And that led us to rethink what Wagtail could be

Speaker 1: and the idea of it being sort of a more content editor-focused replacement for the Django admin. At the time, Model Admin existed as an external third-party module, and this was a major driver for adopting it into Wagtail, with the hope that eventually it would become integrated more into Wagtail's own architecture. And unfortunately, that's been sort of quite a slow process because it's one thing to say, here's a CMS we built, maybe you'll find it useful. And it's an altogether different thing to say, this is how you should be building your apps. In in the course of building Wagtail, we've come up with all kinds of building blocks that would be useful to third-party developers. But at the time we created them, we didn't know that.

Speaker 1: We didn't know which were the ideas that would have legs, which which of these building blocks would actually succeed and be useful and stable over time. The truth is, we've been making it up as we go along. It takes time for these new ideas to be established as, for example, the best way of doing chooser pop-ups. And by the time something has emerged in that direction, we might have had several false starts and halfway implementations that then diverge with their own edge cases. So we've got sort of several different copies of this code base that are slightly out of sync. And Putting that, pulling that back into a piece of stable, reusable code, or while developers are forging ahead, building whole new features and adding

Speaker 1: new edge cases, is it's like herding cats. And uh here's uh a prime example of us making it up as we go along. Um I think Tim sort of talked earlier about uh how every sort of code base has these sources of shame and uh This is the code comment I'm most ashamed of in my entire programming career. And I feel comfortable sharing this now because as of this week I've finally replaced it. Abstract code providing sensible default behaviors for objects implementing the edit handler API, which is a very fancy way of saying absolutely nothing. This is a classical edit handler. You can inherit from it, and then it will do what an edit handler does

Speaker 1: And yeah, the the reason I wrote such a useless comment is that at the time I didn't really know what I was going to write. I just knew that We needed some kind of object to act as a go-between for things like inline panels that the Django form framework doesn't handle by itself as standard. And uh so for the record, this is the new version as of this week. It's taken eight years, but we now know what an edit handler does. It specifies what should be on the form and it renders it. That's a bit less than it did before. And as a result, some of the more esoteric hacks like changing the form class on the fly in response to the current user's permissions, they Some of those won't work, they'll break, but uh the point is now it's well

Speaker 1: defined and where you've got those hacks, it's clear where those hacks should go instead. There's now no ambiguity over whether it's the form that drives how the edit handler works or the edit handler driving the form or why it's sometimes called an edit handler and sometimes a panel. because uh as you see here it is a panel now. Uh they they say naming is one of the hard problems of computer science and I think being able to give it a Clear name is a good sign of the newfound clarity in the uh underlying code. So even though this might not have been code that you've been uh dealing with uh day to day, I think these sorts of details do surface. In some ways. And it has been a slow process to open up these previously internal APIs to a place where we're confident

Speaker 1: documenting them and saying this is now part of the Wagtail platform. If you build your apps in these in this way, they stand a good chance of not breaking in future Wagtail versions. So yes, it's been a slow process, but it's one I think we've now made some big milestones and uh one of them is having a dedicated extending wagtail section in the docks to uh demystify the process. This uh Opening page on creating admin views takes you through integrating custom Django code at a very basic level, starting with a pure Django view and showing how that yeah you can progressively register that into the WhiteL admin URL namespace and then bring in the page furniture and a menu item.

Speaker 1: I have to give uh big credits to uh Daniela Prosida and his uh Dio Taxis uh documentation framework here for helping us realize that this was a missing piece and thinking about the structured way of doing documentation because we had the reference documentation for all of these uh these hooks, but uh nothing that really served as a how-to as saying Yeah, that there is nothing mysterious about this. You just write Django code. And as I say, this here is a very basic kind of integration. And that is a good thing. It's it's right that there should be a continuum from basic Django views doing things the Django way, up to fully embracing Wagtail's building blocks. Because

Speaker 1: right now I think there are quite a few places where you have a binary choice between doing things the Django way and the Wagtail way. And that's something I'd really like to get away from. Things like Django signal handlers versus Wagtail hooks. Django forms versus edit handlers slash panels. And that is probably the big one. As I mentioned, edit handlers are that they have been the major here be dragons bit of infrastructure that uh underpins around half of Wagtail's editing interfaces, despite no one really knowing what they were saying supposed to do. You couldn't really half adopt them and use Django forms up to a point and then edit handles for the rest because no one really knew which half you could remove without the whole thing crashing down.

Speaker 1: And hopefully now this is formally documented what it's meant to do. Hopefully that's going to change from here. So if Plain Django views are the first step on the uh continuum. Then the next one up would be generic class-based views. This is leaning heavily on Django's own implementation, and it and it means that If you've got simple crud views that don't do anything special, like say in this example, the uh site management uh um settings area for of uh of the Wagtail admin. Uh you can get away with writing basically no new code. This is all just configuration. Just make sure you've got other labels for uh for the bits of headings and uh the success messages and that kind of thing.

Speaker 1: Everything else is just plain Django form functionality. And actually we can improve on that a bit more since these views usually come as a group with properties in common like the icon and model. We've uh introduced this concept of a view set so that these properties can be shared and uh everything registered in one go This bit isn't documented yet. These are still evolving as new bits of existing WAGShell code, get ported over to it, and we find new edge cases that aren't accounted for by this. generic implementation. But I think we're quite far along that line and hopefully it should be in a state to call an official API sometime soon.

Speaker 1: This move towards pulling out generic functionality from Wagtail does raise the question of what sort of new building blocks can Wagtail bring to the table that Django didn't have already? Obviously Django has been around for a lot longer and there's been a lot of Accumulated thinking by lots of smart people behind what Django does right now. And it's quite likely that any idea we might have about General purpose, reusable ways of building Django apps out of these building blocks has already existed and been refined in the Django ecosystem. So What can Wagtail bring to the table? What makes Wagtail

Speaker 1: special? And I think one answer to that is uh pluggability. If you take something like the admin dashboard, the information you see here is pulled in from varying various subsystems of Wagtail and potentially third-party modules. That's very different from the traditional Django workflow where you've got one view function that has a global view of all of the information it's dealing with. It runs all of its own query. And you've got a template outputting that from top to bottom. And I think Wagtail is uh arguably, so yeah, this might be controversial statements, but arguably unique in the extent to which you can plug in the functionality from other modules in that nonlinear way

Speaker 1: And that's helped us to notice one recurring pattern in Wagtail that's perhaps uncharted territory in Django as a whole. The evolution goes something like this. You start off with some UI element like a menu or a dashboard that developers want to be able to plug their own items into. So fair enough. We create a hook for registering new items within that thing. So a menu item has a label and a URL, so we pass that to this registration function. But then Some point down the line you're in Kentra's special case. This one particular menu item is styled differently, or it's a button instead of a link, or it has a checkbox, something like that. So, okay, fine, we can make this menu item an object that can return its own HTML.

Speaker 1: But um but building HTML in a string like that is kind of clunky and I've probably already giving you some involuntary twitches from the lack of HTML escaping there. Um so this is always something you want to sort of do as a reusable thing really, uh where which gets those details right. Wouldn't it be nicer if we could use a Django template for that? So uh yeah, a quick bit of refactoring later. We're not building HTML up inside Python code there. And then a voice in the back of my head goes, wait, isn't renders the string the one that recompiles the template on every call? So that'll make it really inefficient. And I go, oh well, but maybe I'll check up on that later. Anyway, so templates are good. Let's make that into a standard pattern for our menu items. Um so then that's uh

Speaker 1: So now we just need to give it a template name and addictive data and the base implementation will handle the rest. But then further down the line you find, oh, we've got this other menu item that has some custom JavaScript behavior and needs to pull in JavaScript library, we're not sure if that's already on the page or not. And uh well uh Django has a solution for that in the forms framework. have these uh media object objects that encapsulate the css and javascript you need for a particular element and you can add these together and have it uh take care of the deduplication So let's uh borrow that idea and uh skip a bit because we're short on time and all of these sort of iterations that uh keep uh keep you keep having to add to this basic concept of an object that

Speaker 1: needs to render itself. We've gone through this cycle maybe half a dozen times, except sometimes we might not go right to the end of this or it might not happen in that order and we'll probably come up with different naming conventions each time. And it creates an inconsistent experience for people building on those APIs. So on the eighth or so time it came up, I went, well, let's build the one true implementation of this. and template components of the results. This is again part of the documented extending white tail API. And again, this is a an ongoing process to get the existing code on board with this pattern. But every element we port over brings us a step closer to Wagtail as a platform.

Speaker 1: It's something that you can learn once and then whenever you work with another area of Wagtail, it's okay, I know this. This is something familiar to me. And once you have a battle-attested solution like this, the really nice thing is that other possible users Start falling into place. You'll see this table style showing up a lot around Wagtail. And it always felt like this should be a reusable component. There's lots of tedious details like clickable column headings that nobody likes implementing and it exists in varying degrees of correctness across the code base. And the tricky bit of making this reusable is the contents of the table are different every time, even if the only difference is that this one has a parent page column and the other one hasn't.

Speaker 1: It's kind of hard to come up with a reusable component where the outside is static and the innards get swapped in and out. But if we say, well, what if each column is an object that knows how to render itself And that gives us the ability to construct a table using a definition like this. There's various different ways that these columns in this example can be rendered, but Anytime we need a fancy custom rendering, that's just a new subclass of column that knows how to render itself and its data in that particular way. So yeah, a common theme here is uh again, it's being able to build things in this sort of definition style rather than uh having to write code. It's only if you're writing custom logic that

Speaker 1: you really need to do that uh um that that you need to actually go down and write code. I think that's a pattern we can take a lot further. There's this this uh kind of mission statement for Wagtail as a platform which uh I I say the aim is to be able to recreate something like Wagtail Media in 10 lines of code and just being able to say um I say, yeah, just pick and choose this functionality that we know and know from something like uh Wagtail Images. Um there's uh various different things that need to be hooked up and uh I think I'm again running short on time, so I'll rush through this. Um, but uh yeah, that yeah, eventually something like Wagtail Media would just be um we have uh a model, we have these crud

Speaker 1: views, we have a chooser, uh, we have stream field block some way of making it embedded into a rich text uh rich text field and being able to bring all of these things together in just configuration and uh just say yes I want these features and then that is uh that that's uh the app that we built. And I think that's about it for me. Thanks very much

Speaker 2: Questions in the audience or on Zoom This gonna be partly gonna be trio , little current trio

Speaker 1: Yeah, so okay, so the yeah the question is yeah, where are we? Is this going to be sort of partially done in in 3. 0 or uh long term thing It's well, I I think the yeah, the whole mission of uh Wagtail as a platform, it's not really going to be something where we can say this is is uh ever going to be complete. Um I think that there's definitely uh sort of certain certain bits um like the uh that template that the uh of tables and things that are ready to go. I think yeah else other things are going to be a more long term thing. The well I think the edit handlers are probably the big thing for 3. 0 and yeah and making those kind of a bit more familiar. So they will now be using the uh template component pattern that I showed.

Speaker 1: And that 's it's uh on on the uh roadmap for 3. 0 that that will be fully documented. So I think that's probably a very large chunk of it. The idea of

Speaker 3: this seeks in the middle of points of the tree All the walking are different and drawing We're all doing how it's really functionality.

Speaker 1: Okay, yeah. So uh the the question is that uh when where we've got uh bits of the admin the turf of drawing information from uh sort of all kind of subsystems, how do we go about sort of planning that uh that process is that that's right. Right. So that is, well, it's, I suppose the the short answer is very carefully. Because where we're going from something that hasn't been very well defined in the past , then you you're never quite sure whether there's sort of bits of code lurking around that are relying on unintended behaviors or APIs that we want to change. So I think having some

Speaker 1: good unit test coverage really helps with that. But I think it's um Making sure that we'd where that any sort of refactoring we do, moving things into the into core is done as a sort of uh slow process with uh if we do have to change any APIs then making sure we've got a path with uh deprecation over a couple of releases And uh yeah, I think just try trying to make sure that uh where possible, we're not just releasing something that is just uh scorched earth and uh it's uh just yeah, you have to rewrite all your things. Intact.