Building plugin ecosystems with Django

Speaker 1: WordPress. Who of you in this room has ever used WordPress in the career? Okay, that's what I thought. Even though we're at a Python developer conference, it's half of the audience and that's no surprise because WordPress is powering a third of the web, which I think is is hugely impressive and it's something where one obviously needs to ask why that is and how a single platform, a single software can be can be so successful. And one part of this is obviously that it's very easy to install WordPress on a commodity web space that you get for a dollar a month everywhere. But I think for the most part it's because of this. There's over 50,000 plugins you can extend WordPress with, most of them free, some of them commercial.

Speaker 1: Um and they range from very simple things like embedding videos easier into your blog posts to very complex things. Example would be WooCommerce, which is a WordPress plugin and at the same time the most successful, the m the most popular e-commerce shop system on the web um which lives inside WordPress. So WordPress is a lot more than a content management system nowadays. It's an application platform if you like. And um You can like that or not, like I'm not sure if it's a good idea to put a shop system into a content management system, but it's possible and it made WordPress hugely successful I'm the founder and main developer of PreTix, which is both a company and an open source project

Speaker 1: It's a ticket shop application for events to sell event tickets and it's based on Python and Django. And from the very beginning in 2014, when I started this project, a key design goal was to be extensible. To not make people fork or patch or monkey patch or modify the software when they want to run it because events are vastly different, they have different needs. Um but I've seen it in other projects what happens if if um versions diverge and if nobody um is able to upgrade anymore and I didn't want to run in the this situation and there's lots of ideas um how you could extend the software in a similar way on how to uh how you could extend WordPress. For example, in a shop system you might want to add additional payment methods or you want to add additional export formats for Weird accounting software in your country.

Speaker 1: Or you want to add in completely different features. For example, instead of putting a shopping system into content management system, we could put a content management system into a shop system, and we did. So how do we go about building such a plugin system and how can we use it? And I want to talk about some things that Django has in store for us and some things that we need to add ourselves. So Django has a concept that you might all know that is called apps. And the Django documentation defines apps as a Python package that provides a set of features. Light's very vague. And it goes on and says applications may be reused in various projects. So most of you have probably one or rather multiple applications in your Django

Speaker 1: projects with your that you build on your own and that are probably tightly intertwined and you're might be using a couple of apps from from other authors that you actually reused. But It's not that simple. Reusable doesn't mean pluggable. Most Django apps out there, like let's take as a very simple example, the Pulse application that you build when you follow the official Django tutorial. You can of course add them into multiple different websites, but it's complex to install and integrate them. You need to add them to your installed apps, you need to set up URLs, you need to somehow overwrite templates to make sure they fit into your page and you can link between them back and forth. And they're meant for developers, no

Speaker 1: fuse, they're fine, they can save you a lot of time. But it's not something that a user of your software, whereby user I mean someone who installs your software on a server, um might be able to do. So Django apps are mostly great for functionality that coexists but is only loosely integrated, like having a news journal website that has a polls section and you it you add in that third-party polse module there. But it's not it's not a plugin system already. The other thing that we have in Django is signals. And if you have never heard of signals, you can signals are a thing that works like this. You basically define a signal. signal is some event that happens and you define it by um instantiating the signal class.

Speaker 1: And then you define receivers, which are functions that will be called whenever the signal is fired. And then you fire the signal by calling the dot send method on the signal. And behind the scenes it's basically a list of functions. So whenever you you use that dot receiver decorator, it adds a function to the list of functions. Whenever you Do the send call, it calls every function in that list and collects responses and gives them back to you. If you talk around to people around here, they will tell you signals are bad and evil and you shouldn't use them. And they tell you that for a reason. Because signals when used wrongly make your code paths hard to follow, make it hard to depack your software, and uh therefore are often discouraged, especially to new users of Django, that you shouldn't overuse them. And I agree you shouldn't overuse them.

Speaker 1: But there are some very valid use cases, especially when you try to integrate parts of applications that are controlled by different entities developing the software which we will end on with. So what we basically want to do is use those concepts we have in Django of of apps and signals. and bring use them to bring them closer together and make it easier for for people to write plugins to your software. And one of the things I want to have for that is uh auto-discovery of plugins and easy installation. So what we do is in every plugin we have an underinit. py file. And um it contains an app config like modern Django apps do.

Speaker 1: It looks like any other Django app config, but we we insert an inner class the It's thous thousands of other ways you could do this, but we do it like the meta classes on on models, where we specify okay, this Django app is a plugin and should behave like our plugin mechanism does, and we can add additional metadata like name or versioning or Whatever. And then we define a ready function because one thing you need to care about when using signals is that the file where your receivers are defined actually gets imported. So the re um decorators actually being executed and the functions actually added to the list of functions. So that's a the a common source of error. And we define that as the default app config for that Python module.

Speaker 1: It's fine. So in itself this does nothing. But when we combine it with a few other things, it will be. And one of those things is is the setup. py file of our plugins. So our plugins are all installable pip packages. And they have a setup. py file which defines the the package metadata. And they have an um user feature of the packaging ecosystem that is called entry points. Entry points are basically a plug-in system on the level of pip or of the Python packaging ecosystem. And when you can define a package in your You can define an entry point in your package definition that basically says I'm providing this feature. And then later you can query the package system to get you give you all packages to implement that feature.

Speaker 1: The syntax is a bit weird but Can ignore that for now. And then in in our main application, in our core application, we can go into our settings file and we can just iterate over all of the installed packages that have said, okay, I'm a plugin, I provide this functionality, and we can just add them to our installed apps automatically. So now installation of those plugins is easy. It's something that every system administrator can do. It's basically pip install the plugin and then migrate. You cannot really do the migrate thing automatically because you might want to do it in a different environment with different database settings and so on. But everything else you can you can put away basically. Um you might want to do some collect static stuff or anything, but you can integrate that if you want.

Speaker 1: So that's nice. We can now easily install plugin without changing any code. However, the they are still not the apps are still not talking to each other. And the the plugins are still not doing anything. So the next thing we do is we want to do URL routing automatically. We want to make it possible for the plugins to register views. But can actually be used. So we just put an URLs. py file into the plugin. Looks like any other Django application, nothing special to it at all. Um but in the URLs file of our main application We um iterate over all apps in our Django project, filter out the ones that have said, okay, I want to be part of this plugin mechanism. Check if they have an URL submodule. If so, import

Speaker 1: it. And then include those URL patterns. And then we include this list again into your main URL patterns. And this way we automatically have namespacing. Like we automatically assign um a namespace based on the name of the plugin to all URLs so your plugin authors don't need to think about clashing names with other plugins and can just easily define their views. And if you want to be do something more advanced and want to be more fancy, you could do things like automatically wrapping all those views. In a decorator, you could just um define a function that walks through a URL tree or a URL table and attaches a decorator to every view and then then call that function when you include it. So for example in this uh in this case you would make sure

Speaker 1: that All views registered by plugins are only accessible by logged in users. But you could do any permission checks you like or any other checks, which makes it harder for authors of plugins to screw things up, especially when things change in the main application. And and allow you to enforce certain constraints on those on those views. The other thing that we need apart from URLs is a way to make those views accessible. We need to make it sure that somehow the link to the custom view of your plugin ends up in the global navigation of your main application. So we need to make it convenient to send signals in places where they can be really useful for simple things. For example, we use the custom template tag That sends out a signal

Speaker 1: and collects all the responses and just outputs the the the responses as HTML. So you could have something like that in your navigation that just collects HTML snippets for the navigation from from every plugin and puts them together. And the implementation of those taggers uh is pretty straightforward. It's mostly concerned with importing the Signal that is specified as a string, so that's not not really interesting. Um then it iterates over all the responses that it gets, um attaches them to a list of HTML snippets and outputs that to the to the template. So this way it's it's easy for both you and the plugin author to extend certain parts of your page. For example the dashboard or the navigation or put additional things into the the header to include additional CSS files and so on. Whatever you need.

Speaker 1: And um With our building blocks, we already got everything to build a pretty effective plugin system that makes it possible to easily inject additional functionality into our main application. without making it completely messy and and and unable to uh unmaintainable. And we've done this for a few years and some of the lessons that we learned Is for example that's really good to use signals like they are in Django for simple useful things like inserting some HTML to a dashboard. Doing an action after some business logic event happened, like doing something after an order was placed, doing something after an email was sent. But it doesn't make sense to use signals for complex interfaces. For example, one of the more complex things that

Speaker 1: plugins do in Pretex is um payment providers, like if you want to add Bitcoin in there, you can create a Bitcoin payment provider through a plugin. And we provide a class-based interfaces that we expect payment providers to implement with a base class that has some utilities and defines the interface And plugins are supposed to subclass that interface and and um and implement their crossing methods. And then we just use um signals as a way to to collect those classes and to discover them. There were uh other methods to discover them you could use meta classes or you could use decorators on the class or whatever, but we decided to have signals as the as of the single point of communication inside the plugins. Um

Speaker 1: so yeah. And the other thing that we learned is that if you do such a thing, you really need to write documentation. Because if you don't, nobody will ever write a plugin for your system. Um another thing that that comes in really handy is providing a cookie cutter template. If you've never heard of cookie cutter, it's basically a way to specify the templates for for source code projects or any projects of that matter. It's a template engine for folders, so you can you can specify all the boilerplate code that you need every time And that can can easily be included again. Um I'm kinda running through this because I'm nervous and speaking quickly, so we have time for an advanced topic.

Speaker 1: That is nice. Our application is a multi-tenant application and both if you self-host it and if you we do the software as a service thing. So we have very different clients using the application and they all share one Django instance. And what we do is we provide them with a list of plugins that are available and we allow them to turn them on of or off per client. And this is really really useful for a number of reasons. First of all, it keeps your user interface simple. If you have a customer who doesn't need a certain feature, you can just switch it off completely. You just switch it off. In that case, no signals of that plugin will be called for that customer and the functionality is just gone. And if they need it, you can just turn it on again. You can use that

Speaker 1: this this approach, even if you're not building an open source application where other people write plugins, you could also use it in closed source software to, for example, easily hide things behind a feature flag. You could use it to easily implement pricing tiers if you have specific functionality that's only um available to to customers who who paid a certain amount. You could just use similar mechanism to just turn off parts of your functionality completely for them. And yeah it's it's it's working really well for us. And the way you implement it is is basically you need to somehow store the list of plugins that are enabled for a client or a tenant. And in our case we just store a comma separated list of strings um because it's simple

Speaker 1: and then we we roll our own version of signal which is a subclass of of Django signals. And it works the same way, except that it um that I only um that it uh requires a client object to be passed as a sender. Sender is kinda a weird concept in Django signals. Signals somehow expect that there's a special argument to the signal which is the sender which makes sense in one some cases but not in others but it's in there. So we reuse it here and only if the the sick uh the plugin is active for that sender we actually send out the signal So how deciding that again is lots of uninteresting code of getting the Python module of a function and then searching if it's in that list.

Speaker 1: Um but you get pro you probably get the basic idea. The and then we use the approach that I talked earlier about um automatically applying a decorator to all views of a plugin to make sure only view views are only accessible if that plugin is enabled for that. Client. So yay, that's a plugin system. It works really well for a couple of years now. So the approach is proven to do to actually work and other people wrote plugins and extended functionality. And there's not a lot missing. There's one thing missing that people often ask for, and it's if you go if you think WordPress, you think

Speaker 1: logging into your web interface and installing a plugin through the web interface And I would say, yeah, let's just not do that. It works really, really badly with modern deployment strategies. In a containerized environment, you cannot modify the source code or you shouldn't really modify the source code of the application that's currently running. Um It works badly if you run on multiple application servers and it's basically remote code execution where network is not really something you want to do. Okay. I think I have a lot of time left for questions and even if those questions are please show that slide again for longer because it was a quick uh Thank you very much, and I'm happy to talk to you about it.

Speaker 2: Thank you. Indeed. We have a bunch of time for questions. So please line up at the microphone here or ask questions online on Slack or IRC or Twitter, which is uh DjangoCon QA.

Speaker 3: Hi, thank you for the great talk. I probably have like a bunch of questions and I might want to talk to you after so that I don't bore everybody else. But my question is with a plugin system, have you ever run into a case where you wanted to add a field to an existing model? And did you uh resort to contribute to class or did you find another way to do it?

Speaker 1: Uh you mean like that a plugin extends an existing model from the base application?

Speaker 3: Yes.

Speaker 1: Yeah. Now we have always avoided to do that. In that case we would create a separate model in the plugin and have a one-to-one relationship. Because you would end up with really weird compatibility problems probably if you if you use contribute to class or something like that. Um I haven't actually explored it I think I wouldn't like it from a design perspective.

Speaker 3: Okay, but if you use one-to-one relationship, are you are you not afraid to end up with really long queries? that like link five models together uh because there are five one-to-one relationships?

Speaker 1: So far not. Since the base application doesn't know about the plugins, the only way such a long query could exist is from the plugin. And it doesn't happen that often that a plugin goes through different plugins to build a query. Just didn't occur in it it's not a problem in practice so far. I see your point though.

Speaker 3: Okay, thank you.

Speaker 1: You're welcome

Speaker 4: Um so first off amazing approach to um enable and disable plugins based on the tenant. Um because I was assuming uh installation in your case means activation because you also add it to installed applications, but nicely done there. On that theme, another question, what could Django do to make your plugin infrastructure any easier? Is there something we could add? following that what would be needed that this plug-in approach because it's probably not that much code but still a little bit of code um to put this into a separate application and open source it

Speaker 1: I'm not sure. Um so the thing is it's the system is very opinionated on the one part, so I'm not sure it would be something that everyone agrees on is the right way to do this. Uh the other thing is that um It's not a lot of code and a lot of that code depends on your special case. For example, this multi multi-tenant thing. I showed it with the client model, but for us it's it's It's tightly integrated with our customer model and yeah, it's it's it's we thought about moving it out into a separate repository and there's someone there was a threat on Django developers a couple of months ago on this topic I think the subject is dynamic dynamic uploading or something like that, and somebody on that thread implemented a different approach to the same problem that

Speaker 1: He put out as a separate library, so that's something that you might want to check out. It's a little bit different, but the basic idea is the same. So I think Django shouldn't do a lot about it. But just not deprecating signals which some people ask for at some point. So that would be nice.

Speaker 3: Hey.

Speaker 1: Hi.

Speaker 3: Um I'm wondering if you could Tell me if your plugin system is pluggable in the sense that I could use it in another project where I would like to have plugins.

Speaker 1: So basically the code examples that I showed are simplified a little, but they are all working and it's not much more going on. So it's basically the same answer as to the previous question. Yes, it's it's hard to to pull it out. It's easy to transfer to another application. The person standing in line in in line after you has Copy it to two other applications so we know it works in in multiple explications.

Speaker 3: Alright, thanks

Speaker 1: Sorry.

Speaker 5: It's all right. Um can you speak to managing dependencies and conflicts with your plugins? Like plugins might only work with a certain version of of PreTix of the base system or might maybe conflict with other plugins or even depend on other plugins.

Speaker 1: That's one of the pain points at the moment. We haven't talked about yet. Um because Um plugins can currently not specify if they're specific to a certain version of the base application and we'd we'd laugh t t to have that checked at pip install time. It would be easy to check it at um at runtime, but it that might be too late to to trigger an error if there is something that yeah if it imports something that doesn't exist anymore so on. We're currently avoiding that problem by all plugins maintained by us or being released on the same date as the releases of the of the main project, which is not a long-term viable solution. We're still looking for a good idea there.

Speaker 6: Hi, real great approach. If you could tell me in terms of migration, like in admin area, when you enable disable plugin, can we migration can be done on that side so we don't need to do on a kind of terminal level.

Speaker 1: So database migrations come pretty much automatically in this approach because every plugin has has their own and you can just run migrate and it does the right thing. Because it's Django apps basically.

Speaker 6: Okay, and did you explore a bit of for Django app registration because some of the like a code that can be used so is that any kind of disadvantage compared to some of the like signaling

Speaker 1: Django app registration. I have never heard of that as a name, so I might not know the project that you're talking about.

Speaker 6: Okay. I'm happy to take a look at it.

Speaker 5: One more question. Um you said that you purposefully do not support one of WordPress's uh main advantages that is just hitting click on a plugin and having it installed, which I get remote code execution isn't fun. But have you considered having a plugin registry where people can at least search for plugins even if they cannot install them directly without help of their systems administrator.

Speaker 1: Yes, I have considered that. I haven't found the time to build it. There's an The nice thing about open source project is someone just a couple of weeks ago created um like the these GitHub repositories awesome. anything for for plugins to pretext, so I'm just referring to that one for now But yeah, there will be something like that.

Speaker 7: Thank you. How can someone uh discover uh new plugins? Something like uh for example in WordPress uh something like a marketplace um we where one can see a list of plugins Yeah, that's

Speaker 1: that's basically something um the same answer to the previous question. We currently don't have a good way for that. I'd love to have a way. Um I'd love to do this in a way that's also like it's Doing this is not so much code. It's maybe not worth it to pull it into a library, but if m more projects use this approach, it might be or similar approaches, it might be worth to have an application for such plug-in marketplaces that is reusable across projects. And I I wanna build that for years, I just don't get around to do it. So if anyone wants to talk about that at the sprints, feel free to approach me.

Speaker 7: Thank you.

Speaker 2: Okay, there don't seem to be any questions online. Um thank you, Rafael. And yeah, that does a