GNU Mailman 3 and Django

Hi, thanks. Um okay, let's start. My name is uh Florian Froch and this talk is about Maiman 3 and Django Uh I don't know if you have noticed that Maimon is you know only you know s minutes or seconds, just a few moments before a new uh version change to version three. Um so this talk is specif specifically about Maman 3. Uh some quick words about me. I um live and work as a freelance web developer in Berlin. I do a lot of work with Python and Django, but also quite a bit of work with front-end technologies. I also do some open source work. I'm on the Maimon3 dev team. uh which I joined a couple of years ago. And

as he said, I mostly work on things that have to do with the web UI for the list management and the REST API. Okay, um so what is this talk about? Um well Mamma three has um Two new web UIs for list management and archiving. List management meaning creating lists, subscribing people to them uh archiving meaning um making messages that get through uh sent through the system searchable and browsable and we decided to use Django for both of them So I'd like to show you how you can use and it how you can use and integrate those Django apps that we have built. And how you can use uh Maiman 3's internal APIs to write to write your own Maiman-related applications in Django.

So um quick warning: this um Mayman 3 is still in better state. Uh but yeah well as I said we're not not far away from a full release. Um okay, uh quick overview over the architecture of Mayman 3 from the view of a web developer. Well I have to tell when I joined the Mayman team a couple of years ago um to start working on a web interface from for version three, um the development of version three was already well underway, so most of the code was there and when I started I had absolutely no idea what Mayman did internally. And the good thing was I didn't have to know that Because in Maiman 3 there are some very clearly defined APIs that you can use as a web developer to write any kind of

interfaces you like. Let's see what the actual components are. There's some module some component that we call the Maiman core. It's um the actual mainman python package that gets imported, imported when you start Maiman. And well, it retrieves emails from list members or people sending emails to mailing lists and it distributes them to the list members. Of course, it does a lot of other things internally, and it's a really extensive piece of software. But right now we don't need to know too much about what it does. Okay. For list management, Mame

3 has a RES API that you can use. It's um JSON-based and it's local. It's a local API to um so it's language agnostic and um it's not intended to be used over a public network. Um But it speaks uh HTTP, which is pretty nice. Um and we also have some Python bindings. There's a library called Mame and Client, which I will tell you something um more uh a little later And uh you know it turns all these HTTP calls to the REST API into nice Python objects that you can operate on. So this RASE API can be used to manage lists, meaning creating lists, subscribing, unsubscribing, moderating messages,

changing preferences, and so on. The other internal API that is provided by the Mayman Core is uh the iArchiver interface. And um, this is a Python API. It's based on ZOP interface. I don't know if you're familiar with ZOP interface or interfaces in general, and it's well it sounds worse than it is, pretty easy to use. It's a Python API that gives you access To every list post that leaves Mailman. So if someone sends a message to a mailing list and the message gets through, so it's not moderated, it's um The person is a legit legitimate sender to that email list. This API will give you access to a copy of that email that gets sent out to all the list members.

We have two Django applications that can be installed that use Maimon3. One is called Postorius. This is the app we use for list management. Uh the funny name has uh well it's a mixture mixture of the fact that um many people use jazz re jazz music references for their when they name their pi uh their their Django apps It also has to do with email and dad bass guitar players. And what's worth noting about Posturiz is um that it has been created in large parts by students who have been participated in the Google Summer of Code, which is a Really nice program that Maimon has been participating in as a suborg of the Python Software Foundation. And there have been a number of students during

the last years who have implemented features for posteriority, which is so kudos to all those students And um yeah actually one of those students has given a uh DjangoCon talk at DjangoCon Europe about form sets because she used form sets, Django Form sets. uh in her posturius um project. So if you don't know about form sets, you totally should uh totally check it out. The other Django application that that's there to use with Mayman is um called HyperKitty and this is the new mailing list archiver. It has been created by a couple of people at Fedora who've done amazing work. It looks really nice, has has a really nice UI. It has some built-in statistics so you can see which um which email lists get the most attention.

Uh the threads are displayed very nicely. And also you can post to uh um to mailing this directly from the archiver interface, which is great because some people just don't like to use the email clients all the time. So that's Nice. So um let me show you some some really random screens of the two interfaces. This is the Post Series list index page. Well, I created some dummy lists today. A page to edit your subscription preferences, whether you like to have um to to to get uh daily delivery. uh as a digest or immediate delivery, stuff like that. So these are very random screens.

This is the archive index for HyperKitty. As you can see you have nice little uh graphs that show you the um the activity index of those lists. This is an overview of all the threads of a certain list. By the way, these screens are taken from a demo that the Fedora people have set up to test their HyperKitty development. So um if you want to check it out further you have to Google Fedora and HyperKitty and demo. So how can you install and integrate those applications? Well There's one installer to get them all.

It's the mainman bundler. It lives on Launchpad. You can download this project. And if you install it, it will download Mayman, Posturius, and HyperKitty with all their dependencies and will install it on your system. And it will make it pretty easy to hook it up to your uh to your local uh main server and to your web server This package also provides a separate Django project for which holds PostSerius and HyperKit with all with all the settings they need. So um you might say well I already have a Django site that I'm running on my server and I don't want to put up another Django project, so could I just can I just please just integrate Posterius and HyperKitty

Of course you can. Both uh live on pip, maiman two of course, and um you can just install them, plus some settings that you have to add to your um existing settings file that Depending on whether you you want to install both of them or only one of them, you can check out these two example projects. Um to see um to check out the necessary settings that you have to add. So other than that, they're behaving just like pretty much um every other um uh reusable Django apps that you can just put into your installed app setting and and and you're good to go So one of the more interesting parts is how you can write your own mainman apps.

How can you actually use those um those APIs that I was talking about for list management and archiv and archiving? So say you have your existing Django site and now you have installed Mayman and it's pretty running pretty smoothly and you have um use Postorius maybe to set up your lists and or uh you'd be just hyper kitty but just on your on your site you want to have maybe a single view which um you can edit yourself which holds a subscription form for a set for a certain list. How would you how would you be able to do that? Well, um maybe remember I was talking with main and client, the Python bindings library that does all the HTTP calls for you. Uh first of the first thing you have to do of course is install it. It lives um You can find it on the cheese shop as well.

And then the next thing you do is you import the client class from the main and client package. By the way, this is not a typo. The project is called maimon. client , but the package is made uh is called maimon client because it doesn't sit in the maiman namespace. So um do you create an incid instance of that client class um by giving it the um providing it with the local uh REST API URL and a username and a password. These are the default values and you can change them in the in your global maim and config file if you want to. You should do, of course So um the first thing you do when you start with the clean slate is create a domain.

Maman can operate a number of domains just like Uh many machines, you know, serve different uh domains for for websites. Um Mayman can uh serve different domains as well. So you add your own domain, I call it mydomain. org, and it will return a domain object that you can operate on further. This domain object, for example, has a create list method And that you can use to create two lists. Well, by the way, you maybe you've noticed I've put some small uh comments below each of these statements um to indicate the URLs that are actually called when you. Execute those statements. So once you've created your lists, those lists show up in the client's list

property. If you want to operate on um on a single list, you can get the list object from the client's getList method. And you can use that list meth uh list object to subscribe and unsubscribe people from that list. You can also provide it with an optional username, which Maman might or might not use when sending out messages. So and once you have subscribed someone, this person, this member

will show up in the in the list objects members property. So This is just a very simple example, but you know the the principle is Mayman client gives you objects with methods that might return other objects that you can operate on. And this is just uh the both most basic use case to uh create lists and um subscribe people to it. You can also moderate messages, change list preferences, and so on So the documentation is um sits in a REST file, uh in the RST file in the Mayman client package. I don't think it's um on Python hosted or read the docs. I have to I have to change that So how would you use that in a in a Django

view? Well, there are obviously many ways to do it. I chose one well example that fit on a p on on a single page. It's a very simple class-based view. It has a get method and a post method. And it has a separate property method that returns the client instance so we don't repeat ourselves. Um well if we take a look at the get method, the first thing that happens, um the list object is retrieved um through REST from the client um from main client from main client And we'll yeah, then there's an imaginary subscription form, which probably holds just an email address, depending on what your needs are.

And um so the list description and the form are um provided um to the to the um request context. So if someone actually posts um uh submits this uh script sub subscription form um The form is instantiated with the request post data , which should be uppercase, not lowercase. And if the form is valid, the person who submitted the form is subscribed as a list member. And then you just usually redirect to the get to wherever whatever your target page is. So um

as I said this is just a very basic example. You could also use a simple uh view function or um If you have several main views, you might maybe put the client instantiation method, which sits in the client property here, in a utility function. So Yeah, this is just one example. Okay, now um what we have done is we have say you have installed Maiman and the web UIs, and now you have Created your own page with your own main functionality. And you say, Well, I'm still not happy. I have one list which is a little bit different than all my all the other mailing lists that I'm hosting.

It's sh it's a private list, it shouldn't go into a public archiver. And um or any usual traditional archiver as well. And um It's say it's a support list and the list members are your the the the support staff of your organization. And uh you want to get a copy of each message that gets sent to the to this list and and processes in a different way. You want to store it into your Django database to display it in Django admin or whatever. So in order to do that, you would um be able to use the iArchiver interface.

So this is a very simple example how to use the IRC interface. This is um, as it said, the Python interface. Basically, what you have to do is create a class that implements three methods and has one name property. The first method is the list URL method, which uh should return a URL. where uh where your list summary page can can be found. It's just something that Maiman puts into the email header. It has another method that you must implement, which is the permalink method.

It doesn't have to return anything. If you want to, you can return a separate URL for each method that you can calculate. But you can also return none. But you have to still have to implement that method. And the really interesting stuff sits in the archive message method. And there you have access to Maiman's internal mailing lists object, the mailing list that this message is sent to. and a message object which is cast to a string but also has get methods to access the message headers. And The way for the Maiman core to know that you have correctly implemented

your your archival class is it uses ZOP interfaces. You've probably noticed noticed the implemented decorator on top of the class. Well it basically says Maiman, okay, this is a class that implements the iArchiver interface. And before you run it, you can check if it's if everything's there that's needed. So that's the one thing you have to do when creating an um an exam uh an an i archiver um class for your custom archiver. The second thing you have to do is, of course, you have to tell Maiman about it. So um there's your central Maiman config file. And all you have to do is add a section

which starts with the archiver and the dot and then the name of your archiver. It has a class property and this is the Python path to your archiver implementation. And then of course we have to enable it. And if you want to, you can add some configuration for your custom archiver. So if you have some Excel configuration that this archiver class might use, might use, you can put it there. So again, how would you use that in the Django context? You might think, well, that sounds good. I can just import my models.

I have, you know, in this example there's an imaginary my list post model And um if you would import it uh at the top of the file, it would raise an um uh improperly configured error because when may when when Maiman imports this um this this module in this class um you're outside of your usual Django environment so um There's no whiskey app instance and the settings are unknown. So what you have to do is that you have to set up your Django environment by setting the Django settings module module environment variable. And um of course you shouldn't probably hard code it, but um you know

this uh Otherwise it wouldn't have uh have fit onto the page if um I wouldn't have you know uh if I wouldn't have put it like that. Okay, and then after you've you've done that you can import your your your your uh Django model and And save whatever information you might want to read from the message file. Again, the message property that is handed to the archive message method. Uh the it has a string represent representation which is the complete message source and you can use Python's built-in email passage to iterate over each line so you get the raw uh email body uh and the subject and the date. So whatever you

information uh you you um you want to to read from that you might probably want to use the email package um for that. Okay. Um So um as with many open source projects, you're welcome to contribute. We are a small group of people that are very enthusiastic, but um also have like also with any many other open source projects, not the time that we'd like to to um to uh um to work on it Also, since we're just you know short before uh before uh new major release of mainman version 3, it would be nice to have more bug reports. So

If you got interested, subscribe to MaminDelopers at python. org, which ironically runs on Maimon 2. We also have an IRC channel on Freenote. net. You can check out a wiki. A lot of the documentation is there. And if you have specific questions for me, my IRC handler is Florin F and I Um I also hang out a lot on May Man on the Mayman channel. So um thank you very much.