Solving your problems by spelunking the Wagtail code (Harris Lapiroff)

Speaker 1: Hi, I'm Harris Laproff. I work for the Freedom of the Press Foundation. We run a number of projects that are using Wagtail. If you're interested in hearing more about those, you can go look up my talk that I gave last year called Using Wagtail to Fight for Press Freedom. Today, I want to talk to you about solving your problems by spelunking the Wagtail code. Now, to start off with, I just want to see how much I'm preaching to the choir here. How many people in this room have spent a fair bit of time reading through the Wagtail code? Okay. So some of you, hopefully you will all still find something interesting in this talk. Um for those of you who did not raise your hands, I'm hoping that I can convince you that reading the Wagtail code is approachable.

Speaker 1: Um and it might solve your problems and it will make you a better developer. So first of all, how do we solve problems? Um I'm talking about when you first have a problem you want to solve, a feature you want to build, and you sit down at your computer and you think to yourself, I have no idea where to get started with this. What do you do then? Well, first thing I hope most of us do is go to the docs. Wagtail has lots of features, there's lots of stuff built into it, and the best way to find out how they work is by reading the docs. The Wagtail docs are great. They're always getting better. You want to improve them. I hope you'll contribute to them. But eventually you're going to want to do something with your projects that is a little more off the beat and

Speaker 1: path, something the documentation writers didn't think to write about. So where might you go then? Well you might go to those community support systems that uh Tom was talking about earlier the Wagtail Slack, Stack Overflow. We've got a very nice, very active community of developers who want to help you out. But of course that's limited by, you know, the people who know what you're trying to work on being available on online. And you're also sort of relying on the fact that even though you might not be familiar with the code that you're trying to interact with with you're hoping that someone else is. So if you yourself haven't read that code, you're hoping someone else online has If you don't find anything there, or maybe before you go to ask other people for help, you might try by experimentation.

Speaker 1: Maybe you throw a bunch of things into a Python file, add in a bunch of print statements, see what it spits out. That's not quite reading the code, but it is getting a little deeper. You're making some reasonable guesses about what you think the code looks like and uh getting a sense of the shape of it. And if you're really good at this, maybe you uh pull up an interactive uh shell and you play around. Uh I really love using IPDB for this purpose, especially because it comes with this nice tab autocompletion. So you can just like pull up an object, tab autocomplete, see all the different methods on that object and play around with it. It's a really great way, if you're not reading the code, to just like get a sense of the shape of all the objects that you're working with.

Speaker 1: But let's say that you try all of those things and you're like still not quite sure how to approach the problem that you're working on. What might you do next? And actually I'll open this up to the room right now. Are there any ways to approach this sort of problem solving that you can think of? I imagine. Yeah. Mmm, yes, that's a good point. Using an IDE. Anyone else? Um one thing that I thought I didn't mention is you know maybe you uh hire someone who's more knowledgeable than you to do it. Sometimes sometimes that's the most efficient option. I don't know how to solve this problem someone out there probably does I'll pay them some money.

Speaker 2: that I can find or sometimes I poke around on like the Torchbox website.

Speaker 1: Yeah, yeah. Yeah. And in in some ways that is uh what I'll be talking about. I mean I'm talking specifically about reading the core code, but reading uh code from other projects also is a great way to do it. All right. So let's say you've exhausted all of those options. and you're still not sure what to do. You know, the documentation doesn't cover your problem. Maybe you're on a timeline. You can't wait for someone to respond to you on Stack Overflow. Um experimentation is like not telling you very much. You don't have the budget to hire someone else to do it. Most of the time, one of those ways will get you the solution that you're looking for.

Speaker 1: But you know, eventually they all have limits and you will run into a problem where you just don't know how to solve it. And what do you do next? You read the code. If you haven't done it before, like I said, I want to tell you that reading the code is easier than you'll expect it to be, and it means that you will never be stuck in it It is the only solution where it really makes your power to complete your project unlimited. And I think that we as Wagtail developers are lucky that we code in a language that is designed to be easy to read. You'll often hear Python developers say the code is the documentation. Well, those developers are wrong and please write real documentation for your projects.

Speaker 1: But they're not totally off. You know, Python is designed to be easy to read and it encourages you to write code in a way that other people can read it. And of course, in particular, well-maintained open source projects like Wagtail and Django are usually pretty well organized and predictable for finding things in them. So why should you read the code? Aside from the obvious that it might solve your problem, I think reading the code also has a few other benefits. For one thing, it makes you a better coder. You'll get a sense of how other people write code. And especially with a project like Wagtail, it can build your repertoire of best practices. It also means that when you're in the habit of reading other people's code, while you're writing code, you'll be thinking about the next person who's going to be reading your code, and so you'll write better code for them.

Speaker 1: When you read the code, you'll understand your tools better. You might come across things that help you understand why something in Wagtail works the way that it does, or you might build knowledge for the next problem you encounter. And when you read the code sorry, I have bullet points for this. And when you read the code, you'll know whose fault it is. You know, every once in a while I come across a problem and it's I've written some code and I think it should be working and I'm like, oh, this just isn't working. I don't know what I'm doing wrong. And I'll go in and I'll read the code and I'll be it's not me. There's something funny in this code. And if you can actually read the code, you can say that say that it's a good idea. And then you can go to Wagtail GitHub and open an issue and everyone's life will be improved. And finally, leading reading

Speaker 1: prepares you to contribute back. When you're in the habit of reading the Wagtail core code, it will feel a little less intimidating when you want to add your own. Um how would you actually get started reading the code? I'm going to go through one example of a problem that we encountered

Speaker 3: or

Speaker 1: that we solved by reading the code. Um but before I do, I do want to do a quick review for anyone who is um you know maybe a little newer to Python uh on just how Python import paths work. So very quick review when you've seen these import statements all over your Python files. What these do is they tell Python to find a file computer and load load file. There are some sort of special rules for how Python interprets those paths into files. As you can see, this import statement could resolve to several different Profiles, it might look for an object called sky blue in nature.

Speaker 1: Or if both nature and sky blue are directories, then we would look for it.

Speaker 3: py.

Speaker 1: Anyway, hopefully at the basic idea

Speaker 3: that we have an import statement, it's

Speaker 1: read and creep file, and if you want to, you can go find that file and see what it says. And in general Python is look for either in the directory that the current script is being run from. So for most of us working on a Django project, that'll be the same directory that manage.

Speaker 3: py is in.

Speaker 1: Or it'll work for them in the package that you've installed using PIP or PIP or whatever research

Speaker 3: management. But in general, that's why you can import, say, home

Speaker 1: done models for your own project, but you can also import WagD top core. models from the WagTel that you've installed. And if you're really, really scary. If you run it from a shell like this, you will get a different result than if you run it from inside of a file file. But enough about

Speaker 3: my problem.

Speaker 1: So many of you may know that Wagtail has an excellent form builder. If you've never used it, it basically lets the developer create a page type. Which content editors can use to build a form basically like a lightweight version of Google Forms or something. Um and then uh we use uh in particular you can build a form that will send an email to a particular email address every time you get a submission to that form. So we use these all over the place in FDX projects and use them for our basic content form. We've used them for running user survey stuff like that.

Speaker 3: But

Speaker 1: the form lets you specify the form less form content content editor specify a subject line. And that subject line will be used you sending the email for every submission that comes on that particular form. Which might be a Tyrion box

Speaker 3: looking a little like this like this.

Speaker 1: We found this frustrating. Firstly, we wanted more informative subject lines, but in particular because we actually have several of those forms feed directly into our support ticketing system. Uh and so it'll automatically send that email to our ticketing system. The ticketing system will open an issue and all of the issues will be called informed submission. Very fine. What we actually wanted is we actually wanted the site visitor to be able to give us a one-line subject when they're filling out the contact form and for that to become the subject of the email. So that's our problem. So that's our problem. So let's start by trying to solve it using a few of the maps that I thought about

Speaker 3: before.

Speaker 1: Let's read the documentation. Now the form builder

Speaker 3: docs are actually actually

Speaker 1: pretty good. They're not too long. They cover most of the basic stuff you want to do with forms, but I don't really see anything in here that has to do with my topic and changing the subject line. Yeah, I can see how to display form information in the admin,

Speaker 3: admin, not too much. Not too much else.

Speaker 1: So that's a bust.

Speaker 3: That's a bust.

Speaker 1: All right, let's try some of that V4 we talked about earlier. Now I could go online right now and post this question to Slack or post it on Stack Overflow, but that would be Take a while, probably be a little boring. So as an experiment, I'm just gonna ask this room. Does anyone in here immediately know how I would solve this problem?

Speaker 3: What's your guess?

Speaker 4: Just control F um what was the subject exactly within the code base and find if there's a file that looks right. So form submission within the code base.

Speaker 3: So your

Speaker 1: suggestion is that we read the code, which is what we will what we will do in a moment. Um like you know we asked the question, nobody responded. I often have this experience. I asked a very complicated question and then I don't get a response. Which is

Speaker 3: yeah, yeah.

Speaker 2: What I would probably do, and maybe this is cheating because I've been playing with the form builder recently, but uh I know that he that he It uses like I forget the plugin, but it like it wraps around David's ability to send emails. So I would probably then look up that documentation to see if there's any and then probably ask among that community looking for like dynamic subject lines instead of wasn't specifically at the Wag telephone builder, but like however it's on the email to see if there's any suggestions there for like customizing. Yeah, yeah.

Speaker 1: Definitely

Speaker 3: a good thought.

Speaker 1: Yeah, and Jangle's forms and does. have like a subject line argument that it takes, which we will see in a minute. But yeah, basically no one immediately can do this. possible someone else would like read the code for me and be like, hey it looks like there's this

Speaker 3: thing.

Speaker 1: But for now I'm going to do the process of reading the code. So I'm gonna like start up our project here. We use Docker Compose. Pull up an interactive Django shell. I'll import our form page model. And then what I'm gonna do is I'm just gonna instantiate um one instance of it so I can play around with it and see what it looks like. Like so here we go, we're creating a page. I'm not actually saving the page from this page. If I get to playing around a little bit more, I might do that. But for now I'm just creating this page object and then I'm gonna look at it and Wow, that's a lot of attributes. I really

Speaker 1: have like no idea where to where to get started exploring here. There's like all sorts of stuff that has to do with the wagta tree. It's a lot. So experimentation. I could maybe bind out what I'm trying to find out here. I think there was like a send mail method on the previous screen, but

Speaker 3: it's not going to be clicked.

Speaker 1: So let's try reading the code. First of all, this line comes straight from the documentation. It's an import statement like I was talking about before. And it basically tells us what file and Wagtail we're going to be looking for this particular code. Um now you could try finding that denial and your own file system on your hard drive. Usually I just go to GitHub. com and like Browse the Wagtail repo until I find the file. Important thing there is make sure that you're browsing whatever version of Wagtail you have installed. Usually things don't change too much between versions, but every once in a while you're like , Um and then you'll realize. So here's the code from that particular file. This is the abstract email form model. that the documentation tells you don't take

Speaker 1: out when developing your forms. And this looks pretty promising. I can actually see that right over there there is a send mail method. And it seems like I can make a reasonably educated guess. But that's probably the method that they've been called when these form

Speaker 3: submissions are being sent out.

Speaker 1: Now if I want to be absolutely sure of that I can read a little more code. I can see that there's also this process form

Speaker 3: submission method.

Speaker 1: And if I read down a couple lines, I can see that yes, if there's a two

Speaker 3: address defined on the model, then it looks like it does call the self

Speaker 1: stop sends

Speaker 3: mail method.

Speaker 1: Um now not quite quite sure this process form submission method is getting so let's just spool around to be sure if I scroll up a little bit I can see the abstract form

Speaker 3: superclass that that one was inheriting from

Speaker 1: And I see it has this serve method. Now from doing a lot of work with Wagtail, I know that the serve method is what Wagtail pages use to send responses to requests. Um and if I read through the serve method I can see right at the top if the request is a post request um which means that it's uh Submission

Speaker 3: submission.

Speaker 1: Then it then we'll take a check that was balanced. Yes, yes. It will call process form submission, which as we saw before. Or also calls send

Speaker 3: now.

Speaker 1: So I can feel pretty confident now that I was correct that the sends mail method is the one I want to be looking at. I went through that quickly. Does anyone have any questions about that? Great. So now

Speaker 3: going back down to this abstract email form, we can read through the send mail method and see what it's doing.

Speaker 1: So it looks like the first line it's taking a Q address field, splitting it by comma, it's a comma-separated field and stripping off the online space, and that's how it's getting an array of email

Speaker 3: addresses.

Speaker 1: Seems pretty predictable and reasonable. The next line has the array of content and then If I look through that four structure there, you know there's

Speaker 3: a lot of stuff going on there, but I think I can basically tell that it's going through every field that's defined on the form.

Speaker 1: And it's creating a string that is the call-in value

Speaker 3: for that field.

Speaker 1: And then putting it into our little content array. Um

Speaker 3: extended email to have the like like

Speaker 1: form question form answer. And then it joins those all up together within the lines. Um that makes that make sense, the content of the email. And then the last line there is the crucial line for solving our problem. I can see that it calls the send mail function, which is different from the send

Speaker 3: mail method.

Speaker 1: And it gives it subject

Speaker 3: content to add two addresses from an address.

Speaker 1: So I can see

Speaker 3: the

Speaker 1: that's where it is controlling the subject

Speaker 3: as the model field. Subject.

Speaker 1: Now what I actually want is I want that to change dynamically per form formation. I want it to pull the subject from one of the form fields that's being addressed in that formula. Do we all understand?

Speaker 3: Yeah, yeah.

Speaker 2: You said that CML function is different from the method,

Speaker 1: where

Speaker 2: is the tax and function?

Speaker 3: That's great. Notice that I did that in that statement.

Speaker 1: We'll get to it later. For now it's enough for me to see that that is called sends mail and I pretty much know what it does. Um so here's the email contact form model that we wrote.

Speaker 3: It inherits a more abstract email form.

Speaker 1: And there might be a number of different ways of solving this particular problem. We chose to solve it just by overriding that send

Speaker 3: mail method with our own custom sendmail method.

Speaker 1: And we basically copied it um line for line. Except we've added a few lines in there that you can see at twelve, twenty twenty one and twenty-five. Um where we basically change the logic that decides that defines

Speaker 3: the subject. Um

Speaker 1: and in particular twenty and twenty-one where the If it encounters a field that has the label subject, so if a content editor has said this form should have a field It's labeled as subject, then it will pull the value from that field that the website visitor has filled in and it will interpolate it into a new subject line, which we'll use for sending the email. If it never encounters that field, there's no field on this form labeled subject, and it just uses the model field subject as it used to. So there's our solution. We solved that problem. So that's an example of solving a problem that we could never have arrived at without looking at the code. That method was not documented.

Speaker 1: You know, no one in this room could tell me that it existed or how it worked. Um but it was like pretty easy for me to go to GitHub, find the files, read through them, understand them, and make the necessary changes. But reading the code is not just about finding undanted methods. You know, let's say we wanted to do something else. Not only do we want to have a dynamic subject line, but we also want to Really spruce up these emails, make them fancy HTML emails instead of plain text and tables and colors and maybe an animated gip or something. How would we go about doing doing that? Let's look at our form

Speaker 3: page code again.

Speaker 1: Now you'll notice this time that I did edit that import

Speaker 3: line that you were asking about.

Speaker 1: Wagtail has its own Csmail function. And uh this is the class that we defeated, but we basically copied that over. Sorry, copied over the same import

Speaker 3: that they were using.

Speaker 1: Um so if I'm looking at this, I don't see any obvious way of sending this email as HTML. You know, it's got this content. Content is fed in so send the mail function there. But the content right now is plain text. I don't have any reason to believe if I fed it in HTML it would send it out as anything other than plain text. So I'm very curious about what's going on there. I want to send a HTML email. I want to know if I can do it without too much modification to this function and

Speaker 3: writing my own mail sending logic.

Speaker 1: So from that in that go see what Web27

Speaker 3: utility looks like.

Speaker 1: So here's that. And before we even get into answering the question of how to solve that particular problem, I want to point out like there's a bunch of things that I learned From le reading this function that I would never have known otherwise. For example, you can see in that top block there that you know you can provide a front email to the send mail function and if you don't provide one there are like three different fallbacks. You know first it'll fall back to the Wagtail notification from email

Speaker 3: setting.

Speaker 1: And if that doesn't exist, it'll fall back to the default from email settings. And if that doesn't exist, it'll fall back to the hardcard webmaster

Speaker 3: at localhost. Yeah.

Speaker 1: None of that's useful information to me right now. But you know, maybe someday down the road I'll encounter a weird issue where we're seeing a bunch of emails coming in from Webmaster at localhost. And when I see that I'll be like, I've seen that before, and I'll have a pretty good idea of where to start looking. We can also see in that second block there that I could give it a custom email connection if I wanted to. Again, it's not something I really need to do right now to solve this particular problem, but it's pretty cool to know that I could do that. And then uh right there in the bottom few lines, I can see some stuff that's actually relevant to my problem. It looks like there is actually a way to send an HTML message. Um and you know it seems like it's only from word on word arguments and uh uh send mail signatures. So if I

Speaker 1: provide ML message to word arguments. Like it will send that as an HTML message. So there's another situation where reading has

Speaker 3: given me the tools I need to solve my problem.

Speaker 1: I'm not gonna

Speaker 3: take you through the whole process of actually doing that, but hopefully you can see how you would get started doing that, and that you can see how reading the code made that knowledge possible. Um

Speaker 1: if you've never known before, I've made the prospect of reading the code to solve your problem

Speaker 3: seeming

Speaker 1: Approachable uh

Speaker 3: like

Speaker 1: it might actually help you um and help you see how it can help you arrive at solutions. Um so I would say whatever you're working with and you run into a brick wall, I recommend going to the Wagtail code or the Django code or whatever you're working with and reading through it. And even if you don't know specifically what you're looking for, it can be a great way to just get like a better perspective on the specific area of functionality that you're working for. Um now if you still feel a little intimidated by it, if you're still like if those code blocks went by and you're like, wow, Harris just like understood those instantly and I had no idea what was going on in them. That's okay. Reading the code can be difficult. And especially in code bases that are more complicated on Wagsail, you know, it can take a lot of time to deal. Um but it's also a skill

Speaker 1: that you can build. And the more you practice it, the better that you're gonna get you're good at

Speaker 3: surfacing the stuff that's relevant to you and understanding it.

Speaker 1: And the more you do it, the better you're going to understand your tools, the better your own code will be, and the more prepared you will be to contribute back to the community. So thanks for listening. Should I take any questions, Tim? Uh

Speaker 5: we're up against the array for the question.

Speaker 1: Well uh we'll take a question if we have uh yeah uh yes.

Speaker 3: Yeah, yeah.

Speaker 2: Um so it looks like you basically copied the existing method and added a couple lines to it. Um So I was wondering would what would you do to refactor that or if you wanted to like you can see you can see that as like a useful thing for other people to use, would you then like Try to write it. How would you how would you try to try to have it back?

Speaker 1: That's a great question. In this particular case, because we were using like a specific subject. Doesn't seem like a general uh a specific field name. It doesn't seem like a general solution. Like it seems like a bit of sort of hidden behavior if you just add this Special subject apply

Speaker 3: to it.

Speaker 1: But I could I could imagine either releasing a third-party package that had a slightly modified abstract uh email form. Um I came up with more general general solution um

Speaker 3: than

Speaker 1: I can imagine. Putting a pull request and that into

Speaker 3: the WACL core.

Speaker 1: Does that answer

Speaker 3: your question? I'll take one more question if anyone has one.

Speaker 1: Great, great.