Tips for Maintaining Wagtail Packages - Tim Allen

Yeah, they just I still have my

Speaker 1: It's really it's really the fact that we're in the right. Oh right. Welcome back everyone. Hope everybody got coffee'd up. Because uh I don't drink coffee, but I tend to have energy of a person who drinks about a hundred coffees a day. So uh I'm gonna dive right in. My name is Timothy Allen, I'm an IT director at the Wharton School Feel free to call me Tim. I just avoid that whole Tim Allen conflict by saying I'm Timothy Allen. I've been using Wagtail since about version, I think it was 0. 4 in 2014. And I'm a recent member of the core team. I just want to take a moment to acknowledge, you know, this feels freaking wonderful to be together with a community that I really deeply care about. For the first time in so long, it hit me that, you know, the last time I gave a conference talk

Speaker 1: was, you know, not during this decade. It was back in 2019 at DjangoCon. And You know, I used to take this for granted, getting all of us together in moments like these. And I, you know, just want to take a moment to really acknowledge uh how wonderful it is to be back together in person at a Django community, Python community, Wagtail Community Conference. It's really great to see everybody and to see everybody online too. It just feels really special and uh wanted all of you to know how much I appreciate being a part of this community. And uh, you know, a big thank you to all the organizers, all the other speakers, um, especially to Vince and the team from Code Red CMS for hosting here in Cleveland. I know how much work this kind of thing takes having hosted the uh first few wagtail spaces back at the Wharton School in Philadelphia.

Speaker 1: And uh thanks so much, especially to the attendees for all taking time to attend and uh and check this all out. So without further ado, I'm going to talk about a bird in the ecosystem, some tips for maintaining wagtail packages. And you know, we can get started by talking about what is a wagtail package. A Wagtail package is a Python package and therefore a Django package. And most often it is something that is pip installable. So when you go ahead and type pip install wagtail code block or pip install wagtail font awesome, you are installing a Wagtail package. And this used to seem like magic to me when I first got into Python back in 2014, that I would type pip install something, you know, all package managers

Speaker 1: It seemed like there was a lot of sorcery and sort of, you know, black magic and voodoo going on that would automatically go out and get something and install it automatically on my local computer. And um You know, the I never really asked the question, well, how and why does pip work? And um, you know, when you pip install something by a name, So if I pip install WagtailFaunes, or even just pip install Wagtail or pip install Django, it goes out to a place called the Python packaging. index or pipi. And it's important to say pi pi when we pronounce it because pi pi is actually something different, the just in time compiled version of Python. So it's PyPI. And PyPI is run by PyPA

Speaker 1: or the Python Packaging Authority And I always like to give this a little shout-out because a lot of the Python packaging authority comes out of the Philadelphia area, which is, you know, where Don and Ann and I hail from. And uh Donald Stuft, who first wrote Pip, is from outside the Philadelphia area. And uh my good friend Dustin Ingram, who does a lot of the current work for the Python Package Authority, is also a Philadelphia guy. So you can thank the birthplace of this nation for also being the birthplace of your Python packaging needs. Some other Python packages you may have heard of over the years. There's a very popular one called Requests. Which I think just about every project I've ever seen ends up using in one way or another. There's Django, there's Wagtail, there's Pandas, there's PyTest.

Speaker 1: These are all Python packages. And a Wagtail package is just one form of Python package. It's something that it's a reusable module that can be installed and it's installed typically by PIP, as we mentioned. And uh, you know, it's a little more than that. So I think it's just important to get that out there, you know, to remove some of the sort of voodoo that I felt as a newcomer at first that When I say a Wagtail package, it's just a pip installable Python package that has been written specifically for Wagtail and therefore also Django. So before we dive in too far, I want to talk about how to publish a package to PyPI. So if you've made a Django app and you have it as a folder within your Django project, That might be something that somebody else wants to use someday.

Speaker 1: That might be a problem that somebody else is trying to solve. Or it might just be something you want to throw out there and get get a hand with. And um It would seem, you know, when I first started thinking about this, I thought it would be a really difficult process to go through and, you know, get something I had written. Onto PyPI. And you know, I now maintain several Wagtail packages, including Wagtail Code Block, which is a syntax highlighter, Wagtail Content Stream, which gives me my most used stream field content at my fingertips whenever I want them. I also helped maintain the Wagtail Error Pages package and the Wagtail Font Awesome package, which were started by my friend Alex. And you know the process for pubbling a package to PyPI so it can be pip installed

Speaker 1: is uh is fairly straightforward. So if you take a look on the screen here, you'll see an actual setup. py file So the first step is we create an account and a token at pypi. org. So we publish. The second step is we create the setup. py file. Which has a lot of package details and information. But you'll see most of it is just metadata. So you'll see I give the name of the package, which here in this case is Wagtail Content Stream, a version number, a description, a long description. The author, the URL, as well as some classifiers, which are really categories for where it will be included. And then just a couple extra steps from there I run pip install twine

Speaker 1: setup tools and setup tools SCM into a local virtual environment, which are tools I can pip install to help me upload my package to the Python package index. And then I run this command to build the source distribution and the binary distribution of my package, which is python setup. py, s dist for source distribution, and b dist underscore wheel, which is binary distribution. A wheel is sort of a binary distributed version of your package, which can be more easily installed by people with actually without them having to compile the source. Then after I've done that, I just run the twine upload dist forward slash star command, and it will upload all those source and binary assets I've created to PyPI using the token that I created at PyPI.

Speaker 1: So I have simplified this somewhat here, but it gives you the basic steps of what it takes to publish the Pi PI. And it really, when I finally saw these steps, it was not as intimidating as I thought it would. And one of the problems we have right now is the tooling around publishing packages to the Python package index has much improved over the past decade. And if you go out on Google and say, how do I put my package up on the Python package index, a lot of the information you get is going to be out of date. A lot of it is flat out wrong and a lot of it is just very cumbersome. And uh with the major improvements that the Python package Union authority has made over the past um over the past couple years, especially. It's not nearly as cumbersome as it used to be.

Speaker 1: These trove classifiers are the categories that are managed by PyPI. So you can also see here within this section at the bottom of the setup. py file, I have a whole bunch of these classifiers. And uh we just actually recently added a couple for Django where you can see it says framework Django 3 and Framework Django 4. We used to only have the specific version numbers, so like 2. 2, 3. 0, 3. 1. Now that Django 's moved to using Semver, just uh just about a month ago. We added these Django major versions to the Trove classifiers, which again was not that not nearly as intimidating a process as I thought it would be. I opened up a pull request. To the Python packaging authority's Trove Classifiers repository, and just said, hey, I want to add these four Trove classifiers for Django's major versions

Speaker 1: And uh, you know, within minutes, my friend Dustin replied and said, okay, that sounds like a great idea. Let's do it. And uh having these major identifiers can reduce the amount of churn we need in these setup. py files. Because rather than having to specify each individual version of Django, I can just say, okay, I support Django 3 and Django 4. And since Django adheres to the Senver numbering convention, That will typically mean that I don't have to change it every time there's a new Django release. So when 3. 2 comes out, or I guess now I should say when 4. 1 comes out or 4. 2 comes out. I won't have to add them to my setup. py file. I only have to do that when you know Django 5 comes out. And then I can probably remove Django 3. So there's just a lot less churn in that setup.

Speaker 1: py file with these triv identifiers. So that's uh that's been that's been a nice win. And I as I've been updating my packages, I've been switching over to them. And uh as you can see, you know, Dustin's really friendly and he waved to me on on there. So don't be intimidated. Dustin will help hold your hand through this process. I will help hold your hand through this process if you if you would like when you're publishing your first package. So my first tip on getting a package published to PyPI is dive right in. It can seem intimidating to release a package to the public. You're putting yourself out there. I know for me, imposter syndrome creeped up big time. And uh, you know, I thought everybody else, you know, was an absolute genius. You know, when I was younger, I was convinced that there was somewhere in this world where there was this panacea

Speaker 1: of beautiful code where everything was perfect and that I did things horribly and nothing was messy. But over the years, you know what I've learned? Everything is messy. I have seen stuff in the Postgres source code and the Linux kernel that you know made me want to cry where you know their comments saying this shouldn't work, but it does. Let's revisit this later. And that's in the source code of this database I trust with all my data. So I don't feel quite so bad about putting my code out there when I know that these things now exist. It helped me get over my imposter code. My imposter syndrome. You know, we're all human. There are mistakes. There's they're bugs. That's okay. But I've learned so much from just putting my code and my ideas out there. My packages have been improved by other people

Speaker 1: with much more domain-specific knowledge than me, which has helped me learn and me become a better developer and improve my package for me, for my employer. And uh, you know, security flaws and bugs are best found with more sets of odds. So I strongly encourage you to, if you've built something, go ahead and put it out there. The worst thing that can happen is nobody else will use it. But you've got a pip installable now. So you've got a portable way of using it within your own projects. And for example, Wactail Content Stream, I don't think many people use it, but I've got a handy way of installing my favorite stream field the way I want it in all my projects in a nice dry don't repeat yourself manner. And uh, you know, it's been great for me. Another tip is to state your intentions with your package.

Speaker 1: So this is something I've started doing that I wish I had done from day one. Um Set realistic expectations. We all have constraints on our time. And uh, you know, give props to your employer if they're letting you work on open source on their time. You know, this is something I'd like to do with Wharton. Wharton is very supportive of us being, you know, more than any other employer I could imagine. It's very, very supportive of us being involved in the open source community. You know, we've hosted the first couple of Wagtail spaces. We've hosted DjangoCon. We host the Philly Python U users group. They allow us to contribute to open source for projects that we have done on their time that become useful to the community, like Wagtail Code Block. was first done for my employer. And then other people said, oh hey, that would be really useful for anybody else sharing code samples.

Speaker 1: So that's how that project was born. It was born out of the generosity of my employer, allowing me to use my time at work for them. So give them credit for that. Um and for many packages of mine on PyPI, I'm the only maintainer. Once a package becomes more popular, It is important to get more maintainers. You know, you never want to run without a backup. If I get hit by the bus, or if you want to be nice about it, if I win the lottery and disappear someday. I want to make sure that there's somebody there to take over the package so it doesn't, you know, just sort of rot on the vine. And if people start relying on your package, that is a bit of a responsibility to make sure that that can happen. But the flip side of that is true. If somebody cares enough about using your package and it's that important to their core business, they should want to invest some resources from their company or their organization to help maintain the package, to help fix bugs, to help improve it.

Speaker 1: And uh the screenshot above on this screen is from one of my more popular packages, which is DRF Excel. Um, it allows you to add Excel export to any Django REST framework endpoint, and it's gotten pretty popular. And the amount of contributors has uh has just skyrocketed exponentially from from one to many. And uh You know, the the two other maintainers were people who just started regularly issuing PRs and using it at their workplace. So I asked them, hey, do you want to be a maintainer? And I gave them access to be able to publish the Pi PI and do merge requests on GitHub. And now it's no longer a one-person operation. I've got help. I've got people who are, you know, shouldering shouldering part of the burden with me. It's great. Another tip is give credit and thanks.

Speaker 1: So highlighting your contributors list is a really good way of getting people involved And I'll tell you a story. My first pull request within the Python community was uh to PyDani. And uh he was the author of the two scoops of Django book. So I'm thinking Django royalty here. I'm super intimidated, but he was incredibly friendly when I issued this pull request. Uh I think it was improving the Windows. installation instructions for Python after they added that lovely checkbox that said add to Windows Path, which I think is the greatest interface improvement in Python history. At least anybody who has ever taught newcomers who are on Windows will appreciate how much time that little checkbox has saved us all. And uh even though my PR was, you know, two lines

Speaker 1: of code and documentation. He took my name and added me to the contributors section of the README. And it felt great. Here's a person I looked up to, a great Django developer, a wonderful author. uh a really nice person who took the time to add me to his contributors even though I thought my contribu my contribution was so minuscule. It was really a wonderful thing. So taking the time to do that is uh is a really good thing. And uh You know, over the years, I've seen many open source projects kind of suffer from the same shortcomings, which I've in the past I've jokingly called open source disease, which is, you know, a lack of UX consideration. Hard to read documentation. Horrible names.

Speaker 1: I'm guilty of this. I mean, the GIMP. Git? Really? Is that the best we can come up with? We need a better marketing department in open source to get people involved. Now granted, Git is popular everywhere. But uh, you know, hardly the hardly the nicest or best name that we could have chosen. And uh, you know, DRFXL was originally named DRF-renderer-xlx. It hardly rolls off the tongue and nobody knows what XLX uh XLSX is outside of the tech community. You know, when people in finance who love their spreadsheets are talking about it, they're talking about Excel. They are talking about Excel. And some sage-like wisdom I got from somebody who has never contributed good to that project was, why don't you just rename it DRF Excel?

Speaker 1: And, you know, that was a brilliant contribution. What I'm saying here is there's a great opportunity here to get people involved who may have skill sets other than writing code. And that is a shortcoming that I have had within my packages and I have seen across the open source community for years. Wagtail is much better than most at doing it, but There's always room for improvement, and there's always room to get people who've got skills in other areas that might not be coding involved in open source. And you know, an important tip is just generally be friendly. We are a community. I can't list. I have no idea how many people have helped me over the years to become a better developer, to teach me things. And you know, I feel it's somewhat my responsibility to pass that on, to pay that forward.

Speaker 1: So getting into a little bit more of the tech end now. These are some of the improved tools that have come out over the past couple years. for uh for setting up packaging. And two of them are called twine and setup tools underscore SCM. So back in the old days, you used to have to maintain a manifest file. This manifest file would list everything you wanted to include in your package. Except for the Python module itself. It could auto-detect the Python module itself by having the dunderscore init. py file within there. It would automatically detect that. I would always forget in the manifest file to include Django templates

Speaker 1: or JavaScript assets. Or anything else that Python would not auto-detect. It would work fine on my local You know Django project just fine because all of them were there locally And they were all under version control and checked in. But within the manifest file, I'd upload it to PyPI, go to install it from PyPI rather than my local version. And ah, I forgot the files again. It's broken. I've put out a broken version. Hopefully nobody's upgrading to it. This is solved by this wonderful package called Setup Tools SCM, which is short for source control management. And it will tie into either Git or Mercurial And basically say everything included in this package is everything under version control. So I have never forgotten an asset since switching over to Setup Tools SCN.

Speaker 1: No more manifest file needed, no more forgotten templates and assets. Another nice thing it really does is it will automatically increase your version number using a GitTag. This is a really wonderful feature. So you can see the new way of doing things in setup. pi. It says setup requires setuptools. scm and use SCM version. Basically says, when I run python setup. py, s disk, and bdesk wheel, get the version number from the tag on the current branch. From version control. So rather than repeating ourselves in multiple places and trying to sync up the version number from Git. to the version number in the setup. py file and having to bump it with a new commit every time.

Speaker 1: We can just use git to our advantage Another thing we do here now, the Python packaging authority has set up the uh ability for the long description to support Markdown. So this makes it very straightforward in the first couple lines you see up there, I think lines three and four, to open up your README and set that to be your long description. And that way your readme in GitHub is exactly consistent in the same as what you see on PyPI as your description. And I'll show that in a minute. And the final tip I'll leave you with is to let GitHub work for you. If you draft your releases on GitHub with tags, like you see here, here's uh uh a tag for Wagtail Code Block 1.

Speaker 1: 25. 0. 2. You can then pull that tag down to create the package. You can also just link to the release notes and the contributors from your README. So within the release note , within the contributors graph, there's a full nice view that GitHub has of the contributors graph. And I just include a link to that from the README with a big thank you. And the release notes, I just link to the GitHub page with a big with a big link rather than again in the README or a separate file including them. Just let GitHub do it for you. That way you keep it in one place. It's nice and dry, not wet, don't repeat yourself. I found it so much easier this way. Automatically adding the version number before you upload to PyPI, that has saved me many, many times as well.

Speaker 1: So that's the basic tips I have for starting to package and get things up on PyPI, but I wanted to leave plenty of time in case anybody had any questions. So If anybody has any questions, I'm going to check on Zoom as well. But if there are no questions, I can also show you what this actually looks like within the browser as well. All right. Doesn't look like we have too many questions. So let me see. Oh, go ahead.

Speaker 2: So a lot of times when you start active, I heard people recommend cookie cutter or some of these other tools. Do you think those tools are helpful?

Speaker 1: You can absolutely use Cookie Cutter for a template for releasing the Python package. I've just always gone uh I So I first learned to do this the wrong way from old tutorials. So that's why I know that Google will show you a lot of things out of date and uh kind of learned the hard way. Cookie cutter does provide different templates, which are pretty nice. I don't know if many of them use setup tools SCM or anything like that But uh I found that to be the easiest way because you know I live and die by version control. So by tying the two together, it's been um a way that's worked for me. So if you want to see how this looks when we're done, can everybody uh see this all right?

Speaker 1: So here's how The page actually looks on GitHub. So you'll see I have the read me for Wagtail code block here. And you'll see on PyPI, We have the same exact README. So the two of them are very much the same. And if we take a look over at uh another package I maintained, DRFXL You'll see down here we have this list of contributors. So I just linked directly to this contributors link from down below. And you'll see, like I said, the release notes.

Speaker 1: I can pop open and there are all the releases. Going back through time And same with the contributors. So if I come down here to my contributors, our wonderful contributors You'll see it shows all the contributors over time. So it's just again a nice way of showing uh who has contributed, who's come together, and uh you know encourages people to continue to contribute in the future. So I think I see a question on Zoom. When starting a package, many people recommend.

Speaker 1: Oh, okay. That was from Vince. Thank you, Vince. Are there any other questions? Go ahead, Diva. But it's good at CM I think too big. I don't see too much. Is there any reason for it? We don't want to do it like that. So So the question was that SetupTools SEM hasn't been seen before by Tebow, and he's wondering if there's any reason for Wagtail not to use it. And um I can't think of any, but I also am not too familiar with the Wagtail build process. Wagtail is a very complex set of packages. So when you pip install Wagtail, you also get quite a few other Python modules like the, I mean, we went through the reorg a couple years ago

Speaker 1: where Wagtail Core became Wagtail. core. We've got the contrib packages, we've got the admin. It is a fairly complex system. I don't know if there is any reason not to use it. But given how complex Wagtail is, there may be some exception that I'm not aware of For most packages though, I think it's it's a day one win if you start with Wagtail SCM from the beginning and Twine from the beginning, because uh they just uh so Twine has worked alongside the Python packaging authority to make the upload process much smoother. and move to token-based uploads rather than username and password-based uploads. And setup tools SCM, you know, the features I've outlined have saved me so many mistakes I made over the years. So

Speaker 1: it uh it might be something worth looking into for Wagtail. Sorry, what that uh Ah, how do you how do you indicate that a package is incompatible with other packages? So Typically I've run into that with different versions. And what I do within the setup. py file, you can say that this package requires at least a version of another package. I tend not to pin the top end of packages within my setup

Speaker 1: because of the new uh so pin has changed the way it resolves dependencies. There's a new version that came out last year on how it resolves dependencies. And that has caused quite a few of us to change the way that we say what dependencies are required. It used to be fairly safe to pin the top end of requirements. So for example, if my package requires requests I can say it requires at least version 2. 3. And it was a good practice to say, but don't install any above 3. 0. But many of us have started removing that top range because it requires, again, a constant churn of updates now that there's a more strict dependency resolver. So what I typically do is say

Speaker 1: that, so if my package doesn't support requests 2. 2 and I need at least version 2. 3, I'll say install requires request greater than or equal to 2. 3 That's one example how to do it. If it's completely incompatible with a package altogether, I don't know if I've ever run into that scenario. And uh the the way I would solve that would be to reach out to my good friend Dustin and ask him how to do it. For very good seats for your project.

Speaker 1: Yeah, so the question was about malicious dependencies possibly entering a project and uh this is a risk. This is always a risk. That's uh It is a big concern, especially if you're doing automatic upgrades. So what I do recommend is that people pin their requirements very tightly within their local project, but not within a package. So what I do within my packages is I say what the minimum version required is. So if somebody tries to pip install, if they're pinned to a lower version. So again, if we use the request example, If my package requires at least 2. 3 and they're pinned to 2. 2, they'll get an error. And then they have to manually intervene to upgrade their version of requests. And it will be up to whoever is doing that upgrade to make sure that the new version of request does not have any malicious code in it.

Speaker 1: It is realistically a bare of a problem. Because none of us have the time or I don't know of anybody who has a job which affords them the ability or time to review every change that goes into requests between each version to make sure that they're absolutely secure There does have to be some trust within the community for any of this to work. And you know, I do like to remind ourselves sometimes that it's a miracle that anything in technology actually works. It really is. So yeah, that's a that's a that you know that could be an entire conference of itself is dependency management and supply chain injection. But I do my best. So the you know, my general strategy is pin at the project level, but uh allow the package dependencies to be a little more wide open.

Speaker 1: And then when I'm doing the upgrade of my own Django projects, I just bring along the packages. Like if I'm upgrading a major version of Django I take the opportunity to also upgrade Django Rest framework and Wagtail and whatever else is going along with it because often those dependencies go hand in hand You know, a new version of Django will come out and about a week later, after Matt doesn't sleep for a week and a half, we'll get the, hey, you know, the new version of Wagtail that's worth Django 4 is out. And we all say, thank you, Matt. And then we go on and upgrade our projects. it's a it's a little different than that these days but a couple of years ago that's really the way it was so i don't know if we're gonna get around that anytime soon but no excellent question thank you

Speaker 3: thank you so much

Speaker 1: all right thanks everybody this is wonderful