Django core demystified

Speaker 1: Thank you for having me. I'm gonna do hopefully a fairly short talk on uh Django core demystified. And I will go into a bit more into it shortly. But for those who don't know, my name is Sarah Boyce I am a Django fellow. I've been a fellow for about three or four months, so not not very long in that role. For those who don't know what a Fellow is, the Django Software Foundation pays a couple of contractors to help maintain the framework. And uh that is basically what I do now.

Speaker 1: So I am very much uh looking after the pull requests and tickets and uh releases and all of the stuff that is uh around making Django better and bigger and secure and all these kind of things. So that is my job. Um and This talk was kind of motivated by the question of how is Django similar and how is it different to other open source libraries and packages Which is an interesting question in and of itself. And to be honest, I have quite limited experience of

Speaker 1: Other open source libraries. So I'm going to talk mostly about Chango and how I think it is different or similar. But I'm more interested in the discussion afterwards. I'm hoping we're gonna get lots of questions and we can discuss all of this a bit better as a group. But just to throw some ideas out of there, out there. We can start with a little bit of theory. And so firstly, there is a book which I will highly recommend called Working in Public by Nadia Erbal , which categorizes open source projects into

Speaker 1: four groups, and these are toys, stadiums, clubs, and federations. And this is based on whether it has high or low user growth or high or low contributor growth Now I don't find this visually very easy to digest, so uh to me this looks a little bit like this. The gray dots represent users and the green dots represent contributors. And essentially if you have a toy, this is an open source project which probably the person who authored it is the main person who is using it. Perhaps there are one or two other users, but the um

Speaker 1: the audience is really, really small And then you will get ones where actually the audience is large. There are lots of users, but still there are not that many contributors or maintainers for that project. There might be reasons for that. Maybe the maintainer or author actually deliberately wants to keep this quite small, or perhaps it's uh Perhaps this is something that Python users use, but maybe it's written in Rust. And so maybe there's a big barrier to entry from the people who actually use the the project to like the skills that are required to uh contribute to it or maintain it

Speaker 1: Another type are these clubs where the overlap between the contributors and the user bases is really high. So It means the group is is really invested in the project in a in a way it sounds great. So it must I'm not really sure about whose projects these are, but it it means that it it it's You can get feedback on what you're doing with it and people must be quite engaged. And I think the example was perhaps if the tool was for um like a a group in the physics community or something like this

Speaker 1: where uh the people who use it are are very invested in in uh contributing to it or or something like that. And then there are federations, and there's a reason I came to that last. And that's where there are lots of contributors, but there are also lots of users And uh in my opinion, Django is a federation. Um and The main reason this is is this is kind of like also an extract from the book as to what a federation is. Federations are similar to companies or NGOs and that Django is a very similar to a company, I'm gonna say uh many times.

Speaker 1: They are more complex to manage from a governance standpoint and so they tend to develop processes Django loves processes. There's voting, there's leadership positions, foundations, working groups and technical councils. that address coordination issues within the broader community. These contributors in turn make decisions for a broader constituency of passive users. If you are already quite familiar with how Django runs, a lot of these things exist in Django. So for example We do have a foundation. We have the Django Software Foundation and we do elect a board.

Speaker 1: So, you know, we the community votes For people to take up leadership positions. We also have a technical council. We have the steering council. Again, this is an elected group of people And uh if I go back, yeah, so this is to help make decisions for a broader constituency of passive users. So There are people who are very engaged in Django and there are a lot of people who are very engaged. And then there are also a lot of users who uh are not engaged in it. And that's fine. But yeah, we have we have voting and mechanisms and processes to help manage this basically.

Speaker 1: And as I mentioned, Django loves processes. This is probably briefly going into some of the governance and I will go into some of the processes that we have. But because Django is a bit like a big company and it has all of this structure and it has lots of processes, this is probably one of the things that people will notice being different to an a different open source project um that doesn't quite have uh the quantity of users or the quantity of contributors there who are you know interacting with it All right, so

Speaker 1: we could talk about that later. I'm looking forward to discussions, but if we now move away from that and go a little bit into the practical stuff of Django Let's say you wanted to get a change into Django. So what are you going to notice about how Django is similar and how Django is different to other open source projects? The first thing you will probably notice is that Django does not use uh GitHub issues. I think GitHub issues Has a very large market share of issue trackers, ticket trackers, whatever you want to call it, for open source projects. Django

Speaker 1: didn't start off in GitHub, it did migrate to GitHub. And the tracker it's been using since I think the start of Django is this tracker called Track. And it is definitely something that people need to get used to as A lot of people have already have some experience using GitHub issues, and I don't think there are that many open source projects that use track. However, it does have a lot of fields in here that you can filter on and order by and it is actually quite a powerful tool.

Speaker 1: But it is definitely something that people get quite overwhelmed with the first time that they come to the Django project and want to make a contribution. But okay, so Django has track, it's an issue tracker. Django also has uh this thing that we call triage. So whenever someone creates a ticket in track, The very first state is called unreviewed and every ticket will be triaged. So this is going through whether it gets closed or accepted. They're basically the the two things that will happen to a ticket quite immediately.

Speaker 1: This is one of the main roles of the fellows, but also of the uh Review and triage team also helps out a lot with this and and actually anybody in the Django community can accept or close a ticket. You can't accept your own tickets. It has to be somebody else. But there are kind of justifications as to what this tick what should happen with this ticket. So if it's a duplicate, that's quite obvious. we we need to find the the duplicate and you close it and you say it's being tracked by number 3,000, blah blah blah blah. But We have uh

Speaker 1: decisions around uh whether if someone is trying to say they've got some issue, we need to be able to replicate it and confirm this is actually an issue in Django and it's not an issue in their code or in a third-party package or in Python itself or something or something something. Blah blah blah blah. So every ticket in track that is accepted went through some level of uh quality control basically. But we can discuss that later. But as I said, Django has processes. It has lots of processes. Very well-defined processes. Okay, so if you

Speaker 1: go And you have a ticket that's accepted. They are categorized into three categories. We have bugs, we have new features. And we have cleanup optimization tickets. So these are the three types of tickets that exist in Django track. I would say this is something that you could probably categorize most requests in any open source project. I'm not sure if they all have the same level of triage process, but but that the core of the request is either it's like a bug fix or it's a new feature or or you're suggesting that something would

Speaker 1: be more readable or be a bit faster or something like this is is probably quite common. Okay, but what's different? So With bug fixes, it has to have an accepted ticket. If you just created a pull request to Django without a ticket and said, oh, this fixes this problem that I had or something um it won't get uh reviewed and merged in without an accepted ticket because uh We need to do the analysis to confirm and validate and all this kind of stuff around the bug fix itself. Another thing is that tests are a must.

Speaker 1: The bug fix needs to have something to make sure that we don't reintroduce this bug in the future. It's not enough to just fix it. We have to have a test around it. And then the other thing which I think is interesting is you might have a test, you might have a fix. And it we might still say no because I'm not sure what your experience is with Django, but I have not come across a bug in the wild using Django myself. I have only ever come across them through investigating

Speaker 1: really weird things with other people that you know and or through the through track itself like when other people have reported it. Uh it's not a buggy application and a lot of these um Things that require a fix, uh it's not that we're so desperate for a fix It needs to happen today and all this kind of stuff. If you have a fix, but it very much is a hack. then it there will probably be more of a discussion of like, oh, but that doesn't feel like the right way to fix it. We should probably look into something, something, something. And I think that is perhaps different to

Speaker 1: certainly maybe how you are at work, maybe, or maybe other open source projects might might be grateful enough that there is a fix. But for for Django , it kind of has to be the right thing to do. And there's this kind of decision. So for me, I think that's quite maybe unique to some. think that's quite big. Uh the other thing uh around Django is that we have um We have a policy around release notes. Bug fixes actually don't usually have a release note. We usually don't talk about bug fixes at all. They just magically

Speaker 1: Never existed or they they got fixed in the background and all this kind of stuff. And we will only write a release note for the The monthly releases, so these are uh the 5. 0. 7s or something And these are called like high severity bug fixes, but usually this is around something that was newly introduced in that version of Django that broke something existing. So that should have never happened. And so we try to uh fix these and there will be a small release note to justify why people have to um

Speaker 1: like upgrade to the newest monthly. Otherwise we don't have release notes and different open source projects have different policies on these I do think released notes are super interesting because it's sometimes it feels a bit of a shame that not all of that work is captured. But this is how we do it. All right, next one is new features. New features I have split into new new features and then like something else. Um one of the things that I think is probably interesting here is uh Django because the Django is like a big company

Speaker 1: um There is a lot more of a culture of cost and risk maintenance then there perhaps is with uh would be a smaller company kind of thing. We we we don't have like the excitement of yeah let's just try it so why not and all this kind of stuff. Um and it's not like it's a very small project where Perhaps someone is so happy that someone is using their project and that they've suggested an improvement that they're like, yeah, that sounds great, let's do it because that they're quite flattered that it's being used. Django has kind of lost all of that um

Speaker 1: uh rosy-eyed kind of thing when people ask for uh adding new stuff to Django uh and therefore the community has to want it we don't want to add something in which Which only one person out of thousands of users is going to use and have to maintain it for forever. And that's why Django has uh some process around encouraging uh that the community tries to engage with each other and get agreement that we we should include this This is not just around new features to Django itself, this also extends to the docs

Speaker 1: because the docs. are just as important as the code. The Django docs get quoted and they are quite a um they're a very important piece of the framework. So even suggesting a new topic in the docs, although that fundamentally sounds like a great idea, there is still uh maintainability and um other forms of costs associated with that and it needs to really make sense that this is something that we can include as being that we know it's correct, that we know people can quote us without you know that being dodgy and

Speaker 1: that this is something that's not going to get out of date in I don't know, a month's time and all this kind of stuff. So new stuff is hard. With contributors, new contributors, I really recommend that you do bug fixes because bug fixes Accepted book fixes are quite nicely defined, and you can probably get progress on that without having to discuss this with the wider community, and discussions are hard. Yeah, new stuff is hard. There's a different type of new stuff and that is changing something existing. So again, the community has to want this change um and the upgrade path becomes quite important here so

Speaker 1: Django does have a deprecation process uh we we want the the developer experience of upgrading Django to be nice and so If something needs to like if you can't deprecate it and you you need to just subtly change it under the hood, um There will be resistance to that. And there is generally resistance to change anyway because we're like a big company and change has cost associated to it. There's lots of risks associated to it. Um and so the if you are suggesting something, you will probably

Speaker 1: um yeah, encounter some resistance and there needs to be a very strong argument as to why we need to do this because it is certainly safer not to. But yes, interesting. Next one. is the cleanup optimizations ones. Right. These these are uh my least favorite group of things. This is when someone goes, I think this is more readable if it was like this, or I think this is uh you know better and it's quite subjective and fuzzy. And the other problem is is it can introduce bugs quite easily.

Speaker 1: There is usually an assumption with these things. That there is sufficient test coverage and that if if the test suite passes, there won't be a problem. But um Actually, that's not necessarily the case. Whereas there is a lot more discipline around the tests for new features and bug fixes than there is around cleanups and optimization. So I have learned that these are actually quite dangerous. The other thing is if you are suggesting that this is faster Uh really you need to try and prove it. So try and provide some benchmarks as to why this is actually

Speaker 1: faster. Uh Django actually does have Benchmarks. This is Django ASV. You can contribute to that repository and run some benchmarks to to help prove it. And there is also, there is, there is always a cost associated with even small changes, even small rewording of the docs. To make that sentence ever so slightly clearer. There is a cost in terms of translations. The translation strings will update for everybody. Things like this. So there are. changes where uh the value has to be there. It has to really be worth it.

Speaker 1: And so Yeah, these are another group of things that we are perhaps a bit more uh pessimistic in Django than perhaps in other uh open source uh libraries. I'm not sure. I I don't know. how people handle these kind of requests. Yeah, so if you want to learn uh more about the contributing to Django process There is a really good talk which even though it's uh in 2016, it hasn't changed much at all and is still very relevant.

Speaker 1: Um And this is uh a talk by Tim Graham mostly. Uh it's a Django under the hood talk, and he goes through uh what it means to be a fellow. It goes through triaging and and contributing to Django and and reviewing tickets and and all this kind of stuff and it it's got lots of information in there and it's definitely in more depth than what I just kind of rattled on through. So I would recommend watching that if you want to learn more about specifically Django. All right, then my last topic, which is also uh I think quite important. I I I didn't have enough time to go into um a great deal of

Speaker 1: detail here. But how is Django similar to other open source libraries? We we deal with people, we're dealing with people. People are hard. People are hard. People are great, but people are hard. And I can make uh oh no. So one of the things that you might notice in Django is that some of the people who are involved are In my opinion, they're like minor celebrities. So some of them have published books, or they have podcasts, or they've been keynote speakers or they've

Speaker 1: like uh you know they they invented the framework itself or something like this and they might seem really uh intimidating and they might be put on like a pedestal and have this kind of hero status and it's quite difficult to um treat them like a normal person. Uh uh believe me they are normal people And you will in time get used to each individual. I think it's worth mentioning that because I know I found that quite intimidating when I started contributing.

Speaker 1: And now I'm just going to say there are lots of talks about people and open source which are really really good and I recommend you uh listen to or watch uh that are gonna do much better than what I'm gonna say right now. So These four, so there are uh there's people, the API users guide by Ned Butchler. This is a uh PyCon US keynote And it's a really great talk about how to self-reflect on different interactions that you have with people online and how to have better interactions, how to have better conversations with people. I think it's a great talk. There is also

Speaker 1: know your limits on surviving open source by Carlton Gibson. I think this is really nice from also a maintainer perspective and Someone giving advice as to how to go into open source with a quite a healthy mindset. There is also Inside Out, My Journey of Understanding Inclusion by Natalia Bilat. This was a uh DjangoCon US talk, and this has got uh some really interesting uh stories and information about what you should bear in mind when you're dealing with um different cultures and communicating uh between different cultures and like

Speaker 1: biases that exist within all of us and um you know how to manage this better And then this is the one I I've listened to recently. It's quite an old talk actually, and this is The Hard Pots of Open Source by Yvonne. Chapel Chapliky. And this is also a great talk and he talks about how people can get uh angry and upset with each other and some ideas as to how to uh not do that basically. And he makes really good reflections on why we see this behavior

Speaker 1: and and what tools might we'd be able to develop in order to see less of it. So all of those are great talks. Much better than this one. But some tips. Um yeah, so I think this is true of pretty much any open source project. Remember that you are interacting with a person, even though it can be hard to know this because perhaps the uh the name that you're looking at it might not be their name, they may not have an image of themselves, and they may seem quite

Speaker 1: cold, but they will still be a person. And if you can try and build a relationship and build trust with the people you're interacting with , it will I think it improves your experience and their experience and uh you're gonna get a better um You're going to get more out of your experience of working with open source And so I recommend that you humanize yourself when possible. Uh if you're nervous or something, say it, you know. I I I think it's worth saying, you know, hello, I'm new here or whatever.

Speaker 1: If if that's how you're feeling then then do say it because it will remind the person who reads it that that's how you're feeling. And try and build empathy also with uh the maintainer themselves. Sometimes they might come across as dismissive or cold or aggressive. And I think if you can zoom out and look at some of the other interactions that's been going on You might be able to tell if actually this person is is not so nice, and maybe that's the case. But maybe they're also very overwhelmed as to what is going on and that they're struggling. So uh if you can try and

Speaker 1: uh relate to them as much as possible I think that will also uh help. And then my next tip is um When I first started contributing to Django, I felt constantly guilty. Because even though I went in um Uh it wasn't in fact I I never went in with like a oh I have this idea of a feature and I I want to get this feature into Django. I I wanted to try and uh be a part of it and I wanted to uh try and resolve some things because I like resolving things. So However, in the experience, I felt like I created more work for other people

Speaker 1: because every ticket I assigned to myself create a PR, it needs a review. And uh the depth of review, in my opinion, at Django, is really strong. And you can see that this person puts a lot of work in uh trying to Help your contribution to be as as best as it can be. And that made me feel awful. I was trying to reduce the workload and in trying to reduce the workload it looks like you you increase the workload that is available. And that it will take some time before you know how to make a contribution that is like of the shape and form of

Speaker 1: what you know is wanted. So I'm not sure if that's a good mindset to have, but uh bear in mind that um It it does generate, like, even if you're trying to help, you you often do create work. So it would be uh quite a nice idea to think about this and to reflect on uh How can I make sure that next time this is better? So if they uh have suggested three or four things, try and reflect on them, learn about them next time you you will you will make sure that those things were done.

Speaker 1: And build it up so that you build that relationship and trust. And uh you can like um yeah, grow from the experience and hopefully uh uh become like an uh a stronger asset to the community because Each individual person there is like a an investment in them to hoping that they're going to learn and grow and come back. It's okay if they don't Um but there is this hope, I guess, that people might uh, you know, also get get the bug and and want to

Speaker 1: want to do more. Which sounds a bit of a downer, but I I meant that in a good way. Yeah. Anyways, so that's roughly some elements of people, of how we do things in Django and a little bit of the theory of things. And I don't have a nice round off of it other than I'm really excited for the questions and the discussion that we're hopefully going to have. And thank you very, very much

Speaker 2: Thank you, Sarah. That was fantastic. Well done.

Speaker 1: Thank you.

Speaker 2: Um I'm gonna start off our questions. When you started as a contributor to Django, how did you, like what were some of the ways that you humanized yourself with the existing contributors? contributors

Speaker 1: um I was so nervous I think it was very obvious um so I guess there are a few things that you it's I I am using my name, you don't have to, and my GitHub profile is picture is of my face you nobody has to do this if they don't want to do this. But those are some subtle uh clues that this is a person. And I think my very first interaction was like some comment on a track uh tickets saying hello is anybody working on this because I would love to work on this and all this kind of stuff um and

Speaker 1: I tend to um I think I showed a screenshot of of Ned's talk and in his tips he says use more words um around how to like have better interactions and i i'm definitely a a person that is quite wordy and scrambling right now. But I do think that there is an element of uh If you add in a little bit of the fluff, uh people can s feel what you're trying to portray a bit better and a bit about you as a person and some fluff is nice so if if if it's your communication style uh

Speaker 1: then go for it use the use the fluff use the emojis and uh Yeah.

Speaker 2: Great. Thank you. Does anyone else have a question? Mo, I saw you had a hand up, not sure if you were clapping or trying to ask a question.

Speaker 3: Yeah, I I had one but it's it's uh far worse than those. But I was just curious on uh the point where uh uh Sarah said like taking the community's opinion. It's always like been kind of vague to me because not all Jengy users are Jang contributors, but like all the contributors are users. So I'm I'm not sure like how representative like the contributors are of like the bigger user base. So can Can taking the community's opinion be biased? Like can like how how how sure are we like Django is moving to the right direction

Speaker 3: or stuff like that compared to like the other ecosystem. I was just curious.

Speaker 1: Can it be biased? Yes. I think it definitely can be. Um and can we be sure? Probably not. Um we do have Uh we do have the elections for the steering council, and in some sense, that is one of the ways that we try to uh show that um the a larger community, so this is I think it's DSF members, are able to uh vote for people that they think uh represent Django quite well and that they trust to help make some of the technical decisions of Django.

Speaker 1: And so that is one mechanism that we have to help with this. we don't have a very good mechanism to get much wider uh feedback. And part of this is uh, you know, back when I was saying about the the structures of different open source projects We do have a group of passive users and we we don't really know how to get in contact with these users. There are definitely some ways where we have larger audiences than the places where we recommend people get feedback for a feature, like one of the larger Audiences that we have is we do have a annual Django

Speaker 1: survey and I would love us to uh develop this in a way that this gets more constructive feedback on uh what what we should develop in Django. I think that would be great But it's a really hard challenge and I I I'm really curious as to what ha what other people what other frameworks, what other large open source uh communities are doing. But I I know that we we try really hard and a lot of people who are engaged they care a lot and Yeah. I don't know, though. I wish I knew.

Speaker 2: Thank you for the question, Mel. Uh Carlton.

Speaker 4: Hi, uh Sarah, great talk, thank you, really nice. Um I I can't remember the exact word you use, so uh I'll stop trying. Um you you said something about um uh new contributors perhaps finding that they you know that that that their their ideas aren't well received perhaps because the bottom line is you can't A new contributor turns up and they've got three or four ideas and they try and make changes and it's like, but we can't just change the API structure of some class deep in Django because that's just not gonna fly. And then after a while, you know, there's this sort of training process for new contributors where it's like, well, you know, you have to why don't you have a discussion first?

Speaker 4: And you know, there's this in the fact how you request a new feature, but nobody reads the fact. Right, no nobody goes as it 's like is there a way you think that we can message better to prospective contributors kind of the how to. Your point was very much like do some bug fixes first because you'll get you'll get the hang of doing the contributions on things that will definitely be accepted. And then you can perhaps suggest a feature. And that's a much better way from a sort of pedagogical point of view. But how can we communicate that? So contributors don't get this kind of um upset when they're when ideas are rejected one after the other at the beginning.

Speaker 1: Um I don't No. What I can tell you is um it is definitely a thing that we we see is happening. We can see that people get frustrated, um, especially when they rattle off three or four ideas in a short period of time and that they get this resistance with all of them. And The process itself can sometimes be frustrating because it's not a quick process getting the feedback either And just because you had like an initially positive feedback from two people, it doesn't necessarily mean it's it's gonna fly a bit later. What I can say we are planning to do is we are planning to

Speaker 1: talk about some of these processes. And some of the times where we've noticed there's a pattern of upset and frustration in the community when certain things are happening. And we want to have a discussion with the Code of Conduct Committee and ideally the Steering Council about how can we make this process by design a little bit better uh because we're we're coming across this same um friction point with people and if we can improve that then it would improve the experience for many people. I I don't know whether I I would love to be able to say when someone comes and be like, oh

Speaker 1: no, but please fix this. It would be great. Maybe they would, who knows? Yeah. Um, I don't know.

Speaker 4: Yeah. I mean, yeah, I there was an example this week that came up and somebody's a new contributor, they've opened two or three PRs without any discussion, and then they those PRs haven't been well received, and then They've been sent to the forum, which is the right thing, but from their perspective, they're massively frustrated. And it's like, how could we have signposted that? They've obviously read the contributing guide to to to make a PR. And then it's and that it's that. And that that happens time and time again. I don't have an

Speaker 1: option. No way that

Speaker 4: thank you.

Speaker 2: Anyone else have a question? All right. Well thank you, Sarah. We'll turn off the recording here, but uh really appreciate the talk. It was fantastic. And thank you for everyone else who was asking questions. Uh till next time.