llms.txt and Django

This video features Thibaud Colas at DjangoCon Europe 2026 in Athens, Greece.

llms.txt and Django
1:28:13
Published July 21, 2026
9 views

WORKSHOP: llms.txt and Django by Thibaud Colas

The llms.txt format is an emerging standard to structure information for Large Language Models. It’s a desirable addition to the docs of Python packages. We have been busy adopting it for Wagtail, and can now share how it all went. Let’s review how to adopt it! We’ll discuss the fundamentals of the format and its benefits as a user of the docs, and as a maintainer. How to produce and consume those files across different tools (Sphinx, mkdocs, Django). How to optimize them for different LLMs with an eval suite. Tools and techniques you should be able to reuse through other engineering tasks with LLMs.

More information: https://2026.djangocon.eu/speakers/Y8QPW8/

Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope

#django #djangocon #djangoconeurope #djangoconeurope2026

Transcript

12,555 words · auto-generated · en Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:03

Speaker 1: We'll get started shortly. This is a bit last minute, so but we'll figure it out. Um if you wanna get a head start. This is a last-minute workshop addition to replace for another uh speaker. I had prepared this as a talk initially, so This is meant to be something you watch, but there's definitely interactive elements here and there. If you're here on your laptop, I would recommend you set up an AI agent, like agency coding tool I would recommend open code or sorry. Open clo via

0:48

Speaker 1: the Kilo Clow platform. They're the most interesting. It's an adventure. There is risks. There might be rewards too, but we'll see. If you're here on your phone It will be more limited, but Google has an excellent open source app. Called Google AI Edge Gallery that has an open source model built in. If you want to try some of the things we're discussing today from your phone, I think this is probably the best option Uh this is a few gigabytes of downloads for the models, so I'm not sure if the conference wifi is up for it, but so far it's been amazing

1:42

Speaker 1: I I hear the main stage was a bit delayed. I don't know when I should start.

1:52

Speaker 2: So welcome all. So as you know, we are in uh Interesting moment of a web development where a lot of LLMs are analyzing, reading the content of the web, but the web is not designed or being in mind to create an LLM or the otherwise as well. The LLM was not created, bear in mind how our website was created. So I'm hand over to Thibaut to explain or put a light on llm. txt similar like a robot. txt but for the AI Thibault.

2:26

Speaker 1: Thank you. Hi, I'm Thibaut. I think I'll start with a brief intro. I'm um product lead for an open source project built on Django called Wagtail. Wagtail is a CMS. We have about on the order of 20,000 projects out there built on Wagtail. And a lot of what I'm going to share today is based on my experience trying to adopt understand and use this this new standard nms. txt on on those projects. So yeah, this is meant to be some somewhat interactive to make it more interesting for everyone. Top of the screen here, wactel. org slash Django dash AI, you'll have access to this doc we're looking at with all of the links.

3:12

Speaker 1: This isn't a must, but if you want to follow along on your own, that will definitely help you. So again, wactel. org. org slash Django dash ai and you arrive on this repository. What not found? Oh the link and just the link is is reverse AI. Django dash ai Yeah. How hard can it be? Django dash AI, AI-Django. Well, we're doing it live already. It's Django dash AI, right? Yes. Thank you. You can tell I I'm very prepared for this. Um yes.

3:59

Speaker 1: Sorry, this isn't this isn't how we do things on Django, but not on Wagtail either, but we'll we'll figure it out. Uh yes, so Um LMs. txt, uh why do we care and why I'm here? Um I came to Wagtail and Django because I value open source, because I value the community we're building together. And um I'm coming now to AI because AI is a fundamental paradigm shift to um our livelihoods, to open source, to our industry, to our society really. Um so I'm I'm here because I think it's an interesting topic. And LMS. txt, I want to cave it this upfront. It's a very teeny tiny part of this paradigm shift.

4:45

Speaker 1: It's a very specific format. There's not that much to it, but I think it's a really good way for us to explore how our jobs are changing and how I suppose the AI uh has an impact on how we find and process information. And yeah, just I wanna I want to plug two things that I'm also part of outside of this one workshop. Um Wacktail, we have a survey running right now on AI and developer experience. If you want to help us understand what developers expect, how our project should adapt to changing expectations, this is the place to be. And tomorrow Laura Laura and I have um facilitated

5:30

Speaker 1: discussion in the open spaces format on uh how AI affects the Django community. It's gonna be in this room uh at the same time. Um yeah so come to that. And uh yes, lms. txt so LM. txt is a file you add to your websites. Uh at a glance that doesn't sound like much. There's actually quite a bit more to it. But the fundamental change here that uh is implicit is you're adding this file because you care about how the LMs, how the AI models understand the information on your websites. You're no longer structuring the pages of your site for humans, you're structuring the content of your site for

6:19

Speaker 1: the intermediary, the AI model that's going to digest that content And turn it into a chat answer in someone's AI agent, someone's deep research prompts. I'm sure you've all heard of Chat GPT before, so I'm not gonna demo that, but maybe just briefly if I show what lms. txt looks like on a real website. This is Wagtail's user guide website. It's our documentation for users of the CMS, people who spend time in the CMS admin interface, just like Django's admin, but a slightly different tool. And in the upper right area of the page we have this new copy page control that's gonna add a markdown version of the page to my clipboard

7:09

Speaker 1: I'm gonna do something a bit risky and switch over to Chat GPT and paste that in. So there what I've done is basically given the ChatGPT model the opportunity to answer my query based on the content of this page. That's kind of what LMZXT is about. Except we don't expect every single user of that site to go and copy that thing. They don't have to care about this. Instead, the file is at a standard location. Uh well-known URL is what it's called, slash lms. txt, so that agents can autonomously find the file. And uh yeah, it's not just the one file. So if we look back at that dropdown, there's also a lm.

7:57

Speaker 1: txt with a dash full suffix. This is a dump of the whole website's content, all 50 or so pages of the site in the one file. So you can imagine if I want to have an answer on my Wagtail CMS experience question It would be very good if I can have the answer based on the official docs. Maybe I don't trust the site's search to work well. Maybe I don't search Google. I don't trust Google to find the right source. But I can use this, I can copy this file and get access to the authoritative information. Um other other platforms have this, so if I look at the cloud docs, for example They have a similar copy page. They have a similar drop-down in the top right.

8:46

Speaker 1: And they have copy as markdown, open markdown, and they have chats in cloud AI. I do not want chat include AI. I do not want our sites to be tied to any one provider. I prefer lms. txt because it's an open standard that any open source tool can consume. So that's kind of the the the reason why we need this, the ethos. Um I'll do a brief demo of how we use this, and it's gonna be the briefest ever. Um who here Uses AIs to code like in the chat mode where you chat chat GPT hey how do I do this? Can I have like a show of hands who does that Okay, who does like agentic coding where in your IDE or CLI you're gonna ask it and it's gonna do a lot of like thinking over time? Yeah, quite a few people still?

9:33

Speaker 1: So you know how important it is for the AI to be autonomous, that it has access to the relevant information. And as a developer, that means developer docs. So you have tools like MCP servers you might have heard of that are there to facilitate your access as a developer to documentation, to maybe quality assurance tools on projects. So Context7 is one of those tools. It's an MCP server that contains documentation for lots of open source projects. basically. So they are there in the middle translating the docs of those projects into a format that is easy to query within an AI agent. And so that's lovely and all, but again, I'd much prefer for us to rely on open standards rather than one specific provider's implementation.

10:26

Speaker 1: But if you want to try um lms. txt, if you want to try documentation uh hooked into your AI agents, right now this is definitely very convenient and uh worth worth a go. Uh there's other options here, so I don't know how to pronounce this. N Ia Nia. Another similar MCP server for docs. And uh more like do-it-yourself, MCP doc is a command line uh MCP server that you can use to have access to the same information with this standard format inside your coding agent. But before we do that, I think it's important to take a step back and talk about why we care about this.

11:15

Speaker 1: I don't know if anyone here will have seen this before. I'm sure you've all heard of um Andrei Karpati, uh lead AI researcher at OpenAI. has since moved on to other ventures, very influential in this field. And he created this dashboard, which is a visualization of the US job market. And it has a it has a filter that I find particularly interesting, which is this digital AI exposure filter. So I'll switch to that. And we have this visualization of the US job market when it comes to digital AI exposure, as in which jobs are most likely to be affected by AI. And we are right there

12:00

Speaker 1: Software developers, that's our official uh US Bureau of Statistics uh information, and we are at nine out of ten. digital AI exposure. That is the highest score you can get, the visualization. I'm sure you've all seen to what degree those tools are changing how we do our jobs. That's why I think we should care. That's because uh you know it's not necessarily gonna reduce the job market. I don't know that. Maybe some Some people would say that. But it's definitely gonna affect the type of work that we do and how we do it. And so yeah, we should uh we should uh unpack this a bit. So earlier I asked who here was using um chat AI tools for their coding and who was using AgenTic

12:46

Speaker 1: tools. Um this uh this uh this uh research here, task completion time horizons of frontier AI models, what a mouthful. That basically means over time, as the degree of AI adoption, diffusion, refinement, as the degree of model capabilities increase How long of a task can the AI tool do? So you could imagine back in 2022 with Chat GPT. You could do a task with it, that's kind of like a web search information retrieval, about 30 seconds worth of task done by the AI. Few years later you start to have connections.

13:34

Speaker 1: Few years, sorry, what am I, who am I kidding? Few months later, start to have integrations in the chat interface. You start to have more powerful models. Maybe you can have the AI instead do ten searches at the same time, give a report back to you. That task might now be more on the horizon of five minutes uh duration. A few months pass again, more powerful models get released, and now you can probably generate a whole page of a website. generate a whole Python module. A few years later again you can generate a whole project. So this here is a logarithm scale. So every increment on the scale is 10 times bigger than the previous one.

14:19

Speaker 1: And um we're at the point now, 2026, where per this research at least AI models can autonomously do on the order of about 10 hours of work. That's not to say that they do it completely correctly, to the equivalent quality as a human would But gives you a ballpark. This is the the 50% success completion of tasks. If we look at the 80% success, we're only at about an hour. So Basically, you have to be mindful of you know expectations here. But we can clearly see that over time AIs make more and more complex tasks. And that also means that fewer and fewer people, fewer and fewer developers, actually spend time looking at documentation because they have the AI do that for them.

15:09

Speaker 1: And uh I'm not sure this pull request did the news, uh made the rounds like a few months ago, weeks ago. This is in uh Tailwind repo. I'm sure lots of you will have heard of Tailwind, but it's basically a CSS framework that is well known because of how AI friendly it is, how well it works with AI tools. They have a do a great job by generating Tailwind code. And um yeah a few months ago someone says, oh, I'm doing lots of work with AI agents. Why don't I propose adoption of this new LMZD format to the Tailwind docs? Seems like a great idea. It'll be even easier to never have to read the Tailwind docs myself ever again. Except if we look at the replies. Adam Watan, one of the lead maintainers of Tailwind, said no.

15:58

Speaker 1: And he said no for a very simple reason. Which is that his whole business model depends on people looking at the documentation of Tailwind to find their training call to action, their paid consultants. call to action, their Tailwind Plus package, those things aren't necessarily part of the Tailwind experience for everyone, but for them it's their business model. They make base Tailwind available for free, they expect you to look at their websites and we'll hear about their other products. And what they found out is that despite Tailwind being more popular than ever, their documentation website visits were down 50-40%. over time and they said that because of this they had to lay off seventy five percent of their engineering team.

16:46

Speaker 1: I'm not necessarily saying that AI is replacing those people that is It's you know a foregone consequence. But definitely right now we have to be aware of the consequences of this adoption and how disruptive it is to both open source and those businesses models that sustain open source. Yeah, I'm sure there's lots of Tailwind users here in the room. I am myself definitely. I definitely haven't seen their website in a while, no matter how much I love the framework. So I do hope Tailwind succeeds I do expect eventually they might start adopting things like this format, but it does mean they have to reinvent their business model in the meantime. And for someone like me, whose whole job is helping other people make websites It's obviously also really impactful.

17:34

Speaker 1: So you know these days people don't really go to Stack Overflow anymore. They go to the AI model yet again. And so it's going to be very important for those reasons that we understand exactly how the AI models find the information and use it. So yeah. I thought it would be a good time for our first little interactive exercise, which is to ask the AI about a specific page on the internet. This tool is called gina. ai and if you have your AI tool of choice at hand now is the time to go to it like I will here and start typing H

18:19

Speaker 1: T P s colon slash slash r dot gina dot ai slash Like that. And we're gonna be able to try out Gina 's Ability to convert on the fly HTML pages to markdown so that the AI can ingest it better. So all I did here again, just to be clear. is type the name of their service, then slash, and then it's URL of any web page on the internet you want to try this on. And I can ask my AI agent of choice, so here, this is OpenClaw

19:09

Speaker 1: , could you recap the contents of this page for me? So I don't know here who has heard of OpenClow before. There's a bit of a mindset change here. We are inside Telegram. Telegram, you might not expect it as a AI app, it's a chat app. But OpenClow is very popular as something that you'll integrate into your existing chat tools rather than dedicated programming tools. Uh the idea here is the AI will be available wherever you are essentially. They'll find you where you are and adapt how it works to in this case your preferred interface being a chat app.

19:56

Speaker 1: Bit of a disclaimer. Earlier in the workshop I was recommending people try out OpenClow. It's a very dangerous tool. uh the whole model of how the tool works is it can do anything that you can do on your computer. It has access to all of your files. It has access to all of your fancy API tokens, all your private keys. So I'm running this inside a virtual private server, completely isolated from my computer. This bot only has access to its own virtual private server and my telegram channel and a GitHub bot account. Uh so just be very mindful, very mindful of that if you were to try the service.

20:43

Speaker 1: I do highly recommend it because in my opinion of how transformative it is. So again if you wanna if you're following this at home or in the room, if you want to try OpenClow, uh the Kilo platform has something called Kilo Clow. And right now they have an offer where they have a free trial for seven days, which is this OpenClaw plus VPS server, so you can try OpenClaw without having to put it on your own machine Which you you definitely should not do, just to be clear. But anyway, this specific demo here, you can do that with any any AI agent, really. I think it's worth being clear, the AI will have no problem reading the HTML of that page, but

21:31

Speaker 1: you can imagine uh Um like you can you can kind of relate as a as a human how unpleasant it would be to want to access the content of this page and have to look at the source code. So the whole point of what we're doing here is to instead have something that's much more optimized to look at the content only. So this is our first taste of I guess why we care in practice about LM ZXT. we care because it's easier for the AI to extract content out of it. But there's a big difference between what we're doing here and LM. txt, which is that LM. txt would not contain all of the site's navigation here. So essentially all of this menu here, the navigation interface, all of the footer.

22:18

Speaker 1: You don't really want your AI agent to have to look at that. You don't really want to have to pay the cost of the data processing of that content. You only want to look at the actual, you know, interesting bits of the page. So that's why we're not just relying on this one provider. We're not just doing HTML2 markdown on the fly. We're publishing another format instead. Just to hammer this point home, I don't know here who has heard of prompt engineering and context windows. If you've used cloud code tools like that, you probably have a good sense of how much of input you can provide it before it starts. Hallucinating and going crazy. So the fundamental unit here is called

23:03

Speaker 1: a token. It's a representation of one bit of information in the model. And The Django docs is way too big to fit in the token uh context window of an AM model. Um I'm sure no one here has ever read through the entire docs of Django end to end. The AI tools, not even them can do that. If you look at the HTML version of the docs, like the whole hundreds of pages in HTML, that's on the order of six million tokens. And flagship models these days, they're on the order of 200,000 tokens that they can hold in memory reasonably well. The 200,000 budget, that's for your whole task, not just the docs.

23:54

Speaker 1: So if you're blowing this budget by few orders of magnitude already, not comfortable. So that's why better format will help us at MZTXT. but also chunking those docs into smaller formats, just like we do for humans. We break it up into separate pages so it's easier to find specific information I'm going back and forth here between like why we care and w how the format works. Any questions before I move on to the next thing? How does it make you feel so far? Good? See some thumbs up? I see some frowny faces. Who here has used lms. txt before? Do we have like early adopters like me in the room?

24:40

Speaker 1: Yeah? Couple? Yeah? I'm sure more of you have heard about it, but maybe thought like oh why do I care? Definitely fair. I'm not saying we should care. I definitely expect those formats could come and go. When I mentioned the 200,000 tokens like context window of models. The window size increases over time as models become more and more capable. So some of the techniques that we had to do, like two, three months ago, they're no longer relevant anymore and there's new techniques emerging. Uh just a bit of a tangent. Who has heard of caveman LLMs? Ah okay. Anyone here using it? You have used it? Wow. Oh yeah, wow.

25:26

Speaker 1: So For those who haven't, it's another type of token optimization where you instruct your AI to respond like uh like a caveman. I feel like I'd I'd probably say a cave person if But no, you you tell it use terse language, respond in this very kind of robotic, crude way. Those people say it's like sixty-five percent more bang for your buck, so to speak. So Yeah, I don't necessarily want us to do that right now, but my point here is the techniques are changing and some of them might feel a bit silly right now. I'm not sure if it's temporary or no. Anyway, check it out later. I'm sure it's fun to try. While we go on to the next step, I'll ask my AI, could

26:15

Speaker 1: you speak like that, please? Give me the same summary as above, but caveman style. See what it makes of it So yeah, back to the Django docs. We want to look at those optimizations because of how big the docs are, because of how hard it is to find information. We have to be very aware though of uh of the pitfalls of this. Um so something I have left unspoken for now is um How much this uh mediation of our information consumption by by AIs can also be a problem. Um I'm sure we've all seen our plenty of share of of slop of AI slop, but that's not what I'm talking about here.

27:02

Speaker 1: What I'm talking about is if you basically leave it up to the AI to explore the projects for you, the AI is gonna make some of the decisions on how to explore things So this report here that I found very interesting is from a company that specializes in understanding how AI models shape adoption of different products. So they've been trying out with Cloud Code Asking Cloud Code, ah, what would you use for this project? Which continuous integration service should I use? Which CMS should I be using if I had a need for this or that and that? If I was on uh on um say a fast API project, which ORM should I be using? It's very important to be aware that the AI is then making those choices. not just

27:47

Speaker 1: based on our own needs, but also how it perceives the world that's affected by how people publish information for those two I don't know here like who has heard that cloud code, like the kind of flagship agent tick coding tool, their source code. So the code of cloud code was leaked like a couple weeks ago. Yeah. It's a super big deal for a ton of reasons, and one of them is those researchers they found that Cloud Code had 400 different tools hard-coded Saying like oh this isn't something you have to be aware of, that is also and when I say like hard coded, it's not necessarily nefarious. It's like oh if the person mentions Django, here is where the Django

28:32

Speaker 1: docs are That said, I can definitely expect the ads coming in six months to a year time where some company might be paying to tell Anthropic, hey, please uh Why don't you push this or that thing I'm using? Um and yeah, so if we compare recommendations with different agents, we we see interesting patterns. So same company, different report. And they have this specific section that I find really interesting. To the right we have Bun. Bun is a node toolchain for JavaScript bundling, formatting, linting, QA, QA.

29:19

Speaker 1: Burn is what Claude Code is built with and Burn was also acquired or acquired by Anthropic I'm sure it makes a lot of uh business sense to do that, but we definitely see in the report here that cloud code is suggesting burn way more than codex is suggesting burn That's interesting, isn't it? I wanna I wonder why that is. So just to be clear, like Cloud code isn't saying on every single project oh you should adopt Burn right now. It's if you have a need that Burn might be a good tool for, then Cloud Code will pretty much systematically mention Burn as an option.

30:06

Speaker 1: Okay, and definitely pick it also. Again, not necessarily nefarious right now, but that's the kind of thing I want us to be aware of as open source technologists, those companies They have those hard-coded tools in their integrations with the models. And um yeah, we can see it in practice. And uh yeah, just to be clear, it's not just entropic and cloud code. So to this side, uh we have statSig. I have no idea what statsig does. All I know is they were acquired by OpenAI. who put the Codex agent tick tool and Codex again recommends StackSig way more than Cloud Code does. So there's there's clearly a pattern here. You might be interested to hear about

30:54

Speaker 1: how they recommend frameworks. So we'll try that next. But um yeah, this It's quite close to home. I'm sure lots of you will have been switching from Peep, Poetry to UV, new package manager. I'm sure lots of you will have been switching from Flake 8, which is a very popular linter, to Rough. Both UV and Rough are by this company called Astral who announced a few weeks ago they were joining OpenAI. I'm sure there's plenty of good reasons for that, but it definitely makes me question how much my tooling starts to be controlled by a single company like that. Anyway, yeah, this is the next like interactive bit that I would like you all to try with your own AI tools.

31:40

Speaker 1: Is asking the AI which web frameworks would you recommend? So again, there's no right or wrong answer necessarily here. I just think it's interesting for us to try this specifically. You might have noted a bit of a pattern here, which is that I'm always recommending open source things. So here again we'll use open source AI models. We're inside a platform called OpenRouter. Router? Is it Open Router or Router? Can never tell. Yes. Um yes, so open open router or open router um is one of the leading providers of kind of uh AI gateway service where you have access to

32:25

Speaker 1: hundreds, maybe at some some point thousands of different AI models, very different qualities, very different sizes of models. But it makes it very easy to see how different models share the same information. They also have excellent statistics that show you which models out there people are using. Uh if I look all the way to the bottom here, there is this very cool app and agents ranking which is showing you when people use open router, which tool do they do that with? So that's how I learned about OpenClaw. That's how I learned about uh Rule code and so on. I don't have a ton of time to keep up with all of that stuff, so it's just nice to see like which ones gain

33:16

Speaker 1: popularity. And um yeah, for for for this demo I'm using the model fusion, which allows you to ask the same query to multiple models at the same time, and then combines the answer. So here we're gonna be on a budget. We're going to be using Gemini 3 from Google, Deep Seek from From DeepSeek in China and Kimi from Moonshot AI. Gemini 3 is a proprietary model. Deep Seek is open weights, or some people people say open source, Kimi also open weights. Which web

34:01

Speaker 1: framework should I use for A new project. Um I need a um additional D B API layer Uh good testing capabilities easy to deploy. Do you think Django will be in the top like three options? No? But but the PI interest would file. How about if I say like oh uh a website and uh built in testing framework. Maybe that will change things.

34:49

Speaker 1: You're all quite cynical. Django's been very popular in my testing so far. Okay, so those three models, and then it's gonna be fused together uh thanks to Cloud Opus. Okay, uh while we'll let them think. Um quick shout out to Kimi K2. 5. It's an open source slash open weight model. I find it really interesting because um it's pretty much as good as CloedSonnet, which is one of the like flagship anthropic models and also because anthropic makers of cloud sonnet have been complaining that Kimi was trained on CloudSonnet

35:35

Speaker 1: output I find it very interesting to hear that knowing how many Django books out there Claude has been trained on. I don't know if anyone knows the answer. It's 165 books. that uh Anthropic, they had uh they had a quite uh well-known lawsuit uh about their their use of pirated books copies and there were 165 Django books in that dataset of thousands of books that uh cloud was trained on. And now they're the ones complaining that uh the Chinese models are trained on CloudSonnet. So yeah just the irony is just oh Any questions before we look at uh the results?

36:20

Speaker 3: Do you have a memo return on for your personal account with the The C Django is there, for example , while would you like a request to take these queries?

36:28

Speaker 1: Do I no, I don't think I have memory. I don't think this has memory support But it's a good question. Yeah, definitely you know what I'm what I'm what I'm telling you here isn't like trust the results right here is try it on your own computer and see how this will be influenced. Okay, so oh top one Django plus Django Race Framework. See you're also negative. Ah No, I mean the thing is people are surprised how good AIs are with Django. Uh I mentioned all of the books that this has been trained on. I'm I'm sure you've also heard of um Sweetbench. Who here has heard of Suibench? Yeah? Do you know like what Sweet Bench is testing?

37:18

Speaker 1: Do you know how much Django is testing? Yes. Yeah, yeah. So I'll repeat for for people who might not hear if you're online. Uh Sweetbench is one of the earliest leaderboards or like benchmarks of AI models And um the original Sweetbench had on the order of about 500 tasks that AIs would complete to see which one solves it best. And I think that sweet bench the original one, it was like 60% of the tasks were Django pull requests. So they literally like pulled 300 pull requests from Natalia, Carlson, our fellows looked at how they would review code, look at how the pull request was created, and were

38:03

Speaker 1: training their models, or at least scoring their models based on that. So yeah, um that's part of the reason why those AI tools are so good with Django. And you know, Python code n generally they're very good at. Django docs are excellent. There's lots of good reasons. It's not just my quirky little snipes that uh that make this good. Anyway, so back to open open router and um cloud summarization. So yeah Django plus REST framework or Django Ninja Our motto displayed exactly as is, very great. Django ORM, one of the most mature, definitely fair. API, testing, deployment, very, very nice. Fast API, huh?

38:50

Speaker 1: Yeah, definitely something we expect as well. What would be very interesting for me is if at some point it says fast API, oh, and also you should probably host it on Fast API Cloud, because now Fast API as well has a business model. That's entirely, not entirely, but dependent on them having their cloud platform successful. So yeah, we're not there right now, but you know, something to be aware of. Nest. js, Ruby on Rails, ASP. NET, and so on. Honorable mention for Go. I'm a bit surprised to see Rails in here. I feel like Rails AIs haven't really picked up on it too much. Don't know exactly why that is. But yeah, I feel like it's all all

39:35

Speaker 1: worth being aware of. There's some something that's absent here that I'm very surprised about. I don't know here like who has a sense of what's the most popular web framework overall right now. Like D one that every single conversation I have mentioned. React computer. Yeah, React and do you know what's built on top of React? NextJS. Next. js, yes. Next. js. It was no it yeah. You know the um JS communities with um we've run out of names for uh JavaScript packages so It's nest, not not next, but uh yeah anyway, next. js. Uh very interesting that it's not mentioned here. Um because Next.

40:20

Speaker 1: js compared to the Fast APIs and the Django's, since they have the support of a venture capital company, Versal, they have millions of dollars to spend in their marketing budgets. push their framework. So I'm really surprised that this isn't picking it up. Anyway, back to our ecosystem, kind of starting to unpack how the AI makes those choices. And um I thought it was worth highlighting highlighting Django packages because part of the strength of Django that isn't necessarily shown too much in those outputs. is the community and the package ecosystem and that's uh something where in particular I think there is a lot of room for us to be adopting LMZXT or changing our practices to better fit

41:08

Speaker 1: those changing expectations. So we're gonna look next at how to measure and test lm. txt files and um I would love for you to consider Maybe during the sprints, see if you can contribute maybe LM. txt on an existing package, assuming the maintainers are okay with that. I think it'll be a really nice task for you to try it out in practice. Okay, measure. Gonna go through this pretty quick. I definitely think it's worth a look. So I just wanna show that This is all for real. This isn't just for enthusiasts like me running OpenClo that this is relevant.

41:53

Speaker 1: Uh we're not looking at the server logs of our websites. So again just to recap the user guide website of Wagtail which has lm. cxt. And the reason we want to look at the server logs is again, people don't necessarily go to the site themselves anymore. Instead, they have their AI fetch those files. And I'll do something very simple. I'll add a filter here for user agent cloud code. Oh. Maybe I misremember what the agent is called. Well no we'll go for one that I know for sure. Chat GPT user.

42:39

Speaker 1: So the user agents that those tools are using, they are not documented anywhere. Oh, now this is definitely something else. Um sorry, I don't know why my data fetching is failing. Because I did equals. You could have told me. Fox, what are you doing? Sorry, definitely a skill issue right there.

43:31

Speaker 1: All right, back in action. So If we look at our server logs, if we filter by user agents, we can see the queries coming through. This is quite a small traffic website, very small audience in the grand scheme of things, but definitely on a daily basis, close to a hundred requests for f from uh chat GPT users. And I can do one further step of filtering and see the paths. No, actually I don't even need to do that. I can just look at the top top pages on the site.

44:06

Speaker 4: T gold. Sorry, can you just

44:08

Speaker 1: clarify?

44:09

Speaker 4: This is visits to the site in response to things that people are doing in ChatGPT. It's not chat B GP chat GPT scraping the web for it. Is that right?

44:22

Speaker 1: Great question. So in this specific case , this will be requests to the site in response to people interacting with Chat GPT. There will be different user agents for OpenAI scrolling of the sites. If I look at GPT bot. OpenAI is kind enough to use different user agents per AI integration. GPTBot is their crawler. I'll see here how they index our sites. If I look at ChatGPT user, this is people inside ChatGPT asking what questions. And I can see here that Across the top things that this navigates to, there is our items.

45:09

Speaker 1: txt file. Just like this top one, that's the site homepage. That's also the homepage because there is a redirect. So basically Thing number one that ChatGPT looks at is our homepage. Thing number two is LMZXT. To me that's pretty clear proof that this is actually used in practice. Yeah, what I leave implicit here is those providers don't necessarily tell you how they're crawling. Don't necessarily anything nefarious there again. It's also because they are changing how they do things all the time. But there is a lot at stake here in understanding which AI tools look at what when. Yeah, so yes? So I think could you switch back to the

45:57

Speaker 1: To the box. The um crawling one?

46:01

Speaker 4: Yes, please.

46:05

Speaker 1: I wasn't sure how to make this interactive, so this is great. Um

46:09

Speaker 4: so I don't see LMMs. txt being picked up there.

46:12

Speaker 1: Yes. So If we look at their crawl pattern, it looks like they've done one big crawl at one specific time. And we might have to filter by path to see whether they load that specific URL.

46:33

Speaker 4: And I wonder if you might try it for a different um a different company as well just to see.

46:41

Speaker 1: Yeah um so what we can see here just to be clear our website GPT bot LMs no match and I know why it's because they index different content So something I've glanced over for now is when we say lm. txt, I've shown you the one file, I've shown you the full file, there's a third format, which is a markdown version of every page of the site. separate. So GPT bots specifically, they only crawl those markdown versions of every page of the site. Does that make sense? And they do that in one burst. And um

47:26

Speaker 1: yeah, Claude is very guilty of these two. Uh traffic pattern is very much like hit you and Yeah, 400. Um anyone have questions at this stage? Yeah

47:42

Speaker 4: I was well five mark a version of your web page.

47:46

Speaker 1: Yeah, good question. Just repeat after clouds. Yes, thank you. Definitely should. How does Cloud find find the version? So There is two uh options that are being floated out there. Um on our site, if I look at the HTML markup of our site inside the head tag We have this link meta rel equals alternate that says there is a markdown version of that same page. And that links from every page to its markdown version. So there is this. In addition, if you look at the lms. txt of the of the sites,

48:31

Speaker 1: It's linking to all those markdown versions. And there's a third option, which I don't know if it's real or no. I've heard that some tools use HTTP content negotiation where the tool will say, oh, I accept markdown responses. And if your server Supports markdown responses, then when the tool goes to the page, your server decides do I give it HTML or Markdown. Does that make sense?

49:01

Speaker 2: Yes. So are you maintaining two versions of the docs? If so, how do you then with drift a little bit?

49:07

Speaker 1: With drift? Okay. So the question is how do we deal with um the docs versions being different? And the answer is we code our way out of this by having the one source that is then converted into the two formats. So have a brief peek at the implementation here. Uh where did I put it? This is how the markdown version is generated. So we're looking at a template here, just like any other Django template, that renders markdown rather than HTML. So the content in this case is managed inside the CMS, inside Wagtail, then we create an HTML version. version and a markdown version. If you are looking at the documentation website

49:54

Speaker 1: where the source is in your repository, I'll show you the other tools later. Any more questions? I feel like I'm going a bit too fast. Yes. Yeah , how exactly is and what they have to do. Yeah, yeah. So Again, it's like a bit of a black box at the moment. So lms. txt some tools find that. That's kind of the sitemap version of the format. lm stash foo that's another version that's the full dump of the whole site content.

50:40

Speaker 1: As far as I know, those two names are like driven by convention. So you know like you can hit those URLs on the site. And also if you look at the adams. txt it will link to the other version. So that's how the tools find out about the files. So what I can't tell you is how much of an improvement to the I guess answers you get from the AI, how much our content perception is improved by those files. I can tell you a hundred percent that those files are being crawled and used. I can't tell you how how they're being used in practice. Yes. Can

51:24

Speaker 3: you please say yes how your backend decides between uh distinguishes between uh the rating version the markdown

51:35

Speaker 1: Yeah, so how how we distinguish. So in the case of our website it's really simple. Two different URLs. You go to one if you want the MacDonald version, you go to the other URL if you want HTML. Okay, um how are we doing for time? Um Okay, so we'll move on a bit to publication. It's gonna be lightning fast. Um if you want to publish in this format for developer documentation It's really simple and there's established tools across the whole Python Docs ecosystem. We're looking at an MK Docs configuration file here for one of the Python packages I help maintain. There's literally all

52:20

Speaker 1: these is here. So we're just saying for this plugin of mkdocs, configure this, configure that, produce the files, and then everything else is done for you behind the scenes. Um so in this case we're looking at this package And based on those five lines of configuration, we'll have lms. txt, lms dash full. txt, and we'll have the markdown versions of every page of the site. Right there. With the same rel equals alternate

53:06

Speaker 1: meta tag. So it's that simple publication in that format for developer docs. You take the source docs You configure the plugin and it produces it like that. In the Sphinx ecosystem, there is Sphinx dash M, exact same story. If I look at um Django. If I look at how the Django docs are built. and the configuration of the Sphinx document builder. All I'd have to do is add Sphinx underscore LLM to this list of extensions here and right there it publishes in this new format.

53:53

Speaker 1: Some of you at this point might be wondering um why bother? Why not just have the LMs look at the source files that are already in Markdown and restructured text. And there's a very simple reason for that, which is that those source files sometimes don't contain what you expect. I'll show you what that means next. So we're now looking at a spreadsheet that has every page in the Django docs. So those are all the pages by path. And then the three numbers, that's the length of the page in tokens. across the three versions of the page. So I mentioned six million tokens for the whole Django docs in HTML.

54:40

Speaker 1: If I look at the source files that are written in restructured text, 1. 5 million. Markdown version 1. 5 million. But then when we look at it page by page, we see quite big discrepancies. So this one for example, this page source is tiny, a fake you as well. That's simply because those pages have restructured text references. to other contents. The documentation content of that page is generated in the documentation build. It's not there in the source. You know you have your Python doc strings in Python code for a reason. And those tools move them over to the published page on builds. So that's why we want to look at

55:26

Speaker 1: a specific format, because the doc strings might be missing if you look at the source the AI won't know to also have to crawl all the Python files. which is uh improving our understanding of how agents read this content, not just on token length, but based on other aspects. So if you want to follow along on your devices Please go to this website, agentreadingtest. com, and we'll be able to see how your own AI agent understands content.

56:14

Speaker 1: So quick disclaimer before you submit this I'm not sure who here has heard of the term prompt injection. There's some dangers there in telling your agent to do this, do that, from what a website tells you to do. The prompt here on this website, the prompt is, hey AI, please go to that page, do whatever the page says. This is the modern equivalent of curl pipe into bash. Some agents will refuse to do this and will score very badly. Some will happily follow and expose you to some some risks

57:00

Speaker 1: uh and yeah maybe rewards as well. But yeah, we're in open claw, we don't mind I'm gonna start a new session. Speaking speaking of that , we play this line that is

57:15

Speaker 4: pen. Uh if you don't use um actual website there's quite a few cultural actions okay uh for sponsoring and things like that uh have any to added those also to another step. Have you noticed my maybe I'm not on the solo making a donation to the light mail?

57:41

Speaker 1: Emma asks uh how naughty you have been and whether I'm adding calls to actions in my lens. txt. Um I haven't yet. Yet. I'm thinking about it. Wagtail is open source. Wagtail is backed by many contributors. The biggest contributor is my employer, Torchbox. Torchbox has a good like 80% worth of Wagtails work is done by us. So definitely we have business incentives there. But in our case we believe that It's important for the product to stand out on its own and so the only like funding we try and get on Wagtail is through sponsorships.

58:27

Speaker 1: So the sponsorship call to action, I might definitely add it in there at some point. And when we say sponsorships, in the case of Wagtail, it's sponsorships of features that are meant to be beneficial for everyone So it's kind of a business sense that we think aligns enough with um community needs. Yeah, I haven't been too naughty yet, but like I definitely want at some point to do something a bit more anyway, moving on Uh okay, so I have my agent. I have warned you about prompt injection. I 'm gonna press enter and see what happens. So while we do this, uh we mentioned donations.

59:14

Speaker 1: Open claw, I learned recently, you can give it your credit card. And it can use your credit card on your behalf, on your behalf in theory at least. So I'll definitely at some point try and donate to the Django Software Foundation via OpenCloud. See what happens. I'll be interesting. Oh. Um We'll switch to something else in the meantime. All right. I do not know what's up with OpenClone.

1:00:03

Speaker 1: We're inside another AI agent called Open Code that I recommend because it's good, it's open source, it's running Kimikate 2. 5 I'll give it the exact same instructions and we'll see which one succeeds first. Hmm. Sure, that would help. Has anyone managed to get this prompt done and see the results? Uh yes, I didn't do. get any of the results that were that would pass anyway. So really? What are you using? No, just uh just not not in agent tools, just in in Gemini Okay. Yeah.

1:00:51

Speaker 1: Yeah. So Daniele says he doesn't trust enough agents or agents enough to do this. And uh Gemini, yeah, so it depends entirely on whether they follow instructions. So OpenClow is letting us down here. We're gonna look at open code instead. And here it says, oh, okay, great, I'll happily fetch all the pages you tell me to look at and process the content per the instructions of the pages. So this site here that we're following along. is a reading test for agents. So the idea is to give it different exercises to understand how the agents fetch the content. And if you've been through this, it should give you a list of um I think they call it tokens. So if you click score your results

1:01:37

Speaker 1: here. You'll be able to paste those uh no not tokens, sorry, canary strings is what they call it. And you'll be able to see how your agent is scoring. So looks like this here embedded markers canary dash strings found so I'll copy this over and it basically gives me a sense of Which parts of the doc the agent will have been able to fetch. So this specific agent with this model scores 11 out of 20. So then I can see from through the different exercises that the site has the agent goes through, which one the agent hasn't been able to figure out. Yes?

1:02:25

Speaker 4: This but it's the general pattern of the things that you're having to do. For example, here it feels like you're triangulating one black box against another black black box. You don't know what's happening in either of those black boxes but all you can do is have them respond to each other so you can get a better fix on on the reality of the situation.

1:02:45

Speaker 1: Yeah.

1:02:46

Speaker 4: Because we don't know what's going nobody's telling us what's going in this inside on inside those black boxes. And earlier on when you were showing the kind of the comparison matches of the um

1:02:58

Speaker 1: different providers

1:03:00

Speaker 4: and AI going round to all the other AIs and getting the other AIs to s to say what they found and then the third AI the other AI is is summarizing it. And Everything that we do has to be through some layer of indirection. And does it bother you? Because you feel that we're not ever getting closer to the truth.

1:03:24

Speaker 1: Yeah, so I'm gonna try my best to summarize this. Uh Daniele asks how I feel about the layers of indirection we have to jump through to understand how All of this is shaping up and what is going on. Yes, it bothers me, obviously. I wouldn't be here talking about it otherwise. I think from my perspective right now, I don't know where we're heading longer term The purpose of doing this right now is just to try and get a layer of understanding, like peel back under the surface, see how those things are shaping up. I'm gonna, you know, leave judgment for another time. Personally at least, what bothers me the most is um the copyright infringements I had mentioned earlier, the energy use that those tools have.

1:04:10

Speaker 1: But I definitely see the benefits in terms of results you can produce with them. So I think I want to know, you know, what's in it for them, what's in it for me, and what's in it for society at large. For now at least I feel like it's more important for me to try this than yeah just step back and feel bad about it. Does that make sense? I'm trying hard hard but yeah there's hits and misses uh yeah so this one again we're back at um context windows sizes of documents And we can see the tests that are failing are the ones where the AI has to fetch quite long documents to find information and it's not able to. Doesn't mean the AI

1:04:56

Speaker 1: tool is bad, it just means that the AI tool has decided I'll try and fetch less, perhaps as an optimization, so it responds faster. So yeah, just want you to be aware of this basically.

1:05:08

Speaker 3: question is there a way in open code to like expand the thinking process? I find it strange that it didn't like decide to just use grab canary

1:05:21

Speaker 1: Yeah, so

1:05:22

Speaker 3: there's the prefix with the same uh starting uh category

1:05:26

Speaker 1: Yeah. So um question is whether open code could optimize for this task and like find the specific tokens. So the point here is that as part of the exercise, the AI isn't aware that it has to find those tokens until the very last step. To try and understand like on a fair basis its ability to find information. So obviously if it was just grepping, if it could fetch the content to grep, yes. But there'll be some content that it can't fetch because it's summarized in another way. There'll be some content that maybe like in a JavaScript rendered page that it can't fetch. So that's why it's not necessarily able to, even if it was aware that it's a task that it's doing right now Um okay, yeah, so you got to see the results.

1:06:12

Speaker 1: I think it's just worth saying um here as well. I tried to compare a bunch of different AI models and the tools around them And interesting to see how they compare, but I can't really say tell you which ones do well or not well, but definitely worth highlighting when I mentioned some tools refuse to follow instructions. This is one of them. Of all the models out there, I definitely feel like the cloud ones tend to tell me how things should be done rather than follow instructions. Just worth being aware of. And um yeah, this reading test, if you're producing documentation for your projects, comes with recommendations just on how to get the best results right now. I don't necessarily feel like you should be jumping through all those hoops because

1:06:57

Speaker 1: they might be changing over time, but just worth being aware of.

1:07:06

Speaker 4: because uh this refused to do the refuse to fall for the comp injection

1:07:11

Speaker 1: yes

1:07:12

Speaker 4: okay so if it's not because it will it did not follow instructions for it.

1:07:17

Speaker 1: It didn't follow my instructions, I suppose. Yes. So yes, the low score is because when I told my cloud code Do this. I was like, no, I will not. So when I did that, it basically didn't go anywhere beyond the first page. didn't come back to mute the results. And I tried multiple times, just to be clear. Yeah, don't let this fool you into thinking that prompt injection is a solved problem. Did a workshop at my company a few months ago, like just a one-hour prompt injection 101 workshop. And within the one hour, first time ever I had tried prompt injection. And

1:08:02

Speaker 1: Just like you know, white text on a white page, text added in CSS, having a text on the page that's inside an image, all those techniques. The AI would find the instructions and follow them while the text was otherwise invisible to site visitors. So you know, just like cross-site security, XS. issues we have to be aware of them prompt injections definitely not a solved problem for any model right now. Okay I have one more thing to show you but I appreciate people are also heading off to the break and the other parts of the schedule. So before I show you this one more thing, I'll just do another mention, another shout-out for Laura and I's workshop tomorrow.

1:08:48

Speaker 1: And on the link you followed earlier, you'll find a link to a little survey. I'm gonna want to have people think ahead of time about which themes they like to discuss in tomorrow's workshop. So just again, this is at the same link, Django dash AI. And this is the link right there. So tomorrow's workshop we want it to definitely be a community discussion driven by the people in the room, this room, but it's a it's a big hairy topic. So just think ahead of time a bit, what themes you want to discuss There's a form.

1:09:33

Speaker 1: If you don't want to follow the form, just think about it in your own mind and come to the workshop prepared with a topic that's interesting to you. Alright. So last thing that I wanted to showcase in this workshop is another way to test how those AI tools are doing and how um The models respond to different prompts for information. Who here has heard of evil suits? Yeah, two of us. Okay. So Evals or Evals suits is kind of unit tests for your AI model or your Agentic harness all all the buzzwords there.

1:10:19

Speaker 1: It's just like bas being able to test re in a repeatable fashion how the AI responds to a given query. And I thought it would be a good interesting thing to end on, just because uh I think it's very foreign for a lot of us, uh, but definitely relevant for people to We can be aware of and potentially try out. So I'm going to run my eval suit. sure this is the best example but let's see. Yeah. So I'm using this tool called Promptfoo. Prompt foo. recently acquired by OpenAI. It's a bit of a pattern there. But it's open source, so you know.

1:11:06

Speaker 1: I'm trying my best. Prompt foo provides um a framework for how to create those tests basically. basically and uh this YAML file we were looking at, I'm sure we all love YAML , you can use it to create those test suits essentially. So you tell PromptVoo which prompts to test So here those are essentially Wagtail support prompts, what I do as a living, like you are a professional support helper for Wagtail CMS users. I'm telling Prompt Foo and via Prompt Foo the EI models, please fetch the docs here and there. And then I see based on my prompt, based on the docs. How the AI will respond to common questions from our users.

1:11:51

Speaker 1: So when it comes to Wagtail, questions like how do I recover a document I just deleted? I'm sure we've all been there. I'm sure lots of you will have wished that Django had a built-in trash or archive feature rather than deletes. Very common thing we can all relate. It's very crucial to us that when people ask this question to their AI, to our docs eventually, they find the right information. So what we're saying here with Prompt Foo is oh when we ask this we expect two assertions to pass. Assertion number one, the results should contain the URL to the page that has this information on our site.

1:12:38

Speaker 1: Because we also always want the AI to cite its sources. And assertion number two. So sorry. Assertion number one, that's regular expression or like string match. Very simple. Assertion number two is we have another AI model judging the quality of the answer. So we tell that AI model, oh, make sure that whatever the other AI says. is clear that this is not supported. Does that make sense? Yeah. So this this type of um Having the AI judge the answer is called LLM as a judge. If you've heard of um job interviews or like CV submission processes.

1:13:23

Speaker 1: really important to be aware of those LM as a judge concepts because they are definitely making decisions on the behalf of their users of what is and isn't good. You know, you can configure exactly what uh criteria the AI is judging based on, but only have so much ability to steer that if you don't know what is going on under the hood Um yeah, so the eval has completed. I ran it with three different providers. Mentioned here to Daniele earlier that one of the things that keeps me up at night is the carbon footprint of AI models. So the reason I'm running three providers and three different models is to try and make sure that the smaller models, the much smaller

1:14:09

Speaker 1: Mistral Small. performs as well as possible, ideally as well as the flagship Cloud Opus. And uh yeah open source, yeah. OpenAI has an open source model. Definitely want to see how that performs as well So rather than doing it with the one model, I try with a bunch. And I can adjust so that they're all on as much of an equal footing as possible. Does that make sense? Daniele question?

1:14:36

Speaker 4: I was trying something similar. I had a folder of 30 or so EDFs that I wanted to run very large prompter gates and to get a score for each one and I was trying to get my local agents to do that and if I gave it to them one One by one, each time it would give me a great result. I could not get, this was in co -pilot, I could not get it to go more than one or two without leaving context from one to the other or starting to take shortcuts and and and and so on. So scaling from a prompt or test that works really well once to

1:15:21

Speaker 4: a machine that can run that in exactly the way that you're demonstrating eluded me so far.

1:15:27

Speaker 1: Yeah, yeah. won't pretend to have the answers, but it's definitely something where um so

1:15:34

Speaker 4: how does this avoid that?

1:15:36

Speaker 1: Um yes just to recap here the use case is um getting the best results from those prompts by running the prompts on a specific I guess in this case document or here specific query. So it's all about how you manage the context essentially And in this case, every single test is a fully separate con context to stack in your unit tests. You'd want to make sure that if they rely on the database. It's all isolated with transactions or features or other fixtures or otherwise. So here every single row here, every single prompt will be fully independent from the other ones And maybe what I should say as well, when it comes to LM as a judge, it's a fully separate model that judges the quality of the results.

1:16:28

Speaker 1: So we have one AI query to get the answer with one model and then a separate model, separate AI query that says this was the user's question This is how you're meant to assess quality. What do you think? Does that make sense? Okay. Yeah. and assertions are uh taken by which instruction sorry can you repeat the tests yeah the test of tests the third test Oh yeah yeah I mean with vibe coding these days you can't really tell anymore which line of code is written by whom.

1:17:19

Speaker 1: Yeah yeah so here this is all written by me all the checks the the quality assessments this is written by me but then the type equals LM rubric here If you look at the prompt foo documentation, it will give you a sense of how the actual checks are run. So it's called model graded in here. And if you do LM rubric , they basically come with the default prompt that is tailored for uh opti optimized for the AI to judge from this criteria right below. So they have written a prompt. that they give to the model in a very specific way to get the best results.

1:18:04

Speaker 1: And the only thing that I have to be in charge of is a definition of done kind of thing where I say say like this is this is good this is what you should look look for. Yeah Emma? So NM up to your computer and this. Yeah, good question. How flaky the tests are. Very flaky. Very flaky because I really try hard to use local models and small models If you work with flagship models and your instructions are like relying heavily on LM as a judge, model graded, you can have really reliable results. There's this temperature parameter here that configures how much

1:18:53

Speaker 1: variation you want between answers. So Those aren't tests that I would run in continuous integration necessarily at this point, but definitely some people do if they have enough time to you know make sure that this is configured in just the right way. So it's flaky, but there's lots of ways to make it more reliable and reproducible. to answer any any further questions or just chat about this. So we're inside the Prompt Foo. Visualization of the results. Again, as I mentioned, every row here is a separate prompt, and every column is a separate model

1:19:40

Speaker 1: responding to that prompt. And so have this kind of recap pass rate. We can see here that the smaller model by um Mistral, so mistral small, 24 billion parameters, only 20% pass rate. This second one, 100% pass rates, same model There's a specific reason why this passes much more than that. Same models, very different answers. This is because one is using the full file, full lms. txt the other just the index. So this is why

1:20:26

Speaker 1: maybe charges GPT fetches the full file, you get much better results out of it. This is why we care about making sure that the small models work well, because they can definitely work just as well as Cloud Opus. If you take the time to make sure the formats are just right. A big difference in fundamental capabilities here between them two is how large of a context window they have. So how much information they can fit into their memory. That's why it's crucial for us to chunk this information in just the right size. At least crucial at this point in time right now. Maybe the models will get better. And so anyway, that's just a recap. Then so I mentioned the rows and the columns.

1:21:12

Speaker 1: Each of the cells here is the results of the specific test scenario. So I can see the response from the AI model and I can can see why it might be failing. So here we have a assertion that's failing about uh some of the things we expect to be there. being missing so this response doesn't mention page types or saving which are fundamental concepts in Wagtail. So as someone who works with this, I might then decide do I change the prompt, do I change the test, or do I change our documentation to have better information This is one of the things I'd love for us to do more of is improve the documentation, find the gaps in

1:21:58

Speaker 1: documentation in a way that's much more reproducible. Because we only have so many contributors in our community who know about tenual writing and have the kind of expertise to conduct user research. But we definitely have a good sense of which questions people ask the docs or ask themselves. And we can definitely write tests like this. So anyway, I hope that was interesting. There's lots to unpack there. There's definitely lots of very interesting things you can do with that format, with AI tooling generally Definitely lots of things that are very concerning to me with how it's shaping our industry, society, our jobs. Don't pretend to have all the answers. I just think it's interesting. to explore. And um yeah I hope I'll be seeing some of you at

1:22:46

Speaker 1: our workshop with Laura tomorrow. It 'll be quite intense. And um beyond that workshop uh I thought I'd give a plug to The Django AI Research Working Group. I have proposed that for us to have an ongoing research group in the community. I have struggled tremendously to get this started because of how intense of a topic this is. But I really think it's great for us to collectively arrive to more of an answer. Yeah, I definitely think you know there's lots of value in having those conversations in person with people around you. Uh keeps us all honest and um yeah important. to have that connection when so much of our online universe is being mediated by

1:23:33

Speaker 1: those tools.

1:23:35

Speaker 2: Thank you. I must say that's a hell of work has been done. And I just want one question, how do you keep top of this, all the research just uh going on? Like it's

1:23:50

Speaker 1: how to keep up. There's no way. I I I I think uh I I think I definitely expect some people will be very happy just staying blissfully unaware of all of this. I'd be very happy to be productive with one or two tools. I think at least for me this is all quite a big source of stress. So definitely worth picking your battles and deciding which ones to explore and not explore. As far as what to recommend, um I think my personal most useful has been to look at Again, the open router rankings, not necessarily because it tells you what's the best, but it tells you what other people might be up to.

1:24:37

Speaker 1: So there's no no bias there. You can just see what's getting traction for what reason. Doesn't say those tools are relevant for you, but it does tell you there's there's lots of usage. Yeah.

1:24:52

Speaker 2: Are you okay to take one or two questions? Like if anyone has If not then I've got one question about this LLM like uh how do you deal with the PDF and you know on a website where HTML can uh content can be converted to markdown how do because some of the website uh some of the like government side they're hello pdf based how How do you push those PDFs to get the best result?

1:25:22

Speaker 1: Yeah, so PDFs, first off, friends don't let friends use PDFs. You should uh you should never make a PDF if you can help it. And if you can't help it, well um definitely publish in multiple formats. And if you still have legacy PDFs to work around of. Definitely that's something that AI models are really good at is um OCR, optical character recognition, um where they can convert the PDFs into more semantic or at least structured information. So this tool called Dockling is quite good in my experience. You know, depending on what the PDF contains you get different results, but that's a good good place to start.

1:26:09

Speaker 1: And I'll convert to Markdown or HTML, no problem.

1:26:13

Speaker 2: Okay, thank you.

1:26:14

Speaker 1: Sure.

1:26:15

Speaker 2: Thank you.

1:26:24

Speaker 5: Uh yeah, one small question. Um you showed that page which had uh uh the how many tokens each page of the Django Docs had that spreadsheet. I was just curious how you found out the number of tokens. Did you use a tokenizer or did you was it in the agent reading test where they tell you And at the end of it, how many tokens were used?

1:26:47

Speaker 1: Yeah, yeah, good question. So remember that uh repository I shared with you earlier? So my ambition with this was to try and explore what the format would look like for Django if we were to adopt lm. txt for the Django framework. And um the the scripts folder in there has my code that goes through this. So without looking at the code, the answer is basically using a tokenizer, as you said. So it's It's an AI model that is meant specifically to give you the count of how many tokens a given document has. And um different AI models, different providers have different tokenizers, so not all token counts are equal, but for what we're doing here, this is

1:27:36

Speaker 1: Clear enough. And so you point the tokenizer at the three different formats, you get the three figures and um spreadsheets, and voila. It's really simple. There's lots of excellent tokenizers out there and wrappers around them that make it more convenient to use them. Um yeah, just to be clear, like this is the English version only, so different languages will have different token counts. because different languages tokenize um shorter or longer.

1:28:03

Speaker 2: Thank you, Tivo. That's what we have all have now. Thank you very much again.

1:28:06

Speaker 1: Thank you all.

Presenters

Conference

DjangoCon Europe

April 15 - 17, 2026

Athens, Greece

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos by Thibaud Colas

More videos from DjangoCon Europe