Wagtail AI and Wagtail Vector Index

There we go. All right. Hi everyone. Welcome. I'm here today to uh spread the world, the word about our new um AI overlords. But on a serious note, I'm Dan, also known as as Zero Lab. I work for Torchbox and you may have seen me in Slack or in GitHub issues. So uh what do I actually want to talk about? Um when we talk Blocktail AI, it's two packages, uh the Bogtail AI and Wogtail Vector Index. They were started by my cold colleague Tom Usher, who um you know wanted to play and had a side project and we've since adapted that uh

adopted that and brought in a number of uh enhancements and iterations and we kind of looked at how might we use them in uh real world uh scenarios and um yeah what what would the be a sensible way to use them. So rather than uh jump in the um LLM, you know, an AI craze, uh we wanted to to to just look at practical solutions. Also I want to note that these are all optional so they're they're not bundled into Wagtail, don't don't worry about that. So Wagtail AI, uh it's a package aimed uh at editors or anyone that has access to the Wagtail admin and it's uh intended to kind of enhance the

the experience and uh the wagtail uh interface. It uses the uh LLM uh package, so that's Simon Willison's uh Sim Simon Willison uh known in the Django world and it allows us to support multiple um multiple LLMs so OpenAI or Claude or even OLAMA It provides configurable uh prompts uh uh from uh the Wagtel admin and uh more recently it supports alt text generation. And that last bit is not new, so Martin put I think it's somewhere here, um had a a a package that did this um for some time uh using

Azure uh cognition and AWS and and uh Google Vision but we wanted to to bring that into uh uh a unified experience So quick demo. I'm I didn't want to uh risk it so I I uh recorded a a screencast. So this is me adding a new prompt um saying well translate to uh to Dutch There you also see two others that come come with it. Here I have a page with some uh some errors, so I I'm asking it to correct those and then uh translating that to uh to Dutch. And it didn't stop. So it kind of does that. That uses OpenAI, but uh you can uh because

because of the LLM uh package you can also use uh a local LLM so you don't send your data anywhere. Um this other uh demo is just generating an an alt uh alt uh text from from uh from an image. Currently it's uh supporting just the open I open AI uh vision uh API. Um but you can uh You can build your own prov provider so you can integrate with other services. The next one is Rocktail Vector Index And that is aimed more at developers to help them build uh

richer sites or richer interactions. So um What are some of the use cases for that? So one of them is natural language search, similarity search, or content recommendations. So first off, um a a note on vector indexes. So for for those of you who don't know This whole thing is really um so vector in uh uh the you may have heard embeddings, basically they're numerical uh representations of the of the of the content. Um or well or of the input. So that might be text, images.

Um so it tries to capture the semantics or the characteristics of the input in in in a numerical form. And then there are specialized databases that store these uh kind of these clusters of numbers and they use different algorithms to basically uh group similar things together so that the you do you you can you can retrieve them more efficiently and then you know the the further the away they are uh the less common they they um though those things are. So the way we went with it, and it's still kind of we're coming to to uh

to an API that would make sense for most people is that we went uh we sort of copied the Wagtail Search Index uh design where you uh have a mix in and then you define some field fields. It supports asynchronous operations, so we it started with the LLM package, but we've integrated the light LLM package to to support that and that's useful if you want to do streaming responses and and kind of chat like interfaces. And it comes with pluggable providers. So NumPy, PG vector, quadrant, and weve8. And these are basically either services or sort of specialized databases that

that know what to do with uh with those numbers. And This is a kind of a quick example how you might implement it. So you uh you have a mix-in, you add it to your uh page model And then you define which fields to be considered for that for for the embedding, so so transformed into these numerical representations. Uh and then the vector index itself um has uh these three public uh interfaces or uh things that it can do and that's uh query, uh search or similarity. So query will take the string, uh

convert that into an embedding, find similar content from your from your uh database or your vector index, then pass that to a an uh an l LLM uh as context and then we'll it will give you a response. Um search uses uh natural language processing to purely look what you have uh look up what you have in uh your index and then uh the similarity one you give it an object and it will try to find you know the closest I think five, but that's configurable. So the closest uh objects uh based on uh magic.

And so um like this is a uh like a practical application. So we built a sort of chat with uh with the internet. Um so that's our Internet torch box. So you just ask questions, it will find the relevant documents and then it uses uh an LLM to just compile the the data. Um so a few considerations when using well any LLM really. So cost if you're using some some service like OpenAI or um Cloud, um or you know, Google's G Gemini and so on. There's some cost associated with with using the API and you know it might be small, but if you

especially if you open that to the public, it will it will ramp up. And if you do, so then you probably want to consider some kind of proxying service so it doesn't cost you an arm and a leg. Prompt injection, it's it's always, you know, that's the big thing, especially with the LLMs and people trying to find creative ways uh to uh you know for the to to let the lm kind of hallucinate or or uh kind of spew uh or or reveal information it's not supposed to to to do so you kind of want to be careful And then if you want to do streaming interfaces, you definitely need on a synchronous server. You want connection pooling and that's coming to Django 5.

1. So basically you don't want to be making lots of requests and and and kind of uh saturating your your database. Of course you could use other you know if you're not using it in in a Django uh and Wagtail context that may not be such such a problem So quickly about the roadmap. So our plan is to further enhance Wagtail AI, so the the user-facing or editorial-facing experience, maybe look at multiple responses. uh a more seamless editorial um process where uh perhaps it looks at the whole page as uh at the whole page rather than uh uh you know one single field And I put there insert your use case here

because we'd like to to hear you know real uh user needs and try to see how we can do that holistically And for uh Wactal vector index, it's it's usable right now, so we have we we're we're we're we're using it in in well you'd say production, but it's uh internal. So we're moving towards a stable release, so a stable um uh API. Um we want to add more out-of-the-box providers so you can use it with you name it, um and then kind of look at the j uh generally like the performance and maybe considerations for things like Pg vector. Yeah, and those are the uh

The URLs um have a play, let us know what you think.