Wagtail Live: bringing a live blogging tool to the Django ecosystem

Hello everybody and welcome to this talk where I present the Wactel Life project. My name is first let me introduce myself. My name is Amoduti Janija. I come from Senegal and I currently I have a bachelor in mathematics and I've started learning web development uh near a year ago. Yes, I started learning with Django and I also learned Woktail because I wanted to build a blog and I found that managing the content It's especially writing code blocks and in the Django admin is not very pleasant, so that's why I started learning Woktail. Wacttail which is a CMS and the goal of a CMS is to help us actually man equip man manage the content in our website. So yes, this is Woktail. It is a powerful CMS for more world websites.

It is very flexible. Yes, and I also wanted to participate to the Google Summer of Code project. The Google Summer of Code project is a project organized by Google every summer. to put in relationship students and mentors in uh open source organizations. The goal being that y organi uh open source organization can have work done and students can also hone these skills and get to know how the open source work world the open source world works. So yes, so I up I have been chosen to do the WorkTech Live project. So yes, the goal in this talk I will present you. So this is how the goal will be organized. First I will explain what a live block is and show some examples and then I will uh show how

how WorkTell Live is designed internally and lastly I will show how the process of actually building a live block with Workchild Live how does it happen So yes. What is a live block? A live block is a block which provides real-time update on an ongoing event. So for example, actually the coronavirus is uh an ongoing event. And here we have a live page from the Guardian. Which provide live updates of the situation all around the world. So here we can see that yes the page has been updated four minutes ago with uh these information. Yes. So yeah, the most live blogs have the similar format, like they have a title and you they this is an example from CNN for the latest election in the US. And here we can see they b both uh they f they all follow somehow the same s

uh format. They may include messy images or maybe sometimes this was for the last final, last Euro final between Italy and England Yeah, because sometimes you get they can include treat embods as and yes. This is how most of the live blocks uh are structured and the difference between live blocks the main difference is that you in with in a live block you don't have to refresh the page to get new content. As soon as the publisher has published his content in whatever tool he uses you will be able to see. it right away without needing to refresh your page. So that being said, I will explain a little bit how Wactail Lab is designed. So yes, this is a little image showing how it happened. So with Wacy Live

the main the the choice we made is that the original idea of the project is from my mentor, Con van der Kemp. So The basically the idea was that you can use Slack and you input you you post your message in Slack and they appear in the live block here. So that's the idea. That's the idea we kept. But you can also use the Wagtail admin. Maybe we imagine that if you are on the field reporting, you know, uh report uh using your mobile phone will be more more easier and maybe the walk tail admin is not very mobile optimized So the main way to add content to a little block powered by Woktail Live is to use a messaging app like Slack or Telegram or so How does it happen? So for example the reporter it's like the image here we have the reporter here who is posting a message about some topic in Slack or in Telegram

Or whatever. And here we have the receivers. What are receivers? The receivers they have two jobs. So first they first job is to retrieve updates from the messaging app or to receive them. For example, for Webhook Technique, they just receive the updates And the second part is to format the updates, you know, uh find which page does this update belong to you and parse the text to see if there are links to convert to YouTube videos or else. That is the job of the receivers. And when their job is done, the page is saved on our database. So our database is modeled by uh Worktail uh a live page So our live page is a worktail page and our worktail page is just a plain Django model. So that's it. Here we save our live post, our database with the information with like the channel ID, the message IDs and all And on the other part, how do the client

receive the how do the when do the client see the updates published by the reporter here? That's the job of the publishers here. So the publishers they use maybe the WebSocket technique or the polling technique and they fetch updates from our database to see what are the new messages if there are an edited message or deleted messages and display them to the front end side. So this is a little bit how WalkTechLive works internally. So now what we will good what we're gonna do is that we'll So this is our GitHub. So yes, this is uh the page of the project. We have also we are also in PyPy. Uh yes, and yes we have the for a good documentation and we are working on it too and yes the project it will test it but just let me go into the documentation and jump into the tutorial

So yes, in this the goal of this tutorial is to build a live block with Wacta Live. So the title of the talk is Building a Live Block within 30 minutes. So yes. I let's jump in. So first we need to install and run Wactain Live. You know, we need to create a virtual environment Yes, install Woktail, install Woktail Live and create a seat. This is similar to the Django Start Projects command. And yes, we add the air. uh the app we just created and worktail life to install app this is uh Django stuff basically this is where the worktail life comes in the when we have to design our first model our first live page model so how do we do it how do we do it First we yes, this is our live log page. It will inherit from page from the Wagtail

models and also live page mixing which is provided by Wagtail Live. Live page mixing provides command functionalities for to bring l live uh to bring live capability to Django models to what TL pages. So yes once that is done so we will check here that we have the that in our model. So here I have my this uh in this talk I will use the Wctel Live demo repository for the to show. So here I is here is our live block model. Here is our models. So we have here our live block page model We are inheriting from life page mixing and from page 2 and here we have the content panels which is properly set. We have the page panels and the live page machine. We are also adding a hero image and hero text. This is just for uh styling thing. Styling. Yes.

So the next step is to create the template which will represent which will render the which corresponds to the model we just created. And in this uh template we will add this line. We will add the uh Woktail live post uh template which will include all the live posts and yes So here we will check that i in our live block app we have the template folder and in our live block we have the live block HTML. So here we are including the yes the live post. html template. So after that that's all we need out now models and We also need to specify the path to the model we are using with with the work till live page model setting.

And yeah we create the database, create an admin user, and now the the next part is to configure a receiver and configure the public. So first we configure an input source and its corresponding receiver. For example, in this image, Slack and Telegram, these are the import source. And each input source has is associated to a receiver that knows how to handle requests coming from that messaging app. So for currently WokTel Live proposed Slack and Telegram as import source, but also the web app that I will talk about later. So here if we i we'll set up Slack first to see. So how do we set up the Slack Events API receiver? The Slack Events API the Slack Events API is the API where we can c

submit a webhook URL and we receive updates. So Wactail Live Provence provides a Slack event API receiver. So first we need to create a Slack app And and uh Slack app is like a boat. It can communicate with your uh with the channels in your workspace. So when you when you click in the link you will be asked to create it from scratch or from an app manifest, but you will just want to create it from scratch. But for this tutorial we will just use the live app which is already set up. So the next step is to retrieve the tokens. So we woke live needs the following information It needs the Slack signing secret because we use it to verify the request because we will provide this endpoint. This will be our webhook URL.

and we will use the Slack signing secret to verify that requests that we receive in this URL indeed come from Slack. And we also want the Slack bot token which we will need to download images from the Slack API. So where do we find those information? We can find the signing secret in the basic information section. It is here just here signing secret and we can find the bot token in permission section It is just this uh starting with X O XB. Yeah, so we grab the token and copy it. So the goal is to save them on our environment like this and then to add the a line like this in our settings. Where we specify the Slack signing secret and the Slack board token. So I have already done it and I've already exported the variable in my environments.

So here are the settings Okay, so the next step is to hook our request URL. So it was the URL that Slack will use to send us mm requ renew updates updates. So currently since we are just using manage. pyrun servers, we will have a local development server which won't be publicly available. So we will use Ngrock which will forward the request to our local server. So to you can see how to set up a local server with Ngrock, basically you just need to install Ngrock and run it. And yes, so the goal is that we need to run NGROC on port uh 8000. 80,000, sorry. So let me start. Andrew Rock here. Yes. So what happens is that

we we will have this URL which is publicly accessible and all the URLs that will be sent to this all requests sent to this URL will be forwarded to our local development server. So the next step is to con is to add the worktail live URLs so that we can Because Wactel Live handles the requests, so we do need to add the Wactel Live URLs, we need to import them and add them add them. So I will check here in the URLs that yes, I have from Wachtel Live import URLs and I have the Vactail Live puff URL button. So the next thing is to specify the Vactail Live receiver. The Vactail Live receiver setting is the setting which defines which receiver do we want to use with our live block And the value that we give to the this setting is the path

to the receiver we want to use. Here we are we want to use the slack events API receiver. So we big the path to the Slack receiver. So I will copy this and I will pass it in my settings. Here Yes. So now we need to make sure that NGREC is running on port uh eighteen thousand thousand and start the develop development server. So yes, Andrew is running here. I will yes, so now most of the messaging app what they will require uh a securized URL, so the one starting with HTTP link here So now what we'll do is that we will start a development server.

Our development server here. Ah, I'm not sure if I have saved yes, I haven't saved this. Yes. That would just to make sure. Right. So now what will happen is that we now what we will want to do is that we will want to hook our request URL in Slack. So to do this we have we'll have to go to the Slack uh user interface and we will go to the event subscriptions uh section so now we see here enable events it's on you can your app can subscribe to be notified of events in Slack at a URL you choose. So now we will take the URL that we from that NGRAC provides

and we will append work live slash slack slash events in the URL. And what will happen is that Slack will send a request to the URL to verify and it sent uh it sends a challenge token to verify that indeed we are we are requesting this change. So yes, the request the URL has been verified. We can check here that yes, we have uh received a request here and we can verify also in NGROG that we have re received a request from uh Slack so we can just save the changes now. This every time we have a new update in our channel we we will want we will receive updates here. Here in our tutorial the next step is to configure the channel and subscribe the board. So now the boat will receive the only events it will have subscribed to. So to subscribe to bot events we will have to go here, the same section as

enabling events and we will go here and add here the event we want to add. So for example here we are adding the message that channel events which is uh which happens when a message is Posted in a channel, which means that when a posted is when a message is posted in a channel where our app is installed, we will receive an update in this URL that forwards to our local development server in this case Actually if you uh also if you want to use maybe a private channel or other type of channel you will have to check the list provided in the documentation to see which scopes you will need to grant your board So now the next step is now that we will make the proper setup adding the channel where we will post content that will appear on our live blog. So here I will come here, I will add a new channel.

Call it test channel Uh we do not add nobody. So now we need to add the here in the documentation It's uh it's in bold because it's very important that you add your app to the channel because if you don't add your app to the channel it your app don't have won't have a way to know that there is something happening in the channel So here we will add the app in our channel. We will go to the integrations. So I will do it again. So yes, we go to the test channels integrations and add an app. The app we will add is the app we'll have configured for. In this case it's the Lav app So yes, it's good.

And that's it. But finally we will need to retrieve our channel ID so we can do the mapping so the in Slack we can the channel ID is this last thing this will be our channel ID so I can just yeah copy it yeah I will come back to this later. So now that we have our import source and our receiver set, we can now go to back to the tutorial and the next step is to configure publisher. To recall the public the goal of the publisher is to fetch updates or to receive updates from the database and display to the client side. So Woktail Life propose different types of publishers. You can use publishers using the polling technique and publishers using the WebSocket technique. For the long polling, yes, we have long interval polling and long polling technique. And for the WebSocket, you can either use Django channels, and Django channels is the classical way to add

WebSocket support to a Django project Or also you can use a publisher based on uh an external service like PySocket or Ably, for example, they provide you WebSocket service, they handle publishing to the WebSocket and to the WebSocket client too. Or you can deploy a separate server next to your Django server, next to your project server, which will handle WebSocket connection. In this case, yes, for example, you can have a publisher, yes, the server will be will have a We'll run with maybe with Starlet, for example. So most of the publishers the configuration is similar. Maybe we 'll have some steps that will be specific to the publisher, but we'll always have these two steps. We specify the worktail life, the path to the work tell life publisher, and we add the publisher in our template.

This is the how all publishers work. So we'll go back to our tutorial. Now we want to set up a long polling publisher. So how to how do we do it? So to use the long polling technique for the publishing part we need to add this in our settings. So this is the path to the long polling publisher. I will add it in my settings. I'll save it. Yeah, you can you can upsuit set uh polling timeout. Uh the default value is thirty seconds So the next step is to add the publisher template, which will be the way to hook the JavaScript, the polling code, into the page. Um just copy it and I will copy it in the live block page HML template

here. Yes. So now we have our long polling publisher set. The next the next part is now we can start live logging. We will go to the I will close this And this two. We'll go to the admin interface of the worktail admin interface and we will create a new live page that we will f that we'll fill. So for example here Uh delete this. Yes, we'll add a child page, a live log page. uh some title f for the talk

so I'll choose an image for the hero and choose this Mm-hmm. Yeah, Django Coin is good. So I will go here and grab the channel ID. That I will put here in the channel ID field from if my page. Yes. And I will take this opportunity to talk maybe a little bit about how the data is saved uh internally so worktail has the concept of stream field stream field is a very flexible way to manage content like if you can mix many type of content so for example here a live post

is represented by maybe its message ID when is work the time and the date and time when it was created, it was sent , is modified and if the message is shown or not. So here we have our stream field. The content of the live post is like a stream field What is the stream field is like we can have here multiple type of contents like we can put images, we can put ember and this is just what I have defined, but if you want to define more types of content you can add more type of content and you can reorder them, you can yes, you can delete, you can copy, you can do whatever. This is very flexible and yes. This is what all life posts are saved in a in a database like like this So now what I'm going to do is that I'm going to publish my page so that it's live and I'm gonna I'm going to view it.

So I will open it in another window. Oh, and I will drop it here. In here. So now what we expect to happen is that if we post a message here in this Slack channel, this Slack channel maps to this live page. So if you post a message here, it will appear here So this is not the following is not specific to WalkTearLive, but it's specific to the WalkTearLive demo repository where the first line is the title. The first line of the message is the title and the the rest is the content. So yes. Now we have the message here. So we can also maybe post With an image

I will Oh I'll add this image and I'll send it. Yes, here is it. I can also maybe add a post with a YouTube video. For example, here uh this is the what's new in Woktail episode four, where I also did a presentation of the Woktail live project, but it also shows what's new in the Wagtail uh ecosystem. So I will if I post a YouTube link or Twitter link or something that can be rendered as a name but in its online, it uh Votel will know how to render it and yes, it will appear like this, like you know, a video. But for example, if I uh if I put the link in line for example like this if like if I put some text

and then the link like this the link will be rendered only like a regular link like you can click on it and you can uh yes you can watch it like this. It will redirect you to the link. So we can also edit messages, delete messages. So for example I will delete this one the post with the YouTube video. I will delete this Yes, it's gone. So what I'm going to do is now that I'm going to edit this one so that the link is in its own line so it's rendered well. Nicely. Yeah, it's here. The post has been edited. So yes, that's it. So here yeah, we have our live blog running. It's here we post in Slack and it appears uh immediately.

Not immediately, but very The quickest possible, I can say so, in in the in the client browser. So yes, this was a setup where we use maybe Telegram and l and and long polling. So now I will show another setup where we'll use this was Slack and long polling. I will show another setup where we'll use Slack we will use Telegram for the import source and starlet for the publisher. So first I will go back to the tutorial page and here what I'm going to do is that I'm going to set up telegram webhook receiver. So how do we set up the telegram webhook receiver? So first we need to create a telegram boat. And after we create a boat, we will have a token. A token with the token will look something like this. And we will need to save this token in our environment and specify it in our environment

like this So I will check that in my settings I have the telegram board token. So as I have already saved in my environment. So yes, I have that line here. And now the next thing is that we have to hook our URL to tele to telegram. Because in in Woktail, in in Slack you you you hook your webhook URL in the interface, so you do it uh manually But here we have to do it programmatically programmatically. They uh telegram doesn't propose a UE to send uh to propose a poke URL. So this is similar to the Slack uh API receiver. We need to hook a public URL so we will use the URL that ngrock has provided us this one and we will paste it here as the telegram

hook URL parameter And we'll save it. The next thing is to add the Octel lab URLs if it's not done yet and we have already done it and now we need to specify which receiver we are using. So now the path to our the path to the receiver we want to use. So we want to use the telegram webhook receiver. So we I will copy this line That I will patch here, cocktail live uh receiver. Rather. Yes, I'm using this receiver Now we need to m run the f server. If everything goes well, the server will start without errors because what happens is that now we that we have provided a URL and that we are using the telegram with

book receiver Uh when we run when we do the run server command, uh Dwctel Live will submit the URL to Telegram so the Telegram send can send us requests. So here I will check that we are not having any errors. So that means that Telegram has received our web URL and it's fine. So now we need to create a channel where we will post messages. So or may maybe we can post uh messages directly to the boat or we can invite the boat in another channel. And then we will need to retrieve the chat ID and that we will use. So for example here, I have a window open with Telegram. It's here. So this is my boat, the Wactor Live board. If you go to the board further, you will see your token. And here I will g get the chat ID. This is provided by

it is a command provided by Octel Live. It gives you the ID of the chat you are currently working on. So we will use this to fill the channel ID field in the Octel admin So now that we have configured the Telegram webbook receiver, what you want is to set up the Starlet Publisher on the other side. So how to set up the Starlet publisher? So the in this architecture I recall I remind you we deploy a separate server next to our Django server that will handle WebSocket connections. So we need a way to communicate between our uh our web socket server maybe and our Django server. So to do this we have to set up an event boost. So in currently we are using the Redis PopSub functionality to render to to implement the current build the event builds. So we need to install Redis and II Redis. And yes you need to make sure that also uh

Redis is running. So I have already Redis running somewhere. So I can Check Redis Clay ping. Yes, Redis is running, so it's good. And yes, and to install I Redis and yes, this is all we need to set up an event base. The next step is to install Starlet, which I have already done. And yes, this is similar to the previous publishers. We have to specify the path to the publisher we want to use, Walter Live Publisher. So I will copy this and I'm going to paste it here. I will pass here. Yep. Yes. And we can uh since we deploy a separate server we can you can specify the server host and the server port. The default is localhost 8765.

So in the development we can Wachtel Live provide this command, the command run publisher and you give run publisher and the command that you are using so that the serve the WebSocket server starts starts running. So here what I'm going to do is that I'm copying this and I'm uh yes. Yeah, here we have the our server now is running on here it will accept WebSocket connections The last step is to add the template which will hook the Starlet JavaScript. I will add this to the live block page, the HTML template here, in place of the long polling template. And I'm saving this. So now what we have though the now what we we can here create a child a new page Like maybe yes, telegram

yeah it choose an image like this one hero text telegram Live block and startlet. So now the channel ID is what we received from the get channel ID command. It's this value. I will copy this and pass it here. So now we can publish our page and view it in live. So I will open it in another. Yes, and here I will open the telegram channel here. So I will just fresh here. Now here we can you we can ensure that there is a new web

circuit connection that is done on the separate server. So what I'm going to do is similar to the to the Slack. This the title is here and we have the content. Yes, it comes here. So we can also add an image. So an image uh i uh really this is a caption, like yeah a caption that's how it's called. So I will send this. Yes, so here is it. I can also post with a YouTube video like it's like I will copy this.

Yes, we still have the same functionality. We can I can edit this for example. I can edit this, post with YouTube video edit it and save Yes, it's it appears here. The w telegram doesn't send an update when you delete a message. So for example when I delete this message here, I won it won't appear in my live page because Telegram doesn't send us an update when we delete a page So there is no way to to to see. So yes, this is we have Telegram uh as an import source and a starlet publisher running uh uh as a publisher. And also I would like to maybe show that you can use there is a setting, this setting here. If I go into documentation. If I go in the reference and the settings, there is this setting, Octel Live, sync with

admin. So if the default is true, you can set it to false. If it's true, uh why you if it's true, it will sync changes. So for example, when there is a change somewhere Uh when you change something in the admin interface, it will appear in the live block. So I will do a demo here. So I will edit the live block page, for example. I will take it here. So we still have the page here. So what I'm going to do for example let's say that that I don't show this page and I just save. So yes, it's gone. So yeah, we can also if the worktail live syncret admin setting is set to true, you can use the admin interface to to manage the content. It will be like an import source. If you post a message it will be li it will appear in live. But if the setting is false, which you may want to set false, if you s know that you won't you don't want to use the admin

because it will have to do some computations some computations to to track. If you don't w if you know that you don't want to use the admin interface, you can set that setting to files and the default it's role. So yes this was for the admin interface and all and also there is an uh basic uh at the start we wanted to provide uh We wanted to provide an alternative uh a debugging tool so that you don't have to set a Slack or Telegram to use WaterLive. So we w uh basically we named it WaterLive Debug, but finally we settled on proposing an alternative so that you called the web app and you can use the web app to post uh like you would use in Slack and Telegram. It is still a work in progress we still need uh to auto need to add authentication and the design styling and all because yes But yes, it is also an alternative.

I will here show some an example. So for example, what I'm going to do is that I'm going to the web update channels. And I'm going to create a new channel. So uh yes, I will delete this channel and I will recall it that way. So I'm creating a new channel here Yes, and I can go to the channel, I can post messages here. So here what I am going to do is that I'm going to create a new page, a new live blog page, that will map to the channel we just created in the web app Web app page. Yes, we chose the same hero image as earlier. Testing the web app.

And here in the channel ID I put the channel ID here, the web channel. Yes, here. I can publish it. Publish Alright, so what I'm going to do is I'm going to close this and open the papes Yes, here. So now we have the web app here. It's still running with the Starlet Publisher. If you want to use the web app you don't have to specify what receiver. It is it will it is the web app works Like that. You don't need to specify the receiver. So what I'm going to do is the web app is an API. So it has this interface, but it has also an API. So I have I have a script running here. I have a script here So I will provide the NGROC URL, which is here.

Copy link. So I will provide here the base URL and what I'm going to do is that I'm going to post to the web app thatapi that message route. So what it will do it will save the message automatically in the web app channel and then in the page. So I will save this And I will run the script here. So what I'm gonna I'm just post this here and the page where is it? Yes, this is the web app page. So now I will run the script. Yes, here. It has been posted and it has been edited and then it has been deleted.

I've not putting I have removed the timed that slip, but if I put the timed slip, you would see that But yes, we still need uh authentication because uh everybody can do this, so it is not very safe and we also need to maybe add more styling here. Yes, uh you can po hi this is uh we can also post with an images for example Uh yes. And I will submit and it appears here. I can for example edit And in this I just delete the images, I can save my changes, yeah. So that's it. The web app can also be used as uh as a receiver, but it still needs like I said

styling and authentication. Yes Yes, we have these issues here up and here. For example, for the web app. Yeah, we have some issues related to the web app, to the documentation. Yes, we have some pull requests. Like I said earlier, the project is 100% coverage. So yes, I think I have said everything I wanted to say. Yes, and I hope that you like the presentation too. And I hope that we'll find many contributors and many users too.