Data internationalization in Django by Raphel Michel

Yeah, I think that's a good thing.

Speaker 1: Hello, good afternoon from my side as well. My name is Raphael. I'm a software developer from Heidelberg in Germany. and I've been the co-chair of DjangoCon Europe this year and um as this conference is coming to to a close shortly I will want to take the opportunity to thank to thank the organizers for putting up this conference. Because once you've done something like this, you get to really appreciate not having to do it and just sitting here and enjoy the wonderful conference that they created for us. Thank you so much for bringing me here to see this. Now let's get to the the topic I want to be talking about, which is data internationalization. in Django. And before we do that, let's recap

Speaker 1: shortly what internationalization in Django without the data could mean. And We've had a talk I think yesterday or the day before on the subject before Django has a set of features that allows you to translate your application into other languages. This means that you can use um certain functions to mark strings in your code or in your templates to say, okay, this is a string string that is different when my application is not run in English or not used in English but in another language and then you can use the getText toolkit which is like the standard translation toolkit in the whole Unix world to extract all of those strings, create a file with all the translations, send it out to

Speaker 1: translators, collect the translations again, compile it and use it. And that's great. But sometimes it's not enough. One of the open source projects I'm maintaining is Pre-Tix, which is an open source ticket shop for events just like this one. And it allows you to create a shop that speaks to a multilingual audience. So if you have attendees coming from different countries or speaking different languages, you might want to present your shop in multiple languages at the same time and that means it's not sufficient if only the application is translated. You need to translate data as well and by data I mean things that are entered by the administrator of the shop, for example the names of the products

Speaker 1: or the description on how to get to the event and so on. That's all something that needs to be entered in multiple languages and then in the output must be shown in the language of the respective user. So we need forms or some kind of input method that allow us to store data in multiple languages. So Uh it's not really suitable to use get text for that because we would need to like every time someone changed something, we would need to generate such a files and the translators and so on. But that 's just doesn't work. And so we cannot really use the tools provided by Django. Surely there is a third-party library that we just need to install and that will solve this problem for us.

Speaker 1: I've got good news and bad news for you. The good news is there is such a library. The bad news is I counted 23 of them until I stopped So who in this room has ever used one of these libraries? Okay, that's not that much. Is there someone here who has written one of these libraries? Okay, that was more people at Django Con Europe last year. Um so disclaimer, I'm the author of Django I18 M Field. Um I put those seven at the top who appear to be actively maintained. By that I mean they are they at least have a development branch that is compatible to Django 2. 1. and they had commits like in the last uh

Speaker 1: six to twelve months. So I will be focusing on giving you A short brief overview over those top seven libraries because the problem is there is not a single best library for this use case. because they're fundamentally different in their approach and fundamentally differently s uh um appropriate for the different use cases. If I get something wrong about one of these libraries, please feel free to correct me later on Slack. I haven't used most of them actually in an actual product, but I've played around with all of them. So to compare these, we want to look at different categories. We want to look at how the data is stored in the database.

Speaker 1: We want to know how their Python API looks and how easy it is to work with them. What other features they might provide, for example, integration with Django Admin , integration with forms, and so on And we are interested in if they have a significant performance impact and um how large that is. So To have an example to work with, let's use a model where we can store a list of movies. And for every movie, we want to store the title of the movie and the year the movie was released. Obviously, the year is something that is not really local dependent, although m it might be, but the title is certainly something that is different everywhere in the world, even though it's the same movie.

Speaker 1: So let's look at how different libraries try to s represent that in a database. And the first approach that we see for see for example in Django Well and Django Palais is to have a separate table that contains the translated strings. So in our main table movies We have just the the untranslated attributes with the ID of the movie and the year, and then we have a second table where every row references An object in the main table and then um says, Okay, for English this is the name of the movie and for Italian this is the name of the movie. So this is in terms of relational databases, this is a very clean approach. It kind of fits the

Speaker 1: normalize until it hurts that we learned yesterday. And if we are interested to like build our shop front end and we want to have a l or our movie list and we want to have l lists of movies and we wanna have the Italian movie f uh Italian title for every movie then this is also very efficient because modern databases are very good at performing joints. However, for example in the back end where we want to have a list of the movies where we want to show every language per movie, this gets really expensive. Because we need to do either a lot of queries or we need to work with the query data a lot. A separate approach seen in Django model translation or Django translated fields is to just have separate columns per language. This way you don't need any joints, and it's very cheap to get all languages at the same time.

Speaker 1: However, every time you add a new language, you need to do a database migration, which can be very annoying. And the third style that we see used in Django I18N field, Django Neche, and Django Model Trends, is to use a JSON-like field. This is less clean in terms of database normalization, but we don't do it need to do any joins. We don't need to do any changes to our schema when we add languages. And um and it's all contained in one field as we had it before. If we're on Postgres and if we use JSON uh Postgres native JSON data type We s can still retain the functionality of filtering by the or searching the name in a specific language or or sorting by a name.

Speaker 1: If we're not on Postgres , We kind of lose the functionality to to index or query that data. That might be a problem or might be totally fine for your use case. Um Django model trends is a bit different than the other two. It um it uses not one JSON column per Per translated column, but only one JSON column for the whole table, no matter how many fields you translate. But apart from that, um those are similar pretty similar niche and model trends only work in Postgres, I 18 N field drops the the indexing and filtering possibilities in their uh but works on all databases. I've I'll be working at the s uh

Speaker 1: sprints on something that uses the Postgres data type when you are on Postgres and gracefully falls back to your text field on all other databases. Next, we want to look at how you define your models. And they're again A couple of different styles. For example, in Django Well, Django Palais, and Django Nechi , you have a custom base class that you inherit your models from. from and they will change your query manager and change a lot of things and how your model work to like as automatically as possible build those joints for you or translate your queries for you Um sometimes you need to wrap your fields to in

Speaker 1: in some wrapper object to to tell the library which ones you translate. Sometimes you have an additional meta option, but in the end it's the same thing. The other style is that you have a custom field type and do not change the way the model in itself works at all. For example, in uh Django I eighteen N field or Django translated fields You just have a custom type that is a translated character field or a translated text field. Whereas in Django model trends, you like have per model you have one Field that is called I-18N or whatever you want to call it and that stores the translation for all other fields.

Speaker 1: The third style is to decouple it from the model definition process um completely and have like a separate registry where you register those options. This is like The most I would call it the most unclean style um of doing it because you it's kinda not obvious where your code lives. On the other hand, this allows you to tr To enable translations for models that are not in the code that you control, which might be something you need. Okay, here. with such registration patterns. And the third thing I want to look at in detail

Speaker 1: is how to interact with your model objects. In some of the libraries, you can only interact with one language at a time. This is mostly because of this um joints that they're performing. If you pull the object from the database, it will pull the information for one language. Like you need to specify that within your query, and then the title attribute of the object will be populated. with the Italian title. And then if you want to access the English title, you need to change the language and will it will perform a new query or depending on the implementation or in case of Nietzsche it will not. It will change like the the the state and title will now

Speaker 1: uh contain an English uh the English title. The other option is to be able to access all properties at once. This comes naturally to the um to the libraries where you have separate columns per language because you Um you have your main um attribute title that usually evaluates lazily to the value of the currently active locale. And you can directly access every other language by just using title underscore and the language code because that's just either because that's just the field that the library creates or because it virtually creates it for you. In Django

Speaker 1: I18N field, it's a bit different. The the title attribute will always contain a special data type, a lazy internationalized string, which is some uh some in some ways like what you get in return from you get text uh underscore lazy it will whenever you you cast it to a string it will coerce to the currently active locale but It is a special data object that contains the information on all language. So you can pass that around as as one object. So, to recap and to add the other features We have a couple of different database

Speaker 1: layouts that um that are in use. We have the the version where we have Everything in its own table, like we have multiple tables to store our model. We have the version where we have a very wide table with multiple columns. um for every language that we have and we have the like embedded version within one column. We have database support for most of the libraries for all databases that Django supports, but in the case of Django Nichi and Django Model Trends, they only run on PostgreSQL. We can we have different levels of support for for filtering the the objects, for example, in those that use

Speaker 1: normalized database layout, it's really easy. Although it might be computationally expensive. Whereas in those that use the um the PostgreSQL JSON field, it should be easy. They don't provide any utility for you to make it even easier, like querying in the currently default language, you need to do that on your own. But it it's conceptually possible, whereas in I18 field it's It's currently not really possible. Like searching somehow works, but ordering or indexing uh is not possible. We have the separate styles of defining the model. either by defining a base class or by registration or by a custom field type. And we have the separate styles of object operation where we can either um

Speaker 1: access one language at a time or all of the languages. at once. I didn't talk in detail about form support. Some of them provide um for most of them form support comes naturally by just like when they generate the uh your your model fields that have um a separate column per language Then um it will just when you use a model form, it was will just automatically get generate that number of fields. So in some of them you will get in a model form you will get a field that just allows you to edit the currently active language, which I don't think is useful in very many use cases.

Speaker 1: In some, you will like get different form fields for each uh one form field per language. Um Nate doesn't have form support at all, it just gives you a take Text widget where you can edit the JSON blob. And in I18N field, you will get a special form field type with a special widget that it works like the compound daytime widget. It just has compound input field fields within one widget to ask for the different languages. Some of them have very elaborate support for the Django admin and do that very nicely. Uh others just present you with the JSON blob or just different fields below. I did a small benchmark

Speaker 1: to have a look at their performance. The benchmark is of course not representative for real life application. It just Um stores a lot of objects into the database, pulls them out again, and tries to access the attributes in various languages. That obviously is rather slow on those that need to do joints and need to to refetch, although at least Django Palais um can make use of caching to reduce this. While the others Quite unsurprisingly, the PostgreSQL uh JSON field back um implementations are very, very fast, and the others are also reasonably fast. I've created a demo app that uses all of the seven libraries

Speaker 1: and also contains the bank benchmark codes uh code in case you're interested in that. And with that, I'm a bit faster than I expected at the end of my talk, and I would be happy if you have any questions on that subject.

Speaker 2: That was a great talk, thank you. Uh just one question. Where did you get the inspiration to do the emoji feature comparison?

Speaker 1: I don't remember. I've seen a lot of emojis at the Django conferences I've attended. So um it's maybe Katie's fault, but

Speaker 3: Hi, great. Thanks for the presentation. Uh right to left support. How easy is that to implement and how do the libraries You mean

Speaker 1: in context of storing the data? I haven't tried. I don't see any problems ahead there because in the end those libraries are just about how to store these Unicode strings that uses input and we out that we output at a later stage. So I think on this, on the on the model level, it's not that important. It might get more interesting if you have like if you're rendering a form and you need to render some of the widgets. with left to right CSS and others with right to left CSS, that might get interesting. But on the database level, I don't think it should be a problem.

Speaker 4: Let's give a big round to Rafael.