Django during the Antarctic Circumnavigation Expedition

This video features Carles Pina Estany at Django London 2020 in Online.

Django during the Antarctic Circumnavigation Expedition
0:49:11
Published April 22, 2021
189 views

At short notice we had to "build a database" for a scientific expedition (Antarctic Circumnavigation Expedition in 2016). I considered different options, I was introduced to Django and chose to use it (initially a bit reluctantly). We did some express learning with a hard deadline: the expedition was starting.

I'll explain why Django was a perfect fit for this case, how we (the data manager and I) developed a system used by the scientists in the expedition, what helped us, how the system grew during the 4 months of the expedition and what the strong points of Django for our case were.

Presented June 2020 at the London Django Meetup: https://www.meetup.com/djangolondon/events/271374566/

Carles: https://twitter.com/carles_pinux

Summary

Carles Pina Estany explains how Django helped build a data-management system for a four-month Antarctic research expedition, where roughly 70 scientists rotated through the ship and needed to record observations with little training. With only weeks to prepare and limited internet access at sea, he chose Django for its models, authentication, permissions, documentation, and especially the admin, which he adapted for data entry, search, reports, and maps. The system tracked metadata and about 28,000 samples, visualized ship routes and instrument data, and supported small tools scientists requested along the way; UTC timestamps and offline documentation and source code were important in practice. Looking back, he would add more tests, organize the project into smaller apps, and build scientist-facing forms outside the admin, but says Django made it possible to deliver much more than he initially expected.

Key takeaways

  • Rotating teams of scientists needed a simple data-entry system they could learn quickly, so the expedition’s Django application was built around adapted admin forms and clear workflows.
  • The system supported metadata and sample records, reports, maps, and visualizations, helping scientists spot data-entry errors such as reversed latitude and longitude.
  • With no reliable web access at sea, downloaded Django documentation, Python’s source code, and other code already on the laptop became key learning and troubleshooting resources.
  • UTC timestamps kept the data consistent despite frequent ship-time changes and crossing the international date line; a utility translated ship time to UTC for users.
  • In hindsight, Estany would write more unit tests, split the growing application into smaller Django apps, and use custom forms outside the admin for the scientists’ interface.

Summarised automatically from the transcript.

Transcript

6,664 words · auto-generated Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:00

Speaker 1: Okay, uh well then I'll shall start. Um I had some photos here, so uh if you want to see them, uh at the end is a link to the slide so you can see all the photos if you want. I'll start the talk. I have some admin to explain first, not Django Admin yet. During the talk, and this is not a joke, you might hear firecrackers. Um because now I'm in Catalonia and today is San John's, well tomorrow St. John 's Day, today's San John 's evening. So the tradition is that we have bonfires, as you can see, and we have firecrackers and fireworks

0:46

Speaker 1: And I know that, well, we're celebrating just I was doing a bit of homework. I knew some of this. The summer's host is that it's two days late for this. and to scare witches away. I've never seen them, but maybe because of the firecrackers. And I know that different countries also celebrate a similar summer solstice event. So might be not new for you, but I thought that if things go off and I'm here talking, I'll feel a bit uncomfortable if I haven't said this before And just to finish this bit, there are more Catalan traditions that some are not noisy, like human towers, as you can see here. Oh that this is the last slide about this, but um eating calcots

1:33

Speaker 1: which is a type of string onion and um here eating this that that's a way to eat them. Uh I'm in London a few years ago, so um look for that. It's very tasty. Anyway, having done the non-jang wadmin , what I'm going to talk about tonight. Four years ago we had uh expedition um and just a few words about the expedition. It started in Africa. I don't know if you can see my mouse pointer And then by boat, uh big boat, we spent a month undersea. We went to Australia. um some scientists changed uh it was about hundred and uh I think

2:18

Speaker 1: seventy seventy scientists more less Many changed, some stayed. We spent another month on the sea to go to Punta Arenas in Chile. Again, some people changed, some stayed. then went back another month small less back to Africa and then the ship went back to Europe and I think 10 scientists uh or eight scientists and data manager me staped and went back to Europe um by ship. So this is the expedition that I'm going to talk a little bit and just to have some context. And I want to explain why Django was selected or why we chose to use Django

3:04

Speaker 1: and what was the good things that we enjoyed of Django And how it was used, which might be different to what I would use now that I've done more Django or what some other people might do as well Um to give a bit of context, that was the big boat. It's a Russian icebreaker. It's about 130 meters, I think, uh length it was. Um and it accommodated a hundred and 80 or 60 people counting scientists and the um again the tripulation of devote the crew Here's a photo at the end of the second leg

3:51

Speaker 1: thing that we're arriving near Chile at this time. The majority of these are the scientists here And an important thing is that the scientists every month were changing. So the system that was developed for the scientists had to be very easy to use. because they would come on board and in two or three days obviously or even before they were collecting different types of data. They were um deploying different equipment and they had to enter this in in the system that they had to use. What I mean is that we didn't have a lot of time to explain and explain them sadly on this expedition. So

4:37

Speaker 1: a bit more of context now about what I was doing at the time. I was a CT software engineer. And I took a sabbatical. Um all of this happened in the winter uh 2016-17 year. And I used to write Python for some scripts at work and what they call evenings and weekends and Python code dojos , site projects. I was not a Python professional developer because I was doing C<unk> QT , but I was quite familiar with Python. And I had been for a very long time uh user of uh Gnor Linux and even before, and this was

5:22

Speaker 1: quite handy for all the expedition. I was um a system administrator, network administrator. I'll not talk about these bits of um the network and the email system on the expedition, but it took more or less half of my time while I was there What I would like to explain is about the data management and IT in a very generic way. So the data management NIT we had um many tasks. Uh it was 22 science projects that were there. Each science project had between two and four people more or less. Some

6:09

Speaker 1: were quite big and had many people working at the same time. Some were more centered on going to an island. and uh collecting some samples from the island, going back to work and then spend five more days to the next island. So it was very different rhythm for different um Teams. The we set up the data management. So we explain well Sorry, this is what the scientists were using to record what they were doing, where and why more less. We had a set uh we set up a data storage and backups. Um lot of data is collected from different equipment and obviously it cannot be lost.

6:55

Speaker 1: It's very hard to go there again and to go there again that day is impossible. So all the effort is to not lose any data. And we work with scientists to back up data. recorded metadata of what was being collected physically or recorded and we have and is still being used to today a database with all of these and um sample records. It was uh 28,000 samples were collected and um this is a way to know where was collected by whom when And on the top of these very data management topics, we were setting up an onboard network, an email system, and a file

7:42

Speaker 1: below them. the internet was very um how to say it could only be used Let's say with a system designed to be used there. Opening Firefox at Chrome and opening a website was not possible. Talk a little bit about this later. We had some connectivity to send files, to send emails in a system designed for this, queuing the emails, retiring, limiting the size of the emails and all of this. And then we solved or we helped scientists connecting equipment on board from a half a snowflake counter, how to get the data out. to

8:27

Speaker 1: I forgot to take the hotspot or the Wi-Fi router that I need for this equipment. Can you do something to get the data now or different types of issues that happened Data management, I had to really talk about this, but to know what's collected, where, and metadata by whom. And now why as I said backups? All the data collected in the Antarctica, it needs to be done, it needs to be made available after a certain time. And also all this data it was useful for the scientist

9:14

Speaker 1: who was collecting because the scientists might be working on a project over this data, but obviously it will be useful or it's been useful for the scientists. So it's very important to make uh all the data as well documented as possible to make it easy to use. And the conversation at before the expedition And be aware that data manager Jen and I were invited or contacted about months before the expedition or six weeks before So in very um kind of last minute or last month obviously, but we had to prepare many things. But Jen knew more or was data manager at the time

10:00

Speaker 1: and had been working on other ships and asked me, can you build the database? Can we help me build it with database? I said, well, yes, of course. And in my mind I had at that moment, yes, I'll have a mask well, I'll do a case table, and I'll start well no creating the tables that we need to collect the data um but then Jen said well we need an interface so the scientists can enter data in this database And coming from my C<unk> QT, I was well, my tools are not the best tools for this. I can build that, but not in a few weeks before we leave and in a way that all the scientists on board can use this easily.

10:46

Speaker 1: So here is when Django appeared. And at that time for different reasons I had used different tools um and I was quite familiar with Python it had built applications with Flask um might as well I know might as well well for these things. I had used a smell alchemy moustache. But I was thinking, well I need to build forums so all these scientists can open a website, authenticate, build um certain for uh enter certain data uh select data do some basic reports search or something so I was like well I don't know what to use um I have almost everything, but I just need something to build forms.

11:38

Speaker 1: So I was wondering how can I do this? I asked a friend who I might might not be online here now. Uh but he said, well, use Django. That's an upgrade. I said, no, no, no, no. Django is too big. I'm living in four weeks. I need to buy all these um storage to uh save all the data. Uh I don't want to learn a new thing now. I just don't have time. I had an impression at the time that Django was big and that I would need a lot of time to be able to do something. Um then Chan that same evening um just created a very small uh Django project with a small admin. I don't know if it was a contact list or to-do

12:23

Speaker 1: list or something one of these um do I not do a model at an admin and and at some authentication. Oh I thought I had one more line here. I loved it. I was like well this is what I need. Yes I'll I'm going to use this because uh seems to do exactly what I wanted and in a very easy way. I I was um salt by the Django admin. I was like, yeah, I'll buy this. Plus it was Python which I knew which was nice. So then I went back uh home and started learning Django. So I still remember my first week of Django when I was there learning the documents, uh the documentation

13:10

Speaker 1: And I found that yes, creating grade tables with all the foreign keys and index by hand, it takes way longer, obviously, than creating um Django models. And for many of these things, long, long time ago, um before my Sidwas Plus Kitty job, I had done um forums, websites. at the time with um PHP, but my main problem is that I hadn't used the framework was 20 years ago And I was very appreciating how Django was making things way easier um integrating, well what they list here authentication permissions admin all of these plus having um templates with the models and

13:56

Speaker 1: security easier or harder to make mistakes, let's say, than not using a framework. So yes, I creating all the documentation and remember asking a few questions to France, but I could build something that I think makes sense. Uh I think that these some new people sometimes hear Django and sometimes I explain Django to some new people or newer than me in Django and I just want to have a word for those of you just joining the IT world now. And I know that you can do Django doing Python and nothing much, but I find that sometimes

14:41

Speaker 1: Um this is a long list of technologies that uh you might not need to know, but for dividing some issues they become very handy. And luckily I knew many of them, not in a very in-depth way, but they knew the concept, but it had used them some more than other ones And sometimes I think like, oh my god, someone starting now if needs to learn all these things. There are many layers now and here I'm not adding SSL, Let's Encrypt, and many other issues that um not issues, but um pieces of a puzzle. And I just want to say that all the beginners just stay and do bit by bit like we all did.

15:27

Speaker 1: And and I feel that as I said for debugging certain issues sometimes it's handy to know something outside Django or how Django interacts. This is when I think that having done some web development without Django, even though it's not pleasant, it might be useful at some point. The Django application that we built, we name it Science Cruise Data Management. still here online. I haven't mentioned but it was in some slide the Swiss Paul Institute was the organizer of this expedition.

16:12

Speaker 1: And this was a four-month expedition. So we had the application before leaving the first port that was South Africa And from the first moment Django admin was used by the scientists to enter search update data of what happens on board. Even before leaving and over the four months, I did modify quite a lot of Jang Guadmin forms and extended them, added validations and um simplified how they can be used in our use case um for our domain And all that we built, it was based

16:58

Speaker 1: and this thanks to Jen Data Manager, on a similar database structure as the YODC and BAS. So the part of the tables obviously were for our use case, part of the tables and how to store metadata, it was following some standards. And before leaving part, we had um a map of the ship's position and data collection locations. I think in my next slide, here I have one of the uh reports um that were this the one that was most used by the scientist to add event reports and also this right -hand side pane for filtering

17:45

Speaker 1: for different uh ways. These in the beginning I didn't have all these options. I'll talk, I mean on the top, I'll talk about this in a bit. But living port we had this map and we had the cruise track um updated every hour, I think, something like this. If I was enabling these show events, it would show where data or samples were collected. Things uh uh lessons learned is that it's very useful to visualize any data as soon as possible It helped a few times to see errors with latitude-longitude being on the other way around, for example, all the data collected in this area in the beginning, and having some points on the other side.

18:41

Speaker 1: um or points outside in the northern hemisphere or different types of issues. So initially data was collect was being collected but not displayed or useful and quite quick we started build building ways to visualize and to work with this data build reports uh with the data the The map out of curiosity was built with leaflet and I found the contour lines, not contouring lines, sorry, the coast lines and converted to JSON um GeoJSON lines. So for all of these I didn't need any online connection.

19:30

Speaker 1: Another approach would have been open script map and visualize this, but open script map um tiles don't have the projection that uh I needed for this case, for example. But all of this is Well, it was quite interesting. I remember spending a few evenings or days trying to put the map together that was working for the Antarctic projection. The system roading expedition, uh I was working at the time in Mendeley for uh was seven years. For once in my life, I had the idea of I'm working on this and I need this soon for the scientists, or scientists

20:15

Speaker 1: need this soon. And I need to maintain this until uh the end of mid-April 2017. Uh then this is all gone. Um it was kind of the opposite of what I usually think that The code will pre-un a long time. So at the end I was thinking, well, this code is getting a bit helly now. I just need two more weeks to end this expedition and just forget about this. Um in as I said initially it was uh for collecting data and then we made more visualizations. Um all the way the we had requests of site from scientists

21:02

Speaker 1: that they wanted some small utilities. Some was as simple as latitude-longitude compared to different formats. Some was that um yesterday I collected the sample, it was um it was at noon yesterday, but I don't know where we were. So uh uh system that they would put today. Um daytime and it would give the latitude-longitude for example for their personal notes. Um some it was um some other systems I'll say but Basically many scientists were coming with small um requests and uh I was building this small request in Django and then it was available uh for everyone.

21:48

Speaker 1: In the nineteen nineties I was thinking that many of these utilities they would come in these magazines, at least in Spain. We had these magazines with city rooms uh when we then have internet full of shareware small utilities to convert units or convert um now sunset and sunrises. So I was kind of building this type of things in Django. And ironically, uh now it's three years later, this system is still running uh for the data manager in one of the internal servers. It's not used or scientists don't have access to this. New data is not being added in the system. I did my read

22:33

Speaker 1: this last year when I joined the Swiss Pol Institute again, but from Django 1. 8 I think to two and now to three. But yeah, it's it's It it wasn't the end of the life at the end of April as I was kind of expecting at the time. Um And for the last year I've been working in two other Django applications, not related to this, but um quite bigger applications in the term in of I've been working on them with for a full year more less. And I would have done, I would do now things different. But what it was done at that time with the resources that we had,

23:19

Speaker 1: it was fantastic. And the credit is for Django. I mean not for me. Like it was fantastic because Django had so many tooling that was very easy to build on all of this. Um after some um weeks and months, I think weeks, we're kind of building this kind of sorry kind of intranet that was giving some information to the um scientist if they wanted to know something quick. And also we're writing some daily announcements. So here we're building these things. I took this screenshot I saw before when I was checking the slides again.

24:07

Speaker 1: At the end of the last month, here scientists could enter the login , sorry, login, the name and contact contact details Basically, some scientists wanted to share emails and phone numbers. And then at the time and at the end where a bit more relaxed, I think, well, no problem, I'll just build Facebook now. I mean just 220 names, photos, and content with other people. Um here, and that was on the second month. This was not built in Django, but a bit of the same approach is like, well, we need a mail system. I I set up an internal on-board mail system with external connections. It was like, well, no, I'm kind of building a small Gmail. And I

24:52

Speaker 1: said, well, if I stay more time here, I'll keep just building small things in in Django. Here's one of the screenshots of utilities about um I think I sending a list of date times on UTC. And then it was sending back, as you can see here, the latitude-longitude of this place or the data and times in different formats, I think. And here more visualization. We were collecting data from different equipment. For example, here's the temperature and salinity of the water.

25:39

Speaker 1: um and scientist well that particular equipment didn't have any display uh and Scientists wanted to know when we're crossing a front, for example, where water was cooler, just to take samples during this period if this was um a thing that they were interested. So Django also was very helpful to just visualize or integrate this with other JavaScript libraries in this case. At the time I remember, well I see that as I'm pretty well like the decimals here. I remember that I was scrolling and each data point had a very long number of decimals I just want to say that some scientists, or it was very easy because they were like, I don't mind.

26:28

Speaker 1: Um don't need to format anything. This is exactly what I need. It was easier than working in some other places where everything needs to be pixel perfect. In a way it was like, oh, they have information, they were happy with this. They didn't need to have fancy formatting or fancy um graphs or anything. I'd like to say a few words about working almost offline. I didn't have any Stack Overflow or Google contact for four months. So Django here came very handy as well. document I downloaded because I knew that I downloaded the Django documentation um

27:17

Speaker 1: the offline HTMLs and they were very useful they were priceless there At the time and now I was using PyCharm and actually reading Python documentation, uh Python code when the documentation doesn't clarify or answer some of my questions. It was another easy way. I think I did it more because I couldn't ask Google or Stack Overflow. Um so I started just digging more in Python code than probably I would have done before, which is quite interesting. You can say, I mean, on a plane maybe or for a week for fun. And I had

28:02

Speaker 1: and this is a third point, I had in my laptop a lot of source code of projects that I've been working or that I had to check out of anything. And this was very useful. I remember that I was quite familiar at the time with the calibre, the ebook reader code, because I had been doing a few things there. and is a massive Python code and I found myself that to know how to use uh certain functions, classes I could just go to Calibre Grep and learn from examples there. So it was kind of instead of Googling things and finding examples in Stackover

28:48

Speaker 1: Flow, I was just Using the things that they had in my laptop and that was sort of source code that they had, and I didn't even plan this. That was kind of how it was. And the Python console to play with things and the help methods. And interestingly, before leaving, I downloaded a few or I bought a few PDF books or free books And I do read books, but usually I read them not as a reference thing. I read them nowadays more as a um uh learning about some topic. But I had some about Python, so about Django, and I found myself using the other ways. So Django documentation, Django code

29:34

Speaker 1: and other code. more than opening a book and searching in the book in in that case, which is how I didn't expect this. Before leaving I spent I don't know an afternoon, a few hours kind of downloading books and then I didn't use them as much as I thought. And in great respect, Django was a very good tool for all that we needed. Um sometimes I think like well without Django I don't know what would have happened but would have I guess way less things in the expedition and it would have taken more effort from me to have fewer things. And initially my um before I knew Django, I thought that was

30:22

Speaker 1: big and that would get a lot on the way of my way of working or of what I wanted to do. And no, it didn't. Um I must say that a few times in the admin probably my thoughts and Django design maybe didn't agree, but it was minimal compared with what I was kind of thinking. And I was happy that the admin forms were possible to adapt and change. This is probably what I would do differently now, differently now, um using for a long time, uh 10 months. So Django KSP forms and I would build all the TV probably outside admin. But at the time it was a lifesaver.

31:09

Speaker 1: And the Django commands. We did many things with Django commands. basically things like copy hard disks when they were connected in the server, validate data or um daily tasks that that was very useful. And I had said in some other conference in a similar talk, but the Django developers, so nice, they get an extra mile and Many error messages don't it's not only an error message but it says and I think I have something here it tells you a possible fix in in in a place. And this is useful when you're at home and is more useful when you're offline and stressed to finish something that it can give you hints.

31:58

Speaker 1: um in this type of things. And again, I've said it but I'll say again that the Django documentation is very well written, so it's one of the parts that I appreciated the most at the time and I still do. And I want to have a final thought. I'm not sure how I'm going with time as yet. I'm okay with finishing. The Django website said, and I had this slide done in 2017 or 2018, it still says, and I still agree, that it makes it easy to build better apps more quickly and with less code, which is completely true. But when the data manager thinks we need to build the database.

32:47

Speaker 1: Yes, Django can be used to build a database, but even I know that uh database front end is a web app, but Some people might not be using Django and they could be using Django for some of these cases. You think, well, I need a database. And who needs a database sometimes of learnings? after this they think as a database not like me that is I need a server with some table schemas and Some front end, but I don't know the front end. What they mean is that they need a front end to enter data. And well, accidentally, yeah, there must be some server, but they don't know about this. But they think of a database as a more front end than the back end, which is usually my um

33:32

Speaker 1: thoughts. So if you know of any data managers that need to build any database or other people, Django. It served us and might serve other other use cases. And uh this was all that I had to or I plan to explain. Um I don't know Adam or Marco should want to do questions now, later.

34:02

Speaker 2: Yeah, sure. We'll do questions now. Uh thank you very much for that talk. Uh nobody else can chat, but we'll um Yeah, so no one's asked anything on the Q<unk>A panel yet. If anyone wants to write there or on the chat, please do. Oh, we've got one clap emoji from Phil Abada. I would like to ask a question. You didn't quite dive into it, so what what was your stack that you were using? Was it MySQL? Is that bootstrap I spotted? Anything else?

34:35

Speaker 1: Yeah, um so it was using Bootstrap, MySQL, because I am when I was more familiar than Postgres. And that was the only reason for choosing MySQL. And I 'm using now always uh Winicorn and Nginx. I'm happy or I can recognize that we use the Django debug server for all the expedition. because we didn't have time and we started this way and it was working. So it was as I said another of the things that have no time and I'll just use this because it seems to be working.

35:22

Speaker 1: And besides that, the server physically actually was a laptop. It was a laptop for the first leg it was overheating and then in the second leg in Australia we changed and we had a proper powerful SSD laptop. The number of concurrent users was usually quite low. It was um I think sixty scientists more or less. I don't think that many of them were working and hitting Django at the same time in a very intensive way. The we prepared another expedition last year and then it got um postponed not because of COVID but because of other

36:08

Speaker 1: diplomatic and reasons and obviously we're planning now to use nginx unicorn and the proper server on on board Cool.

36:18

Speaker 2: And so we have another question from Thomas R. So what was the format of the data you were processing? Were you doing anything odd or were you running a fairly standard Django setup?

36:30

Speaker 1: Sorry, can you

36:34

Speaker 2: what was the format of the data you were processing? And were you doing anything odd or were you running a fairly standard Django setup?

36:43

Speaker 1: So it was a very standard Django setup, processing of data. It was very um kind of low processing. We're not processing a lot of data. It was done usually after the expedition by the scientists. Some data was being processed or visualized and all this data was coming in CSB format. So we had CSV streams. I remember twenty name, but we had CSV streams that were visualizing like the temperature salinity that they showed before. the some heavy data or quite heavy data that we're processing actually

37:28

Speaker 1: or ingesting in the database it was the GPS position We were feeding uh one second um resolution resolution from two GPS. This was being used for the map and also to use this data when scientists were entering data saying at 11am today I deployed this net. and then it was saying where the net was deployed. Um and the format of the GPS that we're collecting it was NMEA NMIA strings And this is a text-based format. Again, it has a checksum, and I think that there are a few Python libraries that can process this.

38:17

Speaker 1: And I think that at the time I wrote a very simple parser for the data that I was interested and we collected all the um original data and I think it was reprocessed after the expedition. um we saw the checksams properly done. The so yes it was NMIA for GPS, CSV files. And the rest of data was more a matter of copying data from hard disks that scientists were blinging, or computers connected to a network into the place and the objects uh not object storage in the network storage that we had.

38:57

Speaker 2: Um so we have another question from David Smith. Uh what did you use to create the charts?

39:04

Speaker 1: The charts.

39:06

Speaker 2: Yes.

39:08

Speaker 1: Oh, that's a good question. Um that I cannot remember now. It was So I used a JavaScript library that someone sent me by mail because I couldn't access webs easily to download. And it It's open source, yes. I think it's um D fly. js. I think it's from the D flee. js I had one of the multiple representations of data.

39:45

Speaker 2: Cool. Another question from Adam Steele. What did you think Django could do better?

39:53

Speaker 1: Oh, I was thinking of this earlier. I was expecting this question. And then I stopped thinking. Oh, I forgot what I thought. The At that time, I found that I could personalize the admin, but sometimes I wanted to go it further. But then since then I learned Django HP forms and other tools, and I wouldn't ask Django to go further on this topic more. Um so yeah, I don't have now any, I mean keep keep with a good documentation. Um

40:34

Speaker 2: Yeah.

40:35

Speaker 1: I mean I'm very excited with the new Django async stuff. I've used Postgres to have JSON fields. I'm very excited now to have them in MySQL and from Django. So I'm very excited with the new things But for that four months I didn't miss a thing that I can say now, like yeah, I wanted to have this teacher

40:55

Speaker 2: Cool. We have a question from Paolo Malchio. Did you have problems memorizing the time data with a ship that was continuously moving between different time zones?

41:08

Speaker 1: Yes. Not I think not many or not any Django daytime and I know that changes are coming on this uh at some point. Not in Django, I think it was more human problem. We had the UTC time and all the science was in UTC time and we had the chip time Uh and I remember two things very vividly. One, the first month from South Africa to Australia. I was very busy setting up all the beginnings of this or maintaining the first weeks. And every three days or every two days I was losing an hour's sleep because we had to in

41:54

Speaker 1: in the month we had to change, I don't know, ten hours or whatever. So it was kind of killing me. But we had the ship data. And then in the second month from Australia to Chile, uh we went back 23 hours when we crossed the um uh day time line or whatever it's called. Uh so basically we lived a day again, which confused um people, even me. It didn't confuse particularly Django because we're using UTC for all of this. And we built uh shift time to UTC utility. So when we're changing the time we had a table with a time change and then scientists could write, okay, it was 1 p.

42:42

Speaker 1: m. yesterday just before lunch, what UTC time was, uh or what UTC date time was. The and there's a um small side topic thing. I remember that that day we had spaghetti and hot dogs and I was thinking maybe they didn't plan a better meal for this day that we live twice.

43:09

Speaker 2: Yeah, okay, uh one more question in the QA. Uh so from Eggby Anderson. We have um after building the project And looking back on it now, uh what would you do differently?

43:21

Speaker 1: Um many things. There will be way more unit tests. to start with uh time problems and I would build all that the scientists use outside admin. Um at the time as a new Django I didn't know better and I had no time to learn. Now this is my kind of better butter of everyday and I would just use Django Krispy form um outside admin authentication and everything. I had I've made the same mistake twice on how to split Django apps.

44:07

Speaker 1: Uh so from my big application and build the small apps. Um somehow I was uh worried or I thought when I was near Django that was um difficult to build new apps and how to interact models from different apps, which is not, but I was afraid and I was staying with the same app that kept growing. It was hard to navigate the models, it was hard to explain to other people when I needed to. So divide this a bit better. Mm and I said I had used some Django package um so some Python pip installation of some things that now I would do maybe by hand

44:55

Speaker 1: because When I did some updates, it was a small thing, so I could have done it, but on the update time years later it was problematic and I had to pre-do something. Some of these things probably.

45:13

Speaker 2: And other any questions from my fellow panelists

45:17

Speaker 3: Uh yes, yeah one. So first of all, thank you for the um the talk. It was extremely interesting. And it makes me want to join an expedition in Antarctica. And how do I do that?

45:31

Speaker 1: I also expected this talk. This question, sorry, I expected this question. The for me it was uh thing that they wanted to do for many many years um was a child glim, I have to say. Um I had I'll tell you how I did and what I would do if I wanted to go. But um years ago I had look at the Spain once and as an outsider the Spanish ships to do this kind of job because there's demand for this job. They are military ships. army ships and at the time I thought like uh this put me off. I talked with scientists that have been there and I don't think it's a problem

46:19

Speaker 1: but when as an outsider I said like That's very strange. It's like going to the army now. Um the way I think to go is that if you look at the British Antarctic survey for the UK and similar things in other countries All of them have jobs obviously. Some are to go to the Antarctica straight away, I think, or kind of targeting this. Some is to work in these institutions. And I've been talking obviously with many people in the last years since I've been there. And even a Liberian who I would think a librarian at the British Antarctic Survey why should need to go there.

47:04

Speaker 1: But they have opportunities over the years for everyone to do some work, admin work , things on the entire in the Antarctica. Um so working on this in one of these institutions, um, I said it helps a lot. Um in our case it came from um the Swiss Pol Institute uh wanted the data manager for this expedition and they contacted Jen who is also my partner asking her if she could go there only six weeks before because things were prepared in the last minute they knew that she was in a sabbatical and then by this way I got in touch with the people on

47:50

Speaker 1: charge and said well um Could I help there? And do you have an ET system and data collection data? And it came a bit from this like, oh no, we need one of you as well, but we didn't think of asking or whatever. But it seems to be very much usually on working on um these kind of institutions. Uh

48:15

Speaker 3: can you share the link uh even later about the UK uh institution?

48:21

Speaker 1: Yes, I'll type on the panel when I'm not sharing the screen. Yeah

48:25

Speaker 3: Thank you very much.

48:27

Speaker 1: Yeah.

48:28

Speaker 2: Thank you.

48:30

Speaker 1: Welcome. Thanks for letting me present this. It when I see the slides again, I almost get um butterflies in my stomach again. Like, oh that was very cool

Questions this talk answers

Why did Carles Pina Estany choose Django for the Antarctic expedition project?

He needed scientists to enter data through forms quickly, and Django’s admin, authentication, models, and documentation let him build a usable system in a few weeks. A small trial project convinced him Django was easier to get started with than he had expected.

Discussed at 12:23

How did the Django app help scientists collect and review expedition data?

The app let scientists enter, search, and update records, and it provided reports and maps showing the ship’s track and where samples or data were collected. Visualizing the records early also helped catch errors such as swapped latitude and longitude.

Discussed at 16:12

How did Carles work on Django while the ship had little or no internet?

He relied on downloaded offline Django documentation, read Python and Django source code, searched code already on his laptop for examples, and experimented in the Python console. He found these resources more useful than the Django books he had downloaded.

Discussed at 26:28

What technology stack did the expedition Django app use?

It used Bootstrap and MySQL; MySQL was chosen because Carles was more familiar with it than PostgreSQL. The team ran Django’s debug server during the expedition, although he later used—and planned to use—Gunicorn and Nginx for production.

Discussed at 34:35

What data formats did the expedition system process?

Most incoming data was in CSV files, while GPS positions came as text-based NMEA strings. The app also ingested GPS readings at one-second resolution for mapping locations and matching scientists’ event times to positions.

Discussed at 36:43

How did the team handle time zones and the ship crossing the date line?

They recorded science data in UTC and built a utility to convert ship time to UTC using a table of time changes. This helped when the ship changed clocks and when it crossed the date line and repeated a day.

Discussed at 41:08

What would Carles do differently if he rebuilt the expedition app?

He would add more unit tests, build the scientists’ interface outside Django admin using Django Crispy Forms, and split the project into smaller apps. He would also reconsider some third-party packages that later made upgrades difficult.

Discussed at 43:21

How can someone get a job on an Antarctic expedition?

He recommends looking for work with organizations such as the British Antarctic Survey or comparable institutions in other countries, since opportunities can arise for many kinds of roles. In his case, he joined after the Swiss Polar Institute contacted his partner about data management and he offered to help.

Discussed at 45:31

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos from Django London