It's about time by Russell Keith-Magee

Well thank you very much for those kind words. Um hi, I am Russell Keith McGee. Uh I come from Majjuk Nunga country, otherwise known as Perth Western Australia. Uh I'd like to begin by recognizing the Kumayai peoples as the traditional owners of the land where we're meeting today. In my day job, I am a senior data engineer at Cervata. We are a market research company. We use Python and data science to help brands to understand their customers. They help me get to conferences like DjangoCon, which is something I'm very, very grateful about. But I am better known for my involvement in the Django community. I've been a member of the Django Core team for almost 13 years. I was president of the Django Software Foundation from 2010 to 2015, and I served on the Django Technical Review Board for the 1. 7 through 1. 11 releases. More recently, I've been doing most of my open source contribution work on the BWARE project.

BWE is trying to bring Python to modern computing platforms like phones and tablets. The aim is to make using Python as simple for developing native applications as Django makes web development, something that's accessible for newcomers but powerful enough for heavy lifting purposes. purposes. But no matter how simple Django makes web programming or BeWare makes mobile programming, there are some problems that are just capital H hard. And one of the big ones is dates and time. Now, something that you may not know about me, I am a horology nut. I love clocks and watches And how that happened is an interesting and conveniently topical story. My undergraduate degree is actually in physics, but I went into university intending to do as much computing as I possibly could.

So I picked up some first-year computer programming units as electives. At this point, I had actually been programming for a few years, so classes covering if statements and while loops wasn't really a technical challenge. And when the time came for our mid-semester assignment, one of the requirements, write a program that will ask the user for a date and return whether it's a leap year and how many days will be in that year. Cool. Now that was meant to be a prompt to do a bunch of keyboard inputs, convert to an integer, take a modulo 4, use an if statement. But I decided I wasn't going to do the obvious thing, I was going to show off and build the best darn calendar utility the world ever seen. Which started me on a bit of a journey because when you dig into it, it's not just mod four. The solar year, the amount of time it takes for the sun to orbit the Earth, isn't exactly 365

and a quarter days. So in addition to having a leap yet leap day every four years, there's a special case every 100 years where you don't add the leap day. And every 400 years there's a leap a special case to the special case where you do add the leap year. Where did these rules come from? Well the rules are called the Gregorian calendar because they were adopted by Pope Gregory III In a papal bull named Intergravissimus in October of 1582. Why did they adopt such a complex set of rules? Well, because the year was shifting. Prior to 1582, the Western Roman Catholic world used the Julian calendar, adopted by Julius Caesar in forty-five BC, but the Julian calendar only has the every four years leap year rule. And so, every 128 years The calendar year gets a day ahead of the solar year.

As a result, by the 1500s, the solar year was ten days out of alignment with the calendar year. So that meant that the vernal equinox, the date in the northern spring where day and night are the same length, didn't happen on March 20th, but on April 1st. So what? Well, it's a problem when you're computing the date of religious holidays. The date of Easter is the first Sunday after the first ecclesiastical full moon falling on or after the vernal equinox. So if the date of the vernal equinox is moving, you've got a problem. The process of calculating the date of Easter is called computus, from the Latin word for computation, and it was the major problem addressed by astronomy and mathematics between the age of Aristotle and the Renaissance. The process of adopting the Gregorian calendar, though, that has some pretty hilarious

consequences. Pop quiz. This year marks the 100th anniversary of Red October, the start of the 1918 Russian Revolution. What month did it start? If you said October, you'd be wrong, because Imperial Russia, as part of the Eastern Orthodoxy, didn't adopt the Roman Catholic Gregorian calendar until after the Revolution. So when the Russian October Revolution happened on October twenty fifth. That was on the Julian calendar, and it was November seventh in Western Europe on the Gregorian calendar. The Roman Catholic world m uh made the calendar adjustment in October of fifteen eighty two, so that year October only had twenty days. But in the only in the Roman Catholic world. Sweden changed in 1700, but they got the math wrong

and overcompensated by a day. And so in 1712 they adjusted a second time, resulting in the one and only example of February 30th. The British Empire changed in 1752. Turkey didn't change until 1926. And so the number of days in the year, the number of days in February, and even the number of days in October can vary depending upon what year you're evaluating and where you are in the world. So that's where my interest in dates and times started. Let's just say my first year computer programming assignment was comprehensive. And Liz Murphy, my first year computer science lecturer, was a very patient woman.

Now, this is an amusing story, but I want to make a point. Like a lot of problems in computer science Computer science, something that seems relatively simple, like what is your name, what is your sex, how many days are there this year, can seem really simple on the surface and trivially easy to implement. But in practice You actually need to understand a lot of human history if you're going to implement a robust solution. But they are solvable problems. These edge cases exist for reasons, and there is logic behind them. Frankly, the results the reasons and logic behind them is fascinating. You just have to be aware that there are edge cases and pay attention to them. The problems we see on a daily basis in computing with dates and time handling is because people either don't understand the complexities of

problems they're trying to solve, or they don't care, or they don't communicate the limitations of the solution they've used, or they're willing to make those limitations someone else's problem in the future. And that's exactly how we got Y2K. A generation of programmers took the vernacular of the time and wrote systems that stored the year using two digits instead of four, seventy-five and still instead of nineteen seventy-five. And in 1975 the math works fine and bytes were actually expensive. But come 1999, the shortcut has transformed from a space-heaving optimization to a major engineering headache. And yet, despite the fact that as an industry we went through this just 18 years ago, I still see official forms that ask for a date and provide two boxes for the year. And we're lining up for a repeat performance in less than 20 years.

Many computer systems store time as Epoch, the number of seconds since midnight UTC January. January 1st, 1970, which will be fine right up until 3. 14 in the morning on January 19th, 2038, when that count will be larger than what you can sign store in a signed 32-bit integer. And if you're thinking, oh, that's a problem we can solve in the future, or we'll be using 64-bit machines by then, so it won't be a problem. Well, firstly, you want to guess how much 1970s COBOL code had to be updated in 1970s And secondly, we have already seen Epoch Bucks. In May 2006, AOL's servers crashed Why? Because their code involved generating events that should never time out. And the programmer used a hack. They just added a billion seconds to whatever the code current epoch timestamp was when

they created the event. Which was fine right up until 127 in the morning. Why is it always the morning of May 13th, 2006, when adding 1 billion seconds overflowed the signed 32-bit limit and never happen events were created using an already expired timestamp. But here's the thing. All computer systems have limitations. All computer systems make assumptions. The problem with ALL server wasn't that they used a hack to make dates non-expiring. The problem with YGK wasn't that the system The problem was that the technique used uh the technique that was used set a hard deadline for the end of life of that code. And the end of life either wasn't understood wasn't institutionally communicated, and as a result, when the clock ran out, hilarity ensued.

Okay, so how do we bide our time with Python? Well, there is, not surprisingly, a Python-built-in model called time. Time is a library for thinking about time the way your computer thinks about it in epoch. In practice, unless you're doing something that interacts with hardware in some way, time is almost certainly not the module you want to be using using. The module you probably want to use for most of your date and time requirements is the date time module. Date time contains tools for dealing with dates and times at a human level. Days, months, years, hours, minutes, seconds, and so on. The constructors all seem relatively straightforward. But if you actually use those constructors, you're going to have a bad time. Take the first one. Sure, that's a date. Where? Every conference call for papers has this problem. The PyCon US call for papers closes on January 3rd, 2019.

January 3rd, where? Okay, so we just attach a time, right? Okay, you attach a time. What time zone? Okay, yeah, but what about things that actually did happen on a date, like you know, births and deaths and things like that? Well, as soon as you don't have time information for a date, you have lost vital context. And if you try to do math without that context, you're going to get bitten. Someone asked Google now, how old is Stephen Hawking? and got the response, Stephen Hawking died tomorrow at age 76. Now, Stephen Hawking did die on March 14th, and he Did do a lot of very interesting things with space-time. But without knowing that he died in England and the person asking the question was in the US, where it was still March 13th, you can't do that math. You're missing vital information. You make an assumption, and the assumption has, in this case, hilarious consequences.

And if you specify time information without a date, the time also loses context and you get similar problems. Okay, so that means we should be using date times. The constructor for date time has one non-obvious part, TZ info. That's the time zone info. If you don't provide a TZ info object when you construct a date , time, it's called a naive date time. And for most practical purposes, it's useless. A date with a date and time without a time zone is an accident waiting to happen. If you do provide a TZ info object, it's what's called an aware date time. And that's something you can work with. But where do you get a TZ info object from? To make date time actually useful, you actually have to use a third-party module. And the module you should be using is called PyTZ. I'd argue if you're doing anything with dates and times and you don't have PyTZ as a dependency, you're almost certainly doing dates and times wrong.

It's a third-party module because it can't be built into Python or Django, and the reason why is another one of those human things. PyTZ is a wrapper around the Olsen Time Zone database. Uh the Olson database contains a list of time zones, the offset from time zone from UTC, the time at the Greenwich Prime Meridian, and it is published regularly, multiple updates per year. The current database is called the 2018 E database, so there's been five updates this year. That regular update cycle is why it can't be part of It has a release cadence that is just not compatible with Python or Django. The 2018 e database was published on May 4th because North Korea gave the world five days' notice that they were going to change their time zone to match South Korea. And sometimes these changes are officially announced as being retroactive

Okay, so why is a whole database format needed? Okay, your time zone is just an integer number of hours from offset from GMT plus an add an hour for daylight savings, right? No. Here are some amusing selections from the time zone database file. Darwin and the Northern Territory of Australia observes a 9 hours and 30 minutes offset from UTC. Adelaide and South Australia observes 9 hours and 30 minutes, but 10 hours and 30 minutes during daylight savings time, which is observed during the southern summer, six months out of phase with the northern hemisphere. Broken Hill is a town in the state of New South Wales in Australia, but even though it's in New South Wales, it observes South Australian time. But it uses New South Wales dates for daylight saving transitions Which most of the time is the same, but not always.

Eucla is a very small border town in Western Australia. It doesn't observe Western Australian standard time of plus eight hours. It uses 8 hours and 45 minutes offset. Lord Howe Island uses 10 hours, 30 minutes offset during the winter and offsets by 30 minutes for daylight saving. We haven't even left Australia yet. This region down here, it doesn't look anything like this. It looks like this. And last I heard there was a Django deployment in Antarctica, so this is a case the Django core team needs to care about. So once you've installed PyTZ, pip install PyTZ, you can reference any time zone by name or use the UTC time zone. The Pacific time zone, which is what's used in San Diego, is identified as America Los Angeles.

Actually using that time zone object, however, that's a little bit more difficult. Like we just create an instance and pass the TZ info in when we create the date time, right? Well no. What's wrong with this picture? Well, 5. 42 pm, 15th of October, 2018. LMT minus one day 1607. What's LMT? And where did that seven minutes bit come in the offset. Turns out, back in the day, time zones weren't something that was shared across the whole of a continental region. They were based upon train timetables and every train station kept its own concept Of time. It was only when we got centrally coordinated time that the idea of Pacific standard time existed. And when you try to map historical times onto modern time zones, you get some interesting offsets.

Los Angeles, for example, used an offset that was seven minutes from the now standard eight hours, which is minus one day plus 16 hours, from LMT. LMT is a local mean time, which is essentially the same as UT. but UTC wasn't formally standard until standardized until the 1960s. So referring to 1900 data UTC prior to 1960 doesn't make any sense. And because LNT minus one day plus 16 hours 7 minutes is the earliest chronological entry in the Olson database for America Los Angeles. In the absence of any other information at time of construction, that's the date that gets used. So time zones themselves are time sensitive. And if you think about it, that kind of makes sense. You can't apply summer time for a date in the middle of winter.

So what you actually have to do is create a naive date time object and then localize that object. When you localize the time zone definition can take into account the date that you are converting and then construct the appropriate teasered info object to pass in to the to the date time. So The datetime module gives us primitives for storing dates, but those primitives can be difficult to construct manually. And unfortunately, it's even harder when you consider that in most cases the data isn't coming in in a nice, clean, well-stored Sorted numerical format. The problem of parsing dates from text is one of the areas where the Python standard library has limitations. And it's not because the standard library is bad, it's because it's a hard problem. And it's a hard problem again because of people. So, today is Monday, October 15th.

Last Monday was the 8th of October 2018. I mean October 8th, 2018. I mean it's 2018's October 8th. Day, month, year, month, day, year, and year, month, day are all common orderings for dates, depending upon where you are in the world. If you are accepting date input. you have to be aware that there are cultural variations in the way people represent dates. And yes, I'm looking at you, America. Date time has a mechanism for parsing dates, STRP time. Now if you happen to know the exact format that your humans will be inputting dates, it generally works fine. The problem is that you probably don't know what format your humans will be providing dates. ISO 8601 is the international standard format for representing dates, and like all good international standards, it's the format no one actually uses

It's it uses year month day hour minutes second and then either a UTC offset in hours minutes or a Z for Zulu or unit UTC time. It's also unfortunately uh the one format that state datetime. strp time doesn't parse natively. There's also a subtle problem lying in wait with ISO 8601 format date times. ISO 8601 uh specifies the date in t hours, minutes, seconds. then specifies a UTC offset in hours minutes. There you've got a problem because a UTC offset isn't the same thing as a time zone. Take my own home time zone, UTC plus eight hours. Now it's not just important because of me, one -sixth of the world's population lives in that time zone.

What UTC A plus 8 hours won't tell you is where they are, which time zone those people are living in. So UTC, sorry, plus eight hours uniquely identifies a point in time, but it doesn't help you work out the right format for a particular user. And it doesn't tell you if any of those places are observing daylight. saving time or when daylight saving time went into or will go out of effect. Dealing with dates is clearly a complex problem and so lots of people Have tried their hands at doing it better. Date Util, Arrow, Moment, Maya, DeLorean are all PyPI modules that attempt to make date time handling easier. Date parsing is one of the areas they've tackled. Whether they succeed or not is a bit of a value judgment. They do tend to be more flexible in what they accept as valid input, but the price paid for that flexibility is occasional inaccuracy.

No amount of fancy logic will tell you whether 810-18 is October 8th, October 18th, or August 10th. So your mileage may vary. If you use these libraries, be aware they are not magic wands for fixing date handling. They make assumptions. Those assumptions have consequences. And that again is not a bad thing. All code makes assumptions. You just need to be aware what assumptions your code is making and validate that those assumptions are reasonable. But once you've been able to parse a date and a time, you've got a time zone, you've got a valid timestamp, now you need to keep it that way. Which again can be harder than you think. So we've created our timestamp. We've used localized to make sure it's in the right place. We can now do some date time math. What time was it 10 minutes ago? Well we can use time delta, which is a piece of the date time

library. And we get a time 10 minutes ago. Fantastic. Sure, but let's try a different date. Say 3. 05 a. m. March 11th, 2018, Pacific Daylight Saving Time. What time was it 10 minutes ago? Well, we just subtract 10 minutes, right? Well no, because that says that 10 minutes ago was 2. 55 Pacific Daylight Time. But at 2 a. m. March 11th, the Pacific Time Zone started daylight savings. So 2. 55 PDT doesn't exist. That time makes no sense. No accurate clock read 2. 55 a. m. on that date. What you need to do is normalize the time zone. Normalization takes a time zone aware object and adjusts The TZ info object to be correct for the time that is being displayed. In this case, the time 10 minutes before 3. 05am was 1. 55 am

PST. And you need to do this every time you do date time math. Another edge case, leap seconds. Just as a leap year exists to adjust the calendar year relative to the solar year, a leap second exists. to adjust the solar day against the calendar day. And in the handling of leap seconds has caused major multiple software crashes in recent history. It's enough of a risk of the New York Stock Exchange routinely stops trading for 61 minutes during leap second transitions to avoid problems. So any math around date times is inherently difficult too. Alright, so we've got ourselves a datetime object. We know the time zone of the person who gave us the data. We've parsed it correctly. We've made sure we've got the math right when we transformed it. It's now time to display that to another user. This means we need to know the time zone of the person who is going to read the data to display it.

And all of the same problems happen all over again. Time zone offsets, formats, all of these problems exist on the display end as well. If you've got a time zone aware datime object, converting it to other time zones is relatively straightforward. You can use the as time zone method on the date time object. You just need to make sure you actually know the time zone. the person you're targeting. But there is another trap that the developers and designers fall into, and they do it with the best of intentions, trying to make numerical dates seem more human. When you say your product is going to be released this summer or for Americans this fall, everyone south of the equator rolls their eyes. If there is any possibility your audience is international, please don't use these phrases. They aren't helpful. They're like an inside joke. They're great if you know the context, but just plain conf confusing if you don't. And if your intention is to communicate effectively, why would you intentionally be confusing?

And you may think, oh yeah, but my audience is all local. They all know this. Really? What about recent immigrants? What about people from outside the country who need to use your service. There is no service more American than the IRS. And I am an Australian citizen who has lived my lived and worked my entire life in Australia. But I have to dig through the IRS website to work out how to sort out my tax affairs with my employer. When they say they're closed on Labor Day, well, Labor Day is the first Monday in March. So uh why is the IRS closed in September? So, my humble request for anyone doing date time-based designs is when you're using dates, consider how they're going to be consumed. If you ever display a date, always display a year. Always use a text version of the month localized for your reader. always display a time zone and in logs always use ISO

8601 format. Details like year and time zone don't have to be front and center in your design. It can be a column heading or as hover text, but include it somehow. Reading a blog post that says published 3rd of May without a year, how do I know if this information is current or not? And when you're trying to correlate between a log of database and web server events, and you've got a timestamp that is accurate to the millisecond And then you work out the two events are eight hours apart because one's in Perth time and one's in UTC, or if you're in Django, more likely six hours out because one's in UTC and one's in Chicago time. Thank you, Adrian. So in summary, a date means nothing without a time, a time means nothing without a date. Both are meaningless without a time zone. An accurate time zone.

Resist the temptation to guess because every time you guess someone's time zone, assume you're going to get it wrong. When you read dates, localize for the author's format, do everything you can to get that right. Once you've got that time zone information, don't ever lose it because you can't get it back. And localize for the reader on output, considering how that output will be considered Consumed, not just now, but in a year from now. In many respects, there's a really strong parallel here with Unicode handling. The fundamental thing about getting Unicode right, you need to know, not guess, know what format your data is in Porting from the one of the big problems porting from Python 2 to Python 3 was that it forced you to pay attention to this detail and verify at every step whether or not what you had was a Unicode string or data encoded in a particular format and if it was encoded

what format it was in. If you've just got a blob of bytes and you don't know its format, it's really, really hard to accurately reverse engineer the format. Similarly, if you don't have or you lose date, time, or time zone information related to a timestamp, it's almost impossible to reverse engineer accurately. You can guess, your guess might even be right, right up until it isn't. The only solution is to be rigorous from the outset and make sure at every step of the process you know what you have. And once you've got it, don't forget what you've got. But my time is running out. So I'd better wrap it up. As I said at the start, my undergraduate degree is in physics. Physics is the study of fundamental forces. And at the very very early phases of a physics degree, you get introduced to MKSA. MKSI is the set

fundamental set of units. Meters, kilograms, seconds, and amperes And because these fundamental units are important, they've all been quantified and standardized. There is an ISO standard kilogram. It's a thing that a physical object that defines what a kilogram is. You can pick it up, you can hold it. You know, not anyone, but People can. You can compare it to other weights, to other kilograms. And you can do the same with two samples of a meter and two samples of an ampere. But you can't do that with time. Is this second the same length as this second? You can't put a standard in a a second in a bottle. There is no formal standard that is physical about a second. Time is ephemeral. It just is. So, you know, don't feel bad about time being hard. It is. That's just physics. And it's really easy to throw your hands up and declare

time zones, how do they even work? And I'll admit even I do that when I've been bitten by them for the first time. Time in a day. But just like physics and humans, you can understand them if you put your mind to it. You consider the full breadth of human experience, treat it as a challenge rather than an impossibility, and plan. And like all planning. When is the best time to plan? Ahead of time. Thank you very much.