Consequences of an Insightful Algorithm by Carina C. Zona

All right, so we didn't do a mic level check. It does work. Yay, okay. Um, so yeah, this talk is meant to be a toolkit for empathetic coding. And we're going to be delving into some pretty specific examples of uncritical programming. So the first thing I want to do, because we're going to be talking about painful results of doing things in ways that are perfectly benignly intended and then sideswipe people is I'm not gonna do that to you. So start with a content warning here. Um we're gonna be delving into examples that include a bunch of things including grief, PTSD, depression miscarriage, infertility, racial profiling, concentration camp, surveillance, sexual history, consent, and assault. And while we won't be dwelling on those, that is topics that are going to come up. All right. So algorithms impose consequences on people all the time.

We're able to extract such remarkably precise insights about any individual. But we haven't really asked ourselves enough, do we have a right to know what they didn't consent to share with us, even when they willingly share other data that ends up leading us there? And we have to be asking ourselves more about how do we mitigate against unintended consequences of that stuff. So it's helpful to step back a little and start with asking just a really basic question. Question, what is an algorithm? And generically speaking, it's just a step-by-step set of operations for predictively arriving at a conclusion. And predictively is a really operative word in that So obviously, when we talk about algorithms, usually we meet in the computer science sense, in the mathematics sense, patterns of instructions articulated in code or in formulas.

But you can also think of algorithms as being part of everyday life all the time. They're just patterns of instructions articulated in other ways, such as, for instance, a recipe. Or a lovely shawl. You know, if your worst code nightmare uh looks like this, you're doing well. Okay, so deep learning is a particular new, actually new old uh style of algorithms. So the technology goes back to, in theory at least, the 1950s, but there's been some recent breakthroughs uh since 2012 and 2013 that have really reshaped the landscape of what's possible with machine learning. It's really just the new hotness because of this. Oversimplified, you can think of machine uh deep learning

as algorithms for fast, trainable, artificial neural networks. It's a technology that, you know, like I said, it's been around in academia, for instance, for a long time, but really mostly at theoretical scale. um breakthroughs in parallelization, uh in uh availability of processors at massive scale, these things are making it much more viable to actually take this now into production use So because of that, deep learning has become realistically able to extract insights out of vast big data in ways that we've never really even been able to imagine about before. Deep learning is a particular approach to building and training artificial neural networks. You could think of them as decision-making black boxes So what does that mean?

Okay, so we have some inputs. You're just giving an array of numbers representing words, concepts, objects in some sense. uh and then executing it by running a series of functions that you write uh and executing against the array and the outputs you get are uh the uh machine learning uh predicting properties that it thinks will be useful in the future for drawing intuitions about similar data sets. So you give it this training data set, it says, hmm, I noticed some patterns here. This is probably how you can figure out similar results next time, and then you can throw much larger data sets at it, and it's able to come up with similar um predictions. So, what it would that look like, for instance? Okay, so there's a few problems here. Black box. Um

we'll get back to that in a second. So Because deep learning relies on an artificial neural network's automated discovery of patterns within that training data set, it gets to apply those discoveries to draw intuitions about the future inputs. However, that means that every flaw or assumption in the training data set or in those original functions that you've written are going to have unrecognized influence on the algorithms and on the outcomes that they generate. So that's something that we're going to revisit, but let's look at a really just basic practical example of deep learning. MARIO. It's an artificial neural network that teaches itself how to play Super Mario World. It starts with absolutely no clue whatsoever. There's a great YouTube uh video of this by the way in action. It spends twenty-four hours simply manipulating numbers and seeing what happens when it changes like pixel locations, for instance.

And discovers that after 24 hours, by golly, it can play the whole game. Um Ray. I told you it was amazing. So it's learning. It's it is not being given rules, it's not being given information, it's exploring its world. So Can you imagine on the data sets that we have access to what we could do with that? Learning to identify patterns and then using patterns to draw insights. This is something that we try to do all the time So, speaking of games. Oops, ruined my punchline here. There we go.

Would you like to play a nice game? Global thermal what is it, global thermonuclear war? Yeah. I love that word. Um we're gonna play a different game. Uh and we'll call it bingo. It is the uh screwyest game of bingo that you will ever play. Uh there are definitely uh odd angles, but um we're gonna be able to cover this board real well. So let's take a look at some of the ways that we have serious pitfalls to deal with. We need to appreciate how many of those exist. why they present new challenges, why we have to be able to consider and deal with them in the very near future. So let's go ahead and play some rounds using some case studies as examples. Target. Many people have heard of this one, right? They had uh

figured out that hey, you can get people to radically change their buying habits if you catch them in the second trimester of pregnancy, something that specific. But how do you find out? that you have a customer who is in her second month of pregnancy. This is I'm not second month, second trimester. You know, that's certainly not something they ask on the visitor cards. So they found that there were a few data points that in fact would reliably predict this. And so they started sending out ad circulars And those ad circulars were very focused on things that you need in your second trimester and as you are planning your little nest. Um, one day a man comes in, he's absolutely furious. Why did my teenage daughter get this? This is outrageous. Are you trying to encourage her to have sex? Um and then

supposedly he came back a day or two later and said, Wow, I'm so sorry. I didn't know what was going on in my household. My daughter is in fact pregnant. Um So we have this, you know, really useful predictability. Um Target took a lesson from that conversation. The lesson they took was we really should disguise the intent of these ads better. So now what you get is you get diaper ad, but you also get like lawnmower ad and uh you know like auto oil ad. So that you don't know that in fact only one of these ads is the one that's being actually targeted at you. So here's here's a lesson about you know mining data is Let's be honest about what we're doing. This was not actually the way to solve this problem that we knew too much.

There are lots of different ways we could deal with it. Shutterfly. Um this one is somewhat less known but in somewhat similar territory. They sent out an ad uh or an email saying, Congratulations, new parent. Uh time to pick out uh cards. To send us thank you notes. Unfortunately, they did not really predict well. Thanks, Shutterfly, for the congratulations on my new bundle of joy. I'm horribly infertile, but hey, I'm adopting a cat, so I lost a baby in November who would have been due this week. It was like hitting a wall all over again. Shutterfly responded, the intent of the email was to target customers who've recently had a baby.

Yeah, I get your intent. Um false positives are a problem. We can't treat them as edge cases. What is the impact of a false positive? What is the impact of a false negative? They didn't really apologize for having targeted wrong, just for you having received something. Mark Zuckerberg of all people has a little something to say about this. A couple of months ago he announced that he's gonna be a father soon What he also wrote about are the three miscarriages that they experienced before that. And he said, you feel so hopeful when you learn you're going to have a child. You start imagining who they'll become and dreaming of hopes for their future. You start making plans and then they're gone.

It's a lonely experience. I can't imagine that the Mark Zuckerberg of ten years ago would have understood this. And I think of all the VCs who focus on hiring. the Mark Zuckerbergs of ten years ago who believe that people who have had not nearly enough experiences of hard things in life are the best people to develop apps for people who have. This is the Mark Zuckerberg who's far more interesting to me. He's got a much better insight on all the possible ways that we can screw up, or at least one of them. Facebook year interview. For years they have been doing this, but each year they've kind of tweaked how.

Just surfacing some posts from the past year. That they feel were kind of the highlights real of your year. Problem is that not everything that had a lot of likes in the past year. is something that necessarily was positive for you. Not everything that people commented on was something you want to remember. And sometimes the things that were back then exactly that exciting and memorable and wonderful have changed over the course of a year. We have changes in our relationships, in our jobs, in our location, in our feelings. Having the belief that we can predict how people are feeling now about something that happened in the past is so absurdly arrogant. Eric Raymond coined the term

inadvertent algorithmic cruelty. And he defines it as the result of code that works in the overwhelming majority of cases, but doesn't take into account other use cases. And the reason that he gets to coin this is because he's one of the people it happened to. His daughter died. And year in review brought that back and kept bringing it back over and over. There's no obvious way to stop it. This is an algorithm too. This was a predictable result, right? Someone didn't predict that not everyone wants to have this come forth unwillingly.

So his recommendation is that we need to increase awareness and consideration of failure modes, the edge cases, worst-case scenarios, and So I'm trying to do that today. What we can do is I hope accomplish a little something, a little bit moving forward that conversation. And with that in mind, my first recommendation here for avoiding this particular specific pitfall, along with a few others, is be humble. We cannot intuit the inner state emotions and private subjectivity of our users. Not yet anyway. Which brings us to issues of consent, especially when de-anonymizing private data.

Fitbit. Uh you know, you think of it as the tracker of all sorts of activities, right? One of the things that it used to track is your sex life. Um it doesn't anymore. And there's a reason for that. Uh yeah, so uh it was in your profile, and the profiles were public. And uh yeah, the profiles were pretty interesting too. I can't decide whether to be really happy or really sad for this person. But I mean if you think about how people are feeling about the Ashley Madison exposure, okay, some people would want to brag and be all competitive, you know.

Uh For other people, this is a horrible violation of their privacy and their expectation of privacy. This was something crucially what's the matter here is default You need to be thinking through what is default private, what is default public, and what are the consequences of making a poor default decision. So hilariously how they dealt with this you know deep technical problem is uh a change to robots. org uh dot. txt. So you know problem solved I'm sure you all use your Fitbits. Yes. Um

so this is why they no longer do sex tracking, because they really weren't willing to put in the effort. to do it well. And you know what? I actually think that that is a fair decision to make. If we're not prepared to handle data really thoughtfully and well, especially if it's something that can harm them, then yeah, opt out of doing it altogether. That's a reasonable product decision. Um so you know Uber had God view, and if you hadn't heard of this, um, you know, in many ways it's similar to anyone else's admin tools. You know, we always need stuff for monitoring, analytics, blah, blah, blah, you know, making sure that the service has uh is doing what you expect. Umbers just happen to be called GodView. Well it didn't happen to be it was called GodView. Um the problem here was not that they had some sort of monitoring system, uh but the way they used it

And the way they used it was for funsies. They used it to show potential investors, for instance, at dinner parties, hey look, let's track all of the guests coming to the party. They don't need to know. Some of the people were not okay with that. They used it for things like an executive was impatient that a reporter was uh on the way and taking a little too long, so he tracked her. uh in order to meet her at the lobby and let her know and share her data with her. These are not necessary uses of data. We need to be thinking about how much access is actually made to private data and does it really be need to be used in the way that we're using it? We make tools and then we expand too much on how they're used.

And Uber was certainly a case of that. I think it's a little bit telling that they do really evil things like this in their code. This is actual Godview code. I mean look at that. Autoplay true. Uh if you remember OKCupid uh 's blog, they they don't really do it anymore, but the research group uh at the dating site okay. They used to blog about things that they were learning from the aggregate data trends and they would just focus on sharing some sort of little insight into simple ways that an OKCupid user could use their dating site better, right? Uber used to blog about their data too. Crucially different, a few things. Theirs was not about improving customers' experience of the service.

Theirs was about invading people's privacy in order to pass judgment on their sex lives. This is not a predictable consequence of signing up for a transport service. you don't expect that you're gonna have someone being drawing conclusions about your private life. And again, you know, it's so easy to look at something like Ashley Madison and say, Okay, well those people have some sort of plausible deniability of I didn't do what you said I did. Uber is saying we know for sure, we think our data definitely predicts that you This is again a lot of opportunity for consequences for the user. Did they expect it? No.

Don't do consent like Uber. Architect for real consent. Architect for informed consent. So what does that mean? Informed consent is permission freely granted, where no is a totally consequence-free alternative and it's the default value. And it's giving informed appreciation and understanding ahead of time. of the facts, implications, and consequences involved in giving a yes. And if that sounds overly burdensome, come on, look at the TOS. We can set aside a little bit of information to genuinely say like, yo, when you sign up, we also do this really fun stuff where we look at your data and we say interesting things about it. You okay with that? This doesn't have to be tremendously hard But we do have to make some sort of effort to give people better disclosure of what they're getting into.

Google AdWords is another really interesting example. There was Harvard research into Uh basically it took two sets of names. One that's highly correlated with people who are black, one that is highly correlated with people who are white. and then found real names of working academics who had those first names and some sort of last name. So if we say that uh LaToya is a name highly correlated with black women. Then we say, okay, so LaToya Adams is a real academic uh and we do some searches on her. And what they found were that when you search for names of white people versus black people You have ads far more likely to show up

that imply that the black person has an arrest record. Now think about what that means because AdWords algorithm is focused on predicting behavior and that's it. It doesn't care about content. It just has a template to fill in and select which one. Its job is just to figure out what was is gonna make us click. What based on what it knows about us and what it knows about others' behavior with those same ads. So what is this reflecting back on us? Um uh deep learning is also really useful for doing photo analysis. We're already seeing some uses, uh photo recognition. Facial recognition. This actually was not using deep learning. But you know, we're familiar with uh some of the interesting trends and possibilities that you can see, right?

Um it's getting a lot better. You know, uh iPhoto is more helpful now. But we're also seeing that even with far more sophisticated implementations that are using deep learning, this is from just two months ago. This is not in fact a jungle gym, it's Auschwitz. This is not an animal. This is Flickr doing auto tagging as well This is Google doing auto tagging. To their credit, they acted on it really quickly once told. Not to their credit. This was a month after Flickr had had that happen with black people being apes.

So learn. Pay attention to the news. They should have been acting on that a month sooner. And they did get it taken care of within 24 hours, but that says that 29 days earlier, this could have never happened. So why does something like this happen at all? And the answer goes back to the 1950s when Kodak was first developing ways to have really calibrated photos. So in order to make sure that photos were consistently rendered with nice accurate colors, they created what are called Shirley cards. And these were cards used to color balance the printer. And you might notice something about them. Shirley cards were of white women. Film stock was optimized to accurately reproduce details in white-skinned people.

The digital sensors of today are still trying to mimic the results of film processing. If we saw something radically different, we'd be complaining that the sensors suck The problem is that that means that we have decade after decade of less legacy data and current data that is all really poor data if it's of dark-skinned people. We think we have objective vast data sets, but the data set is polluted with noise. I'm gonna have to skip a few because we're running low on time, so I'm gonna jump ahead a bit. So rationales for an algorithm can only be seen from inside of that black box that I talked about.

And so that's great. Let's look at the rationale. That's a picture of the inside of the black box. It's in there, I swear The thing about making decisions inside that black box is there's no accountability. There's no way to check our work, and there's no recourse for people when they tell us we've gotten it wrong. So there are some things that we can do to take some lessons from that. And there actually are professional ethicists, including ones in our own profession. Uh I sure hadn't heard about them, and probably you haven't either. I adapted a few rules from various sources, including the Association for Computing Machinery. So let's take a look at a few of those. One, avoid harm to others. This might sound a lot like say medical ethics rules.

How do we do that? Well, consider uh decisions potential impact on others while we're doing development, while we're making decisions. Project the likelihood of consequences Contribute to human well-being, minimize negative consequences to others. It's not enough to just say, ah, this is a problem. Minimize, minimize. What is the least invasive solution? We need to be really honest and trustworthy, even more so because we are gonna screw up at some point. We're gonna need to be able to sincerely apologize and be believed. We need to have engendered enough trust that we can keep moving forward after we've made a fail. We need to provide others with full disclosure of the limitations of what we do and call attention to signs of risk.

We also need to actively counter bias and inequality because code is made by people. It's not objective. It reflects our tunnel vision. It replicates our flaws. Algorithms always have underlying assumptions about meaning, about accuracy, about the world in which they're generated, about how code should assign meaning to data. Underlying assumptions influence outcomes and consequences being generated always. So we need to do things like audit outcomes. You remember that adage, trust but verify? No I doesn't apply here. Um, that's insufficient because photo of black box. Instead, what we have to do is not trust, but audit

constantly. This has to be part of our practices of checking what's coming out, is it what it should be? Is it what's expected? Is it unbiased? So we're in this arms race right now because Google, Microsoft, Apple, Facebook, they're all deeply investing in these technologies. and they're doubling down and they're already rolling out implementations that are fascinating, but they're moving forward very quickly. And the problem with that is that they are both getting more precise in their correctness. and more damaging in their wrongness. So we need to be really thoughtful about how we move forward with this to make sure that we are focused on being more precise in our correctness and using that correct information very carefully. We have to insist on getting this right because we want to be empathetic, right?

So this is what we need to do. We have to have decision-making authority in the hands of highly diverse teams who can anticipate diverse ways to screw up And finally, we have to commit to transparency. We have to be able to apply our own expertise to say no to things that are going to harm users at all We have to be able to say no over and over again to pitfalls like these as much as is necessary because we understand the consequences.