KEYNOTE: Biometric Unsecurity - Carina C. Zona

Hi, I'm Karina Cezona, and this is Biometrics and Security. Biometrics are widely regarded by the public and I think by many developers as heightened security, but the actual track record tells a very different story. Biometric technologies are systematically making the world a less safe place. And I think we truly have an obligation to do something about that. I also really know that we can. So I do want to start with a content warning because this has a lot of stuff going on. So we're going to be talking definitely about surveillance clearly, stalking human rights violations and genocide. Policing, incarceral systems, immigration, border control, religious bias and persecution, refugees, famine, wildfires, transphobia, ableism, psychological abuse,

sexual abuse and there are uh images that are censored uh that are uh semi-pornographic um again censored. And then there are some racial and misogynistic slurs, some of which researchers censored and some that they didn't. So just want to make sure that you're aware that there's some stuff coming up that is potentially offensive and we've done our best to make it uh as as uh discreet as possible. So I think a valid question is why are we talking about biometrics at a Django conference, right? But Python and Django are deeply entwined with biometrics. First of all, if you think about just Python's history.

in research and science and academia and the libraries we have for data science, it was kind of inevitable. It was the place to go. And so we have already so much being built in and through and just starting with authentication and going far beyond that. It's it's deeply integrated into all sorts of things, including Django. And being used in all sorts of ways, including using Raspberry Pis for biometric sensors So taking a step back, when we think about biometrics, typically we're thinking about it as sort of like an upgrade on conventional security, an upgrade on passwords. Typically you want to think of authentication authorization as being that starting point, right?

So password steal and absolutes. So do uh usually the other uh authentication methods we use, like say a security token or um an SMS code. And then you have authent authorization, which is not saying you are who you are, but rather that you are permitted to do something because you are authenticated. So you can spend money or you can get insurance, you can get health care, you can drive a car. And all those things require exact matches. There's no such thing as it's pretty close to your password. You can go through. Whereas with biometrics we're getting into a whole other world of probabilistics where verification and identification are only

looking for is this like it? Is it close? Does it seem about right? And categorization being a percentage possibility of like how confident are we? How how likely is it that this could be an attribute of a thing? Um, so all these turn from exactness into something very different. And It's important to recognize that identification is not the same thing as identity, especially because biometrics is using terms in ways different than we do in everyday life when we talk about, say, an ID as being, you know, something, for instance, like a driver's license or passport. In biometrics, when we're talking about identification, we're talking about comparing credentials.

So that authorization and authentication part, right? Comparing data has to be exact, match one-to-one. Biometric identification, on the other hand, is comparing physical traits. So it's something that a sensor produced, and it's finding things like looking for face or the way you walk or how your heart beats Whereas identity is much more personal. It's who we know ourselves to be, whether that be individually or also part of something greater than ourselves. So our culture, the way we group ourselves with others in terms of things like race or gender or religion or ethnicity.

Oh, sorry. Um unsecurity is not a euphemism euphemism for just poor security. Unsecurity is poor people. losing food rations. It's when the least privileged people lose even more autonomy than they already had lost. And it's to streamline inconveniences to the lives of those who have more privilege, more power. Unsecurity is the undoing of safety, and unsecurity is unraveling autonomy, safety, human rights, and existential threats. So we're trading in that exactness for a percentage. Let's delve into a little bit about what that really is. When we talk about security, we're talking about access control, preventing

loss of something valuable. Bimetric unsecurity is talking about access plus control. Access to the public sphere is conditioned on ceding personal control over some aspect of your body. And what does that look like? What kind of things are you ending up having? Uh what access is being gatekeeped? So uh being able to use a credit card, banking, get insurance, buy real estate being able to enter your home or a friend's home, being able to even enter a neighborhood or in some cases an entire city. And it's being used to control basic services, to get access to a SIM

card, to get food, to get on public transit. to go to school or to use your education, getting a pension, welfare benefits, getting jobs, being able to immigrate, being able to even get your medications, your prescribed medications All these are technologies in use today, programs in use today by government and commercial entities And it's interesting because passwords, of course, you know, all this time we've been telling people don't do password reuse, right? Like every everything should have a distinctive password. And then in biometrics, you know, what do you have here? You can't do a body reset when there's a breach. And data breaches happen all the time.

Biometric data is not some special case, it's not immune. The tightening noose of biometric data linkage makes biometric authentication a really fragile precipice for people to be on. And each time those data sets are linked, the data breach becomes tantamount to having one password for everything. And unlike that password, again, you can't hit reset. Bodies are mutable. Our biometrics change. Biometric insecurity hits from both sides because of that Changes can be natural or unnatural, they can be accidental, they can be unconscious or deliberate, temporary or permanent. For instance, your gait can be altered by an injury, pregnancy, amputation, fingerprints

are worn down all the time by manual laborers to the point even of being undetectable or at least unauthenticatable Um in the new normal of our pandemic, it should be noted too that fever detection um is a really suspect technology because uh skin temperature isn't the only explanation for an elevated temperature. It can also be caused by things as basic as exertion like biking to work. or menstrual cycle or wearing warm clothing, etc. Things like pain, medication, or a stroke can affect your head movements, your gaze, your facial expressions, gestures, vocalizations. All of these are things that are biometrics being used for controlling access. And our behavior also changes when

we know we're being watched. So what are the basic types of data taking a step back? Okay, so we have phenological, which just means visual expressions of genetics. nose, ears, eyes, etc. Actions. So things like the way you walk or the gestures you use or the way you sign your name. Different from how your name looks on a page. For instance Do you cross your T's going this way or this way? And then what's sometimes called cognitive, but is probably better considered biosignals because it goes beyond the mental part, but neurological system responses. And that basic process of probabilistic analysis that we're going through is taking data, pulling out some record

and asking, okay, what kind of object is this? Is there an object even? Oh, okay, there's a person object. What kind of traits can we recognize in the person object? And then analyzing those traits for one of three purposes. And it can be more than one, obviously, but uh verification, which is a one-to -one comparison. Uh is this person who she claims to be? Whereas identifying is is this person somewhere within this larger data set? Is the person in this crowd And then classification is assigning some sort of meaning to traits. And that could be really kind of questionable because again, all of this is essentially educated guesswork.

So we're collecting biometric data from a variety of sensors, some like optical, thermal, and infrared, which are being used particularly for things like temperature detection. Using those biosignals, using electrocardiogram and others. Of course our our you know standard device stuff has like the microphone, camera, accelerometer, things that we're already using for sort of routine everyday biometrics to get into our devices. And then things that we associate more with surveillance in the typical sense. Things like a CCTV, webcam, body cameras. And data sources of record sets are just enormous. You get stuff that's uh sort of conventionally

government records, you know, stuff from police, from intelligence agencies, from various identification services, um, from permitting And then you've got stuff like social media gets scraped all the time. A lot of training data sets actually are scraped Um stock photographs, paparazzi photographs, again, some major training sets are using these as their source data. Uh and then things that are smart home, speakers like Alexa, uh doorbell like ring, smart appliances, there's an Alexa microwave Um and then the stuff that we wear on our body like activity trackers, uh there's now smart glasses and other kinds of wearables. And finally, military

using drones to do facial recognition and make decisions as to what that face is, what that body is, and what to do about that knowledge. And then we have underlying all this ghost workers who are doing the labeling oftentimes of this data, creating data out of, say, photographs or recordings. And finally, generated photos. This one was really interesting to me. For instance, researchers have not actually used multiracial people 's photographs to do labeling instead. They make generated photos, merging people Who have been identified as different races

and creating this sort of imagined composite in order to train on what a multiracial person might look like. So it's starting from a sort of algorithmic hallucination in order to make judgments about something that is incredibly personal and frankly outwardly not knowable at all. So how do we define biometrics in a sort of uh canonical sense? Well supposedly it's unique Um, that it's a measurable, stable kind of body trait that is uh unique in the world, like say your DNA, you're supposed to be the only person with it, right? Fingerprints supposedly are like snowflakes, there's only one each, right? So exactly one person, period.

Uniqueness is absolute. This isn't how biometrics are in practice. It's more just like body traits. body something um and gets a lot less uh precise from there. So let's look at some specific traits to really get an idea. First, look at the the ones in white on the left. Clearly none of these are unique to just one person. More than one person can have the same skin temperature, they can have the same skin color. Um DNA might be unique, uh, but things like say the clothes you wear, the accessories you wear, the tattoo that you have. It's possible that someone else has the same one. Stuff like brain activity and cardiac signature are probably more distinct.

It's really hard to find accurate data on um how precise uh which things are genuinely unique, demonstratively, scientifically unique, and which things are pretty likely unique. So let's put these in the category of leaning heavily towards unique, but it's not clear whether they definitely are. And then we have traits of the face iris, retina, face geometry, voice, right? Ones that are more unusual but are again in consumer use, facial veins, periocular geometry, periocular meaning the area around the eyes and the eyebrows. um earlobe geometry, um various things having to do with the way uh the behavior of our heads.

So the way we tilt it. movements that we make, the expressions, eyes and gaze as well, even the way we laugh, all are considered ways to identify us. Face coverings, eye coverings, head coverings as well. Is this person wearing a veil? A hat? Are they wearing sunglasses? Are they wearing a mask? Um and then we get into hands. So hand geometry itself, fingerprint of course is uh conventional, but also things like palm prints and various veins. So finger veins, palm veins. thumb veins. And then going down to the wrist, we have things like PLS and cardiac electrical activity. And these are pretty standard defined these days on activity trackers, right?

And then the things that you use your hands for, like typing speed, handwriting speed, that stroke order I mentioned of crossing the T, right? In the legs we have gait. How do you walk? What are the mechanics of it? The actual geometry of the leg, footprints, like fingerprints and palm prints. So there's a sense of objectivity, right? This is something that is absolutely measurable and if you're having problems getting exactitude, measure more things. so that you can get closer numbers, right? But the reality is that biometrics are full of biasing variables uh traits have distinctiveness, uh they differ in terms of poise elimination and exposure, all these various things.

are affecting how good the data is, how accurately you can interpret it, and whether you can really interpret it at all. So things like culture, class, gender, race, these are not things that actually can be evaluated externally despite everything else that we think we can do. However, they can be biases. They can change the way we reflect all of these other things. Our gender, our race affect things like, you know, how we how we interact with the rest of the world, sometimes how we speak. For instance, uh context switching, changing the way you talk depending on who you're talking to.

Biometric datasets and platforms like Biometrics as a Service are powering insecurity. Amazon being kind of the classic example of that. Um I think this is ironic because this is uh a new poster in Berlin. But they actually do uh package mass surveillance. Uh they call it Alexa and Ring and recognition. Um so they've got plenty of that already going on. Um So when we talk about technical debt, what are we talking about? We're imposing something, we're talking about something that imposes costs onto a project. Whereas ethical debt imposes costs onto people. It's debt collector comes only after harm has been inflicted on someone.

And this is kind of an example of what I'm talking about, is that we have researchers really committing uh acts of ethical debt. Things like using um external physical attributes in order to predict who supposedly is criminal and where you can find the most criminals. And I do think this is a good point to quote Every time I see this kind of shit, I think about machine learning researchers back in grad school who expressed offense that I wasn't charitable enough in my characterization of their field's widespread ignorance and complicity with racist, sexist, and ultimately deepful deeply harmful work. And it's a valid question. What makes us think that AI won't be mobilized to work towards anything but the detriment rather than the interests of black people

anyway? Why would we go out of our way to more accurately identify black people if it's going to be used against them? And by if I mean is. It is being used against them. Um uh we never thought about misuse is a common excuse for harm having been done. And it's the precursor to that ethical debt. It's when it's the cost of humans to humans from developers just ignoring insecurity and hoping it goes away. So IBM's Diversity in Faces dataset was uh released in 2019 and it was touted as this great innovation

of introducing much more diversity, which would make predictions much more accurate, fantastic. It turns out though that it actually was scraped from Flickr, uh and people had not given their consent. They were Creative Commons licensed, but Creative Commons is a copyright release. It's not a model release. And it's not consent to be defamed, which is exactly what was happening, not just in diversity and faces, but in other data sets as well. First of all, none of the major data sets have any consensual images. This was found just a year or so ago by uh researchers. Oh no, I'm sorry, this was released this summer. So very recent research. And here we have, as I said, slurs

, various ways that people were being labeled in these photographs. with really offensive terms, including as offensive as you can get. And there's a reason obviously why they blurred those photos, but there is enough here to see that there were things that were really indulging in stereotypes in order to come up with labels that were never appropriate to begin with anyway. So we move on to education on security. In schools, in physical to go to school schools, face fingerprint, behavioral, and temperature checks have become a new norm. especially because of COVID nineteen. So at an entrance, uh checking attendance, uh

getting access to meals even doing messaging with teachers are all becoming things that require students to hand over some sort of biometric information, if not several bits of biometric information. And remote ed tech is doing it as well, facial, behavioral, cognitive, also for checking things like attendance, but also doing tech proctoring, sorry, test proctoring or invigilation. Regulators in Sweden, France, Poland, and Norway have all ruled that facial recognition in schools violates GDPR because first of all, uh it's schools abusing their power to withhold children's access to education. and meals that makes consent just illegally coercive. There's too much of a power imbalance. And secondly, that there were less intrusive ways to meet a goal.

If someone can just present, say, their school ID card instead of having to give over their face and their voice and their fingerprints, then that is a more appropriate solution. So a school that used facial recognition in fact did have a data breach. And it ended up exposing teachers and families' communications completely. And now let's look at proctoring, which is exam monitoring, also called invigilation. And it's really become such a strong uh set of mental burdens at this point that even young children are being tested this way. Um proctoring you think of normally as something that's from a bit of a distance, uh, it's in a regulated environment, you have pretty simple uh, you know, clear room, whereas

Online proctoring is in your home and it's close up. It's a camera this close to your face suddenly instead of way over on the other side of the room. It's tracking all sorts of stuff that a an in real life proctor would not get to know. They're not supposed to be staring to watch your every eye movement. or to hear every cough you make and decide just how much your brain is engaged while you're taking the test. And all of these things that are being done remotely then are accumulating bias, invading privacy, inviting abuse, and indulging in ableism The premise really starts with that you're a liar. If you're working from home, you must be dishonest.

If you are uh you know, if you are are not accepting of these terms, then uh get used to it. Uh get used to being psychologically abused in school and at work uh by being called constantly a liar. And now I'm going to just run through a quote here. So all of you but one student violated a major testing protocol. This is from a teacher. Everyone except one student resized their browser, which means that you went to another website. If that happens again, you will get a zero on the next assignment. You have to have a well-lit room. You cannot take the test in the dark. Sit up If I only see your eyes, then that is a violation of the testing protocol. You have to video record. It clearly states in the syllabus. You will get a zero. No head coverings either for both males and females.

This means not what I call skull cap or regular caps And there were a lot of head and eye movements for such a short time period. One student had six minutes of head and eye movements, another had 624 eye movements in eight minutes That's an indication of eyes moving away from the screen. And this was followed by a threat to report people for academic discipline. So psychological abuse has some key features, like name calling, calling someone lazy, suspicious, liar, etc. needing to know where you are at all times, making threats to take away something important, isolating the person from other assistance

support. uh digital spying, keeping track of every little thing that you do with your computer, um, violating boundaries, otherwise invading privacy. All these things collectively are the picture of psychological abuse, they're also the picture of online invigilation Children who have been psychologically abused suffer from anxiety, depression, low self-esteem, symptoms of post-traumatic stress, and suicidality At about the same rate, and in some cases at a greater rate than children who were physically or sexually abused. Among the three types of abuse, psychological maltreatment was the most strongly associated with depression General anxiety, social anxiety, attachment problems, and substance abuse. All of this stuff

is exposing students to racial bias. And if you need more information about why, I've actually done a talk on that. And at the bottom there's a footnote telling you where Um against religious coverings, your yarmaka or your veil are off limits. Neurodiversity, uh the fact that you move around your eyes because you have something like ADHD. or Tourette syndrome, other disabilities, class differences. Now that your whole home can be seen, it becomes much more disconcerting to have the idea of your teacher being able to see that you don't live as, you know, fancily as you would like others to believe. If you're homeless, this is particularly an acute fear to have to have that revealed. Even the privacy of your thoughts and interests becomes something open to inspection by the proctor.

And that means that also your home and everyone in it is also being subject to that same invasion. It also provides opportunities for bullying and stalking. So once you've shown that whole space Sorry, I let me start over. So before taking a test, you must provide a photo ID, which it takes a picture of using your computer webcam, then an example service Proctorio takes a photo of the person taking the exam. Just to make sure that really is you and it flags that the photos don't match. This means anyone who doesn't look the same in their photo for very good reasons is called cheater You have to move very carefully, record your workspace, everything nearby, make sure you're not in the a room with anyone else. If you share a room, that's a problem, because that's going to be flagged as cheating anyway.

And if your roommate has something sitting out, that's also going to flag you as cheating. So you have all these reasons for shame, discomfort. Um, and then once you've finally shown that whole space, you are able to take the exam. But for the duration of the exam, the camera and microphone are on recording the test taker. Not only are sounds picked up by the microphone going to be flagged, but it also flags every time the test taker looks away from the screen at all. In the least problematic cases, it flags for looking away while you're thinking. In the worst cases, it flags folks with physical disabilities as cheaters. So some examples of this software that are in common use are Proctorio, ProctorTrack, ProctorU, and Examity. And all of the first three are using automated proctors.

Xamity uses a human for at least part of it And they variously are using some combination of phenotypical and behavioral biometrics as well as also non-biometric stuff that are really big invasions of privacy. And the kind of stuff they require are things that I was talking about. They also require a fairly high level of technical um availability. So it becomes another issue of I have to spend a whole bunch of money in order to be able to be spied on like this. And all of it really is elevating speculation to some really harmfully false sense of objectivity. It's not just gaze tracking of students, but also of workers.

So we get all of these assumptions, and all of them really have alternate explanations that are not taken for granted. There could be medical reasons that you're tired or you just want some privacy or you're thinking, taking notes off to the side. You're not someone who touched type so has to look down at the the keyboard. Maybe you're connecting via a handhold device There's so many different ways in which your supposed inappropriate behavior is just mundane behavior that is completely honest. but is being now judged in ways that it never was before and the consequences of it are so high. And note also that there's inherent cost to the system. Now teachers have to review hours and hours of video

of students just moving their eyes or doing a little cough or someone just happen to poke their head in and say, hey, are you coming to dinner later? All these things are enormous Um and then we have the notion of phys physiognom. Uh oh. Busyognomy. Okay, sorry. Which is a debunk junk science that the outer appearance can be used to infer inner character. It's complete nonsense. It's long been disproven. But it ends up contributing to things like income unsecurity. HigherView is a platform for recruiters

uh and in it they track things like vocabulary, word choices, expressions, and in a just 20-minute interview they collect all sorts of data and judge it based on things like um oh your use of pronouns that's not at all problematic for trans people is it um so all these things are being used to make an assessment of your worth as a potential employee And here's another one, eight and above. And you can see this is actually done on a 30-second video cover letter is used to make all of these judgments about your character. Uh this is all called the Fetrometric Fectmetrics and it's just

full It has no basis. It's not supported by research. It's mind reading in its latest form. And there's so many ways that this stuff is being applied as if it is genuinely useful, true, and should be attended to. But there is no substantial evidence at all for effectometrics. It's just mind reading, charlatanism, and projection. And projection is what really concerns me because this kind of pseudometric uh pseudo medical stuff has real problems when it's viewed as genuinely medical. So for instance, Apple Watch at least goes through some kind of scrutiny by federal authorities as a somewhat medical device, whereas Amazon Halo, the new

um watch like uh tracker from Amazon does not. They haven't even tried to get that clearance. Um so they make all these extraordinary claims, but when you look down at what they're claiming on behalf of Halo it's really shallow, um, claiming that their expertise in artificial intelligence somehow implicitly makes them experts in psychology that the way you speak can be used to tell you something about how others perceive you. Who is the other? that this can all be used by judging your vocal expression, your vocal tone to improve communication and relationships. More of this, more of it. And third party consent is not in this picture at all So we're using this device to record the tone of your voice

and then to judge from that something that is unknowable really. And here's their research. There's three data points. Um, and the first one uh is not about what they're claiming it's about. The second one is not about what they're claiming it's about, and the third one Doesn't exist. Also, the first two don't even link to any research at all. They link back to the same page. So we have the least amount of evidence, it turns out to be zero amount of evidence There's no rigorous independent science going on here at all. So here's the thing about that projection. It's someone saying, well, I said a thing. And the device saying, well they think that what you feel is

um well the person didn't get any say in this at all. They didn't get to consent to even being judged this way, let alone having that judgment passed back. Projection casts an image onto a blank screen. It's a psychological projection that are external representations that bear little result of little relationship with the person that they're being ascribed to. And abusers love projection. Whether it's positive or negative, it's really feeding into the abusers' tactics. and satisfaction. And something like this is ripe for that kind of abuse. And while it certainly may not be intended that way, again, just because you didn't think about it doesn't mean it's not abusive.

It's up to us to actually think it through. Biometrics are altering power in so many ways. And that hype is a weapon. So, for instance, in India, there was introduced nationwide an identification system that was supposed to be just a number, essentially a primary key in a database. based on face, iris, and fingers. And the idea was that people who don't have an ID can have an ID this way. It's inclusive, it's easy, and it's strictly up to you if you want to use it. Um and so lots of people did and then it became much more compulsory in order to get basic benefits. And the problem with that is that the system has shortcomings.

It didn't always work. It doesn't always to this day work. Um and nearly 300,000 people as a result ended up not getting their pensions who were deserving, who earned them, who were entitled to them, but the system itself, ADHAR, was cutting them off Here's an example. Someone who literally was unable to get a handful of wheat. um a smear sack of wheat because her fingerprints weren't recognized. Well as I mentioned earlier, you know, physical laborers certainly run into this issue all the time of your fingerprints being highly mutable. As of this January, 1. 2 billion Indians already have an adhar, and the ultimate goal is to have every single Indian have an adhar.

very soon. Um so two-thirds of four point seven percent of people whose biometrics failed during authentication We're still able to get rations. That sounds great, right? Um, what a great claim. Except that the remaining one-third remain an urgent action item for India. Okay, well that translates to one point five six percent of highly impoverished people who still, after all that, couldn't get their food rations. This is a really big deal. Those are people who could before, and now they've been forced into using this biometric system, they're much more vulnerable. So Mission Creep, design for privileged bodies, income insecurity, food insecurity, it all becomes issues that contribute to trust and safety unsecurity.

Unsecurity is constantly exploiting right now pandemic and protest. Temperature sensors, I think I already mentioned, are being widely used in schools as well as workplaces. To try to detect whether someone may be infected. It's junk science as well for reasons that I've already mentioned and others. Scientists have weighed in on this. it's not reliable at all, has really very little if no value. As well surveillance to uh prevent contact. So for instance, this is a nurse in uh India Saying at every step there were police present and there were drones being flown overhead constantly all through the night and in the morning Uh all just to make sure that even just going out into the streets to give medical care was something treated with suspicion and with surveillance.

ClearView AI is one of the biggest aggregators of facial recognition data sets. Most of it is scraped. And it's been used repeatedly to identify protesters. They form partnerships with police agencies all over the country, and this is what has resulted. And this is an actual screenshot from a website of a company who makes facial recognition software and they go out of their way to show you exactly what they can do about identifying someone whose face is obstructed. You've chosen these images very pointedly. I can't help but wonder, okay, so you're saying you'll identify people who have religious head coverings, people who are wearing masks.

and people who are protecting themselves from chemical attacks. This is not a benign technology that you are selling today. And unmasking is a really popular new application. So you can use periocular as well as voice, gait, tattoos, clothes, mask, all of them collectively especially by layering them in multimodal. uh analysis become very tightly identifying. Civil insecurity. So being a civilian on the ground during military activity. Killer robots is the term given for lethal autonomous weapon systems. That is systems that are meant to make decisions on their own about who to kill.

and when. So using entirely the system's biometrics and other you know data points to decide uh you choose. Go ahead. As technology becomes more advanced, and especially as things like facial recognition are adapted to drones, you wind up with situations where you can literally be conducting individualized surveillance. on dozens or more people, potentially even hundreds at a time. And this is being used as well by US immigration um and homeland security, all of them th are not military, they are paramilitary, they are civilian uh and yet using this the same way as if it were uh as it was originally intended for military use.

Not so far to kill people, but certainly to engage in practices that have the side effect of being dangerous or deadly. Military biometrics themselves. So Project Maven was a project that formerly was at Google and now is at Palantir. that would uh enable drones operated by the Army and Air Force to automatically identify people, among other things? And this actually got pushback, and I'm really happy to say that, and we'll delve into that more a little bit later. But moving on to refugees. There are over one hundred million I'm sorry, one million Muslim Rohingya in the world's largest refugee camp.

They're in an area of Bangladesh. I'm sorry, in Myanmar. In Bangladesh, having fleed Myanmar , and it's an enormous set of camps. The United Nations investigators detailed atrocities that they were fleeing that were committed by the Myanmar military against the Rohingya. during the two seven two thousand seventeen crackdown and the government called for those responsible to be prosecuted for genocidal intent. Rohingya refugees have gone through atrocities. Um the photographs, the descriptions are really hard to hear. But if you're willing, this article is a really

comprehensive look at how biometrics are endangering the lives of Rohingya who are already fleeing violence. And that's sort of the premise. The United Nations is forcing them to use biometric IDs for its own purposes, which are giving aid, but they are forcing this on populations all around the world that they are there to help. And the premise essentially is give us your biometrics or, you know, your life is at stake. We won't kill you, but we will leave you with no food, no shelter. So without having to say that, everybody understands that that is the price. And so it's very coercive.

When there's nothing left but your body, and now they want that too, how does that feel? When literally you have left everything behind your entire life. And this is all you have is your physical self. And then that too is being stripped from you. 7. 2 million refugees' biometric data has been taken by the UN already, and they continue to increase those numbers. That was two years ago. They're fleeing violence and persecution on the basis of their very identities. So now their most intimate information is being collected and stored in a database that they have no control over. Um and the UN really makes this

uh they sell this. This is their own uh data sheets in which they talk about how great this is. And out of those seven plus million people, the only quote they could manage to find was, um, and now you'll always know me. That's the biggest testimonial. How warm is that? It's about it's about bureaucratic assistance about bureaucratic convenience and there is benefit to that, but for the individual it's taking something while giving their very little back. So their fingers, all the fingers, both irises and their face, all have to be scanned in order to be able to get aid. But you know, this whole notion of consent obviously is an obscene fiction because of that extraordinary power and balance.

And as well the agencies because they're controlling the food uh and in cooperation with governments, the governments themselves, the host governments are also trying to control the data. It's useful to them and the countries they're fleeing also want to control the data. everyone has an argument as to whet why it should not just be the UN's or the people 's but also belong to other governments. And they're weaponizing that access, using it to control people's movements, deny access to driver's license and other privileges. Enforce bans on employment, maintain forcible segregation, surveil refugee populations and individuals, crack down on protests, and even commit atrocities.

All with the UN's own data. Which brings us to genocide. Human rights experts agree that cultural genocide is the accurate term. for what is happening to ethnic Muslim Uyghurs in Chinese Xinjiang province. These are pictures just published a few weeks ago by BuzzFeed, which did a really comprehensive investigation. using um images from web uh mapping service in China, it found blurred out areas and compared them to other satellites images. and discovered uh numerous prisons had been built that were accommodating thousands of people, 268

newly built imprisonment compounds. One million people, which is every person in either Clone or Odessa, that's how many people have been sent into mass detention into these camps with no trial, no charges. They're not considered prim criminals. They're not convicts. They're just people who are suspect and that's the basis for their detention. One million Uyghurs is half the population of Paris or Vienna. This is the context for just how big these numbers really are. It's the largest scale detention of ethnic and religious minorities since World War II. And Muslims in Xinjiang, half the entire population of the region

are under perpetual surveillance as well. So Panapticon is a concept that's pretty old and it's the concept of essentially being able to create an environment in which only one person has to watch. in order for everyone to feel watched. And it's that sense of uncertainty about who is being watched right now that ends up making people self-police themselves. So the system makes it possible to scale greatly with very little effort. Ethnic and religious minorities in China are being faced with that panopticon. so many ways in which their biometrics are being collected and used to constantly surveil them. And collected via things like mandatory physicals, police checks, and constant monitoring by public video on CCTVs.

So I've kind of skipped over data ethics, and that's because it's uh first of all, it's a subject that deserves a whole talk in itself, and there's already quite a few of those talks, so I'm not going to worry that it hasn't been covered. But an interesting new premise has been raised recently that we are in, we've been in a series of waves of how we conceptualize data ethics, and that right now we're in a third wave. It's from this article, which I again recommend by the leader of the Ada Lovelace Institute. And where we've been do what we've been doing is progressing from a sense of, well, what should we do? And in that first wave, it's very philosophical, talking about principles of you know fairness, accountability, transparency, a notion of

In the end, AI is going to solve a lot of problems. We just need to philosophically work out how to do this right And second wave focuses on the technical, that it centers technology and technologists, that we can fix this stuff just by doing a little more tuning, adding some more data to training sets that intervening in some way will finally result in just the perfect algorithm and then everything will be ethical. And third wave, which is where we are now, looks at a much bigger picture of societal impact How is power impacted? How is equity impacted? What action can we take?

So expose, critique, and change systems of power. And these are not mutually exclusive. It's not that the first and second waves are wrong or need to go away. They're all progressive So we have to move along this continuum. And a big problem is that right now the industry is kind of stalled at wave one and a bit at wave two. And we really need to be moving forward. Definitely At least into second wave, but realistically, we need to all get together on third wave because impact is a really big deal Whose power are we reinforcing? Whose vulnerability are we exacerbating? What threats do our actions and inactions contribute to that? And whose problems are we trying to solve? We start off with this premise that everything's going to be solved.

Well, whose? And whose solutions might we be unraveling in the process? The narrow focus on technical fairness is really insufficient. It confines us to thinking only about whether something works, but doesn't permit us to ask whether it should work at all. Uh the movement Tech Won't Build It was started uh in response to Project Maven. Google uh employees uh organized and refuse to build oppressive technologies such as automated uh weapons. And one of the leaders actually ended up resigning and he says, I resigned and I walked out with a big smile on my face after I orchestrated with coworkers a huge campaign to stop Google from building AI drones for the US military.

So taking that kind of big step can feel really scary and legitimately so, but it also can feel really good It can really feel like I have done something that brings meaning to the world. And how does that make me feel? So when we evaluate really hard choices, it helps to evaluate them in a bigger picture and context. Some of the commitments that we've been asked to make are uh to sign this pledge from the Reprogramming War project, uh also called PACS. And you can see their URL there. So setting out some real public commitments and policies for building this stuff and essentially those are we won't, I won't, tech won't build it, our company won't, and I individually won't.

Further asks are to please defund facial recognition as part of the defund police movement. And you can read a lot more about that at this URL at the Atlantic. And then the safe face pledge, which was started by the researchers on various aspects of facial recognition data uh and uh people of color. And you can see it's a pretty long pledge, but it's really um I think the details of itself, regardless of whether you personally sign on to these pledges They give a framework for thinking about third wave and what actions really can be taken and making decisions about what one will take. And then we come to the final and biggest one, existential unsecurity.

Applied biometrics is being built on top of a foundation of junk science, prejudices, abuse. technical debt and ultimately that ethical debt. And the price that we're paying for taking all that power over bodies and endowing unsecurity onto them Our climate change. We're in a year of record fires, record heat, record hurricanes, multiple fire tornadoes at once. Siberia is melting, the Arctic shelf is crumbling. And yet the baseline minimum carbon impact of training a research quality natural language processor is Equivalent to taking three hundred flights

round trip to from Porto to Delhi That's what we're making. We can't wait for regulations. Around the world, legislators and courts are still dithering over whether even those first wave basics like fairness, accountability, and transparency are necessary. let alone whether and how biometrics violate civil rights or human rights. When they do consider biometrics, they're preoccupied foremost with just facial recognition, as if it's the only one or the only one posing threats anyway Their concern focuses on threats posed by privacy and by inaccuracy in policing, but they show little regard for threats imposed by precision. And they disregard the role of consumers, including individuals and non-governmental organizations like UN

They neglect to invite tech workers to the table in any of these crucial discussions. They greet billionaires as spokespeople for every person in our industry. They take for granted that C-levels and academics understand applied technology better than the people who build and use it. They don't ask us whether we want our industry to center the self-interests of BCs or whether we'd rather our labor be used to prioritize humanity. We can't wait for them to draw lines in the sand. They're too far behind and they always will be So it's up to us to make choices, to take stands, to take concrete actions. And the question I have for you is: what will

you do next? Thank you. And I have resources for you as well. I'll be posting the deck, but I've got a lot of stuff for you, including video lists, Twitter lists. and recommendations for a variety of books that are very relevant and wonderful. Thank you.