The Full Story of That Viral AI Face App

If you've been feeling a sense of existential malaise this week, then it's come at a good time, because you now have some handy AI to tell you your purpose.

A facial recognition app (ImageNet Roulette) emerged over the past couple of days that makes assumptions about you based entirely on uploaded photos of your face – everything from your age and gender to profession and even personal characteristics.

While many people say they feel "seen" by the algorithm, others pointed out its absurdly offensive, inaccurate and often racist labelling. Was this thing made by a couple of tech bros with very concerning prejudices just for clout? Not exactly – although, that's not a million miles away from describing how the database on which this app is based came to be.

So what exactly is ImageNet Roulette and why have our timelines been teeming with wildly disrespectful and incorrect categorisations of people? I spoke to artist Trevor Paglen – who created the site alongside distinguished research professor at NYU, Kate Crawford – to find out.

VICE: Hi, Trevor. The AI app is causing quite a stir – has it taken you by surprise?
Trevor Paglen: It's been kind of shocking how much people have used it, and to see how people have been using it, but it wasn't really supposed to be a tool like "haha this is funny", so much as "there's nothing funny about this whatsoever". We vary rarely have access to see what machine learning models are making of the data we put up. That's what ImageNet Roulette is: it's an interface that shows you as a user how a machine learning model is interpreting the data that you put up, and those results are sometimes quite disturbing.

How much are you selling our faces to deep-faking Russians for?
We don't keep the images – they get deleted – but we do keep a record of what classifications have happened. And that's for us to do statistical analysis on the training data itself. We keep none of the images or personally-identifying information.

Data privacy is a hot topic – are you surprised by the amount people have uploaded without truly knowing what was going to happen to their image?
I guess I'm not surprised by that; my understanding is that things like this usually are quite predatory systems, almost designed to like give you a little treat in exchange for often very personal data.

Here's my face: sell it, just give me that sweet sweet dopamine hit.
Exactly… There's no company behind this, there's no business model, it really is meant in earnest and really is just a tool that gives people a glimpse into training sets.

Why did you chose this particular data set?
ImageNet is the most widely used training set in machine learning research, so in terms of publicly available data sets it is really the benchmark that's used in machine learning.

And this is where there's an issue, right? The data set we're talking about here is clearly riddled with prejudice and problematic assumptions.
I do have to say that ImageNet is very vast – it's a dataset of 14.1 million images organised into over 20,000 categories – the vast majority of which are plants, rocks, animals. The goal of the people who created ImageNet was to map out the entire world of objects. Within ImageNet there's about 2,800 categories of people, and that's out of about 20,000 something categories. It's a significant percentage, but it's not the overall purpose of the dataset.

It's easy to forget about the human labour that goes into creating AI. Amazon workers had to sort 50 images a minute into thousands of categories for ImageNet – how could it be anywhere near accurate if that's how it's created?
And for ImageNet, they simply imported those categories from wordnet [an earlier database], including all of these offensive ones and misogynistic ones and racist categories. This is typical in the field where you see a data set propagating from one place to the next without even the people who are building the models knowing what's in them.

To what extent are modern day systems based on ImageNet despite its problematic labelling?
That's a really good question. With ImageNet in particular? We don't know. I know for a fact that it's been used for other machine learning programmes – for example, one called "YOLO 9000", but when something goes commercial it's not in the research community, there's not papers published about it and that sort of thing, so you don't know, really.

Why did ImageNet tell me Jeffrey Epstein was a babysitter?
Recent work in psychology has shown that even the basis for this kind of behavioural assumption really doesn't have a lot of merit to it. Similar to lie detectors. We have a lot of faith in it – like, 'Oh, it's a machine, so it's infallible,' but they are not advisable in court. They've been shown to be quite pseudoscientific.

So AI systems aren't too far away from the pseudoscience of lie detectors? Will Jeremy Kyle soon be refusing me a loan or not hiring me for a certain job?
Well, one of the things that we show is that these approaches towards classifying people – whether that be their emotions or the character they have, or the moral qualities that they have – have histories that in some cases go back to the 19th and 20th centuries. Phrenology and physiognomy would try to divine something about someone's character by measuring their faces or measuring their feet or what have you. There's nothing to that, and there's a politics to that and a very dark politics. One of our fears is that we're seeing a recapitulation of some of those philosophies in current machine learning applications.

What's next for you guys?
I have an exhibition opening at the Barbican, an installation of about 35,000 images from ImageNet. These are ones from the original dataset and not pictures that people have sent in. That opens next week in The Curve. Some guys from my studio are installing it as we speak.

You can read Trevor and Kate’s full essay describing their project here.

@PatrickBenjam