The Washington PostDemocracy Dies in Darkness

Every Asian American has been asked this question. A computer gives the best answer.

No, just don't. (Fox News)
Placeholder while article actions load

To be Asian in America is to be quizzed, constantly, about your ethnicity. What are you? Where are you from? No, but where are your parents from?

Nowadays, such questions are more awkward than ominous. But there was a time when this was a national obsession of sorts — when splashy magazines like Life published guides to help readers distinguish between the “parchment yellow” Chinese with their “finely-bridged” noses, and the “earthy yellow” Japanese, with their “massively boned faces.”

Recently, computer scientists at the University of Rochester tried to teach an algorithm to tell the difference between Chinese, Japanese and Korean faces. They wanted to explore how advancements in artificial intelligence have made it easier for computers to interpret pictures in sophisticated ways. But, intentionally or not, their research taps into the uncomfortable history of how Asians have struggled to fit into American life.

The scientists were inspired by a quiz created by Japanese American web designer Dyske Suematsu. Fifteen years ago, Suematsu decided, half-jokingly, to investigate the stereotype that Asians all look alike. He threw a party in New York City and invited Asian friends. He put their portraits on the Internet and asked strangers to guess their ethnicity.

The website was a huge hit, quickly becoming one of the web’s first viral sensations. Suematsu says that millions have registered and taken the test. On average, people identify 7 out of 18 photos correctly — an accuracy rate of about 39 percent. That's barely better than pure guessing, which would yield an accuracy rate of 33 percent, on average.

“This is a challenging task even for humans,” said Jiebo Luo, a professor of computer science at the University of Rochester. “I asked some of my students to take the test and they all failed horribly — even though all of them were Asian.”

Luo and his students suspected that a trained artificial intelligence might be able to perform as well, or even better. Recently, they collected hundreds of thousands of pictures of East Asian faces and fed them through an algorithm to figure out just what made Chinese, Japanese, and Korean people look different. In a draft report detailing their results, they provide samples of the pictures fed to the computer.

But despite what they expected to be a difficult task, the scientists were surprised to discover that the computer could achieve accuracy rates of over 75 percent. This is far better than humans performed on Suematsu’s quiz. The computer’s advantage is that it could draw on a vast library of faces, Luo explained. “Our machine has seen far more examples than any living person,” he said.

Lack of experience is a major reason humans sometimes struggle to tell foreigners apart. Psychologists call it the “cross-race” effect: We are much better at distinguishing members of our own race or ethnicity than members of other races or ethnicities.

Studies suggest that with training, people can improve at recognizing the faces of people from different ethnic backgrounds. As Luo and his colleagues have demonstrated, computers might even be better than we are at noticing some of these subtle distinctions.

It's not all about physical proportions. When the scientists went to investigate how the computer was making its decisions, they discovered an interesting pattern. Many of the cues that stood out to the algorithm were cultural features, like hairstyles or glasses or facial expressions. This makes sense, since the people of China, Japan and Korea have somewhat shared ancestries, but distinct senses of fashion.

Without being told, the computer seemed to realize that our concepts of race and national identity transcend genetics — they are cultural ideas.

Luo imagines that this kind of research might one day be used in targeted ads or counterterrorism. Being able to discern a person’s nationality from their profile photo would help marketers better tailor online messages. Or, in a more Orwellian context, airports could set up cameras to racially profile people in the name of homeland security.

It may be more interesting though, to view the project almost as a work of conceptual art. Luo and his co-authors are all scientists of Chinese heritage working in the United States, a nation that has not always welcomed Asians, or treated them with respect.

Just a few weeks ago, “The O’Reilly Factor,” the Fox News show, sent reporter Jesse Watters to Manhattan’s Chinatown, where he proceeded to mock the residents with a mash-up of stereotypes. “In a stunning thirty-second clip, Watters asks a man if he knows karate (a Japanese style of martial arts) and then, confusingly enough, proceeds to attempt Tae Kwon Do (a Korean style of martial arts) with nunchucks (which originated in Japan),” writes the New Yorker’s Jiayang Fan. “The point is clear: no one can tell these Orientals apart anyway!”

The work of Luo's lab rebukes the lazy notion that Asians all look the same. If a software routine can be trained to easily recognize the differences between a Chinese person, a Japanese person and a Korean person, then that challenges Americans to pay close attention, to work harder to understand the diverse mix of people living in our nation today.

At the same time, it’s unclear how well the program would fare if it were presented only with pictures of Asian people who grew up in the United States. If the computer is mainly making judgments based on cultural attributes, it might completely fail at distinguishing between Korean Americans, Japanese Americans, and Chinese Americans.

That, too, would be remarkable — it would illustrate how identity is not something we are born with, but something that we build piece by piece.

Ask any Asian American. Every single one has a well-worn reply to the question: So where are you really from? When someone demands your ancestry at the beginning of a conversation — as often happens if you’re Asian — it implies that your genetic history is the most interesting thing about you. This gets tiresome no matter how proud you are of your heritage.

If smartphones could operate some version of the software from Luo’s lab, trained on the different nationalities of the world, we could all avoid a lot of awkwardness. With our new mental prosthetics, we might learn to look at each other in new ways. But we might not master a lesson that truly matters: Sometimes, respecting another person means learning about their differences. And sometimes, it means recognizing how they're really just the same.