A face in a photo tells you far less than you think

Imagine handing someone a photograph of an exaggerated frown or gasp, along with six emotion words, and asking for the match. That person could do it very easily.

That easy setup makes reading faces look easier than it is. In the 1970s, Paul Ekman, who traveled to New Guinea and other non-Western places to study expressions, claimed that six emotions, happiness, surprise, fear, anger, disgust, and sadness, were recognized from faces in every culture, says Ralph Adolphs, a Caltech professor who studies emotion. Many artificial intelligence systems are still built on that idea.

In a review called Emotional Facial Expressions Reconsidered, written with Lisa Feldman Barrett, who studies how emotions are constructed, and three other researchers, Adolphs concluded that the evidence does not hold up nearly as well as people assume.

First, the faces were not everyday faces. They were posed by actors, extreme and stylized, not the ambiguous expressions people wear as they move through the world. Adolphs himself used them in an early study of a patient with amygdala damage.

Second, the task was multiple choice. Matching a cartoonish face to a short list of words is like matching emoticons to labels. It shows a shared convention, not a deep ability to see an inner state.

When researchers instead use real-world expressions and ask people to describe them freely, without a list, answers become much more varied. The same face draws many different words.

Dog owners show the gap between confidence and accuracy. They feel confident they can recognize when their dog feels guilty after doing something bad at home. In controlled experiments, they perform at chance.

In real life, we almost never judge from a frozen face alone. If he saw someone look sad, he would not stop there, he says. He would ask how the person is feeling and what is going on, and might learn that a bad-tasting drink twisted the mouth. We check what else is happening, ask the person directly, and that extra information helps us tell which feeling it is.