Waltham firm helps computers learn empathy by mapping the human face

Long before any human can read a book, he or she can read emotions. By glancing at someone’s face, people know whether someone is happy or angry, frightened, or confused. Now computers are learning to do the same, with help from a Waltham company.

Affectiva Inc., a spinoff from the Media Lab of the Massachusetts Institute of Technology, develops software that scans videos of the human face in search of the same visual cues that humans instinctively recognize – a furrowed brow, the curve of the lips, a wrinkled nose. The company’s Affdex software can instantly measure 15 of these “facial action units,” and use the results to calculate a person’s emotional state.

It’s like the “Voight-Kampff test” that Harrison Ford uses in the classic movie “Blade Runner” to tell humans from androids. Only this test is real, and you can run a simple demo version on an Android phone, or an iPhone too.

The heavy-duty version of Affectiva’s software has been used by 33 Fortune 100 companies. Firms like Coca-Cola and Unilever use it with consumer focus groups, to study their reactions to the companies’ TV commercials. The company is working with candymaker Hershey Co. on a promotional kiosk for retail stores that will spit out a piece of chocolate when a customer smiles at the machine. And Affectiva just patented a new offering that can watch users as they watch TV, and use their reactions to suggest different shows and movies they might like.

But Rana el Kaliouby, Affectiva’s chief strategy and science officer, has even grander goals. She hopes to see “emotion chips” become a standard feature in millions of everyday devices, so that our machines will routinely respond to the way we feel.

“We spend a lot of time with our devices, yet our devices have no clue how we’re feeling,” said el Kaliouby, a former MIT research scientist with a doctorate from Cambridge University in the UK. “There’s an opportunity here. Our devices could be more empathetic.”

If a student got stuck on a tough math problem, an empathetic school computer would recognize the confused look on his face, and instantly offer additional help. An office laptop might see that a worker is bored, and suggest that he take a coffee break or play a simple computer game. A TV that notices that nobody laughed at last night’s Adam Sandler movie might suggest Woody Allen next time.

El Kaliouby’s interest in emotion-sensing machines began when she left her husband behind in her native Egypt to study at Cambridge. Gripped by homesickness, she was frustrated by her inability to fully convey her feelings over the Internet. “Everything was reduced to an emoji,” el Kaliouby said. She decided that to convey emotions, computers must learn to recognize them.

So el Kaliouby delved into the research of Paul Ekman, a psychologist who pioneered the scientific study of facial expressions, and whose work was the inspiration for the Fox Network TV series “Lie to Me.”  Guided by Ekman’s work, Affectiva has built computer vision algorithms capable of recognizing thousands of possible facial configurations, and the emotional messages they convey.

But not all humans are alike, not even in the way we express emotions. To ensure the software’s reliability, Affectiva has built up a video library of 3.2 million faces from 75 countries, including people of every race and age. Along the way, the company has discovered a number of cultural and gender traits. Women, it seems, smile more than men, and their smiles last longer. American women express emotion more freely than American men, but British men and women are equally expressive. And people in general become more emotionally expressive as they pass age 50.

Affectiva is privately held and doesn’t release revenue or profit numbers. The company has raised just under $20 million in funding from Kleiner Perkins Caufield & Byers, Horizons Ventures, Myrian Capital and the advertising company WPP, as well as a $500,000 grant from the National Science Foundation.

But Affectiva doesn’t have the digital emotion market to itself. Emotient Inc. of San Diego has more than 200 clients for its own version of the technology, including research universities and corporate marketing departments. Ekman has signed on as an adviser to the company. And last week, Emotient won a patent for a system which can analyze facial data while protecting the identity of the person whose face is being analyzed.

“We’re trying to find the right balance to move the ball forward and protect people’s right to privacy,” said Emotient’s chief executive Ken Denman during a visit to Boston last week. He said that consumers will reject emotion recognition if companies or governments try to create giant databases of our faces and emotions.

El Kaliouby agrees; in fact, she said Affectiva has turned down business from organizations looking to develop large-scale emotional surveillance systems that would track people without their consent. El Kaliouby said everyone in her company’s facial database gave permission to be included. And when Affectiva software is used in real-world applications, all captured facial data is deleted. For instance, the Hershey’s candy kiosk remembers each visitor’s face temporarily, to keep someone with a sweet tooth from coming back for more. But all data is wiped within 24 hours to protect users’ privacy.

The Affectiva phone apps don’t store data either. Actually, all they do is to instantly recognize expressions of disgust or fear or delight. But el Kaliouby hopes that in a few years, our phones and all our other gadgets will be sensitive enough to know how we feel, and smart enough to respond.

Hiawatha Bray is a technology reporter for the Boston Globe. E-mail him at h_bray@globe.com.
Follow Hiawatha on Twitter - Facebook - Google+