Data privacy Emotion recognition technology could transform retail advertising

A Berlin startup has developed software that can analyze your emotions through facial expressions. The retail and advertising industries are interested, but data privacy experts are concerned.
What you see is what you get.

The surveillance vision from the movie "Minority Report," where the irises of every passerby are scanned in public spaces, is no longer a distant future phenomenon. It's already a reality at the Südkreuz railway station in Berlin, where a facial recognition system is being tested to track down criminals. Now companies are going a step further, and making use of emotion recognition, too.

Cameras and software are already being used in shops, supermarkets and advertisements to analyze who is looking at what, for how long and how they feel from doing so. The software is not only able to determine the viewer's age and gender, but also his or her emotions, and whether an image is repulsive or pleasant. The technology is becoming ever more accurate, as companies around the world further develop it.

Pyramics is among them. Its founder, Thomas Fehn, is a pioneer in the field in Germany. His Berlin-based startup has developed a facial analysis software that can reveal any subject’s age, gender and emotions.

An algorithm fed with thousands of pieces of facial data learns how "to distinguish the faces of men and women and to recognize their age by means of fixed characteristics," Mr. Fehn told Handelsblatt. “It is now even possible to capture other metrics such as a beard, glasses or clothing styles.”

It's important to ask whether measuring emotion might lead to manipulation.... Alexander Niethammer, Data privacy lawyer

The retail industry is interested. According to psychologist and Nobel Prize winner Daniel Kahneman, most of our buying decisions are based on emotions. Emotional decoding, a process developed by psychologist Paul Ekman in the 1970s, makes it possible to understand exactly how customers react to advertisement. Mr. Ekman, known as the "Human Lie Detector" and the real life inspiration for the TV show "Lie to me," coded facial movements in order to draw conclusions about human emotion.

Software can now identify these conclusions in a flash.  "Digitalization makes it possible to decipher emotions in milliseconds," said Andrea Gröppel-Klein, director of the Institute for Consumer and Behavior Research at Saarland University.

Pyramics' customers include large outdoor advertising space providers as well as electronics and food retailers that want to know how effective their digital advertising is, according to Mr. Fehn. Since the start of this year, the company has been offering systems that allow retailers to identify an unprecedentedly clear idea of visitor demographics. The aim is to one day “dive into the field of Big Data and measure flows of people in cities,” said Mr. Fehn. "Shaping demographic change with outdoor advertisers in real time for urban planning – now that would be something."

Such technology, Mr. Fen admits, may worry some Germans weary about data privacy for historic reasons. But he argues the software is ethically sound – Pyramics doesn't store any information and facial images must be deleted or pixelated by law. Emotions belong to no one because they can be attributed to anyone, he says.

German authorities have effectively agreed. Bavaria’s State Office for Data Protection Supervision found that facial analysis in locations of the supermarket chain Real was in line with data privacy regulations. Images were kept for just 150 milliseconds, with only the bulk metadata being transferred. Customers were informed that video surveillance was taking place, and that was considered enough. Nevertheless, the supermarket chain stopped the trial after it triggered a lot of bad publicity.

In Germany, emotion recognition is still an unregulated grey area. "Of course, the argument of anonymized data is intended to put an end to the debate," said Alexander Niethammer, a data privacy lawyer with the law firm of Eversheds Sutherland. "But it's also important to ask whether measuring emotion might lead to manipulation in the end,"

He added there are already legal problems with the parallelism of emotion recognition and video surveillance. "If no personal data is collected, data protection law doesn't apply and the company doesn't have to point this out. But if I don't know what is happening, I can't assert any individual rights to data protection either."

It will be up to the courts to create an appropriate framework when the time comes, Mr. Niethammer said.

Johannes Steger is a Handelsblatt correspondent covering companies and markets. To contact the author: [email protected]