A New Smartwatch App Can Predict Moods and Teach Social Skills

smartwatch mood wearable MIT social
Researchers from MIT used a research-grade Samsung smartwatch coupled with an AI system to detect the tone of a conversation. Jason Dorfman, MIT CSAIL

Researchers have developed a smartwatch integrated with an artificial intelligence system that can serve as a kind of coach in social situations, particularly for people who suffer from social anxiety or Asperger syndrome.

A paper published Wednesday by researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) describes how the wearable AI detects the tone of a conversation through audio and physiological data in order to determine the mood of the speaker. This information can then be used to better understand social situations.

"Technology has done a lot to connect people but even though it helps us communicate, it hasn't done much to improve those communications," Mohammad Ghassemi a PhD candidate who co-authored the paper, tells Newsweek.

"This system can help people with anxiety or conditions like Asperger's understand which moments in a social interaction led to positive outcomes and which led to negative outcomes. Ultimately, it could be used by everyone to improve communication."

Ghassemi and co-author Tuka Al Hanai used a research-grade specialist smartwatch—the Samsung Simband—in order to track certain physiological data from the study's participants, such as movement, heart rate, blood pressure and skin temperature.

Combining this data with audio analysis of the speaker's tone, pitch, energy and vocabulary, the system then provided a specific sentiment score every five seconds during a conversation to determine whether it was happy, sad or neutral.

It is the first time the emotional tone of a conversation has been measured and classified in real-time, according to Ghassemi.

In the future, the researchers hope to be able to integrate the system into more common smartwatches, such as the Apple Watch.

"The team's usage of consumer market devices for collecting physiological data and speech data shows how close we are to having such tools in everyday devices," says Björn Schuller, professor and chair of Complex and Intelligent Systems at the University of Passau in Germany, who was not involved in the study but works in this field of research. "Technology could soon feel much more emotionally intelligent, or even 'emotional' itself."