Text and illustration by Susanne Gold

Spoilt for choice: the music selection on Spotify seems almost endless. Apparently, Spotify wants to help with the music selection – depending on the mood of the listener – in the future. The music streaming service Spotify recorded 155 million paying subscribers worldwide in the fourth quarter of 2020. With ad-supported users, the Scandinavian company has 345 million monthly active users. Podcast hours have also doubled since 2019. The company recently filed and received approval for a patent for a new type of speech analytics software.

Speech and environment analysis allow conclusions to be drawn about mood

This tracking technology is said to make it possible to detect the “mood, age and gender” of users, and to learn details about the mood of app users through the rhythm as well as intonation of their speech. According to Spotify, this could make personalized audio recommendations possible in the future. In addition to the user’s speech, ambient sounds are also analyzed. This is to recognize whether a user is alone or in company.

How does the analysis software work?

Behavioral variables such as the user’s mood, his or her preferred music titles and demographic data are included in the database. These are first used to generate a profile that reflects the music lover’s personality as accurately as possible. Based on the personality traits, it is then possible, according to the company, to advertise personalized recommendations – music and podcasts. In future, the world will be dealing with software that gets deep into the listener’s head: in addition to classic target group characteristics, the emotional state, gender, age and accent are queried and perceived. Based on all the recorded characteristics, the software creates suitable content. The company explains that it is quite common to implement functions in media streaming applications that enable personalized media recommendations. Now, however, the taste attributes are not only queried from the respective user, but determined directly via the speech recognition software.

Software that combines Markov’s mathematics with emotion studies

According to the patent, acoustic information is combined with the “Hidden Markov Model Architecture” so that the software can recommend personalized playlists. The Russian mathematician Andrei Andreyevich Markov invented a model of probability calculation that works with “unobserved states”. Here, in addition to the user’s voice pitch, background noises and conversations of other people, but also natural noises, such as in a park, or technical noises, for example from the office, flow into the probability model. All emotional information is mapped in a structure based on one designed by the emotion researcher W. Gerald Parrott. This means: personalized audio recommendations are created based on the listener’s previous requests, as well as on their rating history, their links to associated profiles, so also on the preferences of friends and colleagues

The perfect playlist: Is every recommendation also correct?

It is doubtful that an app can actually always make the right recommendation. This can be described using the example of mourning. There are events in life that require the ability to grieve. Numerous psychological studies prove that going through crises and grief is an indispensable prerequisite for becoming crisis-proof. In order to develop the capacity to grieve, a special kind of memory work is necessary, which includes the revival of past behaviors, feelings and fantasies. Young people in particular need to learn this. In addition, although Spotify is heard around the world, emotions are dealt with very differently in the regions of our planet. For example, mourning ceremonies in Mexico seem decidedly cheerful to Europeans. In addition to privacy concerns, it can be assumed that this app can correctly interpret emotions, but is not able to really react appropriately.

Some playlists could be the wrong playlist after all

It is unclear if and when this software will be put into operation. A company spokesperson said that the software is only one of many patent applications and does not necessarily have to become a product: You can read more details about the patent here.