(Inside Science) -- When reading a novel, it’s common to let one’s mind wander into the imaginary: What might these characters look or sound like? Now, a new project uses algorithms to translate the emotions conveyed within a text into music that reflects the same sentiments.
TransProse, as the project is called, is a collaboration between Hannah Davis, a New York-based programmer and artist, and Saif Mohammad, a research officer at the National Research Council Canada in Ottawa.
The inspiration for the project came when Davis was in a master’s program for creative communication technology. As a class project, she was translating the grammar of novels into sound.
“Hemingway was really short and staccato,” she said. It led her to wonder how music could tackle a much bigger data set – emotion in novels.
That’s when she found the work of Mohammad, who had created a massive 14,000-word lexicon that associated words with emotions. Ice cream is associated with happiness, while tears linked up with sadness. Together, Mohammad and Davis worked to create an algorithm that could build music based on these associations. They presented their results in a paper at the European Association for Computational Linguistics Workshop on Computational Linguistics for Literature last month in Sweden.
The musical works they’ve made using the algorithm were inspired by great pieces of literature. Some of these works, including Joseph Conrad’s novella "Heart of Darkness" (highest emotion: fear; second highest emotion: sadness) and Harper Lee’s novel "To Kill A Mockingbird" (highest emotion: trust; second highest emotion: fear), are interesting and a little weird.
Davis and Mohammad point to the computer-created music based on the the novel "Peter Pan" as one of their favorites.
“It’s a simple little ditty, but it feels like 'Peter Pan,'” said Davis. They note in the paper that there is no one right way to create music from text. “It’s a little artistic,” said Mohammad. “What I like is that when you think about generating music it’s like a story…there are so many choices.”
The algorithm has trouble with complex storylines and allegorical books that have double meanings.
“'1984' and 'Brave New World' are difficult,” said Davis. “Even in 'A Clockwork Orange,' here we have a character going through the world doing terrible things but talking happily about them.”
The challenge of computers parsing real-life texts, where emotions run deeper than happy and sad, is one that will be tackled in the future. The researchers say that natural-language processing still has a long way to go before it hits the mainstream, and this project is just a start.
“The lexicon only tells you what the words are associated with, but a sentence isn’t just the sum of words,” said Saif. “We have these machine-learning algorithms, and [the technology] tries to generalize what it has learned. We’ve done lots of work in the last 10 years on sentiment analysis, but the emotion work is still in its nascence.”
Connecting text and music is no trivial task.
“We have been reading stories for thousands of years, but haven’t made a connection to music in an automatic way,” said Rada Mihalcea, an associate professor of engineering and computer science at the University of Michigan in Ann Arbor. She points out that the project is also important in drawing attention to the field, but it has implications for research in terms of connecting emotion analysis in text and music.
In the future, it may be possible to analyze emotions in text to figure out how a group of people is feeling, especially during an emotionally heightened situation like a natural disaster or other emergency. This information could potentially predict how people will react in these situations.
In addition, Mihalcea said understanding emotions could lead to better recommendations for products or services.
“Instead of recommending something based on a person’s gender and age, you could include the feelings a person has towards locations, targets or in a certain moment to create richer user profiles,” she said.
Davis and Mohammad say they’d like to create melodic lines for characters who change throughout the novel, or motifs for certain locations in a book. They may also make a mobile application or even incorporate the technology into an online bookstore.
“Right now you can see the front cover of a book, but what if you could listen to the emotional tone of the book and decide if you want to buy it,” Davis asks.
This story was provided by Inside Science News Service. Katharine Gammon (@kategammon) is a freelance science writer based in Santa Monica, Calif., and writes for a wide range of magazines covering technology, society, and animal science.
