New Song Aggregator: 'Black Box' Could Redefine the Search for Music

Voice of San Diego

'Black Box' Could Redefine the Search for Music
By DAVID WASHBURN

Wednesday, April 15, 2009 | Luke Barrington wanted a good example of "funky music with a horn section for listening to at a party."

The University of California, San Diego doctoral student typed the search term into his computer program and it told him to give James Brown's "Give It Up Or Turn It Loose" a try. It also suggested "Onyoghasayo" by Shankin' Pickle and "Super Freak" by Rick James.

For years now, computer-based song aggregators like Pandora and Last.fm have allowed people to compile playlists based on song names or artists. But Barrington's computer at the California Institute for Telecommunications and Information Technology takes things to a new level.

The "black box," as Barrington's professor Gert Lanckriet calls it, can listen to a song and decide, for example, whether it is a romantic song or a dance song. It can figure out what genre it falls into, and know what instruments are being played. And it puts the student/professor team at the forefront of the burgeoning field of machine listening.

"You don't need the artist name; you don't need the song name, all you need to say is 'I want some scary Halloween music,'" said Lanckriet, an electrical and computer engineering assistant professor at UCSD's Jacobs School of Engineering.

Machine listening is part of the latest evolutionary cycle in how people find, compile and listen to music. Personalized playlists, rooted in the mix cassette tapes of the 1980s, have, among younger audiences, largely taken the place of albums put out by record companies.

This change in listening habits has contributed to the destruction of the big record company business model, and to the rise of song-by-song sellers like iTunes and the song aggregators. These sites employ technology that allows users to essentially create their own radio stations derived from what is considered the genetic makeup of songs they enjoy.

Pandora was born out the Music Genome Project, an effort that began in 2000 to breakdown songs to their elemental parts. Musicians were enlisted to "tag" parts of songs such as the type of vocalist, guitar sound and rhythm structures. A typical rock song has around 150 tags, while a jazz song might have 350 tags.

With Pandora, a listener can type into a search engine a song they enjoy, say Bob Dylan's "Mr. Tambourine Man," and the program will create a playlist of songs with similar qualities to Dylan's masterpiece. To avoid copyright infringement, Pandora licenses the music only for streaming over the internet. Apple's iTunes recently added a feature called Genius that performs a Pandora-like function on a person's iTunes library.

The beauty of these technologies, from Lanckriet's and Barrington's perspective, is that it exposes people to artists they likely would have otherwise never heard of, and gives largely unknown artists an audience they otherwise wouldn't have.

Lanckriet and Barrington met for the first time at a barbeque put on by the Jacobs School at the beginning of the 2005 school year. The two immediately hit it off, and, with a little help from a keg of beer, soon reached what Lanckriet calls "a slightly higher level of society."

The evening ended with Barrington inviting Lanckriet to join his band, Audition Laboratory. Taking the Pandora technology a step further was a regular topic of conversation among the members of the band.

"We would talk about how frustrating it is for independent artists to get noticed and how my mom would never use any automated services online because she can't remember artists and song names," Lanckriet said. "Very often people who can describe the kind of music they feel like listening to, but they don't know a specific artists, song name or album that satisfy that desire."

So they set about writing a computer program that could listen to music and learn how to breakdown songs like the musicians who work for Pandora do. And then go a step further and build a Google-like search engine that would allow people like Lanckriet's mom to search for the music they like.

They started by writing code that would allow the computer to break the sound waveform down by its frequencies. A female singer, for example, will produce higher frequencies, while a bass guitar and kick drum will produce lower frequencies.

The next step was a bit trickier. The computer had to be taught how to differentiate between romantic songs and punk songs, and so on. Lanckriet describes the computer's learning process as similar to a human's. "A human learns by being given examples, and then extrapolating to the unknown. We are getting the computer to do the same thing."

But they needed good examples -- songs, for example, that most everyone agrees are romantic or punk or elicit certain types of emotions or images. To start, they went the Pandora route, and paid UCSD undergrads to tag 500 songs. But lacking a multi-million dollar budget, they needed another way to provide the computer with more examples.

So they came up with what is best-described as a word/song association game in which players assign words to describe the songs they hear. Players score points based on how similar their answers are to those of other players. The scientists launched the game, called Herd It, on Facebook this week. The goal is to collect about 1 million word/song associations.

"After the learning process is complete, we basically have a fully-trained black box that now listens to any song out there and basically overnight find millions of similar songs," Lanckriet said.

The black box is still in demo form, and the student/professor team and associate Damien O'Malley are looking for a partner to help with the further development of the program, as well as help with licensing costs associated with web streaming. They feel it could be valuable to any number of potential partners, with companies like Pandora and Apple topping the list.

"The reason we started this project was because it is academically interesting," Barrington said. "But obviously there are a lot of applications in the outside world."

Westergren, Pandora's founder, said machine listening technologies could revolutionize this corner of the music industry much like automation technologies have changed the face of industries from cotton picking to auto making. But he has not been impressed with the technology he has seen so far. He has not seen Lanckriet's and Barrington's invention.

"The promise of machine listening is you can discover a lot more music," Westergren said. "But our experience so far is that there is no substitute for a trained human ear."

Eric Johnson, president of San Francisco-based Wolfgang's Vault, a website that keeps an archive of live concerts and sells music memorabilia, is also skeptical about computers completely taking over the music suggestion business.

"People used to like listening to the radio because you had a DJ you trusted," Johnson said. "Will the computer do that effectively? I don't know."

Please contact David Washburn directly at david.washburn@voiceofsandiego.org with your thoughts, ideas, personal stories or tips. Or set the tone of the debate with a letter to the editor.