Pandora and Last.fm: Nature vs. Nurture in Music Recommenders

Over the past week, there has been some blog talk (Fred Wilson, TechCrunch, David Porter) comparing music-recommendation services Pandora and Last.fm. I've been using both for the past couple months, making notes along the way with the idea that I'd eventually have something to say. That might as well be now.

Both services allow you to specify a favorite artist, based on which you immediately receive an Internet audio stream of similar music. When I tell people that this is possible—that you can have a personalized streaming radio station—most are astonished. So let's start by saying that what these and similar services do is cool. How Pandora and Last.fm do it is an interesting compare-and-contrast.

Nature versus Nurture
Algorithmically, Pandora versus Last.fm is something like the nature versus nurture debate. Taking the nature side, Pandora's recommendations are based on the inherent qualities of the music. Give Pandora an artist or song, and it will find similar music in terms of melody, harmony, lyrics, orchestration, vocal character and so on. Pandora likes to call these musical attributes "genes" and its database of songs, classified against hundreds of such attributes, the "Music Genome Project."

On the nurture side (as in, it's all about the people around you), Last.fm is a social recommender. It knows little about songs' inherent qualities. It just assumes that if you and a group of other people enjoy many of the same artists, you will probably enjoy other artists popular with that group.

Like Last.fm, most music-discovery systems have been social recommenders, also known as collaborative filters. Although much of the academic work in the area has focused on improving the matching algorithms, Last.fm's innovation has been in improving the data the algorithms work on. Last.fm does so by providing users an optional plug-in that automatically monitors your media-player software so that whatever you listen to—whether it came from Last.fm or not—can be incorporated into your Last.fm profile and thus be used as the basis for recommendations. Compared to relying on users to manually provide preferences, this automatic and comprehensive data capture leads to far better grist for the data mill.

A side note: In my years of analytics and data mining, a recurring theme is that better algorithms are nice but better data is nicer. That's because a large number of smart people have evolved the best data-mining algorithms for various scenarios; thus, further improvements tend to be incremental. By contrast, whatever data you happen to be using in a project has probably had no priming for analytical use. Thus, improving how you acquire, clean, and transform that data can have disproportionately large benefits. The catchphrase for the negative version of this is "garbage in, garbage out," although one could just as easily say, "the more signal in, the more signal out."

Surfacing New Artists
Pandora and Last.fm are both about helping people discover new music, so let's consider their approaches in terms of discovering truly "new" music—that is, artists who are just appearing on the music scene. If we assume that both services put new artists into their database at the same rate, Last.fm will be slower in surfacing them as recommendations. This is due to the "cold start" problem that afflicts social recommenders: Before something new can become recommendable, it needs time to accumulate enough popularity to rise above the system's noise level. In contrast, because Pandora is only comparing songs' inherent qualities—not who they're popular with—it should be able to recommend a new artist the first day that artist is in the system. That said, I wouldn't be surprised if Pandora did a little biasing of recommendations by popularity, which it measures as people use the service.

Partisans of Last.fm might retort that, in practice, Pandora will be slower at getting new artists and music into its database because of Pandora's classification bottleneck—that is, the time necessary for a Pandora employee to classify each song on the hundreds of musical attributes. With that bottleneck, Pandora can't just classify everything as it comes in the door. By contrast, Last.fm does not need to do manual classifications. With its software plug-in continually updating people's preferences, Last.fm has a virtual army of talent scouts constantly finding new things, which Last.fm can integrate into its database automatically.

(Leaky) Locked Loops
Pandora people might counter that Last.fm's army of talent scouts is compromised by its relative uniformity. That is, a social recommender tends to reward people who are like those who already use the system. If there are already many people in Last.fm with similar tastes to you, you'll get good recommendations; if not, then maybe not. And if you don't get good recommendations, are you going to keep feeding the system data? Probably not, and thus we have a self-perpetuating in-group/out-group situation. The result is a "locked loop," whereby a social recommender gets stuck in certain genres and styles.

But with a social music recommender, a truly locked loop is unlikely. The reason is "leakage": A population that shares the same core musical tastes will have enough variance in secondary tastes to allow for a continually expanding spectrum, albeit with much slower expansion in certain genres than others. Here's an example of the problem. When I checked Last.fm's similar artists to the reggae legend Bob Marley, first on the list was James Brown, followed by The Chemical Brothers, then Aerosmith. (If you're reading this well after January 30, 2006, beware that Last.fm's system is continually evolving, so the lists these links point to will probably have changed.) Other reggae acts appear further down, but the unlikely top choices suggest that Marley has been brought into the system more as a distant secondary choice than as a primary choice with other acts in his genre. A quick check of Aerosmith's similar artists confirms this: Marley is 41st on the list, way behind various likelier suspects.

While better non-reggae recommendations are easy to imagine for Marley, they probably won't appear until Marley's primary fans are better represented on Last.fm. Then the quality non-reggae choices can emerge from his core fans' secondary choices.

For the sake of comparison, when I put Marley into Pandora, I got something like a reggae radio station at first, which then drifted into other stuff over time.

Why versus What
Pandora is less subject to the echo chamber of overly like minds, but it has its own fundamental challenge in its reliance on matching songs' "genes." This rules out connections between songs or artists that don't fit Pandora's modeling and matching of musical qualities—which, in turn, puts enormous pressure on Pandora's specific approach to be correct. In other words, Pandora's success hinges on a theory, and a specific implementation of that theory, about why music recommendations work. By contrast, Last.fm simply describes what goes together according to its audience and then makes relatively simple inferences from that. So if there are hidden factors that Pandora isn't explicitly capturing, Last.fm is at least capturing them indirectly.

It's not hard to find cases where Pandora's approach runs aground, although the system's lack of transparency makes it difficult to know where the problem lies. For example, it's hard to explain Pandora's initial choices for Gary Numan (he of "Cars" fame). With Numan as the seed, Pandora gave me syrupy pop tunes by Orchestral Maneuvers in the Dark and the Human League. Yes, each artist's most famous material was from the same time and was primarily electronic, but the latter two really miss the Numan aesthetic, which is more like supercooled liquid metal than warm syrup. Pandora went on to do somewhat better, but not great, with subsequent tunes.

In comparison, Last.fm immediately delivered Numan-appropriate songs from Assemblage 23, Killing Joke, Kraftwerk, and Skinny Puppy, eventually drifting into less relevant territory. Still, Pandora partially redeemed itself with an inspired connection: "Out of Control" by Ric Ocasek (former leader of the Cars), an obscure cut from an artist that is far from obvious as a connection for Gary Numan.

Last.fm's Delivery versus Pandora's Promise
I raise the Numan example because it exemplifies my experiences with Last.fm and Pandora. Having used a wide range of artists as seeds, I found Last.fm better than Pandora at delivering songs that I liked or at least didn't feel compelled to skip, which is the most important thing when I'm listening while doing something else. The exception was when the seed artist had not hit critical mass in the Last.fm system, per the Marley example. Meanwhile, Pandora had more misses but was more likely to surface something truly out of left field, as with the Ric Ocasek example.

As a result, both Pandora and Last.fm have maintained a place in my music-listening world. However, ultimately I think Pandora has greater promise because it is far easier for Pandora to incorporate Last.fm's functionality than the other way around. This point is important because, just as with the nature versus nurture argument, the best answer is likely to involve elements of both camps. That said, Pandora's advantage comes at a significant cost to its business, with all the manual work it entails. At this point, Pandora is not delivering proportionally more benefit for that cost—which is why I used the word "promise" above.

Pandora Possibilities
The key to Pandora's changing the game is to take better advantage of its exclusive, hard-to-replicate metadata about music. Users may never be able to objectively judge the quality of recommendations among different services, but they can definitely tell the difference between services with unique ways of getting to recommendations. For example, I'd like to see Pandora expose some of its internal attributes as dials for the user to control. If I put in the singer Paul Westerberg (former leader of the Replacements), I'd like to tell the system to match more strongly along his lyrical style rather than by the fact he has a "gravely male voice" (which is one of the things Pandora said it was matching on). It's easy to picture many other creative uses of Pandora's metadata, both in terms of a recommender and other applications.

Finally, I wonder why Pandora continues to employ hundreds of attributes. In the world of modeling preferences, hundreds of variables typically can be consolidated down to a much smaller number with nearly the same predictive power. Typically, you start with a large number of variables as a kind of fishing expedition and then, over time, reduce the set down to those that are doing most of the work. The reduced set can be part of the original set and/or new variables derived specifically for predictive power. For a manual-labor-intensive business like Pandora's, being able to cut the number of variables in half (or a lot more) would help contain the costs. And if there's good reason not to consolidate attributes, I would still be wondering how to innovate in streamlining the production process just as much as how to innovate in the customer-facing part of the business.

Bowling or Batting?
A final thought: What Last.fm and Pandora do is hard, and the people who built these services deserve a lot of credit. Given the ambitious scope, it's easy to find examples where each of the services comes up short. However, it's worth considering what the yardstick should be. Should we expect spot-on recommendations like a pro bowler expects a strike every time? Or is this more like the baseball batter, who is happy to get a hit one in three times? Whatever the metaphor, the fact that these services do enough right to retain a substantial number of users is good news, because the features and quality will only get better. So when you try Last.fm and/or Pandora, be sure to give them enough time—and enough different starting points—to show their best stuff.

Comments

[Brendan]

Curiously, Steve didn't notice the most conspicuous thing of all about Pandora - how ridiculously AMERICAN all the music is !!!

That's the real flaw in the Pandora model - they can only think of themselves, their own tastes, and their own extremely limited worlds.

As good as Pandora is, it's deeply flawed until they start to understand and address the inexhaustibly rich music universes outside Fortress America.

Get it?

[bontzy]

that's funny, pandora.com isn't working for me "it's taking longer than expected ... bla bla bla" (and i tried several artists and tracks) whilst last.fm is and has always been working for me (they had a one day streaming downtime because of an upgrade)

[eric casteleijn]

The Bob Marley example is striking, but I do not think the problem is lack of critical mass, but overabundance of critical mass, so to speak. Since Bob Marley is the one reggae artist hugely popular among people who wouldn't touch most other reggae with a 10 foot pole, the recommendations actually make a warped kind of sense: Bob Marley is mainstream, and so is James Brown, but not so much, for instance, Lee Perry.

That is also why The Beatles have a tendency to show up everywhere, at least on other hugely popular artists' entries, like Coldplay and U2.

I've given up worrying about it: last.fm has features which let you get around all that very easily: you can set the stream to only play music you've never heard before, and there's an obscure vs. popular slider.

And re: John Hart: An unfortunate coincident had Last.fm upgrading its servers yesterday, which may explain your outage.

[Simon Snook]

I am a user of Last.FM (formally Audioscrobbler) and I did try Pandora whilst it was in beta. I have to say the big point of Last.fm is to create a community (user buy-in I think they call it), where I keep going back to Last.FM to check the groups/forums and nose around the site. I have the choice to check out the charts and see what my (musical) neighbours are listening to.

I do agree on the quality of data being submitted to Last.fm it looks like a headache to people that run it and they ask that users run all their tracks trough the Musicbrainz software which straightens out any incorrect metadata.

I will continue to use Last.fm I like what they do and they seem like a nice bunch. Oh and it's free.

[Kyle M]

It's worth noting that Last.fm gives paying members priority when it comes to loading pages and delivering streams. There was also a radio upgrade going on around the time the first comment was left according to their news page.

I prefer Last.fm for two main reasons:

1) I've never stumped it with an artist. No matter how obscure I've tried to be somebody else has already listened to that band and put them in the system. While Pandora frequently responds to my inputs by saying "who?"

2) The social aspects. In last.fm I can read the journals of my neighbors (users with similar tastes according to their algorithms) and participate in forums based on my general criteria. Such as "Radiohead is not our top artist" or "we love Radiohead" or even "Firefox users." And each of those groups get their own recomendations and their own radio stations.

So for your Bob Marley example you could've listened to the radio station for one of the Bob Marley groups or the radio station attached to a related tag such as "Marley" or "reggae"

[walter]

I like the randomness of Pandora--I've been using it for about three months. My tastes happen to focus on bands that aren't widely known so I think pandora is a better fit for me--Plus I have heard some awesome music I would have never heard on Lastfm just because it's so "underground".

I don't have anything super important to say except that you wrote "january 30th 2005" instead of 2006. You might wanna change that.

Liked the article, it was a good read.

[twohorses]

One thing I've noticed about Pandora, is that it does much better when you use specific songs as seeds rather than artists.

[Ant]

Good article. Have been using last.fm for a while now (with no problems at all in page loading times). Might check out Pandora too after this.

Btw, you put 2005 in there (just after the bob marley link) when I assume you meant 2006. Might want to fix that. :)

[Barry Burton]

I think you are missing the reason many people use Last.fm -- not primarily for music recommendation, but for its data collection and presentation abilities. Each user gets a public page where the music they like is presented within an attractive, well-designed web page. Many people would like to add information on what music they are currently listening to as well as music they most like to their blog, but are either too lazy or lack the technical competence to do so. Last.fm makes this easy and automatic.

[phil miller]

Steve-
this is the first time i have been to your blog. based on this post i will be sure to look at some previous posts and checking back for the stuff in the future.

I just recently have been exposed to last.fm and Pandora (listening to Pandora right now), I found Pandora to be much better at playing songs that "matched" what i was looking for. I would not say that it has exposed me to much new music though. I am not sure about all the techinical stuff that goes on behind the curtin to make it all work, but i do know that if i want music to play with out much work from me to get songs that i will not want to skip i will listen to Pandora. It works for me.

Thanks for the comparison and all the technical background.

[Michael C Murray]

I tried Pandora but ditched it as too many of the 'predictions' were misses for me.. Their algorithms didn't seem to work very well for some electronica subgenres. I will try last.fm.

Thanks for the write-up though, nicely done.

[T. Roberts]

AMG, the people behind allmusic.com have just launched a similar service demo called "Tapestry" at http://tapestry.allmusic.com/ . Besides doing such mundane tricks as finding similar songs, albums, or artists, Tapestry uniquely creates playlists bases on "Themes", "Tones", and "Styles". One can choose from giant lists of Themes such as "Road Trip", "Spring", "Dinner Ambiance", and "Divorce". After playing with various services extensively, I can say with confidence that, in my humble experience, nothing even gets close to Tapestry in the accuracy and relevance of the playlists that are generated. Tapestry is getting around the trials, tribulations, and inaccuracies of other approaches such as collaborative filtering by using the Allmusic.com editors to hand review each song using a list of more than 6200 descriptors (according to the Tutorial documentation). Hopefully AMG will come out with a commercialized service for on-demand streaming radio based on Tapestry since it would instantly make subscription streaming music services hugely popular.

[John Bailo]

Well, some of this has to do with the concept of "new" and that has to do with the "state" or "intent" of the listener.

For example, when I was a college dj, in 1980-82, and punk/new wave was cool, I liked to find 60's classics that mapped back to similarities. So, I played things like the "nuggets" collection and garage band rock opposite new wave.

Today, a 13 year old would find classic Elton John "new" and a system that mapped back from chronologically new, to all things similar and maybe obscure might be more what the listener was looking for.

[Mike S]

Each service seems to answer a different question. Pandora answers "What other songs sound like this song/arist?". Last.fm asks "Assuming my taste in music is not based just on the inherent qualities of music, but on my personal demographics (age, where I grew up, where I've lived, etc.), what other songs are associated with the demographics of my life?".

Sometimes people a prefer song because it reminds them of other songs, while other times they like a song because it reminds them of other times/people/things/experiences.

[David Thatcher]

Well... Last.fm requires a LOT of software, and doesn't work with my music store of choice, Rhapsody. Not only that, the Last.fm people actually DISCOURAGE writing plugins for streaming media services.
OTOH, Pandora is beautifully simple, what a breath of fresh air compared to trying to get Last.fm to work. I've been listening to Pandora for an hour and can't get myself off of it to go do stuff that needs to get done!

[Marti Holguin]

You really hit my main issue with Pandora!
They need to expose more of the prperties.
If I give a song a Thumbs Down because of lyrics or vocal style, but really liked the beat or the melody, there is no way to distinguish, thereby diluting the value of the "Personalized Station"

[Steve Krause]

Update: Per several comments, lest I be accused of living in the past, I fixed a date reference to 2005 that was supposed to be 2006. Also, in addition to this post's comments section, there are more than 100 comments on Slashdot:

http://slashdot.org/articles/06/02/01/0553233.shtml

Thanks for the kind words and interesting opinions.

[Keith Spragg]

Just a quick note - you've written the date wrong, I presume - 2006, not 2005, right?

Personally, I've been using Last.fm as a tool, mostly because I like the fact it records my tunes, more than using it as a recommnedations engine - I have VERY wide rangnig tastes, so find Pandora too closed.

Great post though!

K

[Tom Conrad]

Steve this is an incredibly thoughful and articulate post. Thanks for taking the time to consider all of this and contribute to the conversation that Fred kicked off a couple of weeks ago.

We've just tonight pushed out a big new Pandora release, so I've been very caught up in the logistics around that. As a result, I've have been kind of watching all of this from the sideline. Maybe that's for the best; it's been fascinating to watch and learn from the passions, observations, and critiques that are fueling the discussion.

Suffice to say that while I'm the CTO at Pandora, I'm also a fan of what the gang at last.fm are doing; I'm a paying last.fm subscriber and I maintain a musical profile there. At least in my case, the services scratch two very different itches.

Thanks again for the insightful post.

Tom
CTO @ Pandora

[Mike J]

Very nice breakdown and comparison. I personally use pandora, and have been for quite some time. I think there are two distinct sides to the classification system: The good side is that it is genre ignorant. Sometimes I really like that, since I listen to most musical styles. The bad side is that it is genre ignorant. Some genre combinations just aren't suitable on the same station. It would be nice if they added "omg no more of this genre on this station" buttons for rating. I've also found that as I add seeds to the stations they get become appropriately refined.

I would also say that I think the commercialization aspect could end up being a bad thing. With pandora's algorithmic model, you're more likely to get things fed off pure data. With last.fm, I could see the easier potential for preference seeds to be bought (kinda like they are for radio now) - which means that in the end you're really just hearing what record companies want you to.

Overall, I've been much happier with pandora and discovered good new music from it. After reading this I will give last.fm another chance, but I don't see pandora leaving my regulars (especially if they get to the point of opening an API for me to develop against).

[oscar]

Here's another music recommendation service which I think its worth to mention it, its called Foafing the Music http://foafing-the-music.iua.upf.edu

BTW, I tried (in the demo box) with artists: Bob Marley, Gary Numan and Paul Westerberg and the results are quite interesting to have a look at!

[Bo Williams]

I wonder how many comments to this entry were placed by viral marketing firms- a good half of them triggered my B.S. filters.

[zrenneh]

last.fm works fantastically for very fringe artists. Bob Marley was used effectively to make a point, but everyone's heard of him. Hence, since reggae is under-represented, you get people who mostly listen to aerosmith, and some Bob Marley on the side. However, if you search for Autechre you get a very effective selection including Aphex Twin, Squarepusher and Boards of Canada. People who aren't into electronic music probably won't have heard of Autechre, so the results are 'undiluted'.
As long as last.fm keep a Google-style independence (no preference-for-money), they will interest me...but have they already sold-out to the Dave Matthews' Band?
Pandora don't interest me as much: their classification system is too obvious. I love music which is non-obvious and hard to second-guess.
I would love it these tools could suggest interesting comparisons: for example Amy Winehouse's October Song and Sarah Vaughan's Lullaby of Birdland. I have no idea how this could be done. I guess that's what real DJs are for!

[Howard Dickins]

I agree with Andy Farnsworth on his mention of the Taxonomy/Folksonomy comment. After becoming a fan of the tagging idea as modelled by Flickr and Del.icio.us - I went on an active search of musical folksonomies - and so I was sort of glad to discover Last.fm. But also a little disappointed - because the tagging feature only seems bolted on as an afterthought. I suspect that most users don't bother tagging things with any diligence. And who can blame them - it's not a very prominent feature of the site - nor is it very easy to navigate to enter lots of tags for a diverse playlist. What I wanted & still want - is a way of searching for music by a large variety of criteria: time-signature, instruments, country-of-origin (and language?), subject-matter, and of course genres and sub-genres... (How else can I find a Hungarian Rock-n-roll Czardas Ballad featuring Hurdy-Gurdy? (Ok - there probably aren't too many of those - but if they're out there - then I do want to find them!)

[Dan]

I've taken a slightly different approach to comparing the two services. I review them both from an end-user, web-developer, and entrepreneur perspective on my blog here - http://www.streampad.com/blog/?p=24 . In the end, I prefer last.fm