Pandora and Last.fm: Nature vs. Nurture in Music Recommenders

Over the past week, there has been some blog talk (Fred Wilson, TechCrunch, David Porter) comparing music-recommendation services Pandora and Last.fm. I've been using both for the past couple months, making notes along the way with the idea that I'd eventually have something to say. That might as well be now.

Both services allow you to specify a favorite artist, based on which you immediately receive an Internet audio stream of similar music. When I tell people that this is possible—that you can have a personalized streaming radio station—most are astonished. So let's start by saying that what these and similar services do is cool. How Pandora and Last.fm do it is an interesting compare-and-contrast.

Nature versus Nurture
Algorithmically, Pandora versus Last.fm is something like the nature versus nurture debate. Taking the nature side, Pandora's recommendations are based on the inherent qualities of the music. Give Pandora an artist or song, and it will find similar music in terms of melody, harmony, lyrics, orchestration, vocal character and so on. Pandora likes to call these musical attributes "genes" and its database of songs, classified against hundreds of such attributes, the "Music Genome Project."

On the nurture side (as in, it's all about the people around you), Last.fm is a social recommender. It knows little about songs' inherent qualities. It just assumes that if you and a group of other people enjoy many of the same artists, you will probably enjoy other artists popular with that group.

Like Last.fm, most music-discovery systems have been social recommenders, also known as collaborative filters. Although much of the academic work in the area has focused on improving the matching algorithms, Last.fm's innovation has been in improving the data the algorithms work on. Last.fm does so by providing users an optional plug-in that automatically monitors your media-player software so that whatever you listen to—whether it came from Last.fm or not—can be incorporated into your Last.fm profile and thus be used as the basis for recommendations. Compared to relying on users to manually provide preferences, this automatic and comprehensive data capture leads to far better grist for the data mill.

A side note: In my years of analytics and data mining, a recurring theme is that better algorithms are nice but better data is nicer. That's because a large number of smart people have evolved the best data-mining algorithms for various scenarios; thus, further improvements tend to be incremental. By contrast, whatever data you happen to be using in a project has probably had no priming for analytical use. Thus, improving how you acquire, clean, and transform that data can have disproportionately large benefits. The catchphrase for the negative version of this is "garbage in, garbage out," although one could just as easily say, "the more signal in, the more signal out."

Surfacing New Artists
Pandora and Last.fm are both about helping people discover new music, so let's consider their approaches in terms of discovering truly "new" music—that is, artists who are just appearing on the music scene. If we assume that both services put new artists into their database at the same rate, Last.fm will be slower in surfacing them as recommendations. This is due to the "cold start" problem that afflicts social recommenders: Before something new can become recommendable, it needs time to accumulate enough popularity to rise above the system's noise level. In contrast, because Pandora is only comparing songs' inherent qualities—not who they're popular with—it should be able to recommend a new artist the first day that artist is in the system. That said, I wouldn't be surprised if Pandora did a little biasing of recommendations by popularity, which it measures as people use the service.

Partisans of Last.fm might retort that, in practice, Pandora will be slower at getting new artists and music into its database because of Pandora's classification bottleneck—that is, the time necessary for a Pandora employee to classify each song on the hundreds of musical attributes. With that bottleneck, Pandora can't just classify everything as it comes in the door. By contrast, Last.fm does not need to do manual classifications. With its software plug-in continually updating people's preferences, Last.fm has a virtual army of talent scouts constantly finding new things, which Last.fm can integrate into its database automatically.

(Leaky) Locked Loops
Pandora people might counter that Last.fm's army of talent scouts is compromised by its relative uniformity. That is, a social recommender tends to reward people who are like those who already use the system. If there are already many people in Last.fm with similar tastes to you, you'll get good recommendations; if not, then maybe not. And if you don't get good recommendations, are you going to keep feeding the system data? Probably not, and thus we have a self-perpetuating in-group/out-group situation. The result is a "locked loop," whereby a social recommender gets stuck in certain genres and styles.

But with a social music recommender, a truly locked loop is unlikely. The reason is "leakage": A population that shares the same core musical tastes will have enough variance in secondary tastes to allow for a continually expanding spectrum, albeit with much slower expansion in certain genres than others. Here's an example of the problem. When I checked Last.fm's similar artists to the reggae legend Bob Marley, first on the list was James Brown, followed by The Chemical Brothers, then Aerosmith. (If you're reading this well after January 30, 2006, beware that Last.fm's system is continually evolving, so the lists these links point to will probably have changed.) Other reggae acts appear further down, but the unlikely top choices suggest that Marley has been brought into the system more as a distant secondary choice than as a primary choice with other acts in his genre. A quick check of Aerosmith's similar artists confirms this: Marley is 41st on the list, way behind various likelier suspects.

While better non-reggae recommendations are easy to imagine for Marley, they probably won't appear until Marley's primary fans are better represented on Last.fm. Then the quality non-reggae choices can emerge from his core fans' secondary choices.

For the sake of comparison, when I put Marley into Pandora, I got something like a reggae radio station at first, which then drifted into other stuff over time.

Why versus What
Pandora is less subject to the echo chamber of overly like minds, but it has its own fundamental challenge in its reliance on matching songs' "genes." This rules out connections between songs or artists that don't fit Pandora's modeling and matching of musical qualities—which, in turn, puts enormous pressure on Pandora's specific approach to be correct. In other words, Pandora's success hinges on a theory, and a specific implementation of that theory, about why music recommendations work. By contrast, Last.fm simply describes what goes together according to its audience and then makes relatively simple inferences from that. So if there are hidden factors that Pandora isn't explicitly capturing, Last.fm is at least capturing them indirectly.

It's not hard to find cases where Pandora's approach runs aground, although the system's lack of transparency makes it difficult to know where the problem lies. For example, it's hard to explain Pandora's initial choices for Gary Numan (he of "Cars" fame). With Numan as the seed, Pandora gave me syrupy pop tunes by Orchestral Maneuvers in the Dark and the Human League. Yes, each artist's most famous material was from the same time and was primarily electronic, but the latter two really miss the Numan aesthetic, which is more like supercooled liquid metal than warm syrup. Pandora went on to do somewhat better, but not great, with subsequent tunes.

In comparison, Last.fm immediately delivered Numan-appropriate songs from Assemblage 23, Killing Joke, Kraftwerk, and Skinny Puppy, eventually drifting into less relevant territory. Still, Pandora partially redeemed itself with an inspired connection: "Out of Control" by Ric Ocasek (former leader of the Cars), an obscure cut from an artist that is far from obvious as a connection for Gary Numan.

Last.fm's Delivery versus Pandora's Promise
I raise the Numan example because it exemplifies my experiences with Last.fm and Pandora. Having used a wide range of artists as seeds, I found Last.fm better than Pandora at delivering songs that I liked or at least didn't feel compelled to skip, which is the most important thing when I'm listening while doing something else. The exception was when the seed artist had not hit critical mass in the Last.fm system, per the Marley example. Meanwhile, Pandora had more misses but was more likely to surface something truly out of left field, as with the Ric Ocasek example.

As a result, both Pandora and Last.fm have maintained a place in my music-listening world. However, ultimately I think Pandora has greater promise because it is far easier for Pandora to incorporate Last.fm's functionality than the other way around. This point is important because, just as with the nature versus nurture argument, the best answer is likely to involve elements of both camps. That said, Pandora's advantage comes at a significant cost to its business, with all the manual work it entails. At this point, Pandora is not delivering proportionally more benefit for that cost—which is why I used the word "promise" above.

Pandora Possibilities
The key to Pandora's changing the game is to take better advantage of its exclusive, hard-to-replicate metadata about music. Users may never be able to objectively judge the quality of recommendations among different services, but they can definitely tell the difference between services with unique ways of getting to recommendations. For example, I'd like to see Pandora expose some of its internal attributes as dials for the user to control. If I put in the singer Paul Westerberg (former leader of the Replacements), I'd like to tell the system to match more strongly along his lyrical style rather than by the fact he has a "gravely male voice" (which is one of the things Pandora said it was matching on). It's easy to picture many other creative uses of Pandora's metadata, both in terms of a recommender and other applications.

Finally, I wonder why Pandora continues to employ hundreds of attributes. In the world of modeling preferences, hundreds of variables typically can be consolidated down to a much smaller number with nearly the same predictive power. Typically, you start with a large number of variables as a kind of fishing expedition and then, over time, reduce the set down to those that are doing most of the work. The reduced set can be part of the original set and/or new variables derived specifically for predictive power. For a manual-labor-intensive business like Pandora's, being able to cut the number of variables in half (or a lot more) would help contain the costs. And if there's good reason not to consolidate attributes, I would still be wondering how to innovate in streamlining the production process just as much as how to innovate in the customer-facing part of the business.

Bowling or Batting?
A final thought: What Last.fm and Pandora do is hard, and the people who built these services deserve a lot of credit. Given the ambitious scope, it's easy to find examples where each of the services comes up short. However, it's worth considering what the yardstick should be. Should we expect spot-on recommendations like a pro bowler expects a strike every time? Or is this more like the baseball batter, who is happy to get a hit one in three times? Whatever the metaphor, the fact that these services do enough right to retain a substantial number of users is good news, because the features and quality will only get better. So when you try Last.fm and/or Pandora, be sure to give them enough time—and enough different starting points—to show their best stuff.

Comments

[John Hart]

Implementation is also critical - as of right now (the first time I've checked out either service), Pandora is loading & working beautifully whereas Last.fm cannot serve even static pages in less than 30 seconds.

For the moment, at least, this means that I'll play with Pandora but cannot even try out Last.fm ... which I may never visit again.

Steve, I'm sure you recall that Firefly originally started as Ringo, a collaborative-filtering based music recommendation site in the mid-nineties. It's taken a long time for these ideas to find an implementation that can meaningfully draw users.

[AMA]

I think your insight is spot on, Steve. I also think that implementation is extremely important, as noted above. I believe ease of use for the end user is just as important as the content you receive. Pandora is a very slick, web based app that any user can jump right into, immediately. Last.FM has a confusing interface that requires more thought, and they ask me to install their proprietary software on my machine. Simple and good will always beat complex but slightly better. (or in this case, arguably better)

I'm a musician myself whose spent many years trying to market my music. It's really tough in the music industry, especially for unknowns like my band 'Atomic Mint'. But the good news is that companies like Pandora and Last.FM in conjunction with the accessibility the Internet provides, will lead to a revolution of sorts.

I think it's also important to realize that there is lots of competition in this space today. Companies like Last.FM and Pandora are beginning to emerge as the first players in the personal digital radio market, but what model will really grab the public by the horns?

I just started a new music community with a few friends called BlueO2. We are still in Beta at the moment, please check us out! http://www.blueo2.com

What's clear is this much. There are many evolving technologies in the music space. That much, coupled with a general public dissatisfaction with traditional music distribution models tells me, change is coming. In fact, it's already here! Who ever can figure out how to simply give quality recommendations to music lovers AND solve the distribution piece of the puzzle at the same time will come out on top.

[chris]

I cannot do anything but agree to this brilliant review of these services. I haven't tried last.fm, but will definately be toying with it.

The Pandora drawbacks are precisely those that I encountered in trying to "teach" it my musical likes and dislikes. Often Pandora would give me something that is theoretically similar to what I seed it, but it lacks the inherent qualities I listen for in Music.

Hence I haven't really been able to construct a channel in Pandora that won't have me skip more than half the songs.

Somehow this all reminds me of Zen and the Art of Motorcycle Maintenance. The concept of what one percieves as Quality cannot be readily defined, and thus never captured by a mere algorithm. This means that in the short run Last.fm's approach makes more sense. In the long run I cannot but agree that it should be a property of Pandora's approach in order for Pandora to "learn" about what humans percieve as quality music.

[Cain Adamsson]

Hey!

I have used last.fm for quite a while now (http://last.fm/user/thul) and I'm quite satisfied with the service! they have given me a lot of new inputs and most of them have been right on target!

[Rhods]

Great article, which made me try out Pandora (I already use Last.fm)

So,I loaded up Pandora for the first time and decided to create a station based on JJ72 (fairly obscure band). Woo it found them!

But their profile says 'aggressive female vocalist'. Their lead singer is a MAN! Admittedly, a somewhat high-pitched singer, but still definitely of the male variety.

Then I try a more mainstream artist: Manic Street Preachers. Again their description is problematic. Although their last two albums fit the 'harmony, emphasis on production' profile, their first four albums are utterly different, with a noisy raw punk influenced sound.

You can use all the fancy coding that you like, but if the basic data is wrong then the whole thing is pointless!

[Anthony]

I'va also been using both services for a while. Where I've come out is that I'm using last.fm more than Pandora, but because of the "tag" based stations. One stupid example was over Christmas, I was able to get a great channel going on the last.fm service, playing a succession of tunes from Frank Sinatra's version on White Christmas through to Slade's "Merry Christmas Everybody". No musical link between the two that could get caught by Pandora but a link which makes perfect sense in terms of my personal use of the service.

[Guido Hosman]

I am afraid that commerce will have to say a lot about which model will prevail. My guess is the last.fm model. Record companies will pay to have "matches" inserted into the listeners streams, which they hope will create interest with some listeners. Once a song gets known, the last.fm model will do the rest in making it a hit.

[William Thomas]

Regarding the lock loop issue that you mention above, there is another trend which could possibly help Last.fm. Specifically, people with non-mainstream/eclectic tastes wanting to show how eclectic their tastes are.

[Shayaan Faruqi]

I am rather curious, why not include the best service there is like this right now -- Yahoo's Launchcast? I have been using that service for years and it's helped me to find lots of new bands that I would never have found otherwise.

Algorithmically, I can't say whether it's better or worse than any of the others... but it works, and quite well.

[StopIE]

Great article, lets hope that Pandora and Last's people are reading too :-)

[Andrew Strader]

I found that Pandora works better if you give it tons of seeds. I initially entered "Bjork", got one Bjork song and then 8 out of 10 songs that I did not like. Then I browsed Yahoo Launch for my other favorite artists and began seeding Pandora with them. After about 10 seeds, I started getting more songs that I liked.

[Chris Stiles]

I prefer pandora to last.fm, for a number of reasons; It loads faster, it falls over less often, and it doesn't keep playing the &^%^%"!" Dave Matthews Band no matter which artist I plug in.

[Seamus]

I read about both these programs and having tried them both I would agree with your prognosis, although I feel the music gene idea is a little dubious in itself.

http://www.liveplasma.com/

This is another interesting take on the music reccomendation idea.

[.:DataWhat?:.]

Wow, great analysis.

I found the same thing in Last.FM that trying contemporary artists was spectacular, but stuff like Reggae and Jazz was not nearly as accurate.

I did a similar analysis in December (also including Soundflavor as another option, and an AllMusic.com project called Tapestry that I've been involved with).

More Music Nerdery here:
http://datawhat.blogspot.com/2005/12/places-to-discover-music-online.html

[Sveinn R Jóelsson]

I noticed in your examples that you seem to focus on 'feeding' Pandora artists. I've found it much more rewarding (more hits than misses) If I just put in a collection of favorite songs. This makes sense to me since the metadata (to my knowledge) is focused on songs. How the people at Pandora derive/assign metadata on whole artists(a collection of songs?) is not known to me. Not that I know exactly how they rate songs but according to the website they do focus on songs.

I rather suspect that they rate artists like they rate me, i.e. create a user which has all the songs the artist is responsible for in the database as his/hers favorite songs. This can be a very versatile(depends on the artist) collection of song profiles indeed.

It may be that a social filter scheme is more suitable when dealing with artist or genre.

Im my experience using Pandora "stations" based on a collection of favorite songs within a genre will work rather well, where a few artists will yield more misses than hits.

So currently I create a few pandora "stations" which I browse through according to what type of music I'd like to listen to, and currently it's working very well for me.

Thanks for an excellent text on the subject (feel free to correct/ignore spelling and/or grammar errors since that is not my strong side).

[Michael Salsbury]

I'd never heard of Last.fm until I read this article, but you seem to have analyzed the differences very fully. I agree that Pandora would benefit from allowing users to dial in the influence certain musical "genes" should have on its recommendations. While I might, for example, like Phil Collins for his vocals, I might be more interested in Bowling for Soup on the basis of their lyrics or energy.

[Wrathe]

I have been a user of Pandora now for 6 months. At first I started using pandora just because the whole idea of a music "genome" is very interesting; and in fact a few years back a friend of mine and I had discussed the possibilities that could come from a system that could ascertain musical relationships based on the music itself, and not its popularity...especially in realtionship to music copyrights.

Anyhow, back to using Pandora, I began to realize that it really depended on the starting artist that you choose when you create your station and I firmly believe (like with your Newman expierence) that the closer you can get to a song that has exactly what your looking for by a certian artist, the better the system matches it with other artists.

Example...just putting in "The Cure" the system will pull a random song by that artist and start the "comparisons". However by actually stating "Fire In Cairo" as opposed to "lullaby", it comes up with different results (as well as some crossovers). That is where I think Pandora shines, because then at least 1 - 2 in 5 songs is a song you have never heard or an artist you never heard of.

If it was Last.fm, no matter what song I put in there by "The Cure", the whole of "the cure"'s songs are used, therefore you get the same recomendations, very rarely any new stuff...which is good when I want to hear all the usualls in more like a "underground late 70's early 80's" station.

Just my thoughts, I'm not saying that Pandora is better then Last.fm, just in my case I prefer a little mix up that Pandora seems to provide easier than Last.fm.

[Andy Farnsworth]

Another "service" similar to this that works on your own music collection is the Predixis MusicMagic Mixer (http://music.predixis.com/). This "fingerprints" your music by analysing the actual music itself rather then manually entering variables. This may or may not be better but it would certainly be less man power intensive. I have had good luck with them, however, it does have the downside of never introducing you to new artists. I don't mean artists who are new on the scene today, I mean anyone who is not already in your personal music collection. The advantage of MagicMusic Mixer is that you can take your mixes with you on your personal music player. I have suggested to Pandora that they allow users, especially paying users, to create playlists in addition to just streaming.

[evano]

Some very good points -- especially liked the bowlers v. batters and nature v. nurture analogies. But one set of opposing theories was left out: taxonomy v. "folksonomy".

Pandora and Last.fm are on opposite sides of this divide, too. Last.fm has implemented (unfortunately not very well) tagging of the music which members enter. Used properly, along with the massive amounts of data collected, this could conceivably provide some of the filtering which you find lacking in Pandora. Pandora, however, has built its service on a fixed taxonomy, and, if "lyrical style" is not in their taxonomy, you'll probably not get to filter on that attribute. Now, of course, their analysis of their "genetic" data may turn up some unsuspected correlation between attributes which accurately predicts and delivers musical style, but that is hardly assured.

Finally, even if Pandora could adapt Last.fm's data model more easily, Last.fm seems to be a more complete service, incorporating so many other aspects of community into the site that complement and extend the music recommendation core of the service.

Now, excuse me, but I've got the Replacement's "Can't hardly Wait" stuck in my head and I've gotta go listen!

[Ralph Gonzalez]

Thanks for that, I've been using Pandora for a while and even emailed them to suggest they go to a collaborative filtering approach -- I wasn't aware of LastFM. (I wrote a collaborative filtering system several years ago which I sold to IdeaLab! for it's short-lived RecoMentor website.)

Another point of comparison: Pandora is far, far more listenable than 99% of internet radio stations in my experience.

-Ralph

[Matt]

Just pointing out a quick mistake:
"(If you're reading this well after January 30, 2005, beware that Last.fm's system is continually evolving, so the lists these links point to will probably have changed.)"
It's not 2005 any more =]

[mriswyth]

I've been using Pandora for about a month now, but I started at a different place than you; I entered a SONG that I liked as the seed and worked from there. Pandora's ratings are based on a song's "genes", and supposedly doesn't seem to care about performer's relationships too much. So I think I would expect your Numan results if he ever recorded anything that was "warm syrup." Songs that you may not like by that artist get used as seed songs when you enter an artist name.

By entering single songs I have found that I have much less variation in my recommendations, and I can gradually increase the variation by adding slightly different single songs.

[Jasen]

I've been using Last.fm for about a week. I found it when I bought my Squeezebox from Slim Devices. There is a plugin to Slim's server software that will report your listening habits to Last.

Anyway, I went to Pandora and entered in two artists whom I like - "Daniel Amos" and "Phil Keaggy". These are both fairly popular artists in the Christian music arena. Pandora did not know either of them, while Last provides an extensive list of tracks played, albums, and similar artists.

So in my case, having musical interests that fall outside of what Pandora's controllers have chosen to enter, Last.fm wins hands down.

So, perhaps contrary to your "critical mass" example above, Last.fm works better because if your musical tastes are "esoteric" you at least have a chance that others with similar tastes have signed up, where with Pandora one is completely at the mercy of the operators.

- Jasen.

[Patrick Neville]

I completely agree with your viewpoint on Pandora. I do enjoy the loading time and fluid response in music playback. However, I also have had serious problems with the way it connects certain groups. With selections from Pearl Jam or Moe. I have found various times that I would have to skip 5 to 6 songs to get to one good song.

This can be extremely annoying when I am doing something else on my PC or in my office. Especially if Pandora plays 3 consecutive songs that match the taste of sound I want to hear, followed by 5 songs that have no connection to the style of music outside of certain guitar rifts or artist "influences". Last.FM is a great engine for people who want to listen to music while multi-task, but Pandora seems to be more of the gem at finding new artists.

There are times where Pandora has delivered several groups that are new to my ears and pleasing. Thoug h I agree some sort of reform for it is needed. I would prefer, maybe not everyone, to see a better pattern in its presentation of artists I enter in. If a music channel I define is supposed to have Pearl Jam as the main group, I would like to hear more consecutive tracks of that artist. I do like that it makes the connection from Pearl Jam to Mad Season (a band that consisted of members of Pearl Jam/Alice in Chains/Soundgarden collaboration), and I hope it still makes those type of connections.

In time, I can see Pandora expanding from user feedback, and I hope the constructive criticism your article provided as well as others helps in doing that.

[Svenjick]

Hi,

I find your article excellent, and very much to the point as for the observations you have made on the pros & cons of these two recommender systems. However as a musician, I must point out that neither of the two systems satisfy me truely (although I use them with pleasure, and am comvinced of their success for 95% of the listenners).

Pandora is somewhat frustrating, as you pointed out, since you cannot refine your liking of a song/artist, i.e. I like mild syncopations, but I don't like accoustic arrangements. As a musician I was frustrated by Pandora, not so much by it's poor choice (which I found often "good enough to listen to"), but because I quickly found I couldn't give accurate enough indications about what music I liked. The best way I found to imrpove this was by adding multiple songs and artists to a channel, rather than just having one artist that I critiqued a lot.
Last.fm (which I have just discovered through your article-thx) really has a HUGE cold-start problem. Having used it now for the whole afternoon (and having critiqued at least 50 songs) I am still getting 9 bad songs for 1 good one (according to my taste). However the tag-system works really well, which is not surprising: tags combine a mix of "user-preference" and "content value". Is this the future of recommendation systems? ...