My buddy IB sent this article to me…very interesting. Netflix is running a contest for data crunchers and offering $1M to anyone (or any team) that can beat their current recommendation system by 10%. One of the leaders is a psychologist working by himself who is looking less at raw data and more at human nature.
One such phenomenon is the anchoring effect, a problem endemic to any numerical rating scheme. If a customer watches three movies in a row that merit four stars — say, the Star Wars trilogy — and then sees one that’s a bit better — say, Blade Runner — they’ll likely give the last movie five stars. But if they started the week with one-star stinkers like the Star Wars prequels, Blade Runner might get only a 4 or even a 3. Anchoring suggests that rating systems need to take account of inertia — a user who has recently given a lot of above-average ratings is likely to continue to do so.
I think this guy is onto something, and I’d like to see this move a step further. Associating movies using k nearest neighbor is relatively straightforward, but attacking the other side of the equation (the viewer) is a lot tougher. Here’s an example…
“The Outlaw Josie Wales” is one of my favorite movies, but that doesn’t mean that an algorithm could spit out a bunch of westerns and give me something I like. Clint Eastwood movies wouldn’t do it either, but it would be a little closer. The real way to suggest movies for me would be to look at some other factors that aren’t so obvious. You need to be able to draw conclusions from my other favorites–“Fight Club”, “Pulp Fiction”, “Smoky and the Bandit”, and “Swingers”. You may peg all of these as “guy movies”, but that doesn’t mean I’m going to like “Gladiator”. In fact, I hated “Gladiator”. A movie like “Thelma and Louise” is a much better suggestion for me than “Gladiator”. Why? Because it is much more quotable, and that’s something my favorite movies suggest that I like.
Just an example, but that’s the direction we’re going. In order to make a powerful suggester for anything (books, movies, music, raincoats, etc.), it is now necessary to consider the individual making the purchase instead of a one-size-fits all approach. How else can you help a guy like me who hates sci-fi but loved “The Matrix” and can’t stand to watch horror flicks but has seen “Scream” several times?
I’m oversimplifying it a bit, but this is a very difficult problem. You’re basically tasked with generalizing a solution which has to consider literally millions of individual problems within the problem. It’s very tough to quantify so many parameters in so many dimensions.
What amazes me most is that this is such a simple task for us to complete in our heads. Computers are still so far behind us in our ability to do something as simple as watch a movie and think to ourselves, “That movie sucked, but my buddy really likes movies like this…I think I’ll suggest it to him.”
interesting, and waaay more complex from a programmatic standpoint. You’re right, to do this right, somebody will have to break AI threshold and be able to build programs/systems that can adapt and learn.
A certain Hungry Mother (f-er) I know took a shot at this contest at one point last year or the year before.
I did take a look at it, but my approach was too computational for the amount of data in the data set provided. I was looking at some statistical methods, but my short attention span moved me to other things. I like your approach; it reminds me of the approach that Isaac Asimov talks about in the Foundation Trilogy (I thought you’d enjoy a sci-fi reference).
Here’s a problem: BillyMac and I have almost identical tastes in movies, but we disagree on “Juno.” I would rate it 3 out of 5.
To me these are the real problems that need to be solved in CS, and they’re the most interesting. I’d love to work on a problem like this…it’s the kind of thing I’d like to do in retirement.
HM, it’s really strange to think about the movies I don’t like, but it seems like I would. I’ve even thought to myself after watching a movie…”Why didn’t I like that?”