I want to look a little more at the Yahoo! Answers story and add what I’ve learned about gaming a system. When you look back at the history of Epinions, the site made a big bang when it debuted because it offered to pay writers for their product reviews. The idea was the company would make money off of ads (this is when banners were big) and they would share that money with the community of writers.
At the time, your “Income Share,” as it is known, was based on the number of people that looked at your review. At the time, I believe that the going rate was something in the $.30 per-page-view range. You can just imagine how this got gamed.
How it works
Now, given that the site’s raison d’être was product reviews, there is a relatively finite amount of products to review and ideally, from a shopper’s perspective, you would want the “best” review at the top. So to accomplish this, Epinions built a reputation system called the Web of Trust. In theory, Epinions’ “Web of Trust,” or WoT what an early social network where your “friends” were people whose reviews you trusted and respected.
The second component of review ranking was how helpful someone found your review. At the bottom of every review, users are asked to rate the review on a 4 point scale from “Not Helpful” to “Very Helpful.” Review order is determined by a combination of the two criteria, WoT and review ranking. The basic idea is that if you are trusted more than another member and your review received a higher rating, that review would appear higher up in the list.
The story goes that writers on the site formed rating and trust circles to boost their own reviews over others. The stakes for this were high. If you were the top review on a popular product, most shoppers would see your review first. If you collected all the page views, you received considerably more money than anyone else who wrote the review. And while points are nice, money is a lot better.
So, what did Epinions do to thwart gaming?
There were three ways that Epinions practically eliminated gaming on the site.
1. Epinions gave some users more power.
If those users trusted you and/or rated your review, their vote counted more than a regular user. These members are nominated and selected by other members of the community and they have a limited tenure in their position.
2. Changed the system.
Members no longer received Income Share on a simple x views = y dollars, but rather Epinions created a system that was a little more opaque and paid out based on the number of non-members viewed the site, thus eliminating what would be now known as click fraud.
3. Dot-com bust.
The third way wasn’t really on purpose. In late 2001, the company was in trouble. Times were tough for all advertising based web sites. Epinions was in the process of laying off staff and reducing costs. The amount of Income Share, as a result, was also greatly reduced.
Ultimately, when stakes are lowered, the incentive to game is also greatly reduced.
So, what happened next? As you might expect, the number of reviews on the site diminished a great deal, but interestingly, many feel that overall quality on the site has increased. Also, at least while I was there, a majority of products have at least three “Helpful” reviews on them.
It seems that getting rid of the writers that were purely writing for the money weren’t as good as the folks doing it for the love of writing and community. Just ask Amazon. They don’t pay their reviewers a dime.