Book Review: Building Web Reputation Systems by Farmer & Glass
By Eric Goldman
Building Web Reputation Systems by F. Randall Farmer & Bryce Glass (O’Reilly 2010) [affiliate link]
As you may know, for the past couple of years, I have been researching how we regulate reputation systems. My most recent recap of my progress-to-date. As part of researching other disciplines’ approaches to reputation systems, I was pleasantly surprised to find this book, which discusses web reputation systems from a technical/product development standpoint. I’m not aware of other books directly on point, so that alone makes the book noteworthy. [If you know of analogous books that I should look at, I’d be grateful for the references.]
The word “reputation” is a complex and nuanced word. This book defines reputation as “information used to make a value judgment about an object or a person.” Notice how this definition treats reputation as actionable information (i.e., making a “judgment”). I favor that approach; my work also uses an actionable definition of reputation.
Their definition equally treats both objects and people as having “reputation,” and this does not work. In general, people are dynamic, i.e., they can change behavior; while content is static, i.e., an item of content does not change its character unless subsequently edited. This single definition of “reputation” created significant tension throughout the book. Recognizing this, the authors often bifurcated the discussion to separately address the process of establishing a person’s “reputation” (which they confusingly called “karma”). However, the book primarily focuses on grading and sorting content items, especially user-generated content, and I personally would not describe content items as having a “reputation.” As a result, I think the book is mistitled. It principally addresses content filtering, not “reputation” as I use the term.
Although this analytical tension pervades the book, the book nevertheless contained a lot of useful insights about both content filtering and establishing user trustworthiness. The authors have a lot of experience building filtering systems for different websites, so the book is packed with the kind of first-hand observations that only an insider can offer. There’s no substitute for the voice of experience when designing Web 2.0 UGC systems, and this book provides an easy and accessible way to learn some tips and tricks.
The book emphasizes the authors’ contributions to the reputation system at Yahoo Answers, and rightly so. Yahoo Answers has emerged into a bona fide success story and recently trumpeted its billionth answer. In my opinion, the book’s high point is Chapter 10, a case study of how Yahoo Answers developed a new filtering and reputation system that helped turbocharge the Yahoo Answers community.
Although the book doesn’t say this directly, two key lessons from Yahoo Answers’ evolution are:
1) UGC websites should let users vote on content, but not all user votes should be weighted equally.
2) UGC websites do not need to publish all user-supplied content items in an equally prominent manner. Perhaps some content should be obscure/hard-to-find until other users validate it.
The book pitches these conclusions as novel, but they seemed fairly intuitive to me. We implemented a very similar system embodying these two points back in 2000-01 at Epinions. Epinions allowed users to grade each others’ content; we weighted votes differentially based on users’ credibility; and we displayed ungraded and poorly graded content only to registered users (a small fraction of our readers). The fact that the authors “discovered” these conclusions at Yahoo Answers shows the dire need for books like this to help websites implement best UGC management practices without reinventing the wheel.
The fact that the authors didn’t acknowledge the Epinions precedent (and other systems like it) highlights another weakness of the book. There is a deep academic literature addressing the book’s topics (especially on content filtering and user incentive systems), but the book barely acknowledges this literature. For example, several times the authors cite Dan Ariely’s Predictably Irrational for descriptions of human psychology and foibles. That’s a perfectly credible citation, but it should be one of many literature citations, not the only citation. Instead of dipping into the rich academic literature, the book almost exclusively relies on the authors’ experience-based impressions. These impressions are a valuable information source that makes the book worth reading. However, because those impressions aren’t tempered with more rigorous academic findings, it’s not clear to me at all that the authors’ conclusions represent true best practices…or even state-of-the-art.
Because of its many structural flaws, this edition will not become a classic. Nevertheless, I have enthusiastically recommended the book to several UGC start-ups because the book provides a good repository of high-value experience-based perspectives that are not readily available elsewhere. Even if the book’s recommendations are debatable, it’s a debate worth having.