A List Apart

Menu

The Wisdom of Community

Issue № 283

The Wisdom of Community

by Published in Community · 18 Comments

It’s one of the most important concepts on the web today—perhaps the most important for social media—but it’s one of the least understood. When James Surowiecki wrote The Wisdom of Crowds in 2004, he explored the stock market and other classic social psychology examples, but “web 2.0” was still nascent. It’s time to connect his ideas to the social web, where they can reach their full potential.

Article Continues Below

The Wisdom of Crowds (WOC) theory does not mean that people are smart in groups—they’re not. Anyone who’s seen an angry mob knows it. But crowds, presented with the right challenge and the right interface, can be wise. When it works, the crowd is wiser, in fact, than any single participant.

The standard example is this: Imagine you have a jar of pennies. Ask a few hundred people how many there are inside. When you tally the results, chances are, all the guesses will be wrong. But if you average all the answers, the result will be almost perfect, almost all the time.

The web, with its low barrier to entry and permeable social boundaries, is the ultimate medium through which to explore the finer points of the wisdom of crowds. You’re surrounded by online examples: Google’s search results. BitTorrent. The “Most E-mailed” stories on your favorite news site. Each is powered by wisdom gleaned from crowds online.

You need a few things to enable online crowds to be wise.

Simplicity

In the penny jar example, you ask each participant for a number. Google didn’t ask anyone anything to create their search results, though they just intuited the importance of every page in their index by considering how often it was linked to.

Note that in each case mentioned above—Google, BitTorrent, “Most E-mailed” lists, and the penny jar—the inputs are not conversational. Conversational inputs are too complex for Wisdom of Crowds systems. Online discussion systems do not lead to wisdom on their own.

The simplicity of the individual task is also important. Systems based on the Wisdom of Crowds can tackle surprisingly complicated projects, but each project must first be broken down to its simplest possible components.

Interface

Complicated interfaces can be great for complicated tasks, but not for WOC systems. The more complicated the interface is, the less participation you can expect, and the more muddled that participation will be. Soliciting WOC feedback is about tapping the zeitgeist. Don’t make participants think too much.

WOC interfaces are often voting mechanisms that use a spectrum or a thumbs up/down system, but they could just as easily involve selecting a point on a map or drawing a shape.

Aggregation

One of the reasons discussions do not lead to wise results is that there’s no aggregation—the conversation just happens. But WOC systems are there to produce a result. This requires an aggregator (like you) and an algorithm.

In the penny jar example, the aggregator is the person tallying the guesses, and the algorithm is a simple median. With Google’s search results, Google is the aggregator, their algorithm is called PageRank, and it’s a constantly evolving, closely guarded secret.

Participation

A defining element of any WOC system is that the more participants it has, the better it gets. Discussion systems and chat rooms fall apart when too many voices get involved. If your community feature gets worse the more people use it, it’s not a WOC system.

For a crowd to truly be wise, it also has to be diverse. The Wisdom of Crowds works because the people on the edges balance each other out. Recruit too many people on one side of any spectrum and your results will suffer.

Selfishness

It’s counter-intuitive, but the wisest crowds are the ones made up of individuals who are thinking about their own needs, not the needs of the group. In the stock market, the participants are all motivated to buy low and sell high. Yet the markets are usually wise about finding the value of a company. Each person is thinking about their bottom line, not the health of the company or the market, but it works.

Similarly, website creators were not consciously voting for certain sites to be highly ranked, but the collective linking decisions did produce wise results. Nowadays, link spammers do try to manipulate Google’s results, which is akin to stock manipulation. Both practices are fought by the institutions that depend on unmanipulated results.

Selfishness also fights a larger problem. Group-think is when the members of a group put the group’s needs above their own. As soon as this happens, the group is in danger. The stock market collapse, the NASA Challenger disaster, and many other examples can be attributed to group-think.

In the penny jar example, participants were told that the correct guesser would win all the money. Their participation was entirely selfish — they wanted to win the money. The fact that their participation could be averaged to create an answer is just a fortunate byproduct.

Designing for selfishness does not necessarily mean paying cash prizes. A news site with a Most E-mailed Stories box that displays the stories that have been e-mailed to people doesn’t have to pay anyone to pick their favorite stories—the information is easily gleaned from server logs. The key factor is the user’s motivation for participating: They’re doing it for their own personal reasons (“Heather would love this story!”). They’re not consciously voting for the story—the vote is a byproduct.

This method should be contrasted with news voting sites like Digg, where users do explicitly vote for top stories. As a result, these sites face a constant battle against people trying to game the system. Indeed whole businesses have sprung up in order to place a link on the highly trafficked Digg homepage.

Scores

Keeping score is part of any game, and any website with community input is going to be used as a game. So think very carefully before assigning your participants a score for their participation.

For example, Slashdot had an internal “karma” score that it assigned to every member based on their participation. The system then used that score to determine certain features (like the ability to moderate other users’ comments and their default comment score).

That’s all fine. But then they disclosed the user’s score to them. The moment they did that, they invented “karma whores”—users who post comments they know will be rated highly by the community, creating a unique kind of group-think.

Games are fine, so long as the goal of the gamer benefits the site. But Slashdot’s games either promoted group-think (by agreeing) or trolling (by disagreeing). Both probably existed before the scores, but disclosing the scores certainly fueled them.

Leaderboards

Leaderboards create a problem for Wisdom of Crowds systems. On the one hand, a well-tuned WOC system can create an excellent leaderboard. Feedback from users is collected, the algorithm scores the content, and the result is a list of items in a fairly accurate good-to-bad order (think Google results).

And there’s the rub: Disclosing the ranked list to the community amplifies group-think. The highly rated items get even more highly rated, the low rated items fall off the radar. Showing the list destroys its accuracy.

So what to do? Here are a few suggestions:

  • Go in phases. Allow voting for a set period of time. When that time is up, close voting and display results. Threadless does this for its design submissions.
  • Mix it up. Instead of showing a ranked list (aka leaderboard), show a selection that includes highly rated items in a random order. This is what Flickr does with it’s “interestingness” view.
  • Make users earn disclosure. Show the voice of the community only after your vote has been cast. This is what many online polling systems do to avoid letting the current tally influence the voters.
  • Use an algorithm. When you have to show a current, or a ranked list, use a recipe that takes lots of data into account, in addition to votes. This is what Google does with its search results. It’s also why they have to constantly tweak the results in an ongoing arms race with people who try to figure out their system.

How you display the wisdom of your crowd can be as important as how you ask for it in the first place. It can be tempting to just put up a ranked list, but in most cases doing so will harm the very wisdom you’re trying to glean.

Explicit vs. implicit feedback

In many of the examples I’ve used, wisdom is gleaned from user behavior. In these cases, the feedback is implicit. In other cases we ask users outright for feedback, as in voting systems. That’s explicit. Whether you use explicit or implicit feedback, or some combination of the two, is an important decision in designing any WOC system.

In working on your own WOC systems, pay attention to when you can glean implicit feedback without having to ask for it directly. Implicit feedback is usually more honest and less prone to gaming. There are also ways to mix the two, for example, asking for explicit voting, but comparing it to implicit data (such as page views, comments, or other recordable user actions).

Voting

This sounds undemocratic, but voting does not have to rule all in WOC systems. In many cases, it shouldn’t.

Just because you’re collecting votes doesn’t mean you have to crown the item with the most votes the winner. You could just as easily look for items that are controversial (high percentage of both good and bad votes) or are undiscovered (low number of total votes).

And remember that not all votes have to be equal. Votes from “good” members (however you determine good) can have a higher impact.

Studies show that when rating a series of items, users are more likely to vote bad in succession. In other words, once you start voting bad, you’re more likely to keep voting bad. So it would be fair to count a bad vote less if it comes after another bad vote by that user. Or just keep the length of time in that voting session as a variable, and see what happens when you alter how you weigh early votes vs late ones. If someone’s been voting for an hour, does that make them more valuable or less? Experiment to find out.

Wiser together than we would be alone

These aspects of the Wisdom of Crowds are just the start — there’s a lot more to learn. Be sure to pick up Surowiecki’s book. And remember, WOC systems must evolve: you’re never done. But done right, they can change the way we live online, and maybe make us all a little wiser.

18 Reader Comments

Load Comments