Does what they see affect what you get with crowdsourced input?

To get truly valid data on crowd demand do you have to remove the risk of a popularity contest? That’s one of the common questions we get asked around crowdsourcing and voting.

To shed some light on this, we thought we’d share what we discussed with good friend Veer Gidwaney, Founder of Humanity Calls – a platform where crowds of people assemble online to evaluate charities and make donations to those organizations which perform best – set to launch early 2010.

Here’s what we’ve seen as the affects of transparency in the crowds we’ve worked with:

Crowd votes totally hidden

Pros

• Makes people think for themselves & eliminates group think
• Produces incredibly valuable data

Cons

• Seeing “group think” is interesting, engaging and can foster participation
• Without ranking of ideas/solutions into top voted, it can be overwhelming for people to sort through and parse items for voting

Vote totals shown, individual votes hidden

Pros

• Presenting highest vote listing engages crowd, gets them voting
• Crowd’s time gets focused on crowd-deemed highest quality ideas/solutions

Cons

• Biases crowd energy towards top voted ideas/solutions – other high potential ideas never get seen
• Earliest in ideas/solutions biased to get most votes

Individual votes and vote totals seen by all

Pros

• With full transparency, the amount of malicious votes drops dramatically
• Crowd has means to spot and report voting irregularities

Cons

• People swayed to vote for ideas/solutions by the most popular crowd members
• As with just showing vote totals – attention biased to top voted/earliest in ideas

So with limits on all voting models – how do you best limit bias? First, pick the model with the strengths best suited to your crowd and purpose. Second, consider other means to reduce the risk of bias:

• Broaden what’s filtered & presented: Would you get more useful results and participation displaying not only top voted, but most viewed, most commented or a random display of ideas and solutions?
• Get experts to aid in filtering: Might an expert panel reviewing all ideas and filtering to top ideas for a tournament get to the most valuable winning idea?
• Pace/Batch the voting: Consider weekly or monthly showdowns to keep the volume of ideas/solutions manageably viewable by the crowd– with a tournament of finalists at the end.

These are some of our best practices & lessons learned. What have you seen work to limit crowd bias in voting?

Thanks Veer for reaching out to us!

photo by: starbright31