Improving the Aggregated Estimates of Ordinary People (and Experts too).

In the Journal of the Royal Society Interface a team of researchers has published a new method of aggregating estimates produced of non-experts to improve the accuracy of the collective estimate – that is, the so called “wisdom of the crowd.”

Estimates produced by non-experts and experts experience several forms of error. Specifically, individual estimates are rarely independent which can influence the accuracy of the collective estimate.

For instance, every day both experts and non-experts interact socially with their networks before being asked to provide their perspective on situations. In an experiment described in the paper that involved asking folks to guess the number of gumballs in a jar, only 38 percent of the participants discounted social information that was feed to them with the intent of skewing their view (truly fake news). In other words, their estimates of the number of gumballs both before and after receiving false information from others in the crowd were identical.

The study also found that the larger the social group the more important the information in skewing the independence of the estimates. Therefore, if you’re dealing with large social groups with good information sharing it actually shrinks the distribution of estimates.  

Other studies have found that incorporating experts into the population can improve estimates. In other words, including experts in the group providing estimates (rather than apart or above the group) can improve the aggregated estimates of the entire group.

However, the new methods in the paper using a maximum likelihood aggregator appears to improve the crowd estimates irrespective of knowing anything about the background of the crowd.