Beyond the statistical effect in crowd predictions: What is the impact of discussion intensity on the prediction performance of groups?

The aggregated numerical predictions from large groups have been found to be wiser than those of most individuals. Starting with the seminal study of Galton (1917), numerous experiments in a variety of knowledge domains have... [ view full abstract ]

The aggregated numerical predictions from large groups have been found to be wiser than those of most individuals. Starting with the seminal study of Galton (1917), numerous experiments in a variety of knowledge domains have advocated the independence of individual judgments as a key underpinning of the “wisdom of the crowds”. This makes perfect sense from a statistical perspective as independent judgements increase the likelihood that individual errors will cancel out through averaging. In reality, however, many group-based prediction tasks are actually characterized by dependency, i.e. the individual raters know the judgements of other group members and even enter into discussions about the ground truth (e.g. online problem solving communities; project teams in organizations). Even in formal numerical prediction processes, such as prediction markets, participants receive feedback on the aggregated predictions of all participants. It is therefore reasonable to relax the boundary conditions and to investigate the wisdom of crowds in situations characterized by non-independence, judgmental biases, and social influences.
This study examines the accuracy of predictions made by participants of the Transfermarkt.de online community (www.transfermarkt.de) which attracts more than 3 million unique users per month. On this platform, users debate and individually assess the virtual market values of professional soccer players. The predicted market values substantiate whenever a player is transferred to a new club. The transfer fee payed by the new club therefore constitutes the true value that is to be predicted by the community. There is a thread for each player in which the participants can post, reply to other posts and, most important, make their own prediction of the transfer value. We propose that prediction performance will increase with (1) discussion intensity in the group and with (2) the accuracy of the first prediction submitted in a thread. Further, we hypothesize that (3) the positive effect of high discussion intensity will be stronger for less accurate initial predictions and weaker for more accurate initial predictions.
Our first results are based on the analysis of all soccer player transfers that took place in the German “Bundesliga” and the Italian “Serie A” in summer 2015. The analysis provides support to all three hypotheses. The findings contribute to our understanding of the role of the interaction between group members in arriving at accurate estimations in complex judgment tasks. By concentrating on the power of “statisticized” groups and by setting the independence of judgments as a key boundary condition, so far, communication between subjects in the prediction process has been primarily seen as a source of systematic assessment errors. Our empirical setting, however, allows studying “open” predictions and the impact of group communication. The findings suggest that organizers of crowd predictions can hope to benefit from interaction if they manage to stimulate an intense discussion among the predicting subjects.