Credible Serendipity in an Age of Data Mining: Using the Wisdom of Experts with Pre-Analysis Plans and Split-Sample on a Randomized Trial in Kenya

Prevalent data mining and publication biases question the credibility of social science research. Pre-Analysis Plans (PAPs) are a leading method for increasing transparency and reproductibility. The use of multiplicity... [ view full abstract ]

Prevalent data mining and publication biases question the credibility of social science research. Pre-Analysis Plans (PAPs) are a leading method for increasing transparency and reproductibility. The use of multiplicity corrections, however, presents researchers with a dilemma: how many and which tests to perform? Researchers are incentivized to pre- specify only certain (high-prior) hypotheses, limiting serendipitous (unexpected) discoveries. The split-sample technique for remaining uncertain but relevant (high-utility) hypotheses constitutes a potential solution. Measuring priors and utilities can guide researchers when choosing hypotheses and methodologies. This study innovatively measures these parameters by averaging the knowledge of academic experts. Using a combination of PAP and split-sample on a randomized trial in Kenya yields credible serendipitous discoveries, which would have been ignored by a conventional PAP.