We combine a randomized behavioral field experiment with big data and machine learning techniques in a U.S. state unemployment office to reduce unethical behavior. Using machine learning to identify who is likely to commit unethical behavior in the future, we enter those individuals into a randomized controlled trial, in which we test behavioral interventions aimed at reducing unethical behavior.
Methods
This study was carried out on a U.S. state government platform where former employees (“claimants”) apply for unemployment benefits (N=33,727). The outcome variable in our study was a self-reported measure of whether or not a claimant earned income in the past week. If the claimant reported earnings, the amount was deducted from the weekly benefits payment, thus creating an incentive to behave unethically and not report earnings.
We used a two-step methodology involving (1) machine learning and (2) a randomized controlled trial (RCT). First, we used machine learning based on claimants’ behavior from past years (N=527,854 claims) to predict which claimants were most likely to be flagged for misreporting income in the future, assigning each a Risk Assessment Rating (RAR) between 1 and 100 (where higher values indicate greater likelihood to behave unethically).
Second, claimants with a high RAR were entered into a RCT, in which they were either part of a control group or one of six treatment arms that included previously-identified behavioral messages (e.g. norms, penalties, reminders, or a variation of messages) on the screen where they reported whether they had any earnings last week.
Results
First, we analyzed whether our algorithm placed claimants who actually ended up intentionally misreporting income in higher RAR deciles: indeed, claimants with a RAR between 1 and 10 were 89% less likely to behave unethically than the average claimant. Conversely, claimants with a RAR between 80 and 89, and claimants with a RAR between 90 and 100, behaved unethically at 74% and 256% higher rates than average.
Next, we look at the effect of behavioral messages on likely unethical claimants. Pooled across all messages, we find a significant effect of messages on the likelihood to report earnings (p < 0.05). In particular, a social norm message (“99 out of 100 report earnings honestly”) as well as an intervention combining several messages over time significantly increase earnings reported (both ps < 0.05). Other messages had positive effect sizes but were not statistically significant.
Discussion
Our investigation makes several theoretical and practical contributions to public policy research. Behavioral interventions have been shown to have had success in affecting behavior, often through relatively small changes. While we find similar effects for some interventions, we argue that a “one size fits all” approach is unlikely to affect behavior for everyone equally. Using new technologies such as machine learning enable customization to target behavioral interventions, while still allowing for causal inference through randomization of messages. We have demonstrated this in a two-step methodology combining behavioral RCTs with machine learning, which leads effective and efficient policy interventions.
