My research will contribute towards estimating the generalizability of randomized controlled trials used to evaluate social and economic policies or interventions. The narrower research question pertains to estimating the external validity bias that can arise when sites are non-randomly selected on the basis of implementation costs, the willingness of the sites to participate, etc. It is important to understand the magnitude of external validity bias since the estimate obtained from the randomized controlled trial may not be the estimate for the population we are interested in.
This estimation is conducted by carefully reviewing the policy impact evaluation literature and modelling different hypothetical scenarios of selecting sites for a multi-site randomized controlled trial. These sites are selected as different subsets of a randomized controlled trial with randomly chosen sites, which is taken as the "gold standard."
Sources
Olsen, Robert B., et al. “External Validity in Policy Evaluations That Choose Sites Purposively.” Journal of Policy Analysis and Management : [The Journal of the Association for Public Policy Analysis and Management], vol. 32, no. 1, 2013, pp. 107–21. PubMed Central, doi:10.1002/pam.21660.
Robert A. Moffitt. “The Role of Randomized Field Trials in Social Science Research: A Perspective from Evaluations of Reforms of Social Welfare Programs.” American Behavioral Scientist, vol. 47, no. 5, Jan. 2004, pp. 506–40. SAGE Journals, doi:10.1177/0002764203259292.
Rothwell, Peter M. “Factors That Can Affect the External Validity of Randomised Controlled Trials.” PLoS Clinical Trials, vol. 1, no. 1, May 2006. PubMed Central, doi:10.1371/journal.pctr.0010009.
