General Seminar - Rafe Meager

Implementation Matters: External Validity of Treatment Effects in Education

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet reported impacts vary by an order of magnitude. We study this variation via meta-analyses of randomized trials across contexts and use the results to inform a new randomized trial and evidence aggregation model. We find two factors explain most of the heterogeneity in effects across settings: accounting for the rate of implementation (intention-to-treat or treatment-on-the-treated effects) and program delivery model (teachers or volunteers). There is debate on whether randomized trial results in one setting, which have high internal validity, are also externally valid in new settings; we find high external validity when considering implementation. Yet, formal accounting for implementation in the literature remains rare: less than 12% of impact evaluations currently account for quantitative metrics of implementation. Our study motivates more frequent estimation and reporting of treatment-on-the-treated effects to enhance external validity and generalizability. Given the importance of implementation, we also introduce a new Bayesian framework to formally incorporate implementation uncertainty and various dimensions of implementation into evidence aggregation. Results show targeted instruction delivers average learning gains of 0.43 SD when taken up and 0.85 SD when implemented with high fidelity.