by Jennifer Gauck, University of Kent, Canterbury – Department of Politics, 2011. APSA 2011 Annual Meeting Paper. Available as pdf
Abstract:
“Recent debates over the quality, quantity and purpose of development aid has led to a renewed emphasis on whether, and in what circumstances, aid is effective in achieving development outcomes. A central component of determining aid effectiveness is the conduct of impact evaluations, which assess the changes that can be attributed to a particular project or program. While many impact evaluations use a mixed-methods design, there is a perception that randomized control trials (RCTs) are promoted as the “gold standard” in impact evaluation. This is because the randomization process minimizes selection bias, allowing for the key causal variables leading to the outcome to be more clearly identified. However, many development interventions cannot be evaluated via RCTs because the nature of the intervention does not allow for randomization with a control group or groups.”
“This paper will analyze the methodological challenges posed by aid projects whose impacts cannot be evaluated using randomized control trials, such as certain democracy and governance (D&G) interventions. It will begin with a discussion of the merits and drawbacks of cross-sectoral methods and techniques commonly used to assess impact across a variety of aid interventions, including RCTs, and how these methods typically combine in an evaluation to tell a persuasive causal story. This paper will then survey the methods different aid donors are using to evaluate the impact of projects that cannot be randomized, such as governance-strengthening programs aimed at a centralized public-sector institution. Case studies will be drawn from examples in Peru and Indonesia, among others. This paper will conclude by analyzing how current methodological emphases in political science can be applied to impact evaluation processes generally, and to D&G evaluations specifically.”
RD Comment: See also the 3ie webpage on Useful resources for impact evaluations in governance which includes a list of relevant books, reports, papers, impact evaluations, systematic reviews, survey modules/tools and website