3ie and the Funding of Impact Evaluations

A DISCUSSION PAPER FOR 3IE’S MEMBERS. by Rick Davies, July 2011. Commissioned by the Office of Development Effectiveness, AusAI. Available as pdf.

The purpose of this discussion paper is to inform AusAID’s and other 3ie members’ engagement with 3ie (the International Initiative for Impact Evaluation). It precedes the forthcoming evaluation of 3ie, and is more limited in scope. It is expected to be complementary and useful to the larger Department for International Development (DFID) study now underway, Developing a broader range of rigorous designs and methods for impact evaluations, as the author of this report is also a member of that study team.

AusAID is a member of 3ie and provides core funding to 3ie to contribute to the global public good of policy-relevant evidence on what works in development. Direct benefit to AusAID is not the purpose of the membership. However, it is important to AusAID that 3ie’s work is relevant to AusAID’s partners, particularly partners with low income and/or in fragile countries. AusAID’s Office of Development Effectiveness (ODE) manages AusAID’s membership of 3ie and has commissioned this discussion paper.

The focus of this discussion paper is on 3ie methodological approach, used in both the funded impact evaluations and systematic reviews, and how this has changed over time.

Contents
1. Introduction
1.1 Purpose and approach
1.2 A summary of 3ie’s stated methodological approach
1.3 Debates on impact evaluation
1.4 What we don’t know
2. The Issues
2.1 Are RCTs being promoted to the exclusion of other approaches?
2.2 Is attention being given to how it works, as well as what works?
2.3 Are RCTs focusing on the simple questions and ignoring the more difficult?
2.4 Do the specific requirements of RCTs limit their range of application?
2.5 Who can afford RCTs?
2.6 How valid are the results?
2.7 How generalisable are the results?
2.8 Risks of unintended effects
2.9 Ethical issues
2.10 A reminder of the value of experimental approaches
3. A Wider View: Methodology in the Context of Strategy
4. Future Directions
4.1 Obtaining more value from existing investments
4.2 Development of new methodologies
4.3 Focusing on the strategic objectives