Impact Evaluation in Practice

Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch, World Bank, 2011

Impact Evaluation in Practice is available as downloadable pdf, and can be bought online.

“Impact Evaluation in Practice presents a non-technical overview of how to design and use impact evaluation to build more effective programs to alleviate poverty and improve people’s lives. Aimed at policymakers, project managers and development practitioners, the book offers experts and non-experts alike a review of why impact evaluations are important and how they are designed and implemented. The goal is to further the ability of policymakers and practitioners to use impact evaluations to help make policy decisions based on evidence of what works the most effectively.

The book is accompanied by a set of training material — including videos and power point presentations — developed for the “Turning Promises to Evidence” workshop series of the Office of the Chief Economist for Human Development. It is a reference and self-learning tool for policy-makers interested in using impact evaluations and was developed to serve as a manual for introductory courses on impact evaluation as well as a teaching resource for trainers in academic and policy circles.

CONTENTS
PART ONE. INTRODUCTION TO IMPACT EVALUATION
Chapter 1. Why Evaluate?
Chapter 2. Determining Evaluation Questions
PART TWO. HOW TO EVALUATE
Chapter 3. Causal Inference and Counterfactuals
Chapter 4. Randomized Selection Methods
Chapter 5. Regression Discontinuity Design
Chapter 6. Difference-in-Differences
Chapter 7. Matching
Chapter 8. Combining Methods
Chapter 9. Evaluating Multifaceted Programs
PART THREE. HOW TO IMPLEMENT AN IMPACT EVALUATION
Chapter 10. Operationalizing the Impact Evaluation Design
Chapter 11. Choosing the Sample
Chapter 12. Collecting Data
Chapter 13. Producing and Disseminating Findings
Chapter 14. Conclusion

Evaluation Revisited – Improving the Quality of Evaluative Practice by Embracing Complexity

Utrecht Conference Report. Irene Guijt, Jan Brouwers, Cecile Kusters, Ester Prins and Bayaz Zeynalova. March 2011. Available as pdf

This report summarises the outline and outputs of the Conference ‘Evaluation Revisited: Improving the Quality of Evaluative Practice by Embracing Complexity’’, which took place on May 20-21, 2010. It also adds additional insights and observations related to the themes of the conference, which emerged in presentations about the conference at specific events.

Contents (109 pages):

1 What is Contested and What is at Stake
1.1 Trends at Loggerheads
1.2 What is at Stake?
1.3 About the May Conference
1.4 About the Report
2 Four Concepts Central to the Conference
2.1 Rigour
2.2 Values
2.3 Standards
2.4 Complexity
3 Three Questions and Three Strategies for Change
3.1 What does ‘evaluative practice that embraces complexity’ mean in practice?
3.2 Trade-offs and their Consequences
3.3 (Re)legitimise Choice for Complexity
4 The Conference Process in a Nutshell

Purposeful Program Theory: Effective Use of Theories of Change and Logic Models

by Sue C. Funnell,  Patricia J. Rogers. March 2011. Available on Amazon.

Product Description

“Program Theory in Evaluation Practice is a ground–breaking reference that teaches how to develop an explicit causal model that links an intervention (project, program or policy) with its intended or observed impacts and using this to guide monitoring and evaluation. Peerless in its explanation of why and how to use and develop program theory, the book is rich with examples and alternative approaches. The book is an invaluable resource to faculty and students as well as professionals in professional development programs, education, social work, and counseling. “

From the Back Cover

“Between good intentions and great results lies a program theory—not just a list of tasks but a vision of what needs to happen, and how. Now widely used in government and not–for–profit organizations, program theory provides a coherent picture of how change occurs and how to improve performance. Purposeful Program Theory shows how to develop, represent, and use program theory thoughtfully and strategically to suit your particular situation, drawing on the fifty–year history of program theory and the authors? experiences over more than twenty–five years.

“From needs assessment to intervention design, from implementation to outcomes evaluation, from policy formulation to policy execution and evaluation, program theory is paramount. But until now no book has examined these multiple uses of program theory in a comprehensive, understandable, and integrated way. This promises to be a breakthrough book, valuable to practitioners, program designers, evaluators, policy analysts, funders, and scholars who care about understanding why an intervention works or doesn?t work.” —Michael Quinn Patton, author, Utilization–Focused Evaluation”

“Finally, the definitive guide to evaluation using program theory! Far from the narrow ?one true way? approaches to program theory, this book provides numerous practical options for applying program theory to fulfill different purposes and constraints, and guides the reader through the sound critical thinking required to select from among the options. The tour de force of the history and use of program theory is a truly global view, with examples from around the world and across the full range of content domains. A must–have for any serious evaluator.” —E. Jane Davidson, PhD, Real Evaluation Ltd.

Promoting Voice and Choice: Exploring Innovations in Australian NGOAccountability for Development Effectiveness

– Exploring innovations in Australian NGO accountability for development effectiveness

by Chris Roche, ACFID research paper, 2010

From the Preface

“This research paper represents the latest chapter in a body of work, led by ACFID’s Development Practice Committee (DPC), focused on Australian NGO program quality and effectiveness. Over the past 10 years DPC has engaged the sector in a series of consultations and discrete research phases to define our effectiveness and identify the principles, program strategies, standards of engagement and organisational management practices which underpin it.

The objective of the current research was to capture and share cutting edge practice in demonstrating Australian NGO effectiveness through innovative forms of accountability and social learning, in which the views of those who are ultimately meant to benefit were central. ACFID member agencies participated through submitting examples of their attempts to improve downward accountability.

The findings presented in this report will contribute to ACFID member agencies’ journey of continual improvement of our collective effectiveness. It will do this through engaging with senior NGO managers and AusAID in the analysis of the findings, as well as contributing to the international work on CSO Development Effectiveness. The next research phase will be in partnership with an academic institution to undertake a more rigorous examination of a sample of the case studies and the organisational enablers and obstacles to improving our effectiveness.”

See also Chris Roche’s new guest posting on the (Australian based) Development Policy Centre’s Development Policy Blog, titled “Changing the rules of the game?” In this blog he follows up on issues raised in the above paper.

Learning how to learn: eight lessons for impact evaluations that make a difference

ODI Background Notes, April 2011. Authors: Ben Ramalingam

“This Background Note outlines key lessons on impact evaluations, utilisation-focused evaluations and evidence-based policy. While methodological pluralism is seen as the key to effective impact evaluation in development, the emphasis here is not methods per se. Instead, the focus is on the range of factors and issues that need to be considered for impact evaluations to be used in policy and practice – regardless of the method employed. This Note synthesises research by ODI, ALNAP, 3ie and others to outline eight key lessons for consideration by all of those with an interest in impact evaluation and aid effectiveness”.  8 pages

The 8 lessons:
Lesson 1:  Understand the key stakeholders
Lesson 2:  Adapt the incentives
Lesson 3:  Invest in capacities and skills
Lesson 4:  Define  impact  in ways  that  relate  to  the specific context
Lesson 5:  Develop the right blend of methodologies
Lesson 6:  Involve those who matter in the decisions that matter
Lesson 7:  Communicate effectively
Lesson 8:  Be persistent and lexible

See also Ben’s Thursday, April 14, 2011 blog posting: When will we learn how to learn?

[RD comments on this paper]

1.     The case for equal respect for different methodologies can be overstated. I feel this is the case when Ben argues that “First, it has been shown that the knowledge that results from any type of particular impact evaluation methodology is no more rigorous or widely applicable than the results from any other kind of methodology.”  While it is important that evaluation results affect subsequent policy and practice their adoption and use is not the only outcome measure for evaluations. We also want those evaluation results have some reliability and validity, that will stand the test of time and be generalisable to other settings with some confidence. An evaluation could affect policy and practice without necessarily being good quality , defined in terms of reliability and valdity.

  • Nevertheless, I like Ben’s caution about focusing too much on evaluations as outputs and the need to focus more on outcomes, the use and uptake of evaluations.

    2.     The section of Ben’s paper that most attracted my interest was the story about the Joint Evaluation of Emergency Assistance to Rwanda, and how the evaluation team managed to ensure it became “one of the most influential evaluations in the aid sector”. We need more case studies of these kinds of events and then a systematic review of those case studies.

    3.     When I read statements various like this: “As well as a supply of credible evidence, effort needs to be made to understand the demand for evidence” I have an image in my mind of evaluators as humble supplicants, at the doorsteps of the high and mighty. Isn’t it about time that evaluators turned around and started demanding that policy makers disclose the evidence base of their existing policies? As I am sure has been said by others before, when you look around there does not seem to be much evidence of evidence based policy making. Norms and expectations need to be built up, and then there may be more interest in what evaluations have to say. A more assertive and questioning posture is needed.

    NAO report: DFID Financial Management Report

    NAO Press Release 6 April 2011…

    “Sound financial management will be essential at the Department for International Development as its spending increases by a third over the next four years, according to the National Audit Office.

    The Department has improved its core financial management and has an ambitious programme underway to improve its focus on value for money. It has put important building blocks in place; however its financial management is not yet mature.   The Department cannot yet assess important aspects of the value for money of the aid it has delivered, at an aggregated level.

    The Department’s programme budget will grow by £3.3 billion from 2010-11 to 2014-15 (34 per cent in real terms). At the same time, its administration budget is going to reduce by a third. The Department will face significant financial and operational challenges, making sound financial management essential.

    The Department has increased the number of finance professionals it employs, but this expertise needs to be used more effectively across the business. In addition, new financial information systems do not yet provide the data needed to support well-founded decisions and forecasts are still an area of weakness.

    Having conducted a thorough review, the Department now has a high level plan allocating its resources on the basis of the results it aims to achieve.  Along with actions to strengthen measurement of aid projects, this has the potential to help strengthen the focus on aid results and value for money. But key risks need to be managed and the Department should now develop a coherent, single strategy for doing so.

    With greater spending in higher risk locations and more fragile states, the Department must do more to assure itself that it minimises fraud and corruption risks. Although the level of reported fraud is low, it is likely to be under-reported. The NAO has found that the investigation of fraud is reactive and the Department does not attempt to quantify its estimated likely fraud losses.

    Amyas Morse, head of the National Audit Office, said today:

    “The Department knows its increase in funding, and new approach to aiding developing countries, brings challenges. This report shows considerable progress is being made, but a better information environment is needed to deal with the heightened levels of assurance required in targeting future aid at higher risk locations”

    [RD comment] The Executive Summary ends with a section titled: Conclusion on value for money, which says:

    • We recognise that the Department has been improving its core financial management and has also been strengthening its focus on value for money at all levels of the organisation, including through a step change in its approach to the strategic allocation of resources based on expected results. Important building blocks have been put in place, but key gaps in financial management maturity remain. The changes the Department has introduced to-date are positive, and provide a platform to address the challenges that will come with its increased spending.
    • At present, however, the Department’s financial management is not mature. The  Department’s forecasting remains inaccurate and its risk management is not yet fully embedded. Weaknesses in the measurement of value for money at project level, variability in the quality and coverage of data, and lack of integration in core systems, mean that the Department cannot assess important aspects of value for money of the aid it has delivered, at an aggregated level. The Department now needs to develop a coherent single strategy to address the weaknesses identified and the key risks to meeting its objectives.

    Sound expectations: from impact evaluations to policy change

    3ie Working paper # 12, 2011, by Center for the Implementation of Public Policies Promoting Equity and Growth (CIPPEC) Emails: vweyrauch@cippec.org, gdiazlangou@cippec.org

    Abstract

    “This paper outlines a comprehensive and flexible analytical conceptual framework to be used in the production of a case study series. The cases are expected to identify factors that help or hinder rigorous impact evaluations (IEs) from influenc ing policy and improving policy effectiveness. This framework has been developed to be adaptable to the reality of developing countries. It is aimed as an analytical-methodological tool which should enable researchers in producing case studies which identify factors that affect and explain impact evaluations’ policy influence potential. The approach should also enable comparison between cases and regions to draw lessons that are relevant beyond the cases themselves.

    There are two different , though interconnected, issues that must be dealt with while discussing the policy influence of impact evaluations. The first issue has to do with the type of policy influence pursued and, aligned with this, the determination of the accomplishment (or not) of the intended influence. In this paper, we first introduce the discussion regarding the different types of policy influence objectives that impact evaluations usually pursue, which will ultimately help determine whether policy influence was indeed achieved. This discussion is mainly centered around whether an impact evaluation has had impact on policy. The second issue is related to the identification of the factors and forces that mediate the policy influence efforts and is focused on why the influence was achieved or not. We have identified and systematized the mediating factors and forces, and we approach them in this paper from the demand and supply perspective, considering as well, the intersection between these two.

    The paper concludes that, ultimately, the fulfillment of policy change based on the results of impact evaluations is determined by the interplay of the policy influenc e objectives with the factors that affect the supply and demand of research in the policymaking process.

    The paper is divided in four sections. A brief introduction is followed by an analysis of policy influence as an objective of research, specifically, impact evaluations. The third section identifies factors and forces that enhance or undermine influence in public policy decision making. The research ends up pointing out the importance of measuring policy influence and enumerates a series of challenges that have to be further assessed.”

    IMPACT AND AID EFFECTIVENESS: Mapping the Issues and their Consequences

    [from the IDS Virtual Bulletin, March 2011]

    Introduction
    In this virtual Bulletin we bring together ten articles dating from across three decades. They all address Impact. From the outset, we note that there are a number of common threads and ideas that stretch across all the articles:

    • The implicit emphasis of all the articles on complexity
    • The breadth and depth of impact analysis, from the national level to the individual
    • The importance of knowing the audience for any evaluation or impact assessment
    • The virtuous cycle that can be created by using insights into impact to adjust interventions
    • The dependency of that virtuous cycle on participation and engagement of programme staff and clients.

    What we notice, however, is how the articles framing these issues vary according to discipline and research site. We also see how some ongoing preoccupations have been shaped by their proximity to other debates or policy concerns. Our hope is that hindsight will provide some perspective for practice and policy going forward.
    View Full Introduction

    Articles
    A Revolution Whose Time Has Come? The Win-Win of Quantitative Participatory Approaches and Methods
    IDS Bulletin Volume 41, Issue 6, November 2010
    Robert Chambers

    Impact of Microfinance on Rural Households in the Philippines
    IDS Bulletin Volume 39, Issue 1, March 2008
    Toshio Kondo, Aniceto Orbeta, Clarence Dingcong and Christine Infantado

    You Can Get It If You Really Want’: Impact Evaluation Experience of the Office of Evaluation and Oversight of the Inter-American Development Bank
    IDS Bulletin Volume 39, Issue 1, March 2008
    Inder Jit Ruprah

    The Role of Evaluation in Accountability in Donor-Funded Projects
    IDS Bulletin Volume 31, Issue 1, January 2000
    Adebiyi Edun

    Micro-Credit Programme Evaluation: A Critical Review†
    IDS Bulletin Volume 29, Issue 4, October 1998
    Shahidur R. Khandker

    Macroeconomic Evaluation of Programme Aid: A Conceptual Framework
    IDS Bulletin Volume 27, Issue 4, October 1996
    Howard White

    Measurement of Poverty and Poverty of Measurement
    IDS Bulletin Volume 25, Issue 2, April 1994
    Martin Greeley

    Developing Effective Study Programmes for Public Administrators
    IDS Bulletin Volume 8, Issue 4, May 2009
    Ron Goslin

    Improving the Effectiveness of Evaluation in Rural Development Projects
    IDS Bulletin Volume 8, Issue 1, July 1976
    B. H. Kinsey

    Managing Rural Development
    IDS Bulletin, Volume 6, Issue 1, September 1974
    Robert Chambers

    Behavioral economics and randomized trials: trumpeted, attacked and parried

    This is the title of a blog posting by Chris Blattman, which points to and comments on a debate  in the Boston Review, March/April 2011

    The focus of the debate is an article by Rachel Glennerster and Michael Kremer, titled Small Changes, Big Results:  Behavioral Economics at Work in Poor Countries

    “Behavioral economics has changed the way we implement public policy in the developed world. It is time we harness its approaches to alleviate poverty in developing countries as well.”

    This article is part of Small Changes, Big Results, a forum on applying behavioral economics to global development. This includes the following 7 responses to Glennerster and  Kremer, and their response.

    Diane Coyle: There’s nothing irrational about rising prices and falling demand. (March 14)

    Eran Bendavid: Randomized trials are not infallible—just look at medicine. (March 15)

    Pranab Bardhan: As the experimental program becomes its own kind of fad, other issues in development are being ignored. (March 16)

    José Gómez-Márquez: We want to empower locals to invent, so they can be collaborators, not just clients. (March 17)

    Chloe O’Gara:  You can’t teach a child to read with an immunization schedule. (March 17)

    Jishnu Das, Shantayanan Devarajan, and Jeffrey S. Hammer:Even if experiments show us what to do, can we rely on government action? (March 18)

    Daniel N. Posner: We cannot hope to understand individual behavior apart from the community itself. (March 21)

    Rachel Glennerster and Michael Kremer reply: Context is important, and meticulous experimentation can improve our understanding of it. (March 22)

    PS (26th March 2011: See also Ben Goldacre’s Bad Science column in today’s Guardian: Unlikely boost for clinical trials (/When ethics committees kill)

    “At present there is a bizarre paradox in medicine. When there is no evidence on which treatment is best, out of two available options, then you can choose one randomly at will, on a whim, in clinic, and be subject to no special safeguards. If, however, you decide to formally randomise in the same situation, and so generate new knowledge to improve treatments now and in the future, then suddenly a world of administrative obstruction opens up before you.

    This is not an abstract problem. Here is one example. For years in A&E, patients with serious head injury were often treated with steroids, in the reasonable belief that this would reduce swelling, and so reduce crushing damage to the brain, inside the fixed-volume box of your skull.

    Researchers wanted to randomise unconscious patients to receive steroids, or no steroids, instantly in A&E, to find out which was best. This was called the CRASH trial, and it was a famously hard fought battle with ethics committees, even though both treatments – steroids, or no steroids – were in widespread, routine use. Finally, when approval was granted, it turned out that steroids were killing patients.”

    Making Evaluations Matter: a Practical Guide for Evaluators.

    Authors: Cecile Kusters with Simone van Vugt, Seerp Wigboldus, Bob Williams and Jim Woodhill. 2011

    “Too often evaluations are shelved, with very little being done to bring about change within organisations that requested the evaluation in the first place. This guide will explain how you can make your evaluations more useful. It will help you to better understand some conceptual issues and appreciate how evaluations can contribute to changing mindsets and empowering stakeholders. On a practical level, it presents core guiding principles and pointers on how to design and facilitate evaluations that matter. Furthermore, it shows you how you can get your primary intended users and other key stakeholders to contribute effectively to the evaluation process. This guide is primarily for evaluators working in the international development sector. However, if you are a commissioner of an evaluation, an evaluation manager or a monitoring and evaluation officer, you too will find it useful”.

    The book can now be found here as a PDF version, http://www.cdi.wur.nl/UK/resources/Publications/