Tag: Guides

How to monitor and evaluate anti-corruption agencies

Guidelines for agencies, donors and evaluators

By Jesper Johnsøn, Hannes Hechler, Luís De Sousa, Harald Mathisen (2011)  Bergen: Chr. Michelsen Institute (U4 Issue 2011:8) 84 p. Available as pdf

“The number of Anti-corruption agencies (ACAs) around the world has increased dramatically over the past decades. Nevertheless, the value of ACAs is increasingly being questioned by international donors and national governments. Frequently, ACAs are not considered to deliver on the high expectations bestowed upon then.

Evaluations of individual agencies were collected and analysed to assess the evidence underlying the assumptions about the effectiveness of ACAs. Surprisingly, few evaluations had actually been done, and even fewer measured the actual outcomes and impacts of the ACA. Thus, whilst opinions about ACAs are many, the actual evidence about their performance is scarce. To develop this body of evidence, ACAs need to do a better job at establishing results-based indicators for their work, showing how activities lead to impact, and collecting data.

To which extent the perceived failure of ACAs is an issue of measurement or design can therefore not be answered with any certainty. The value of ACAs can only be determined once evidence-based evaluations are conducted.

To this end, the report provides technical, methodological, and practical guidance to assist staff of ACAs in undertaking monitoring and evaluation and shows how the outcomes and impact of the work of ACAs can be evaluated in an objective, evidence-based manner.”

 

Impact Evaluation for Development: Principles for Action

IE4D Group, January 2011. Available as pdf

“The authors of this paper come from a variety of perspectives. As scholars, practitioners, and commissioners of evaluation in development, research and philanthropy, our thematic interests, disciplines, geographic locale, and experiences may differ but we share a fundamental belief that evaluative knowledge has the potential to contribute to positive social change.

We know that the full potential of evaluation is not always (or even often) realized in international development and philanthropy. There are many reasons for this – some to do with a lack of capacity, some methodological, some due to power imbalances, and some the result of prevailing incentive structures. Evaluation, like development, needs to be an open and dynamic enterprise. Some of the current trends in evaluation, especially in impact evaluation in international development, limit unnecessarily the range of approaches to assessing the impact of development initiatives.

We believe that impact evaluation needs to draw from a diverse range of approaches if it is to be useful in a wide range of development contexts, rigorous, feasible, credible, and ethical.

Developed with support from the Rockefeller Foundation this article is a contribution to ongoing global and regional discussions about ways of realizing the potential of impact evaluation to improve development and strengthening our commitment to work towards it.”

Patricia Rogers is Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology, Australia. Her work focuses on credible and useful evaluation methods, approaches and systems for complicated and complex programs and policies.
Sanjeev Khagram is a professor of public affairs and international studies at the University of Washington as well as the Lead Steward of Innovations for Scaling Impact (iScale).
David Bonbright is founder and Chief Executive of Keystone (U.K., U.S. and South Africa), which helps organizations develop new ways of planning, measuring and reporting social change. He has also worked for the Aga Khan Foundation, Ford Foundation and Ashoka.
Sarah Earl is Senior Program Specialist in the Evaluation Unit at the International Development Research Centre (Canada). Her interest is ensuring that evaluation and research realize their full potential to contribute to positive social change.
Fred Carden is Director of Evaluation at the International Development Research Centre (Canada). His particular expertise is in the development and adaptation of evaluation methodology for the evaluation of development research.
Zenda Ofir is an international evaluation specialist, past President of the African Evaluation Association (AfrEA), former board member of the American Evaluation Association and the NONIE Steering Committee, and evaluation advisor to a variety of international organizations.
Nancy MacPherson is the Managing Director for Evaluation at the Rockefeller Foundation based in New York. The Foundation’s Evaluation Office aims to strengthen evaluative practice in philanthropy and development by supporting rigorous, innovative and context appropriate approaches to evaluation and learning.

Measuring Results: A GSDRC Topic Guide

Available as linked pages on the Governance and Social Development Resource Centre (GSDRC), website as of August 2011

The guide is designed to provide a quick and easy way for development professionals to keep in touch with key debates and critical issues in the field of monitoring and evaluation. It will be updated on a quarterly basis.

About this guide
“How can the impact of governance and social development programmes be assessed with a view to improving their efficiency and effectiveness? What particular challenges are involved in monitoring and evaluating development interventions, and how can these be addressed? How can the ‘value for money’ of a particular intervention be determined?

Monitoring and evaluation (M&E) is vital to ensuring that lessons are learned in terms of what works, what does not, and why. M&E serves two main functions: 1) it builds accountability by demonstrating good use of public funds; and 2) it supports learning by contributing to knowledge about how and why programmes lead to intended (or unintended) outcomes. There can sometimes be a tension between these functions.

This guide introduces some of the core debates and considerations for development practitioners involved in designing and managing M&E activities. It introduces key tools and approaches, provides case studies of applying different methodological approaches, and presents lessons learned from international experience of M&E in a range of developing country contexts. While the guide focuses on M&E for governance and social development programmes, it has relevance for all programmes.

The guide was originally prepared by Claire Mcloughlin, and a comprehensive update was undertaken by Oliver Walton in July 2011. The GSDRC appreciates the contributions of Claire Vallings and Lina Payne (DFID) and Hugh Waddington and colleagues at 3ie. Comments, questions or documents for consideration should be sent to enquiries@gsdrc.org.”

Evaluating Legislation and Other Non-Expenditure Instruments in the Area of Information Society and Media

March 2011, DG INFORMATION SOCIETY AND MEDIA

[This manual]”… addresses the pressing need for more guidance in evaluating legislation, which will become an increasingly important part of the [EC] policy officers’ work in the coming years, just as impact assessment is now. More focus on evaluation should be seen as an opportunity to enhance the quality of policy making.

It explains what the evaluation of legislation and other non-expenditure instruments is and gives simple and straightforward guidance on how to go about it. After reading the manual a reader will:

  • understand more fully what the purpose of evaluating legislation is and how it can help with policy work;
  • be able to structure well and efficiently conduct the evaluation process to obtain relevant and high-quality output and to maximize the use of the results of an evaluation in the policy cycle;
  • predict risks and difficulties in the evaluation process and be able to minimize them;
  • understand what the basic evaluation methods and techniques are, and how to use them effectively in the specific context of evaluating legislation
  • be able to find additional guidance and support

The manual is divided in two volumes. Volume 1 will guide a reader step-by-step through the entire evaluation process, from preparation and planning to data collection and analysis, to the dissemination and the use of evaluation results. Volume 2 contains a toolbox on methods and techniques tailored to the specific needs of evaluating legislation”

Documents:

Additional Information/documentation

Mr Bartek Tokarz
European Commission – Directorate General Information Society and Media
Policy Coordination and Strategy
Evaluation and Monitoring
Tel no.:  +32 2 298 58 90
email: infso-c3 in the domain ec.europa.eu

The “Real Book” for story evaluation methods

Marc Maxson, Irene Guijt, and others, 2010. GlobalGiving Foundation (supported by Rockefeller Foundation). Available as pdf.  See also the related website.

[“Real Book” = The Real Book is a central part of the culture of playing music where improvisation is essential. Real books are not for beginners: the reader interprets scant notation, and builds on her own familiarity with chords. The Real Book allows musicians to play an approximate version of hundreds of new songs quickly]

About this book
“This is a collection of narratives that serve to illustrate some not-so-obvious lessons that affected our story pilot project in Kenya. We gathered a large body of community stories that revealed what people in various communities believed they  needed, what services they were getting, and what they would like to see happen in the future. By combining many brief narratives with a few contextual questions we were able to compare and analyze thousands of stories. Taken together, these stories and their meanings provide a perspective with both depth and breadth: Broad enough to inform an organization’s strategic thinking about the root causes of social ailments2, yet deep and real enough to provoke specific and immediate follow-up actions by the local organizations of whom community members speak.

We believe that local people are the “experts” on what they want and know who has (or has not) been helping them. And like democracy, letting them define the problems and solutions that deserve to be discussed is the best method we’ve found for aggregating that knowledge. Professionals working in this field can draw upon the wisdom of this crowd for understanding the local context, and build upon what they know. Community efforts are complex, and our aim is not to predict the future, but help local leaders manage the present. If projects are observed from many angles – especially by those for whom success affects their livelihood – and implementers use these perspectives to mitigate risks and avoid early failure, the probability of future success will be much greater.”

See also:

RD comment 1: See also a different perspective on the Global Giving experience: Networks of self-categorised stories, by Rick Davies

RD comment 2: What I like about this doc: 1. Lots of warts and all descriptions of data collection, with all the problems that occur in real life, 2. the imaginative improvement on Cognitive Edge’s use of triads as tools to enabling self-signifier tools, a circular device call the story marbles approach. This enables respondents to choose which of x categories they will use and then indicate to what extent each of these categories apply to their story. It meets the requirement the author described thus: “What we need is a means to let the storyteller define the right question while also constraining the possible questions enough that we will derive useful clusters of stories with similar question frames.”

Theory of Change: A thinking and action approach to navigate in the complexity of social change processes

Iñigo Retolaza Eguren, HIVOS/DD/UNDP, May 2011 Available as pdf.

“This guide has been jointly published by Hivos and UNDP, and is aimed at the rich constellation of actors linked to processes of social development and change: bilateral donors, community leaders, political and social leaders, NGO’s representatives, community-base organizations, social movements, public decision makers, and other actors related to social change processes.

The Theory of Change approach applied to social change processes represents a thinking-action alternative to other more rigid planning approaches and logics. When living in complex and conflictive times, we need to count with more flexible instruments that allow us to plan and monitor our actions in uncertain, emergent, and complex contexts from a flexible and non-rigid logic. As known, this thinking-action approach is also applied to institutional coaching processes and to the design of social development and change programs.

In general terms, the Guide synthesizes the core of the methodological contents and steps that are developed in a Theory of Change design workshop. The first part of the Guide describes some theoretical elements to consider when designing a Theory of Change applied to social change processes. The second part describes the basic methodological steps to develop in every design of a Theory of Change. For reinforcing this practical part, a workshop route is included, illustrating the dynamics in a workshop of this kind.

The approach and contents of the guide emerge from the learning synthesis of the author, Iñigo Retolaza, as facilitator of Theory of Change design processes where social change actors from several Latin American countries have been involved. His two main bodies of experience and knowledge are: (i) the learning space offered by Hivos, where he could facilitate several Theory of Change workshops with Hivos partner organisations in South and Central America, and (ii) his professional relation with the Democratic Dialogue Regional Project of UNDP, from a research-action approach around dialogic processes applied to various areas of the socio-political field: national dialogues on public policy making and adjusting and legislative proposals, facilitation of national and regional dialogue spaces on several issues, capacity building on dialogue for social and political leaders from several countries in the region”

 

 

Impact Evaluation in Practice

Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch, World Bank, 2011

Impact Evaluation in Practice is available as downloadable pdf, and can be bought online.

“Impact Evaluation in Practice presents a non-technical overview of how to design and use impact evaluation to build more effective programs to alleviate poverty and improve people’s lives. Aimed at policymakers, project managers and development practitioners, the book offers experts and non-experts alike a review of why impact evaluations are important and how they are designed and implemented. The goal is to further the ability of policymakers and practitioners to use impact evaluations to help make policy decisions based on evidence of what works the most effectively.

The book is accompanied by a set of training material — including videos and power point presentations — developed for the “Turning Promises to Evidence” workshop series of the Office of the Chief Economist for Human Development. It is a reference and self-learning tool for policy-makers interested in using impact evaluations and was developed to serve as a manual for introductory courses on impact evaluation as well as a teaching resource for trainers in academic and policy circles.

CONTENTS
PART ONE. INTRODUCTION TO IMPACT EVALUATION
Chapter 1. Why Evaluate?
Chapter 2. Determining Evaluation Questions
PART TWO. HOW TO EVALUATE
Chapter 3. Causal Inference and Counterfactuals
Chapter 4. Randomized Selection Methods
Chapter 5. Regression Discontinuity Design
Chapter 6. Difference-in-Differences
Chapter 7. Matching
Chapter 8. Combining Methods
Chapter 9. Evaluating Multifaceted Programs
PART THREE. HOW TO IMPLEMENT AN IMPACT EVALUATION
Chapter 10. Operationalizing the Impact Evaluation Design
Chapter 11. Choosing the Sample
Chapter 12. Collecting Data
Chapter 13. Producing and Disseminating Findings
Chapter 14. Conclusion

Reflexive Monitoring in Action: A guide for monitoring system innovation projects

“Researchers at Wageningen University and the VU University Amsterdam, the Netherlands, have been working together on a type of monitoring that they have called reflexive monitoring in action (RMA).  RMA has been developed especially for projects that aim to contribute to the sustainable development of a sector or region by working on system innovation.   Sustainable development demands simultaneous changes at many levels of society and in multiple domains: ecological, economic, political and scientific. It requires choices to be made that are radically different from the usual practices, habits, interrelationships and institutional structures. But that is precisely why it is not easy. System innovation projects therefore benefit from a type of monitoring that encourages the ‘reflexivity’ of the project itself, its ability to affect and interact with the environment within which it operates. If a project wants to realise the far-reaching ambitions of system innovation, then reflection and learning must be tightly interwoven within it. And that learning should focus on structural changes. RMA can contribute to this.   In the guide, -aiming at supporting the work of project managers, monitors and clients-, the authors present the characteristics and the value of Reflexive Monitoring in Action, together with practical guidelines that will help put that monitoring into practice. At the end of the guide the authors provide detailed descriptions of seven monitoring tools.”

The guide can be freely downloaded in pdf format, in English or Dutch, from http://tinyurl.com/wurcispubs or http://tinyurl.com/vupubs.

The guide is also available in printed version (Dutch only), through Boxpress (http://www.boxpressshop.nl). Price: € 49,95 (full colour) or € 29,95 (black-white with pictures in full colour). For more information please contact: barbara.vanmierlo@wur.nl

%d bloggers like this: