Month: August 2011

Measuring Impact: Lessons from the MCC for the Broader Impact Evaluation Community

William Savedoff and Christina Droggitis, Centre for Global Development, Aug 2011. Available as pdf (2 pages)

Excerpt:

“One organization that has taken the need for impact evaluation seriously is the Millennium Challenge Corporation. The first of the MCC programs came to a close this fiscal year, and in the next year the impact evaluations associated with them will begin to be published.

Politicians’ responses to the new wave of evaluations will set a precedent, either one that values transparency and encourages aid agencies to be public about what they are learning or one that punishes transparency and encourages agencies to hide findings or simply cease commissioning evaluations.”

The Canadian M&E System: Lessons Learned from 30 Years of Development

by Robert Lahey, November 2010, ECD Working Paper Series, Independent Evaluation Group, World Bank. Available as pdf

Foreword

As part of its activities, the World Bank Group’s Independent Evaluation Group (IEG) provides technical assistance to member developing countries for designing and implementing effective monitoring and evaluation (M&E) systems and for strengthening government evaluation capacities as an important part of sound governance. IEG prepares resource materials, with case studies demonstrating good or promising practices, which other countries can refer to or adapt to suit their own particular circumstances (http://www.worldbank.org/ieg/ecd).

World Bank support to strengthen M&E systems in different countries has grown substantially in the past decade. There is intense activity on M&E issues in most regions, and IEG has provided support to governments and World Bank units, particularly since 1997, on ways to further strengthen M&E systems, with the objective of fully institutionalizing countries’ efforts.

While several World Bank assessments have been done on the strengths and weaknesses of developing countries’ M&E systems, fewer analyses have looked at OECD country experiences with a view to help identify and document approaches, methods, and “good practices,” and to promote knowledge sharing of those cases as key references for developing country systems in the process of design and implementation.

This Evaluation Capacity Development (ECD) paper seeks to provide an overview of the Canadian model for monitoring and evaluation developed over the past three decades. The Canadian M&E system is one that has invested heavily in both evaluation and performance monitoring as key tools to support accountability and results-based management in government.

The paper tracks the evolution of Canada’s M&E system to its current state, identifying key lessons learned from public sector experience. It offers insights from officials’ own perspectives, highlights key initiatives introduced to help drive the M&E system, and discusses the demands for public sector reforms and the emphasis they have placed on M&E in public sector management.

It is hoped that the lessons and practices identified here will benefit officials undertaking similar tasks in other countries.

This paper was peer reviewed by Anne Routhier, head of the Center of Excellence for Evaluation (CEE) at the Treasury Board Secretary of Canada; Keith Mackay and Manuel Fernando Castro, M&E experts and former World Bank Senior Evaluation Officers; and Nidhi Khattri and Ximena Fernandez Ordonez, IEG. The paper was edited for publication by Helen Chin, IEG. Their comments and feedback are gratefully acknowledged. The views expressed in this document are solely those of the author, and do not necessarily represent the views of the World Bank or of the government of Canada.

Australasian Evaluation Society 2011 International Conference: Evaluation and Influence

 

Date: 29 August – 2 September (workshops on 29-30th)
Venue: Hilton, Sydney, NSW, Australia

View the Conference website here

View the detailed outline of the program and please click here to view the detailed Pre Conference Workshop Program. Please note the Conference Program is subject to change.

Read more about the Keynote Speakers, their presentations and Conference main streams and ‘hot topics’.

Evaluation and influence

Evaluation claims to influence public policy, professional practice and the management of organisations. What is the nature and extent of this influence? How can evaluations be made more influential?  And conversely, in a rapidly changing world, what are the main influences on evaluation? To what extent is evaluation responding by taking on new approaches and technologies?

With the focus on influence, the conference builds upon the two previous conferences with their themes of evidence (Canberra 2009) and reflections on evaluation (Wellington 2010).

The conference will focus on three sub-themes:

The influence of evaluation on society

How much and in what ways does evaluation impact upon policy, practice and organisations? Where and in what circumstances does it have the most impact, and why? What are other important sources of influence, and how do they compare with evaluation?

Making an evaluation more influential

How can an evaluation be designed and conducted to increase its use and influence?  What are the most persuasive ways of communicating the results of an evaluation?What role can evaluators play in implementing evaluation results? What are the lessons for evaluation from theories of influence and diffusion?

Influences shaping evaluation

How is evaluation changing in response to emerging social, economic and political issues, to increasing complexity and uncertainty, and to new approaches and technologies? What are the important influences on evaluation, and how are they shaping evaluation?

The conference can explore its theme in streams around fields such as education and research, health, human services, justice, international development, Indigenous peoples, natural resource management and the economy. We also expect a stream on design and methodology.

International Conference on National Evaluation Capacities.

 

Date: 12-14 September 2011
Venue: Johannesburg, South Africa

UNDP Evaluation Office and the Public Service Commission (PSC) of South Africa are co-hosting the second International Conference on National Evaluation Capacities. See the official website here.

This is a follow up conference to the 2009 International Conference on National Evaluation Capacities held in Casablanca, Morocco, which was organized by the UNDP Evaluation Office in partnership with the Moroccan National Observatory for Human Development.

Objectives

1. To share experiences from countries with different levels of development of national M&E systems including those who may be considering creating one and have important experiences with other types of evaluation efforts;

2. To identify lessons and constraints in implementing national M&E systems; and,

3. To identify supply and demand for technical assistance in strengthening institutional capacity for national M&E systems under the umbrella of South-South cooperation.

If you have any questions please send your inquiry to: nec.2011@undp.org

Mr. Indran A. Naidoo, Deputy Director General, Monitoring and Evaluation
Office of the Public Service Commission, South Africa

Ms. Azusa Kubota, Evaluation Specialist, UNDP Evaluation Office

Follow in Twitter: @NEC_2011 for the latest information on the International Conference on National Evaluation Capacities – 12-14 September 2011.

The Evaluation of the Paris Declaration: Phase II Report

“After the landmark international Paris Declaration on Aid Effectiveness, endorsed in 2005, what have been the improvements in the quality of global aid and its effects on development? And how can a political declaration, implemented across widely varying national contexts, be robustly evaluated?

An independent global evaluation – a fully joint study involving countries and donor agencies – has assessed these efforts to improve the effectiveness of international aid, especially since 2005. The evaluation is the largest aid evaluation ever conducted. It has been a major international effort in itself, comprising more than 50 studies in 22 partner countries and across 18 donor agencies, as well as several studies on special themes, over a period of four years. It has broken some new boundaries in the study of complex evaluation objects.

The study had the unusual object of a political declaration, implemented across very diverse national environments to varied timescales. Its methodology rejected a traditional linear approach to cause and effect, emphasising instead the importance of context. It opted to draw out the programme theory of the Declaration itself (and that of the Accra Agenda for Action) and their statement of intent. Recognising the limits of aid in development, it applied contribution analysis to assess whether and how the commitments, actors and incentives brought together by the Declaration and the Accra Agenda have delivered on their intentions. The methodology traces the logic of how the Declaration is supposed to work and illustrates the complex pathways from development objectives to results. Recognising that development is a journey, it focuses on assessing the direction of travel on each key point, and the pace and distance travelled so far.

The study concludes that the global campaign to make international aid programmes more effective is showing results, giving the best hope in half a century that aid can be better used to help developing countries raise their economic and living standards. Improvements are slow and uneven in most developing countries, however, and even more so among donor countries and aid agencies. The Evaluation report, all the component studies and the Technical Annex – which describes the methodology and process – can be found at www.busanhlf4.org and www.oecd.org/dac/evaluationnetwork/pde. The second phase of the study was managed by UK-based development consultancy IOD PARC.”

For more information email // julia@iodparc.com  IOD PARC, 16-26 Forth Street. Edinburgh EH1 3LH

RD comment: Of additional interest ” Given the importance of the Evaluation of the Paris Declaration, the Management Group commissioned an independent assessment – a meta evaluation – of the evaluation process and outcome to determine whether the evaluation meets generally accepted standards of quality and to identify strengths, weaknesses, and lessons. The Report, by Michael Quinn Patton and Jean Gornick, can be downloaded here” “Evaluation of the Phase 2 Evaluation of the Paris Declaration”

 

Measuring Results: A GSDRC Topic Guide

Available as linked pages on the Governance and Social Development Resource Centre (GSDRC), website as of August 2011

The guide is designed to provide a quick and easy way for development professionals to keep in touch with key debates and critical issues in the field of monitoring and evaluation. It will be updated on a quarterly basis.

About this guide
“How can the impact of governance and social development programmes be assessed with a view to improving their efficiency and effectiveness? What particular challenges are involved in monitoring and evaluating development interventions, and how can these be addressed? How can the ‘value for money’ of a particular intervention be determined?

Monitoring and evaluation (M&E) is vital to ensuring that lessons are learned in terms of what works, what does not, and why. M&E serves two main functions: 1) it builds accountability by demonstrating good use of public funds; and 2) it supports learning by contributing to knowledge about how and why programmes lead to intended (or unintended) outcomes. There can sometimes be a tension between these functions.

This guide introduces some of the core debates and considerations for development practitioners involved in designing and managing M&E activities. It introduces key tools and approaches, provides case studies of applying different methodological approaches, and presents lessons learned from international experience of M&E in a range of developing country contexts. While the guide focuses on M&E for governance and social development programmes, it has relevance for all programmes.

The guide was originally prepared by Claire Mcloughlin, and a comprehensive update was undertaken by Oliver Walton in July 2011. The GSDRC appreciates the contributions of Claire Vallings and Lina Payne (DFID) and Hugh Waddington and colleagues at 3ie. Comments, questions or documents for consideration should be sent to enquiries@gsdrc.org.”

RCTs for empowerment and accountability programmes

A GSDRC Helpdesk Research Report, Date: 01.04.2011, 14 pages, available as pdf.

Query: To what extent have randomised control trials been used to successfully measure the results of empowerment and accountability processes or programmes?
Enquirer: DFID
Helpdesk response
Key findings: This report examines the extent to which RCTs have been used successfully to measure empowerment and accountability processes and programmes. Field experiments present immense opportunities, but the report cautions that they are more suited to measuring short-term results with short causal chains and less suitable for complex interventions. The studies have also demonstrated divergent results, possibly due to different programme designs. The literature highlights that issues of scale, context, complexity, timeframe, coordination and bias in the selection of programmes also determine the degree of success reported. It argues that researchers using RCTs should make more effort to understand contextual issues, consider how experiments can be scaled up to measure higher-order processes, and focus more on learning. The report suggests strategies such as using qualitative methods, replicating studies in different contexts and using randomised methods with field activities to overcome the limitations in the literature.
Contents
1. Overview
2. General Literature (annotated bibliography)
3. Accountability Studies (annotated bibliography)
4. Empowerment Studies (annotated bibliography)

 

Cultural cognition and the problem of science communication

[also titled “Cultural dissensus over scientific consensus”]

These are the titles of a very interesting 50 minute presentation by Dan Kahan of the Yale Law School, available here on YouTube. It is part of a wider body of work by The Cultural Cogition Project, also at the Yale Law School.

It is about how what might be described as some core cultural values affect people’s attitudes towards evidence, both new evidence and perceptions of where the consensus lays in regard to existing evidence, in relation to a number of fields of scientific inquiry which have been the subject of some public debate. It is very relevant to evaluation because it could be argued that at the heart of much evaluation work is  “rationalist”  theory of change, that if people are presented with evidence about what works, where and when  and how, then they will adjust their policies and practices in the light of those findings. The findings presented by Dan Kahan suggest otherwise, quite dramatically. Fortunately, he also touches on some ways forward, about how to deal with the problems his work has raised.

“Its a deliberative climate that needs environmental protection, just as much as the physical environment, and providing it as a kind of public good…So this is a science of science communication, to create conditions in which people, the likelihood, of converging on the scientific truth has no connection to these kinds of values , how to do that is kind of complicated, but I do want to start by appealing to you that is the kind of goal we should have”

There is also an associated paper, available as pdf: “Cultural Cognition of Scientific Consensus” by Dan M. Kahan,Hank Jenkins-Smith, Donald Braman, Journal of Risk Research, Vol. 14, pp. 147-74, 2011, Yale Law School, Public Law Working Paper No. 205