International Conference on National Evaluation Capacities.

 

Date: 12-14 September 2011
Venue: Johannesburg, South Africa

UNDP Evaluation Office and the Public Service Commission (PSC) of South Africa are co-hosting the second International Conference on National Evaluation Capacities. See the official website here.

This is a follow up conference to the 2009 International Conference on National Evaluation Capacities held in Casablanca, Morocco, which was organized by the UNDP Evaluation Office in partnership with the Moroccan National Observatory for Human Development.

Objectives

1. To share experiences from countries with different levels of development of national M&E systems including those who may be considering creating one and have important experiences with other types of evaluation efforts;

2. To identify lessons and constraints in implementing national M&E systems; and,

3. To identify supply and demand for technical assistance in strengthening institutional capacity for national M&E systems under the umbrella of South-South cooperation.

If you have any questions please send your inquiry to: nec.2011@undp.org

Mr. Indran A. Naidoo, Deputy Director General, Monitoring and Evaluation
Office of the Public Service Commission, South Africa

Ms. Azusa Kubota, Evaluation Specialist, UNDP Evaluation Office

Follow in Twitter: @NEC_2011 for the latest information on the International Conference on National Evaluation Capacities – 12-14 September 2011.

The Evaluation of the Paris Declaration: Phase II Report

“After the landmark international Paris Declaration on Aid Effectiveness, endorsed in 2005, what have been the improvements in the quality of global aid and its effects on development? And how can a political declaration, implemented across widely varying national contexts, be robustly evaluated?

An independent global evaluation – a fully joint study involving countries and donor agencies – has assessed these efforts to improve the effectiveness of international aid, especially since 2005. The evaluation is the largest aid evaluation ever conducted. It has been a major international effort in itself, comprising more than 50 studies in 22 partner countries and across 18 donor agencies, as well as several studies on special themes, over a period of four years. It has broken some new boundaries in the study of complex evaluation objects.

The study had the unusual object of a political declaration, implemented across very diverse national environments to varied timescales. Its methodology rejected a traditional linear approach to cause and effect, emphasising instead the importance of context. It opted to draw out the programme theory of the Declaration itself (and that of the Accra Agenda for Action) and their statement of intent. Recognising the limits of aid in development, it applied contribution analysis to assess whether and how the commitments, actors and incentives brought together by the Declaration and the Accra Agenda have delivered on their intentions. The methodology traces the logic of how the Declaration is supposed to work and illustrates the complex pathways from development objectives to results. Recognising that development is a journey, it focuses on assessing the direction of travel on each key point, and the pace and distance travelled so far.

The study concludes that the global campaign to make international aid programmes more effective is showing results, giving the best hope in half a century that aid can be better used to help developing countries raise their economic and living standards. Improvements are slow and uneven in most developing countries, however, and even more so among donor countries and aid agencies. The Evaluation report, all the component studies and the Technical Annex – which describes the methodology and process – can be found at www.busanhlf4.org and www.oecd.org/dac/evaluationnetwork/pde. The second phase of the study was managed by UK-based development consultancy IOD PARC.”

For more information email // julia@iodparc.com  IOD PARC, 16-26 Forth Street. Edinburgh EH1 3LH

RD comment: Of additional interest ” Given the importance of the Evaluation of the Paris Declaration, the Management Group commissioned an independent assessment – a meta evaluation – of the evaluation process and outcome to determine whether the evaluation meets generally accepted standards of quality and to identify strengths, weaknesses, and lessons. The Report, by Michael Quinn Patton and Jean Gornick, can be downloaded here” “Evaluation of the Phase 2 Evaluation of the Paris Declaration”

 

Measuring Results: A GSDRC Topic Guide

Available as linked pages on the Governance and Social Development Resource Centre (GSDRC), website as of August 2011

The guide is designed to provide a quick and easy way for development professionals to keep in touch with key debates and critical issues in the field of monitoring and evaluation. It will be updated on a quarterly basis.

About this guide
“How can the impact of governance and social development programmes be assessed with a view to improving their efficiency and effectiveness? What particular challenges are involved in monitoring and evaluating development interventions, and how can these be addressed? How can the ‘value for money’ of a particular intervention be determined?

Monitoring and evaluation (M&E) is vital to ensuring that lessons are learned in terms of what works, what does not, and why. M&E serves two main functions: 1) it builds accountability by demonstrating good use of public funds; and 2) it supports learning by contributing to knowledge about how and why programmes lead to intended (or unintended) outcomes. There can sometimes be a tension between these functions.

This guide introduces some of the core debates and considerations for development practitioners involved in designing and managing M&E activities. It introduces key tools and approaches, provides case studies of applying different methodological approaches, and presents lessons learned from international experience of M&E in a range of developing country contexts. While the guide focuses on M&E for governance and social development programmes, it has relevance for all programmes.

The guide was originally prepared by Claire Mcloughlin, and a comprehensive update was undertaken by Oliver Walton in July 2011. The GSDRC appreciates the contributions of Claire Vallings and Lina Payne (DFID) and Hugh Waddington and colleagues at 3ie. Comments, questions or documents for consideration should be sent to enquiries@gsdrc.org.”

RCTs for empowerment and accountability programmes

A GSDRC Helpdesk Research Report, Date: 01.04.2011, 14 pages, available as pdf.

Query: To what extent have randomised control trials been used to successfully measure the results of empowerment and accountability processes or programmes?
Enquirer: DFID
Helpdesk response
Key findings: This report examines the extent to which RCTs have been used successfully to measure empowerment and accountability processes and programmes. Field experiments present immense opportunities, but the report cautions that they are more suited to measuring short-term results with short causal chains and less suitable for complex interventions. The studies have also demonstrated divergent results, possibly due to different programme designs. The literature highlights that issues of scale, context, complexity, timeframe, coordination and bias in the selection of programmes also determine the degree of success reported. It argues that researchers using RCTs should make more effort to understand contextual issues, consider how experiments can be scaled up to measure higher-order processes, and focus more on learning. The report suggests strategies such as using qualitative methods, replicating studies in different contexts and using randomised methods with field activities to overcome the limitations in the literature.
Contents
1. Overview
2. General Literature (annotated bibliography)
3. Accountability Studies (annotated bibliography)
4. Empowerment Studies (annotated bibliography)

 

Cultural cognition and the problem of science communication

[also titled “Cultural dissensus over scientific consensus”]

These are the titles of a very interesting 50 minute presentation by Dan Kahan of the Yale Law School, available here on YouTube. It is part of a wider body of work by The Cultural Cogition Project, also at the Yale Law School.

It is about how what might be described as some core cultural values affect people’s attitudes towards evidence, both new evidence and perceptions of where the consensus lays in regard to existing evidence, in relation to a number of fields of scientific inquiry which have been the subject of some public debate. It is very relevant to evaluation because it could be argued that at the heart of much evaluation work is  “rationalist”  theory of change, that if people are presented with evidence about what works, where and when  and how, then they will adjust their policies and practices in the light of those findings. The findings presented by Dan Kahan suggest otherwise, quite dramatically. Fortunately, he also touches on some ways forward, about how to deal with the problems his work has raised.

“Its a deliberative climate that needs environmental protection, just as much as the physical environment, and providing it as a kind of public good…So this is a science of science communication, to create conditions in which people, the likelihood, of converging on the scientific truth has no connection to these kinds of values , how to do that is kind of complicated, but I do want to start by appealing to you that is the kind of goal we should have”

There is also an associated paper, available as pdf: “Cultural Cognition of Scientific Consensus” by Dan M. Kahan,Hank Jenkins-Smith, Donald Braman, Journal of Risk Research, Vol. 14, pp. 147-74, 2011, Yale Law School, Public Law Working Paper No. 205

A comic’al perspective on methodology issues

Courtesy of XKCD

As the saying goes, “If you torture the data for long enough, it will tell you what you want to hear”

On the risks of data mining, and the management of those risks, see Walking the talk: the need for a trial registry for development interventions, also on this site.

DPC Policy Discussion Paper: Evaluating Influencing Strategies and Interventions

A paper to the DFID Development Policy Committee. Available as pdf  June 2011

Introduction
“1 The Strategy Unit brief of April 2008 envisaged that DFID should become more systematic in planning and implementing influencing efforts. Since then, procedures and guidance have been developed and there is an increasingly explicit use of influencing objectives in project log frames and more projectisation of influencing efforts. Evaluation studies and reports have illustrated the wide variety of DFID influencing efforts and the range of ambition and resources involved in trying to generate positive changes in the aid system or in partner countries. These suggest that being clear and realistic about DFID’s influencing objectives, the stakeholders involved and the specific changes being sought, is the fundamental requirement for an effective intervention. It is also the basis for sound monitoring and evaluation.
2 To support this initiative, the Evaluation Department organised a series of workshops in 2009 and 2010 to further develop the measurement and evaluation of influencing interventions producing a draft How to Note with reference to multilateral organisations in September 2010. However, with the changes to DFID’s corporate landscape in 2010 and early 2011 this work was put on hold pending the conclusion of some key corporate pieces of work .
3. An increase in demand for guidance is also noted given the changing external environment. DFID is now positioning itself to address the demands of the changing global aid landscape with new initiatives, such as the Global Development Partnerships programme. This has a relatively small spend, however its success will be measured largely by the depth and reach of its influence.
4. The Evaluation Department is now seeking guidance on how important the Development Policy Committee considers the evaluation of influencing interventions, and the direction in which it would like this developed.
5. This Paper sets out why evaluation of influencing interventions is important, why now, key theories of change and an influencing typology, value for money of an influencing intervention and metrics, and finally , the challenges of measuring influence.”

See also the associated “Proposed Influencing Typology”

The paper also refers to “Appraising, Measuring and Monitoring Influencing: How Can DFID Improve?” by the DFID Strategy Unit April 2008, which does not seem to be available on the web.

RD Comment: I understand that this is considered as a draft document and that comments on it would be welcomed. Please feel free to make your comments below

Measuring National Well-being: Measuring What Matters

[UK] National Statistician’s Reflections on the National Debate on Measuring National Well-being July 2011

Foreword
Introduction
Chapter 1: What is national well-being? … 4
Chapter 2: Why measure national well-being and who will use the measures? . 9
Chapter 3: Measuring national well-being…..15
Chapter 4: Partnerships and next steps..20
References. 24
Notes. 25

Foreword
“On 25 November 2010, I accepted an invitation from the Prime Minister, David Cameron, to develop measures of national well-being and progress. I am convinced that this is something that can only be done with an understanding of what matters most to people in this country.

In response to this invitation, the Office for National Statistics (ONS) undertook a national debate on ‘what matters to you?’ between 26 November 2010 and 15 April 2011. I was impressed by the number of people who were willing to take part in discussions and also by the depth of responses. In total, ONS held 175 events, involving around 7,250 people. In total the debate generated 34,000 responses, some of which were from organisations and groups representing thousands more. The quotes on each page of this report were taken from online contributions, where permission was given to reproduce the participant’s words anonymously. I am grateful to everyone who took the time to take part in the debate, and to those who organised and hosted events.

The debate has helped us identify the key areas that matter most and will help to ensure that the measures we use will be relevant not only to government but also to the wider public. This is crucial to allow for effective development and appraisal of policy for individuals to use information to identify ways of improving well-being, and to allow for assessment of how society is doing overall.

The term ‘well-being’ is often taken to mean ‘happiness’. Happiness is one aspect of the well-being of individuals and can be measured by asking them about their feelings – subjective well-being. As we define it, well-being includes both subjective and objective measures. It includes feelings of happiness and other aspects of subjective well-being, such as feeling that one’s activities are worthwhile, or being satisfied with family relationships. It also includes aspects of well-being which can be measured by more objective approaches, such as life expectancy and educational achievements. These issues can also be looked at for population groups – within a local area, or region, or the UK as a whole.
Developing better measures of well-being and progress is a common international goal and the UK is working with international partners to develop measures that will paint a fuller picture of our societies. This is a long-term programme and I am committed to sharing our ideas and proposals widely. This will help to ensure that UK well-being measures are relevant and founded on what matters to people, both as individuals and for the UK as a whole as well as being reliable and impartial and serving to improve our understanding of UK society.

This report summarises the contributions made to the debate and explains how ONS is using the findings to develop measures of national well-being. I look forward to your further comments and advice in response to this report. These should be sent to nationalwell-being@ons.gov.uk.”
Jil Matheson
National Statistician

See more on the ONS website

A list of M&E training providers

Update 2014 12 20: The contents of this page have become woefully out of date and it would be more than a full time job to keep it up to date.

My advice is as now as follows:

If you are looking for M&E training opportunities visit the MandE NEWS Training Forum, which lists all upcoming training events. There are many training providers listed there, along with links to their websites

Please also consider taking part in the online survey of training needs.

If you are a training provider, please look at the cumulative results to date of that survey.

I have now deleted all the previous training providers that were shown below

A results take-over of aid effectiveness? How to balance multiple or competing calls for more accountability

Date: 25 July 2011 12:00-13:30 (GMT+01 (BST))
Venue: British Academy, London

This debate will explore possible tensions – and opportunities – when donors seek to reassure domestic publics that aid is being spent well, while also endeavouring to support the needs and priorities of aid recipient countries and their citizens.

The language of results is not new – it is integral to the aid effectiveness agenda. But against the backdrop of growing financial constraints, it is receiving renewed emphasis in many donor countries. This debate will explore possible tensions, as well as opportunities, where donors seek to reassure domestic publics that aid is being spent well while they also endeavour to support the needs and priorities of aid recipient countries and their citizens. How can domestic accountability to both these constituencies be supported more effectively? Are there tensions between these different stakeholders and forms of accountability, and how can they be addressed?

Speakers:
Sarah Cliffe – Special Representative and Director, World Development Report 2011: Conflict, Security, and Development
Sue Unsworth – The Policy Practice, and ODI Board Member
Alan Hudson – Senior Policy Manager, Governance (Transparency & Accountability), ONE
John Morlu – former Auditor General, Liberia
Chair:  Alison Evans – Director, ODI

An ODI and BBC World Service Trust public event in the Busan and beyond: aid effectiveness in a new era series.

Click for more details           Register to attend this event

%d bloggers like this: