Monitoring and Evaluation NEWS
A news service focusing on developments in monitoring and evaluation methods relevant to development programmes with social development objectives. Managed by Rick Davies, since 1997
Courtesy of XKCD
As the saying goes, “If you torture the data for long enough, it will tell you what you want to hear”
On the risks of data mining, and the management of those risks, see Walking the talk: the need for a trial registry for development interventions, also on this site.
A paper to the DFID Development Policy Committee. Available as pdf June 2011
Introduction
“1 The Strategy Unit brief of April 2008 envisaged that DFID should become more systematic in planning and implementing influencing efforts. Since then, procedures and guidance have been developed and there is an increasingly explicit use of influencing objectives in project log frames and more projectisation of influencing efforts. Evaluation studies and reports have illustrated the wide variety of DFID influencing efforts and the range of ambition and resources involved in trying to generate positive changes in the aid system or in partner countries. These suggest that being clear and realistic about DFID’s influencing objectives, the stakeholders involved and the specific changes being sought, is the fundamental requirement for an effective intervention. It is also the basis for sound monitoring and evaluation.
2 To support this initiative, the Evaluation Department organised a series of workshops in 2009 and 2010 to further develop the measurement and evaluation of influencing interventions producing a draft How to Note with reference to multilateral organisations in September 2010. However, with the changes to DFID’s corporate landscape in 2010 and early 2011 this work was put on hold pending the conclusion of some key corporate pieces of work .
3. An increase in demand for guidance is also noted given the changing external environment. DFID is now positioning itself to address the demands of the changing global aid landscape with new initiatives, such as the Global Development Partnerships programme. This has a relatively small spend, however its success will be measured largely by the depth and reach of its influence.
4. The Evaluation Department is now seeking guidance on how important the Development Policy Committee considers the evaluation of influencing interventions, and the direction in which it would like this developed.
5. This Paper sets out why evaluation of influencing interventions is important, why now, key theories of change and an influencing typology, value for money of an influencing intervention and metrics, and finally , the challenges of measuring influence.”
See also the associated “Proposed Influencing Typology”
The paper also refers to “Appraising, Measuring and Monitoring Influencing: How Can DFID Improve?” by the DFID Strategy Unit April 2008, which does not seem to be available on the web.
RD Comment: I understand that this is considered as a draft document and that comments on it would be welcomed. Please feel free to make your comments below
[UK] National Statistician’s Reflections on the National Debate on Measuring National Well-being July 2011
Foreword
Introduction
Chapter 1: What is national well-being? … 4
Chapter 2: Why measure national well-being and who will use the measures? . 9
Chapter 3: Measuring national well-being…..15
Chapter 4: Partnerships and next steps..20
References. 24
Notes. 25
Foreword
“On 25 November 2010, I accepted an invitation from the Prime Minister, David Cameron, to develop measures of national well-being and progress. I am convinced that this is something that can only be done with an understanding of what matters most to people in this country.
In response to this invitation, the Office for National Statistics (ONS) undertook a national debate on ‘what matters to you?’ between 26 November 2010 and 15 April 2011. I was impressed by the number of people who were willing to take part in discussions and also by the depth of responses. In total, ONS held 175 events, involving around 7,250 people. In total the debate generated 34,000 responses, some of which were from organisations and groups representing thousands more. The quotes on each page of this report were taken from online contributions, where permission was given to reproduce the participant’s words anonymously. I am grateful to everyone who took the time to take part in the debate, and to those who organised and hosted events.
The debate has helped us identify the key areas that matter most and will help to ensure that the measures we use will be relevant not only to government but also to the wider public. This is crucial to allow for effective development and appraisal of policy for individuals to use information to identify ways of improving well-being, and to allow for assessment of how society is doing overall.
The term ‘well-being’ is often taken to mean ‘happiness’. Happiness is one aspect of the well-being of individuals and can be measured by asking them about their feelings – subjective well-being. As we define it, well-being includes both subjective and objective measures. It includes feelings of happiness and other aspects of subjective well-being, such as feeling that one’s activities are worthwhile, or being satisfied with family relationships. It also includes aspects of well-being which can be measured by more objective approaches, such as life expectancy and educational achievements. These issues can also be looked at for population groups – within a local area, or region, or the UK as a whole.
Developing better measures of well-being and progress is a common international goal and the UK is working with international partners to develop measures that will paint a fuller picture of our societies. This is a long-term programme and I am committed to sharing our ideas and proposals widely. This will help to ensure that UK well-being measures are relevant and founded on what matters to people, both as individuals and for the UK as a whole as well as being reliable and impartial and serving to improve our understanding of UK society.
This report summarises the contributions made to the debate and explains how ONS is using the findings to develop measures of national well-being. I look forward to your further comments and advice in response to this report. These should be sent to nationalwell-being@ons.gov.uk.”
Jil Matheson
National Statistician
See more on the ONS website
Update 2014 12 20: The contents of this page have become woefully out of date and it would be more than a full time job to keep it up to date.
My advice is as now as follows:
If you are looking for M&E training opportunities visit the MandE NEWS Training Forum, which lists all upcoming training events. There are many training providers listed there, along with links to their websites
Please also consider taking part in the online survey of training needs.
If you are a training provider, please look at the cumulative results to date of that survey.
I have now deleted all the previous training providers that were shown below
Date: 25 July 2011 12:00-13:30 (GMT+01 (BST))
Venue: British Academy, London
This debate will explore possible tensions – and opportunities – when donors seek to reassure domestic publics that aid is being spent well, while also endeavouring to support the needs and priorities of aid recipient countries and their citizens.
The language of results is not new – it is integral to the aid effectiveness agenda. But against the backdrop of growing financial constraints, it is receiving renewed emphasis in many donor countries. This debate will explore possible tensions, as well as opportunities, where donors seek to reassure domestic publics that aid is being spent well while they also endeavour to support the needs and priorities of aid recipient countries and their citizens. How can domestic accountability to both these constituencies be supported more effectively? Are there tensions between these different stakeholders and forms of accountability, and how can they be addressed?
Speakers:
Sarah Cliffe – Special Representative and Director, World Development Report 2011: Conflict, Security, and Development
Sue Unsworth – The Policy Practice, and ODI Board Member
Alan Hudson – Senior Policy Manager, Governance (Transparency & Accountability), ONE
John Morlu – former Auditor General, Liberia
Chair: Alison Evans – Director, ODI
An ODI and BBC World Service Trust public event in the Busan and beyond: aid effectiveness in a new era series.
Click for more details Register to attend this event
Date: 17-21 October 2011
Venue: John Knox Centre, Geneva, Switzerland
Dear Colleagues:
A proposal prepared by the Evaluation Capacity Development Group (ECDG) and the Joint Committee on Standards for Educational Evaluation (JCSEE), in partnership with the International Organization for Cooperation in Evaluation (IOCE), to create an International Workshop Agreement (IWA) on evaluation capacity development (ECD) was recently approved by the International Organization for Standardization (ISO).
Everyone agrees that there is an acute need to develop evaluation capacity. However, resolution of the problem has not been possible because there is no agreement on HOW to develop evaluation capacity. Some think that individual evaluators should be better trained through workshops and seminars. Others think that organizations should be redesigned to enable the achievement of a shared vision for evaluation. And, yet others think that evaluation should be institutionalized in national governments to promote accountability to their citizens.
We are now organizing a workshop that will be held 17-21 October 2011 at the John Knox Centre, Geneva, Switzerland. The workshop will use a systems approach to develop an IWA that integrates ECD at the individual, organizational and national levels. I am particularly pleased to inform you that a leading expert in systems-based evaluation, Bob Williams, has consented to facilitate the event.
As per the procedures explained in Annex SI of the Supplement to the ISO/IEC Directives, ANY organization with an interest in evaluation capacity development can register to send a representative to the workshop to participate in the preparation of this important document. Limited support may be available. To learn more about the workshop and to register please go to http://www.ecdg.net/
Best Regards,
Karen Russon
President
Evaluation Capacity Development Group
Date: 3-5 October 2012
Venue: Helsinki, Finland
EVALUATION IN THE NETWORKED SOCIETY: NEW CONCEPTS, NEW CHALLENGES, NEW SOLUTIONS
The Tenth Biennial Conference of the European Evaluation Society will be the international evaluation event of the year. It will be held in Helsinki, Finland during 3-5 October 2012 (pre-conference workshops 1- 2 October). Mark your calendars!!
Evaluators are living in times of unprecedented challenge and opportunity. The networked information environment is inducing fundamental changes in culture, politics and society. Whereas the industrial society was reliant on centralised, hierarchical, high cost information systems, the networked society is characterised by decentralised, voluntary and cheap information exchange.
The advent of social networking without borders will have fundamental implications for evaluation agendas and methods. First, it will redefine the value and legitimacy of evaluation in global social accountability networks and accelerate the internationalisation of evaluation. Second, evaluation cultures, structures and processes will have to deal with the limitless quantity, speed and accessibility of information generated by new technologies, e.g. drawing useful meaning from huge data bases, assessing the validity of an exploding number of rating systems, league tables, etc. in ways consistent with democratic values of freedom of expression and protection of privacy.
The new information technologies offer new ways of making authority responsible and accountable as well as bringing real time citizen involvement and reliable information to bear on public policy making. What are the implications of an information economy that allows instant connectivity to thousands of program beneficiaries suddenly able to make their voices heard? Will the spread of mobile telephony to the weakest and most vulnerable members of society and the rising power of social networks act as evaluative and recuperative mechanisms or will they merely aggravate social instability? What are the risks of network capture by single or special interest groups and cooptation of evaluation?
The rise of the evaluation discipline is inextricably linked to the values central to any democratic society. How will these values be protected in a context where weak links and increasing inequalities have created new fissures in society? How will evaluation independence be protected against the pressures of vested interests intent on retaining control over the commanding heights of the society?
To help explore these and other issues relevant to the prospects of evaluation in Europe and beyond the Conference will stimulate evaluators to share ideas, insights and opinions about a wide range of topics that will throw light on the future roles of evaluation in the networked society. The Conference will help draw evaluation lessons learnt in distinct sectors and regions of the world. It will also examine the potential of alternative and mixed evaluation methods in diverse contexts and probe the challenges of assessing public interest in complex adaptive systems and networks.
To these ends the Conference will offer participants a wide choice of vehicles for the transmission of evaluation experience and knowledge: keynote speeches, paper presentations, panel debates, posters, etc. As in past years the EES Conference will aim at a pluralistic agenda that respects the legitimacy of different standpoints, illuminates diverse perspectives and promotes principled debate. The Conference will also provide an opportunity for evaluation networks to interact and improve the coherence of their activities.
We look forward to welcoming you in Helsinki. It is one of the world leaders in modern design and it provides Europe with a a world class high tech platform. It also boasts a 450 year history and lays claim to being the warmest, friendliest, most “laid back” city of Northern Europe. Its nearby archipelago of islands offers an ideal environment for sea cruises and its neighboring old growth forests provide an idyllic setting for restful nature walks. We promise you an enjoyable as well as a professionally rewarding time!!
Ian Davies, President, European Evaluation Society
Maria Bustelo, Vice President and President Elect, European Evaluation Society
The ‘Independent Review of Aid Effectiveness’ and the Government’s response were released on 6 July 2011 by Foreign Minister Kevin Rudd, in an official launch at Parliament House, followed by a Ministerial Statement to Parliament. For an overview, see this page on the AusAID website
Independent Review of Aid Effectiveness:
Commissioned in November 2010, this was the first independent review of the aid program in 15 years. It made 39 recommendations to improve the program
Australian Government response:
The Government has agreed (or agreed in principle) to 38 of the recommendations. Including that the agency develop a three-tiered results framework for reporting on agency-wide performance.
See also
RD Comment: The following section on Independent Evaluation is of particular interest [underlining added]:
ii) Independent Evaluations
“AusAID’s Independent Completion Reports and Independent Progress Reports are another key part of its Performance Management and Evaluation Policy.
Under current guidelines, a report must be completed for an activity every four years, either during its implementation (a progress report) or at completion (a completion report). Reports are required for projects above $3 million and are meant to be made public. They are independent in that they are done by individuals not involved in the project. Typically, but not always, they are written by non–AusAID staff.
By international standards, this policy is thorough. For example, at the World Bank, independent completion reports are done only for a sample of projects
But a study of AusAID evaluation reports commissioned by the Review Panel found that implementation of AusAID’s evaluation policy is patchy:
• Of 547 projects that should have had a completion or progress report in 2006–10, only 170 were recorded as having been done.
• Of the 170, only 118 could be found.
• About 26 per cent of the completion and progress reports were assessed to be too low quality to publish.
• Only about 20 have been published on the AusAID website.
Clearly, the policy is not being fully followed. Other problems were also evident. None of the 118 completion or progress reports reviewed provided an unsatisfactory rating. This raises questions of credibility. In comparison, 20 per cent of World Bank projects are rated unsatisfactory by its independent evaluation group.
There is also a structural issue with the policy: AusAID program managers must approve the publication of an independent report. This risks conflicts of interest and long delays in publication. The low rate of publication suggests these problems may be occurring.
Independent completion reports, when done and published, can be very useful. For example, the completion report on the first phase of the Indonesia Basic Education Project is in the public domain and helped to inform recent public debate about the second phase of the project (AusAID 2010b). In contrast, several useful completion reports have recently been done for the PNG program, but only one has been released.
Given the problems described above, it is not surprising that the Review Panel has seen little evidence that these reports inform and improve aid delivery.“
