September 2011 – Monitoring and Evaluation NEWS

On evaluation quality standards: A List

The beginnings of a list. Please suggest others by using the Comment facility below

Normative statements:

DFID Evaluation Quality Assurance templates for Entry level and Exit level. Entry level refers to TORs or evaluation plans, Exit level refers to draft evaluation reports. Circa 2011.
USAID Evaluation Policy, January 2011. See Section 4 on Evalation practices, which lists 6 quality criteria
American Evaluation Association’s PROGRAM EVALUATION STANDARDS. Available online in Summary Form. The full text is Yarbrough, D. B., Shulha, L. M., Hopson, R. K., and Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage
Quality Standards for Development Evaluation, DAC Guidelines and Reference Series, OECD, 2010
Quality control in the evaluation reports, EuropeAID Co-operation Office, 2006. (Note: a quality rating is given by the EC to every final evaluation report.)
Norms for Evaluation in the UN System, UN Evaluation Group, 2005
Standards for Evaluation in the UN System, UN Evaluation Group, 2005
UNICEF’s Evaluation Standards, based on those also used by AEA above, developed by the American Joint Committee on Standards for Educational Evaluation (AJCSEE), undated
UK Evaluation Society Good Practice Guidelines, undated

Standards for specific methods (and fields):

The CONSORT Statement is intended to improve the reporting of a randomized controlled trial (RCT), enabling readers to understand a trial’s design, conduct, analysis and interpretation, and to assess the validity of its results. It emphasizes that this can only be achieved through complete transparency from authors. 2010
The TREND statement “Transparent reporting is crucial for assessing the validity and efficacy of these intervention studies, and, it facilitates synthesis of the findings for evidence-based recommendations. Therefore, the mission of the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) group is to improve the reporting standards of nonrandomized evaluations of behavioral and public health interventions” 2009
Systematic Reviews: CRD’s guidance on undertaking reviews in health care. Centre for Reviews and Dissemination, University of York, 2009
Methodology checklist: Qualitative studies, National Institute for Health and Clinical Excellence, UK, January 2009
Quality in Qualitative Evaluation: A framework for assessing research evidence A Quality Framework, by Liz Spencer, Jane Ritchie, Jane Lewis and Lucy Dillon National Centre for Social Research, UK Cabinet Office, 2003

Meta-evaluations:

Are Sida Evaluations Good Enough?An Assessment of 34 Evaluation Reports” by Kim Forss, Evert Vedung, Stein Erik Kruse,Agnes Mwaiselage, Anna Nilsdotter, Sida Studies in Evaluation 2008:1 See especially Section 6: Conclusion, 6.1 Revisiting the Quality Questions, 6.2 Why are there Quality Problems with Evaluations?, 6.3 How can the Quality of Evaluations be Improved?, 6.4 Direction of Future Studies. RD Comment: This study has annexes with empirical data on the quality attributes of 34 evaluation reports published in the Sida Evaluations series between 2003 and 2005. It BEGS a follow up study to see if/how these various quality ratings correlate in any way with the subsequent use of the evaluation reports. Could Sida pursuaded to do something like this?

Ethics focused

Australasian Evaluation Society

Guidelines for the Ethical Conduct of Evaluations , 2010
Code of Ethics, 2000
Policy on the application of the Code of Ethics, 2000

Journal articles

Assessing Quality in Applied and Educational Research: A Framework for Discussion, by John Furlong, Alis Oancea, 2008
Evaluation Standards in an International Context, Love & Russon, New Directions for Evaluation, 2005

Checklists:

Evaluation checklists prepared by the Western Michegan University ,covering Evaluation Management, Evaluation Models, Evaluation Values and Criteria, Metaevaluation, Evaluation Capacity Building / Institutionalization, and Checklist Creation

Other lists:

See also the European Evaluation Association’s lists of:
- Evaluation Standards of National and Regional Evaluation Societies and
- Evaluation Standards of International and Supranational Organisations

Workshop: Case Studies in Development Evaluation: Validity, Generalisation and Learning

Venue: University of Copenhagen
Date: May 21-23, 2012
Invitation and Call for Papers to International Workshop
Centre for Social Science Development Research

The Evaluation Department of the Danish Ministry of Foreign Affairs and the Centre for Social Science Development Research at Institute of Food and Resource Economics at the University of Copenhagen are pleased to invite you to submit an abstract (preparatory to a full paper) to an International Workshop, which will focus on methodological and practical considerations when using case studies in evaluations of development. The workshop will be organised in collaboration with the journal Evaluation:the international journal of theory, research and practice.

A large number of development evaluations are broader, learning-oriented evaluations based on cases at country, sector or project level. A key challenge in these evaluations is how to deal with the question of external validity. Within the field of development evaluation the methodological and practical debate on how to address this issue has been relatively limited.

The Organising Committee of the International Workshop is seeking abstracts that address theoretical/methodological challenges as well as more practical experiences when using case studies in learning-oriented development evaluations. Abstracts could e.g. focus on: Continue reading “Workshop: Case Studies in Development Evaluation: Validity, Generalisation and Learning”

Asian Development Bank: 2011 Annual Evaluation Review

Available at ADB website

Background

This report summarizes the key findings and lessons of evaluation studies carried out in 2010, and provides trends in the success rates of ADB operations. It also reviews the recommendations from evaluation reports and the status of actions taken by ADB Management in response to these recommendations. The report also reviews the work program accomplishments of IED in 2010.

Key Findings and Issues

Declining performance in terms of success rates. According to the past data, success rates have not reached 80%, although this is ADB’s corporate target for 2012. Performance began to decline in approval year 2000 after peaking at over 70%. Although the declining trend is supported by a limited sample size, the project and program performance report (PPR) system indicates that this trend will continue unless significant corrective measures are taken. Based on the new PPR system, about 25% of ongoing projects are facing implementation challenges and are at risk of not meeting their objectives (which confirms IED’s previous findings that portfolio performance ratings were overrated in PPRs)-an important issue that needs to be addressed.
Continue reading “Asian Development Bank: 2011 Annual Evaluation Review”

Essentials of Utilization-Focused Evaluation

Michael Quinn Patton, August 2011. Sage publications

Publisher’s description:

“Based on Michael Quinn Patton’s best-selling Utilization-Focused Evaluation, this briefer book provides an overall framework and essential checklist steps for designing and conducting evaluations that actually get used. The new material and innovative graphics present the utilization-focused evaluation process as a complex adaptive system, incorporating current understandings about systems thinking and complexity concepts. The book integrates theory and practice, is based on both research and professional experience, and offers new case examples and cartoons with Patton’s signature humor. ”
Continue reading “Essentials of Utilization-Focused Evaluation”

Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,

Date: 17-18 October 2011
Venue: ANU, Canberra, Australia

Concept Note, Chris Roche & Linda Kelly, 4 August 2011

The Developmental Leadership Program (DLP)[1] addresses an important gap in international thinking and policy about the critical role played by leaders, elites and coalitions in the politics of development. At the core of DLP thinking is the proposition that political processes shape developmental outcomes at all levels and in all aspects of society: at national and sub-national levels and in all sectors and issue areas.

Initial findings of the DLP research program confirm that development is a political process and that leadership and agency matter. This is of course not new, but the DLP research provides important insights into how, in particular, leadership, elites and formal and informal coalitions can play a particularly important and under-recognized role in institutional formation (or establishing the ‘rules of the game’), policy reform and development processes[2].

International aid therefore needs to engage effectively with political processes. It needs to be flexible and be able to respond when opportunities open up. It needs to avoid the danger of bolstering harmful political settlements.

Furthermore Monitoring & Evaluation (M&E) mechanisms need to be improved and made compatible with flexible programming and recognize the importance of ‘process’ as well as outcomes. Donors should invest in a range of monitoring and evaluation techniques and approaches which are more aligned with the kinds of non-linear and unpredictable processes which characterise the kinds of political processes which drive positive developmental outcomes. This is important because it can be argued that, at best, current approaches are simply not appropriate to monitor the kinds of processes DLP research indicates are important; or, at worst, they offer few incentives to international assistance agencies to support the processes that actually lead to developmental outcomes Continue reading “Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,”

Impact Evaluation for Development: Principles for Action

IE4D Group, January 2011. Available as pdf

“The authors of this paper come from a variety of perspectives. As scholars, practitioners, and commissioners of evaluation in development, research and philanthropy, our thematic interests, disciplines, geographic locale, and experiences may differ but we share a fundamental belief that evaluative knowledge has the potential to contribute to positive social change.

We know that the full potential of evaluation is not always (or even often) realized in international development and philanthropy. There are many reasons for this – some to do with a lack of capacity, some methodological, some due to power imbalances, and some the result of prevailing incentive structures. Evaluation, like development, needs to be an open and dynamic enterprise. Some of the current trends in evaluation, especially in impact evaluation in international development, limit unnecessarily the range of approaches to assessing the impact of development initiatives.

We believe that impact evaluation needs to draw from a diverse range of approaches if it is to be useful in a wide range of development contexts, rigorous, feasible, credible, and ethical.

Developed with support from the Rockefeller Foundation this article is a contribution to ongoing global and regional discussions about ways of realizing the potential of impact evaluation to improve development and strengthening our commitment to work towards it.”

Patricia Rogers is Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology, Australia. Her work focuses on credible and useful evaluation methods, approaches and systems for complicated and complex programs and policies.
Sanjeev Khagram is a professor of public affairs and international studies at the University of Washington as well as the Lead Steward of Innovations for Scaling Impact (iScale).
David Bonbright is founder and Chief Executive of Keystone (U.K., U.S. and South Africa), which helps organizations develop new ways of planning, measuring and reporting social change. He has also worked for the Aga Khan Foundation, Ford Foundation and Ashoka.
Sarah Earl is Senior Program Specialist in the Evaluation Unit at the International Development Research Centre (Canada). Her interest is ensuring that evaluation and research realize their full potential to contribute to positive social change.
Fred Carden is Director of Evaluation at the International Development Research Centre (Canada). His particular expertise is in the development and adaptation of evaluation methodology for the evaluation of development research.
Zenda Ofir is an international evaluation specialist, past President of the African Evaluation Association (AfrEA), former board member of the American Evaluation Association and the NONIE Steering Committee, and evaluation advisor to a variety of international organizations.
Nancy MacPherson is the Managing Director for Evaluation at the Rockefeller Foundation based in New York. The Foundation’s Evaluation Office aims to strengthen evaluative practice in philanthropy and development by supporting rigorous, innovative and context appropriate approaches to evaluation and learning.

Commons Select Committee to Scrutinise the DFID’s Annual Report & Resource Accounts

13 September 2011

“The International Development Committee is to conduct an inquiry into the Department for International Development’s Annual Report and Accounts 2010-11 and the Department’s Business Plan 2011-15.

Invitation to submit Written Evidence

The Committee will be considering

Changes since the election to DFID’s, role, policies, priorities and procedures;
The implications of changes for management styles, structures, staffing competences and capacity to deliver; and
The overall impact on the efficiency, effectiveness and cost-effectiveness of DFID’s activities.

The Committee invites short written submissions from interested organisations and individuals, especially on the following areas: the implementation of the structural reform plan: the bilateral, multilateral and humanitarian reviews; DFID administration costs; expenditure on, and dissemination of research; and the use of technical assistance and consultants.

The deadline for submitting written evidence is Monday 10 October 2011. A guide for written submissions to Select Committees may be found on the parliamentary website at: http://www.parliament.uk/commons/selcom/witguide.htm

FURTHER INFORMATION:
Committee Membership is as follows: Malcolm Bruce MP, Chair (Lib Dem, Gordon), Hugh Bayley MP (Lab, City of York), Richard Burden MP (Lab, Birmingham, Northfield), Sam Gyimah MP (Con, East Surrey), Richard Harrington MP (Con, Watford), Pauline Latham MP (Con, Mid Derbyshire), Jeremy Lefroy MP (Con, Stafford), Michael McCann MP (Lab, East Kilbride, Strathaven and Lesmahagow), Alison McGovern MP (Lab, Wirral South), Anas Sarwar MP (Lab, Glasgow Central), Chris White MP (Con, Warwick and Leamington).
Specific Committee Information: indcom@parliament.uk / 020 7219 1223/ 020 7219 1221
Media Information: daviesnick@parliament.uk / 020 7219 3297 Committee Website: www.parliament.uk/indcom

The Oslo Governance Forum – Governance Assessments for Social Accountability

Date: 3-5 October 2011
Venue: Oslo, Norway

ABOUT THE OSLO GOVERNANCE FORUM

The Oslo Governance Forum (OGF) is an initiative of the Oslo Governance Centre and the Democratic Governance Group of UNDP. The Forum will facilitate exchange of innovative experiences, knowledge and policy options among international development practitioners, academic institutions, government representatives and civil society from the global south.

The Oslo Governance Forum will take place from the 3-5 October 2011. The overarching focus is on governance assessments and their current and potential contribution to improving social accountability within developing countries. For the purposes of the OGF, social accountability has a wide meaning and relates to the mechanisms and instruments that are used by communities, groups and ordinary people to make governments and their agents, answerable and responsive in terms of the commitments that they have made. Governance assessments are an increasingly important tool for monitoring whether governments are failing or succeeding in terms of their commitments in legislation, government policies and international law.

To date, much of the focus of the development community on governance assessments, has been on the “supply side”, that is, improving the methodological aspects of an assessment and getting the right indicators. The OGF will focus on the “demand side”, examining, discussing and sharing experiences on how governance assessments are used by stakeholders as a basis for dialogue on governance deficits, as an instrument to monitor performance and as an input for revising and correcting policies. One of the key elements of democratic governance and accountability is empowerment of the people and the fostering of demand- and people driven accountability as opposed to accountability to external actors like donors.

The world is changing rapidly and never before has democratic governance and accountability been so visibly important on the global stage. The Arab Spring revolutions have shown that governments must take people’s calls for accountability and their rights to be governed democratically more seriously. These events have also added to the growing number of case studies that attest to the potential of social media and related technologies for mobilizing people for change.”

See the Forum home page for more information as well as the Concept Note for the Forum

Evaluation of Governance – A Study of the Government of India’s Outcome Budget

by Anand P. Gupta, Economic Management Institute, New Delhi, India
in Journal of Development Effectiveness, 2:4, 566-573, December 2010.

[Found courtesy of Public Financial Management Blog]

“In 2005, the Government of India launched an apparently excellent initiative – the Outcome Budget – with the objective of changing the culture of measuring performance in terms of the amount of money spent against the budgeted allocations, to one of measuring performance in terms of the delivery of the outcomes that people are concerned with. This paper describes how the Outcome Budget was launched, articulates the theory of change underlying the Outcome Budget, presents a case study of the Outcome Budget of the Government of India’s Accelerated Power Development and Reforms Programme, and discusses the lessons that the Government of India may learn from its experience with the Outcome Budget.

The paper argues that the Outcome Budget has failed. This has happened because the assumptions of the theory of change underlying the Outcome Budget have not been satisfied. The failure of the Outcome Budget has extremely important lessons for the Independent Evaluation Office, which the Government of India has decided to set up. The paper articulates the theory of change underlying the Independent Evaluation Office. This theory assumes that policymakers in India currently demand rigorous impact evaluations of public interventions and will continue to demand such evaluations in future, not because they have to comply with any requirement but because they really want to know the answers to the impact evaluation questions of ‘what works, under what conditions does it work, for whom, what part of a given intervention works, and for how much?’, so that they may draw appropriate lessons from these answers and use these lessons while designing and implementing public interventions in future. However, given Indian public officials’ current culture, the Independent Evaluation Office may not make any visible difference in development effectiveness in India.

The paper, published in Journal of Development Effectiveness, Volume 2, Number 4 (December 2010), is amongst the Journal’s “most read” (downloaded) papers, and is currently on the free download list of most read papers.”

Measuring Impact on the Immeasurable? Methodological Challenges in Evaluating Democracy and Governance Aid

by Jennifer Gauck, University of Kent, Canterbury – Department of Politics, 2011. APSA 2011 Annual Meeting Paper. Available as pdf

Abstract:

“Recent debates over the quality, quantity and purpose of development aid has led to a renewed emphasis on whether, and in what circumstances, aid is effective in achieving development outcomes. A central component of determining aid effectiveness is the conduct of impact evaluations, which assess the changes that can be attributed to a particular project or program. While many impact evaluations use a mixed-methods design, there is a perception that randomized control trials (RCTs) are promoted as the “gold standard” in impact evaluation. This is because the randomization process minimizes selection bias, allowing for the key causal variables leading to the outcome to be more clearly identified. However, many development interventions cannot be evaluated via RCTs because the nature of the intervention does not allow for randomization with a control group or groups.”

“This paper will analyze the methodological challenges posed by aid projects whose impacts cannot be evaluated using randomized control trials, such as certain democracy and governance (D&G) interventions. It will begin with a discussion of the merits and drawbacks of cross-sectoral methods and techniques commonly used to assess impact across a variety of aid interventions, including RCTs, and how these methods typically combine in an evaluation to tell a persuasive causal story. This paper will then survey the methods different aid donors are using to evaluate the impact of projects that cannot be randomized, such as governance-strengthening programs aimed at a centralized public-sector institution. Case studies will be drawn from examples in Peru and Indonesia, among others. This paper will conclude by analyzing how current methodological emphases in political science can be applied to impact evaluation processes generally, and to D&G evaluations specifically.”

RD Comment: See also the 3ie webpage on Useful resources for impact evaluations in governance which includes a list of relevant books, reports, papers, impact evaluations, systematic reviews, survey modules/tools and website