Category: Uncategorized

Sustainable development: A review of monitoring initiatives in agriculture

(from DFID website)

A new report has just been released on the Review of the Evidence on Indicators, Metrics and Monitoring Systems. Led by the World Agroforestry Centre (ICRAF) under the auspices of the CGIAR Research Program on Water, Land and Ecosystem (WLE), the review examined monitoring initiatives related to the sustainable intensification of agriculture. Designed to inform future DFID research investments, the review assessed both biophysical and socioeconomic related monitoring efforts.

With the aim of generating insights to improve such systems, the report focuses upon key questions facing stakeholders today:

  1. How to evaluate alternative research and development strategies in terms of their potential impact on productivity, environmental services and welfare goals, including trade-offs among these goals?
  2. How to cost-effectively measure and monitor actual effectiveness of interventions and general progress towards achieving sustainable development objectives?

An over-riding lesson, outlined in the report, was the surprising lack of evidence for the impact of monitoring initiatives on decision-making and management. Thus, there are important opportunities for increasing the returns on these investments by better integrating monitoring systems with development decision processes and thereby increasing impacts on development outcomes. The report outlines a set of recommendations for good practice in monitoring initiatives…

DFID welcomes the publication of this review. The complexity of the challenges which face decision makers aiming to enhance global food security is such that evidence (i.e. metrics) of what is working and what is not is essential. This review highlights an apparent disconnection between what is measured and what is required by decision-makers. It also identifies opportunities for a way forward. Progress will require global co-operation to ensure that relevant data are collected and made easily accessible.

DFID is currently working with G8 colleagues on the planning for an international conference on Open Data to be held in Washington DC from 28th to 30th April 2013. The topline goal for the initiative is to obtain commitment and action from nations and relevant stakeholders to promote policies and invest in projects that open access to publicly funded global agriculturally relevant data streams, making such data readily accessible to users in Africa and world-wide, and ultimately supporting a sustainable increase in food security in developed and developing countries. Examples of the innovative use of data which is already easily available will be presented, as well as more in-depth talks and discussion on data availability, demand for data from Africa and on technical issues. Data in this context ranges from the level of the genome through the level of yields on farm to data on global food systems.

 

Improving the Evaluability of INGO Empowerment and Accountability Programmes

Shutt, C. and McGee, R. CDI Practice Paper 1 March 2013 Publisher IDS Available as pdf (109kb)

Abstract
This CDI Practice Paper is based on an analysis of international NGO (INGO) evaluation practice in empowerment and accountability (E&A) programmes commissioned by CARE UK, Christian Aid, Plan UK and World Vision UK. It reviews evaluation debates and their implications for INGOs. The authors argue that if INGOs are to successfully ‘measure’ or assess outcomes and impacts of E&A programmes, they need to shift attention from methods to developing more holistic and complexity-informed evaluation strategies during programme design. Final evaluations or impact assessments are no longer discrete activities, but part of longer-term learning processes. Given the weak evaluation capacity within the international development sector, this CDI Practice Paper concludes that institutional donors must have realistic expectations and support INGOs to develop their evaluation capacity in keeping with cost–benefit considerations. Donors might also need to reconsider the merits of trying to evaluate the ‘impact’ of ‘demand-side’ NGO governance programmes independently of potentially complementary ‘supply-side’ governance initiatives.

See also: Tools and Guidelines for Improving the Evaluability of INGO Empowerment and Accountability Programmes Centre for Development Impact, Practice paper. No.1 Annex March 2013

Livestreaming of the Impact, Innovation & Learning conference, 26-27 March 2013

(via Xceval)

Dear Friends
You may be interested in following next week’s Impact, Innovation and Learning conference, whose principle panel sessions are being live-streamed. Keynote speakers and panellists include:
  • Bob Bob Picciotto (King’s College, UKES, EES), Elliot Stern, Editor of ‘Evaluation’, Bruno Marchal (Institute of Tropical Medicine, Antwerp), John Grove (Gates Foundation), Ben Ramalingan (ODI) ,Aaron Zazueta (GEF),Peter Loewe (UNIDO), Martin Reynolds (Open University),Bob Williams (Bob Williams), Richard Hummelbrunner (OAR), Patricia Rogers (Royal Melbourne Inst of Technology), Barbara Befani (IDS, EES), Laura Camfield and Richard Palmer-Jones (University of East Anglia), Chris Barnett (ITAD/IDS), Giel Ton (University of Wagenigen) ,John Mayne, Jos Vaessen (UNESCO), Oscar Garcia (UNDP), Lina Payne (DFID), Marie Gaarder (World Bank), Colin Kirk (UNICEF), Ole Winckler Andersen (DANIDA)

Impact, Innovation and Learning – live-streamed event, 26-27 March 2013

Current approaches to the evaluation of development impact represent only a fraction of the research methods used in political science, sociology, psychology and other social sciences. For example, systems thinking and complexity science, causal inference models not limited to counterfactual analysis, and mixed approaches with blurred ‘quali-quanti’ boundaries, have all shown potential for application in development settings. Alongside this, evaluation research could be more explicit about its values and its learning potential for a wider range of stakeholders. Consequently, a key challenge in evaluating development impact is mastering a broad range of approaches, models and methods that produce evidence of performance in a variety of interventions in a range of different settings.
The aim this event, which will see the launch of the new Centre for Development Impact (www.ids.ac.uk/cdi), is to shape a future agenda for research and practice in the evaluation of development impact. While this is an invitation-only event, we will be live-streaming the main presentations from the plenary sessions and panel discussions. If you would like to register watch any of these sessions online, please contact Tamlyn Munslow in the first instance at t.munslow@ids.ac.uk.
More information at:
http://www.ids.ac.uk/events/impact-innovation-and-learning-towards-a-research-and-practice-agenda-for-the-future If you are unable to watch the live-streamed events, there will be an Watch Again option, after the conference.
With best wishes,
Emilie Wilson
Communications Officer
Institute of Development Studies

Rick Davies comment 28 March 2013: Videos of 9 presentations and panels are now available online at http://www.ustream.tv/recorded/30426381

Evaluability – is it relevant to EBRD? (and others)

EBRB Evaluation Brief, June 2012 by Keith Leonard, Senior Adviser (EvD), Amelie Eulenberg, Senior Economist (EvD) Available as pdf.

RD comment: A straightforward and frank analysis.

CONTENTS
Conclusions and recommendations
1. Purpose and structure of the paper
2. Evaluability and why it matters
2.1 What is evaluability?
2.1.1 Expression of expected results
2.1.2 Indicators
2.1.3 Baseline
2.1.4 Risks
2.1.5 Monitoring
2.2 How and by whom is evaluability assessed?
2.3 Why evaluability matters
2.3.1 Relationship between evaluability and project success
2.3.2 More reliable and credible evaluations
2.3.3 Telling the story of results
2.4 What is quality-at-entry and how does it differ from evaluability?
3. How other IFIs use evaluability
3.1 Asian Development Bank
3.2 Inter-American-Development Bank
3.3 International Finance Corporation, World Bank Group
4. Current practice in the EBRD
4.1 Structure of Final Review Memorandum
4.2 EvD evaluation of the Early Transition Country Initiative
4.3 EvD synthesis of findings on a decade of evaluations of technical cooperation
4.4 Grant Co-financing Strategic Review
4.5 The findings of the Besley Report

EvalPartners International Forum on Civil Society’s Evaluation Capacities (report on)

3-6 December 2012, Chiang Mai, Thailand. Available as pdf

Exec Summary excerpt: The EvalPartners International Forum on Civil Society’s Evaluation Capacities, co-sponsored by the International Organisation for Cooperation in Evaluation (IOCE) and the United Nation’s Children Fund (UNICEF), was held December 3-6, 2012 in Chiang Mai, Thailand with the intention of enhancing the role of civil society to support equity-focused and gender-responsive country-led evaluation systems. The forum, attended by 80 high-level evaluation professionals  representing 37 countries, included regional and national presidents and chairs of voluntary organizations for professional evaluation (VOPEs), and directors of evaluation from various bilaterals, multilaterals, and government ministries. The associated discussions represented the first assembly of all regional and national VOPE presidents, all of whom expressed formal commitment to the goal of establishing an international partnership and movement to strengthen civil society and capacities  of VOPEs.

Contents
1. Opening Remarks
2. EvalPartners and National Evaluation Capacity Development
3. The Role of VOPEs in Influencing an Enabling Environment for Evaluation
4. Working Group Summaries
5. Institutional Capacities in Voluntary Organizations for Professional Evaluation
6. Institutionalizing Sustainable Learning Strategies
7. Equity-Focused and Gender-Responsive Evaluation
8. Panel Discussions

Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis

Carsten Q. Schneider, Claudius Wagemann Cambridge University Press, 31 Aug 2012 – Political Science – 392 pages Available on Amazon and Google Books

Publishers blurb: “Qualitative Comparative Analysis (QCA) and other set-theoretic methods distinguish themselves from other approaches to the study of social phenomena by using sets and the search for set relations. In virtually all social science fields, statements about social phenomena can be framed in terms of set relations, and using set-theoretic methods to investigate these statements is therefore highly valuable. This book guides readers through the basic principles of set theory and then on to the applied practices of QCA. It provides a thorough understanding of basic and advanced issues in set-theoretic methods together with tricks of the trade, software handling and exercises. Most arguments are introduced using examples from existing research. The use of QCA is increasing rapidly and the application of set-theory is both fruitful and still widely misunderstood in current empirical comparative social research. This book provides an invaluable guide to these methods for researchers across the social sciences.”
Book reviews:

Collaborative Evaluations Step by Step

by Liliana Rodriguez-Campos and Rigoberto Rincones-Gomez, Stanford Business Books, 2013 (2nd edition, first was in 2005)

Book website here and available on Amazon. But neither sites show the contents pages or exerpts

Book website says “Collaborative Evaluations is a highly comprehensive and easy-to-follow book for those evaluators who want to engage and succeed in collaborative evaluations. The author presents the Model for Collaborative Evaluations (MCE) with its six major components: (1) identify the situation, (2) clarify the expectations, (3) establish a shared commitment, (4) ensure open communication, (5) encourage best practices, and (6) follow specific guidelines. In clear and simple language, the author outlines key concepts and methods to help master the mechanics of collaborative evaluations.

Each section deals with fundamental factors inside each of the six collaborative evaluation components. In addition, each section provides practical tips for “real-life” applications and step-by-step suggestions or guidelines on how to apply this information. The MCE has emerged from a wide range of collaboration efforts that the author has conducted in the private sector, nonprofit organizations, and institutions of higher education. The author shares her experience and insights regarding this subject in a precise and easy-to-understand fashion, so that the reader can use the information learned from this book immediately.”

Related blog posting: Josey Landrieu on Collaborative Evaluation, on the AEA365 | A Tip-a-Day by and for Evaluators website

What counts as good evidence?

by Sandra Nutley, Alison Powelland Huw Davies,  Research Unit for Research Utilisation (RURU), School of Management, University of St Andrews, www.ruru.ac.uk November 2012

Available as pdf. This is a paper for discussion. The authors would welcome comments, which should be emailed to smn@st-andrews.ac.uk or Jonathan.Breckon@nesta.org.uk

In brief

Making better use of evidence is essential if public services are to deliver more for less. Central to this challenge is the need for a clearer understanding about standards of evidence that can be applied to the research informing social policy.  This paper reviews the extent to which it is possible to reach a workable consensus on ways of identifying and labelling evidence. It does this b y exploring the efforts made to date and the debates that have ensued . Throughout, the focus is on evidence that is underpinned by research, rather than other sources of evidence such as expert opinion or stakeholder views .

After setting the scene, the review and arguments are presented in five main sections:

We begin by exploring practice recommendations: many bodies provide practice recommendations, but concerns remain as to what kinds of research evidence  can or  should underpin such labelling schemas.

T his leads us to examine hierarchies of evidence: study design has long been used as  a key marker for evidence quality, but such ‘hierarchies of evidence ’ raise many  issues and have remained contested. Extending the hierarchies so that they also consider the quality of study conduct or the use of underpinning theory have  enhanced their usefulness but have also exposed new fault – lines of debate.

More broadly, in beyond hierarchies, we recognise that hierarchies of evidence have  seen most use in addressing the evidence for  what works .  As a consequence,several  agencies and authors have developed more complex  matrix approaches for  identifying evidence quality in ways that are more closely linked to the wider range  of  policy o r practice questions being addressed.

Strong evidence, or just good enough? A further pragmatic twist is seen by the recognition that evaluative evidence is always under development. Thus it may be  more helpful to think of an ‘evidence journey’ from promising early findings to  substantive bodies of knowledge.

Finally, we turn to the uses and impacts of standards of evidence and endorsing  practices .  In this section we raise many questions as to the use, uptake and impacts of evidence labelling schemes, bu t are able to provide few definitive answers as the research here is very patchy.

We conclude that there is no simple answer to the question of what counts as good evidence. It depends on what we want to know, for what purposes, and in what contexts we en visage that evidence being used. Thus while there is  a  need to debate  standards of evidence we should be realistic about the extent to which such  standard – setting will shape complex , politicised,  decision – making by policy makers,  service managers  and local practitioners.

 

The impact of statistics classes ;-)

(found via Duncan Green at Oxfam)
 

What she should then have said: “Well, let’s look to see if there is any plausible causal mechanism underneath this correlation” “Can you remember where you were when you first changed your mind? Can you remember what the discussion was about at that time?”

See also many other similar comics at the XKDC website, including:

Duggan & Bush on Evaluation in Settings Affected by Violent Conflict: What Difference Does Context Make?

From AEA365:| A Tip-a-Day by and for Evaluators. Posted: 08 Feb 2013 12:51 AM PST

“We are Colleen Duggan, Senior Evaluation Specialist, International Development Research Centre (Canada) and Kenneth Bush, Director of Research, International Conflict Research (Northern Ireland).  For the past three years, we have been collaborating on a joint exploratory research project called Evaluation in Extremis:  The Politics and Impact of Research in Violently Divided Societies, bringing together researchers, evaluators, advocates and evaluation commissioners from the global North and South. We looked at the most vexing challenges and promising avenues for improving evaluation practice in conflict-affected environments.

CHALLENGES Conflict Context Affects Evaluation – and vice versa.  Evaluation actors working in settings affected by militarized or non-militarized violence suffer from the typical challenges confronting development evaluation.  But, conflict context shapes how, where and when evaluations can be undertaken – imposing methodological, political, logistical, and ethical challenges. Equally, evaluation (its conduct, findings, and utilization) may affect the conflict context – directly, indirectly, positively or negatively.

Lessons Learned:

Extreme conditions amplify the risks to evaluation actors.  Contextual volatility and political hyper-sensitivity must be explicitly integrated into the planning, design, conduct, dissemination, and utilization of evaluation.

  1. Some challenges may be anticipated and prepared for, others may not. By recognizing the most likely dangers/opportunities at each stage in the evaluation process we are better prepared to circumvent “avoidable risks or harm” and to prepare for unavoidable negative contingencies.
  2. Deal with politico-ethics dilemmas. Being able to recognize when ethics dilemmas (questions of good, bad, right and wrong) collide with political dilemmas (questions of power and control) is an important analytical skill for both evaluators and their clients.  Speaking openly about how politics and ethics – and not only methodological and technical considerations – influence all facets of evaluation in these settings reinforces local social capital and improves evaluation transparency.
  3. The space for advocacy and policymaking can open or close quickly, requiring readiness to use findings posthaste. Evaluators need to be nimble, responsive, and innovative in their evaluation use strategies.

Rad Resources:

  • 2013 INCORE Summer School Course on Evaluation in Conflict Prone Settings , University of Ulster, Derry/ Londonderry (Northern Ireland. A 5-day skills building course for early to mid-level professionals facing evaluation challenges in conflict prone settings or involved in commissioning, managing, or conducting evaluations in a programming or policy-making capacity.
  • Kenneth Bush and Colleen Duggan ((2013) Evaluation in Extremis: the Politics and Impact of Research in Violently Divided Societies (SAGE: Delhi, forthcoming)