Horizontal Evaluation: Fostering Knowledge Sharing and Program Improvement within a Network

Authors: Thiele, Graham; Devaux, Andre; Velasco, Claudio; Horton, Douglas
American Journal of Evaluation, v28 n4 p493-508 2007

Abstract: Horizontal evaluation combines self-assessment and external evaluation by peers. Papa Andina, a regional network that works to reduce rural poverty in the Andean region by fostering innovation in potato production and marketing, has used horizontal evaluations to improve the work of local project teams and to share knowledge within the network. In a horizontal evaluation workshop, a project team and peers from other organizations independently assess the strengths and weaknesses of a research and development (R&D) approach being developed and then compare the assessments. Project team members formulate recommendations for improving the R&D approach, and peers consider ways to apply it back home. Practical results of horizontal evaluation have included strengthening the R&D approaches being developed, experimenting with their use at new sites, improvements in other areas of work, and strengthened interpersonal relations among network members. (Contains 2 tables.)”

Also available as ILAC Brief: http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief13_Horizontal_Evaluation.pdf

And a  Spanish version of the same Brief

Evaluation Of Citizens’ Voice & Accountability – Review Of The Literature & Donor Approaches Report

O’Neill, T., Foresti, M. and Hudson, A. (2007) Evaluation of Citizens’ Voice and Accountability: Review of the Literature and Donor Approaches. London: DFID.


1.3 A core group of DAC partners are collaborating on a joint evaluation of development
aid for strengthening citizens’ voice and the accountability of public institutions. The
Overseas Development Institute has been contracted to undertake the first stage of
this evaluation, which involves the development and piloting of an evaluation
framework. This literature review is the first output from this first phase. It aims to: (i)
review the theoretical debates on voice and accountability and how they relate to
development; (ii) review the different donor approaches to supporting voice and
accountability and identify commonalities and differences across contexts; (iii)
provide an overview of evaluation theory and practice in relation to voice and
accountability interventions; and (iv) identify key knowledge gaps in relation to the
effectiveness of donors in supporting voice and accountability.

1.4 This review has three main sections. Section 2 surveys the academic literature to
present current thinking on what voice and accountability means, how they operate in
practice and how they relate to the achievement of broader development objectives.
Section 3 turns to the donors’ own understanding of voice and accountability as set
out in their relevant policy and guidance documents. It discusses how the donors see
voice and accountability contributing to their poverty reduction mandates and what
approaches they have adopted to strengthen them, including in different contexts.
Section 4 considers the main issues relating to the evaluation of interventions to
strengthen voice and accountability. It first reviews some of the methodological
debates in the theoretical literature before summarising the donors’ own evaluative
efforts in this field, identifying both common findings and key gaps in their

1. Introduction 1
2. Voice and Accountability: A view from the literature 3
Voice and accountability: a basic static model 3
Voice and accountability: a complex dynamic reality 5
Relating voice and accountability to other key concepts 6
Voice, accountability and development outcomes 9
3. Voice and accountability: A view from the donors 13
Why do donors want to strengthen voice and accountability? 13
What strategies do donors adopt for strengthening voice and accountability? 18
Do donor approaches take account of context? 25
4. Evaluating voice and accountability 29
Approaches and frameworks for evaluating voice and accountability interventions 29
What have donors learnt about their effectiveness? 36
5. Conclusions 47
Annexes 49
References 53

Negotiated Learning: Collaborative Monitoring for Forest Resource Management

(via Pelican email list)

Dear all

Niels has asked me to make you aware of a new publication that some
‘Pelican-ers’ might find relevant.

I have edited a book on how learning and monitoring can become better
‘friends’ than is currently usually the case. The book comes off the press
tomorrow. The full reference: Guijt, Irene, ed. (2007). Negotiated
Learning: Collaborative Monitoring for Forest Resource Management
Washington DC, Resources for the Future/Center for International Forestry
Research. Although the cases in the book focus on natural resource (forest)
management, the issues about how to create genuine learning through the
construction, negotiation and implementation of a monitoring process will
have much wider relevance.

Full details on how to obtain the book can be found at :
http://www.rff.org/rff/RFF_Press/CustomBookPages/Negotiated-Learning.cfm ,
where the book is described as follows :

“The first book to critically examine how monitoring can be an effective
tool in participatory resource management, Negotiated Learning draws on the
first-hand experiences of researchers and development professionals in
eleven countries in Africa, Asia, and South America. Collective monitoring
shifts the emphasis of development and conservation professionals from
externally defined programs to a locally relevant process. It focuses on
community participation in the selection of the indicators to be monitored
as well as in the learning and application of knowledge from the data that
are collected. As with other aspects of collaborative management,
collaborative monitoring emphasizes building local capacity so that
communities can gradually assume full responsibility for the management of
their resources. The cases in Negotiated Learning highlight best practices
but stress that collaborative monitoring is a relatively new area of theory
and practice. The cases focus on four themes: the
challenge of data-driven monitoring in forest systems that supply multiple
products and serve diverse functions and stakeholders; the importance of
building upon existing dialogue and learning systems; the need to better
understand social and political differences among local users and other
stakeholders; and the need to ensure the continuing adaptiveness of
monitoring systems.”

PS: Links to full texts of some chapters






Learning by Design

Bredeweg 31, 6668 AR Randwijk, The Netherlands
Tel. (0031) 488-491880 Fax. (0031) 488-491844

Stories of Significance: Redefining Change – An assortment of community voices and articulations

(via the AIDS Alliance India website)

“A report based on an evaluation of a programme on “Community Driven Approaches to Address the Feminisation of HIV/AIDS in India” by means of the ‘Most Significant Change’ Technique: Using the participatory evaluation technique, Most Significant Change (MSC), this report derives its findings from the MSC evaluation of work from Alliance India’s recently concluded DFID -supported programme on community-driven approaches for addressing the feminisation of HIV/AIDS in India. The MSC technique is a participatory monitoring tool based on gathering and analysing stories of important or significant changes from a cross-section of target groups, to provide a richer picture of the impact of programme interventions. This document is written in a lucid manner and contains many new insights for the purposes of learning for future programming in relation to sexual and reproductive health and HIV/AIDS integration and HIV/AIDS programming for women.”

(English) Stories of Significance Redefining Change.pdf

For information about how to get a printed copy of this report, please contact: India HIV/AIDS Alliance info@allianceindia.org

The utilisation of evaluations.

Chapter 3:ALNAP Review of Humanitarian Action in 2005

Peta Sandison

This chapter draws upon a literature review, four case studies of evaluation
utilisation volunteered by CAFOD, MSF(H), OCHA and USAID, semi-structured
interviews with 45 evaluators, evaluation managers and evaluation ‘users’, a
review of 30 sets of terms of reference, and an electronic survey sent to ALNAP
Observer and Full Members (19 evaluation managers and 27 evaluators
responded to the survey).

The researcher was supported by an advisory group composed of representatives
from the British Red Cross Society, CARE International, MSF(H), the Netherlands
Ministry of Foreign Affairs (Policy and Operations Evaluation Department), OCHA
and ODI, plus an independent evaluation consultant and the ALNAP Secretariat.

Results-based Management in CIDA

[from CIDA website]
CIDA uses results-based management (RBM) to better manage Canada’s international development programming from start: investment or project planning and implementation, to finish: evaluations, reporting and integrating lessons learned into future programming. This page provides a set of comprehensive guides, linked to the Results-Based Management Policy Statement 2008.

Note: The Performance Management Division of CIDA’s Strategic Policy and Performance Branch is updating results-based management (RBM) guides and supporting documents according to CIDA’s 2008 RBM policy. If you need access to one of the documents below, or if you have other requests regarding RBM at CIDA, please contact the Performance Management Division.

  • Annex 4 of the Guide for Preparing a Country Development Programming Framework: The Performance Measurement Framework
  • A Results Approach To Developing the Implementation Plan (March 2001)
  • RBM Handbook on Developing Results Chain (December 2000)
  • Guide to Project Performance Reporting: For Canadian Partners and Executing Agencies (May 1999)
  • Results-based Management in CIDA: An Introductory Guide to the Concepts and Principles (January 1999)
  • The Logical Framework: Making It Results-Oriented (November 1997)

Monitoring government policies A toolkit for civil society organisations in Africa

(identified via Source)

The toolkit was produced by AFOD, Christian Aid, Trocaire

This project was started by the three agencies with a view to supporting partner
organisations, particularly church-based organisations, to hold their governments to
account for the consequences of their policies. This toolkit specifically targets African

partners, seeking to share the struggles and successes of partners already monitoring

government policies with those that are new to this work.
The development of this toolkit has been an in-depth process. Two consultants were
commissioned to research and write the toolkit. They were supported by a reference group
composed of staff from CAFOD, Christian Aid and Trócaire and partner organisations with
experience in policy monitoring. The draft toolkit was piloted with partners in workshops
in Malawi, Sierra Leone and Ethiopia. Comments from the reference group and the
workshops contributed to this final version of the toolkit.


1.1  Core concepts in policy monitoring 5
1.2  Identifying problems, causes and solutions 8
1.3  Beginning to develop a monitoring approach 10
Interaction  13
2.1  Different kinds of policies 15
2.2  Which policies to monitor 18
2.3  Access to policy information  22
2.4  Collecting policy documents 24
Interaction   27
3.1  Stakeholders of government policies 29
3.2  Target audiences and partners  31
3.3  Monitoring by a network of stakeholders 34
Interaction  37
4.1  Analysing the content of a policy 39
4.2  Defining your monitoring objectives 42
4.3  What kind of evidence do you need? 44
4.4 Choosing indicators 47
4.5  Establishing a baseline 50
Interaction  52
5.1  Budget basics  55
5.2  Resources for policy implementation 59
5.3 Budget analysis 61
5.4 Interaction  67

6.1 Interviews  69
6.2 Surveys 72
6.3  Analysing survey data and other coded information 77
6.4  Workshops, focus group discussions and observation 84
Interaction  89
Interaction  98

Networks and policy processes in international development: a literature review

Publisher: Overseas Development Institute, London, 2005

Authors: E. Perkin; J. Court Making networks work in international development and influencing policy

When, why and how do networks function best for policy impact in international development? The objective of this paper is to review and synthesise existing literature in an effort to answer these questions. The authors take policy processes as the starting point instead of focusing on types of networks themselves.

The three main objectives of the paper are to:

  • outline why networks work
  • identify how networks can influence policy, focusing on their impact on four key components of policy processes: agenda setting; policy formulation; implementation; and monitoring and evaluation
  • identify lessons for capacity building, communications and policy influence activities and areas for further study

Continue reading “Networks and policy processes in international development: a literature review”

Who Measures Change

Who Measures Change?”  is a detailed introduction to the participatory monitoring and evaluation of communication for social change. It explains the value of a participatory approach and outlines key PM&E principles and stages in the process. It also includes tools on M&E methodologies, indicators and questions relevant to CFSC, and a sample of relevant data collection techniques.

Measuring Change is an abridged 12p. version of Who Measures Change? produced for practitioners and community groups interested in further developing their skills in communication for social change and in participatory monitoring and evaluation.

Communities Measure Change is an at-a-glance reference guide to the participatory monitoring and evaluation of communication for social change, designed to be distributed during CFSC and related training workshops.

All are produced by the Communication for Social Change Consortium

Experiences of Uganda’s PPA in implementing and monitoring poverty reduction

[from Eldis Poverty Reporter]

Authors: Ssewakiryange,R.
Produced by: International Institute for Environment and Development (2005)

Uganda’s Poverty Eradication Action Plan (PEAP) incorporates poor people’s voices and perspectives. This paper focuses on the lessons learnt in the implementation of the PEAP by using examples from the Ugandan participatory poverty assessment process.

Key recommendations based on lessons learned include:

  • success of a partnership between civil society, government and other actors depends on each actor’s understanding of their role in the partnership
  • mechanisms for sustained engagement of the poor and other actors in the process are still very weak, it is therefore important design ways to ensure continued engagement
  • practitioners should ask themselves if empowerment of the poor is still a central objective of the poverty reduction strategy, and how it can be attained
  • invest in innovative ways of spending more in poverty reduction areas without expanding the debt volume
  • align monitoring frameworks to enhance coordin ation and to allow for more inclusion of actors

Available online at: http://www.eldis.org/go/topics/resource-guides/conflict-and-security&id=39402&type=Document