“The DAC Network on Development Evaluation is a unique international forum that brings together evaluation managers and specialists from development co-operation agencies in OECD member countries and multilateral development institutions. Its goal is to increase the effectiveness of international development programmes by supporting robust, informed and independent evaluation.

A key component of the Network’s mission is to develop internationally agreed norms and standards to strengthen evaluation policy and practice. Shared standards contribute to harmonised approaches in line with the commitments of the Paris Declaration on Aid Effectiveness. The body of norms and standards is based on experience, and evolves over time to fit the changing aid environment. These principles serve as an international reference point, guiding efforts to improve development results through high quality evaluation.

The norms and standards summarised here should be applied discerningly and adapted carefully to fit the purpose, object and context of each evaluation. This summary document is not an exhaustive evaluation manual. Readers are encouraged to refer to the complete texts available on the DAC Network on Development Evaluation’s website: www.oecd.org/dac/evaluationnetwork. Several of the texts are also available in other languages.”

“Introduction

In June 2009, the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Network on Development Evaluation agreed to undertake a study of its members’ evaluation systems and resources. The study aims to take stock of how the evaluation function is managed and resourced in development agencies and to identify major trends and current challenges in development evaluation. The purpose is to inform efforts to strengthen evaluation systems in order to contribute to improved accountability and better development results. It will be of interest to DAC members and evaluation experts, as well as to development actors in emerging donor and partner countries.

To capture a broad view of how evaluation works in development agencies, core elements of the evaluation function are covered, including: the mandate for central evaluation units, the institutional position of evaluation, evaluation funding and human resources, independence of the evaluation process, quality assurance mechanisms, co-ordination with other donors and partner countries, systems to facilitate the use of evaluation findings and support to partner country capacity development.

This report covers the member agencies of the OECD DAC Network on Development Evaluation.1 See Box 1 for a full list of member agencies and abbreviations. Covering all major bilateral providers of development assistance and seven important multilateral development banks, the present analysis therefore provides a comprehensive view of current policy and practice in the evaluation of development assistance.

The study is split into two sections: section I contains an analysis of overall trends and general practices, drawing on past work of the DAC and its normative work on development evaluation. Section II provides an individual factual profile for each member agency, highlighting its institutional set-up and resources.”

Preamble

Participation has been widely taken up as an essential element of development, but participation for what purpose? Many feel that its acceptance, which has extended to even the most conventional of institutions such as the international development banks, has resulted in it losing its teeth in terms of the original ideology of being able to empower those living in poverty and to challenge power relations.

The more recent emergence of the rights-based approach discourse has the potential to restore the ‘bite’ to participation and to re-politicise development. Enshrined in universal declarations and conventions, it offers a palatable route to accommodating radicalism and creating conditions for emancipatory and transformational change, particularly for people living in poverty. But an internet search on how to measure the impact of these approaches yields a disappointing harvest of experience. There is a proliferation of debate on the origins and processes, the motivations and pitfalls of rights-based programming but little on how to know when or if it works. The discourse is messy and confusing and leads many to hold up their hands in despair and declare that outcomes are intangible, contextual, individual, behavioural, relational and fundamentally un-quantifiable!

As a consequence, results-based management pundits are resorting to substantive measurement of products, services and goods which demonstrate outputs and rely on perception studies to measure outcomes.

However, there is another way. Quantitative analyses of qualitative assessments of outcomes and impacts can be undertaken with relative ease and at low cost. It is possible to measure what many regard as unmeasurable.

This publication suggests that steps in the process of attainment of rights and the process of empowerment are easy to identify and measure for those active in the struggle to achieve them. It is our etic perspectives that make the whole thing difficult. When we apply normative frames of reference, we inevitably impose our values and our notions of democracy and citizen engagement rather than embracing people’s own context-based experience of empowerment.

This paper presents the experience of one social movement in Bangladesh, which managed to find a way to measure empowerment by letting the members themselves explain what benefits they acquired from the Movement and by developing a means to measure change over time. These measures , which are primarily of use to the members, have then been subjected to numerical analysis outside of the village environment to provide convincing quantitative data, which satisfies the demands of results-based management.

The paper is aimed primarily at those who are excited by the possibilities of rights-based approaches but who are concerned about proving that their investment results in measurable and attributable change. The experience described here should build confidence that transparency, rigour and reliability can be assured in community led approaches to monitoring and evaluation without distorting the original purpose, which is a system of reflection for the community members themselves. Hopefully, the reader will feel empowered to challenge the sceptics.

Dee Jupp and Sohel Ibn Ali

CONTENTS
Foreword.
Preface.
Preamble.
Acknowledgements.
List of Abbreviations and Glossary.
Summary.
1. Introduction.
Women’s drama
Men’s drama
2 The Problem of Measuring Empowerment
Empowerment – what is it really?
Empowerment as a process and outcome of participation
Empowerment is a contested concept
Empowerment is a moving target
Attempts to measure empowerment lack rigour
3 The Challenge for Social Movements
4 The Evolution of this Empowerment Measuring Tool
5 How this Monitoring Tool Works
6 Using the Empowerment Monitoring Tool
Analysis at group level
Analysis at programme level
Performance management
Comparing models of scaling up
Analysis to satisfy donor requirements
Costs
7 Facing the Challenges
Attempts to measure empowerment lack rigour
Other reasons people were not happy with the reflection process
The Mother organisation
8 Applying the Approach to Your Programme
Timing start-up
The process
9 Lessons Learned
Bibliography

“Purpose: This paper reviews the current field of network monitoring and evaluation with the goal of identifying where progress has been made and where further work is still needed. It proposes a framework for network impacts planning, assessment, reporting and learning that can help to close some of the current gaps in network evaluation while building on the advances that have been made. This document is written for practitioners undertaking network evaluation and foundation program staff working to support networks.

Introduction and Context: Networking and networks have become increasingly and consciously utilized as organizing strategies and structures for creating social change in this world. The growing realization that no single actor, no matter how effective they are, is capable of tackling today’s social problems has spurred a flurry of international interest and investment in networks from a wide range of actors and sources. The World Bank began funding networks through its Global and Regional Partnership Program
in the mid-1990’s and is now currently supporting approximately 175 partnership programs, having spent $3.5 billion in 2006 alone.  The Climate Works Foundation has organized more than ten funders and organizations from across sectors and geographies as part of a billion dollar coordinated campaign to fight climate change. The Global AIDS Alliance has launched an
advocacy network across six African countries in a Campaign to End  Pediatric HIV/AIDS. Grant makers like the Ford Foundation, Anne E. Casey Foundation, MacArthur Foundation and many others have made the funding of networks a key part of their portfolio and grant making strategy. At the same time, the growing demand for more success from social change public and private initiatives has created an explosion of interest in and demand for increased and improved monitoring and evaluation and impact evaluation. Governments, donors and practitioners are all feeling pressure to demonstrate and report on the impact of their work. Together, these two trends have created a growing appetite for the monitoring and evaluation and impact evaluation of networks. An increasing number of methods, tools and metrics have been proposed, developed and piloted in response to this demand. While important steps have been taken, the field of network monitoring and evaluation is still, in theory and even more so in practice, in its infancy.

This paper is divided into four sections. Section one provides a literature review of the current field of network monitoring and evaluation describing various current metrics and tools for network monitoring and evaluation. The section concludes with a discussion of the key gaps in the current field. Section two examines the key characteristics of networks and the implications these have for evaluation. Section three further elaborates the specific challenges of network evaluation and desirable characteristics for a network monitoring and evaluation system. Section four introduces a framework for network monitoring and evaluation currently being developed by iScale and its partners and explains how it can be implemented.”

For questions and/or comments on this paper please contact Catrina Lucero at clucero@scalingimpact.net

“Editor’s Note: The 2010 Issues and Prospects of Evaluations for International Development employs systems concepts as clues to re-assess the conventional ways of conducting evaluations and to explore how development evaluation can potentially be made more useful.

In Japan, development evaluation predominantly relies on the Logical Framework (logframe) when conducting evaluations. Evaluations based on a logframe often face difficulties. One such difficulty arises from the futile attempt to develop an evaluation framework based on a logframe, which, in many cases, was prepared as part of the early-stage planning of the project and which then does not necessarily reflect a project’s real situation at the time of evaluation. Although a logframe can be utilised initially as a tentative project plan, logframes are rarely revised even when the situation has changed. By the end of the project, the original logframe may not be an accurate embodiment of what the project is about and therefore logframes do not particularly help in terminal or ex-post evaluations.

Still, having been institutionalized by clients, logframe-based evaluations are common practice and in extreme cases, evaluators face the danger of evaluating the logframe instead of the actual project. Although widely used for its simplicity, logframes can end up becoming a cumbersome tool, or even a hindrance to evaluation.

Various attempts have been made to overcome the limitations of the logframe and some aid organizations such as USAID, UNDP, CIDA and the World Bank have shifted from the logframe to Results-Based Management (RBM). Now GTZ  is in the process of shifting to a new project management approach designed on RBM and systems ideas.

In the first article, “Beyond logframe: Critique, Variations and Alternatives,” Richard Hummelbrunner, an evaluator/consultant from Austria, sums up the critique of logframe and the Logical Framework Approach (LFA), and explores some variations employed to overcome specific shortcomings of LFA. He then outlines a systemic alternative to logframe  and introduces the new GTZ management model for sustainable development called “Capacity WORKS.” Richard has dealt with LFA and possible alternatives to LFA at various points along his career, and he is currently involved in GTZ’s rollout of Capacity WORKS as it becomes the standard management model for all BMZ 5 projects and programmes.

What does he mean by “systemic alternative”? In the second article, “Systems Thinking and Capacity Development in the International Arena,” Bob Williams, a consultant and an expert in systems concepts, explains what “thinking systemically” is about and how it might help evaluation. He boils down systems ideas into three core concepts (inter-relationships, perspectives, and boundaries), and relates these concepts to various systems methods.

In December 2009, FASID offered a training course and a seminar on this topic in Tokyo. Through the exchange of numerous e-mails with the instructors prior to the seminar, it occurred to me that the concepts might be more easily understood presented as a conversation. That is what we tried to do in the third article, “Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –.” These two instructors of the FASID training course and workshop explain in simple conversational style where and how we can start applying systems concepts in development evaluation.

This issue also carries a report of two collaborative evaluations of Japanese Official Development Assistance (ODA) projects. The first case presents an innovative joint evaluation conducted collaboratively with Vietnamese stakeholders. The evaluation took place in 2009 – 2010 as the last year of a three-year evaluation capacity development project coordinated by the Japan International Cooperation Agency. The second case covers a joint evaluation study of another Japanese ODA project in Lao PDR with a local Lao administration for which neither logframe nor OECD DAC five criteria was used. Instead, an evaluation framework was developed from scratch, based entirely on the beneficiaries’ interests and perspectives. In both cases, a partner country’s participation in the evaluation necessitated considerable changes in perspectives of evaluation practice. I hope they provide examples of how boundaries and perspectives, as discussed theoretically in the first three articles, relate to development evaluation in practice.”

“Making Evaluation Matter” is the theme of the conclave. The theme embodies an idea of evaluation that starts with relevance and context, with a strong understanding of who evaluation should serve.

The event will attract global thinkers engaged in cutting edge evaluation research, theorizing, or practice who seek opportunities to push their thinking in new directions and are interested in applying ideas in a South Asian context. It will include leading development theorists, activists and policy makers from South Asia to embed discussions in current development issues and contexts that evaluation must respond to.

The event will be a space to engage with, and test knowledge. The conclave will include around 200 leaders from the global development and evaluation community.

50 + speakers, 24+ hours of knowledge building and networking, 4 power packed days, 1 Great venue, Countless new evaluation opportunities.

Programme schedule | Read workshop description
Click here to download registration form

[Please post your Comments below and/or on the Guardian Katine website ]

“British taxpayers will see exactly how and where overseas aid money is being spent and a new independent watchdog will help ensure this aid is good value for money, International Development Secretary Andrew Mitchell has announced.

In his first major speech as Development Secretary, Mr Mitchell said he had taken the key steps towards creating an independent aid watchdog to ensure value for money. He also announced a new UKaid Transparency Guarantee to ensure that full information on all DFID’s spending is published on the departmental website.

The information will also be made available to the people who benefit from aid funding: communities and families living in the world’s poorest countries.

These moves come as part of a wider drive to refocus DFID’s work so British taxpayers’ money is spent transparently and on key priority issues such as maternal mortality and disease prevention.”

In Mr Mitchell’s speech, delivered at the Royal Society with Oxfam and Policy Exchange, he argued that overseas aid is both morally right and in Britain’s national interest but that taxpayers need to see more evidence their money is being spent well.

Andrew Mitchell said:

“We need a fundamental change of direction – we need to focus on results and outcomes, not just inputs.  Aid spending decisions should be made on the basis of evidence, not guesswork. That is why we have taken the first steps towards creating a new independent aid watchdog.

“The UK Aid Transparency Guarantee will also help to create a million independent aid watchdogs – people around the world who can see where aid money is supposed to be going – and shout if it doesn’t get there.”

Andrew Mitchell highlighted the results of well-spent aid, saying:

“Development is good for our economy, our safety, our health, our future. It is, quite simply, tremendous value for money: the best return on investment that you’ll find anywhere in government.

“British aid pays for five million children in developing countries to go to primary school every day. That’s roughly the same number as go to primary school in Britain yet it costs only 2.5 per cent of what we spend here. That’s real value for money.”

And he gave this pledge to UK taxpayers:

“To the British taxpayer I say this: our aim is to spend every penny of every pound of your money wisely and well. We want to squeeze every last ounce of value from it. We owe you that.

“And I promise you as well that in future, when it comes to international development, we will want to see hard evidence of the impact your money makes. Not just dense and impenetrable budget lines but clear evidence of real effect.”

Barbara Stocking, Oxfam Chief Executive, said:

“Andrew Mitchell’s announcement of a watchdog to improve the quality of British aid is welcome and something Oxfam has requested for years.

“The new watchdog must be truly independent of government if it is to achieve the best results for the British taxpayer and poor people alike.

“As our report launched today argues, strengthening transparency and accountability will be important in further improving the quality of British aid but they should be pieces in a larger jigsaw. Ensuring aid is poverty focused not politically driven and strengthens the public services poor people rely on are all vital to ensuring that it makes the biggest possible difference to the lives of the world’s poorest. We look forward to working with the new government on these issues.”

Richard Miller, ActionAid UK Director, welcomed today’s announcement on an Aid Transparency guarantee.

“ActionAid has long campaigned for citizens in poor countries to have information about aid provided in their name. We are therefore delighted with the Secretary of State’s announcement of increased transparency in aid spending. It will hopefully establish a new standard of openness and encourage scrutiny by citizens in countries receiving UKaid, as well as assuring UK taxpayers that aid is reaching the poorest.”

Dr Amy Pollard, CAFOD’s aid effectiveness policy analyst said:

“CAFOD are delighted that the Secretary of State today announced that Britain’s aid is to become more transparent.

“Whilst it is not a silver bullet, transparency is a vital condition for improving aid.  Proper information on aid funds is critical in the fight against corruption and essential for promoting accountability in development work.

“Transparency underpins the effort to make aid better – it is impossible to co-ordinate aid, assess results or build real partnerships between donors and developing countries without it.”

Hear the full speach

Read some public comments on the Guardian Katine website

In order to achieve this goal Impact 2.0 has two components. The first involves three pilot projects that seek to combine to combine current theory on the relationship between research, policy and advocacy with advances in Web 2.0/Social networking technologies and practices. The second is a fund to support research into the use of Web 2.0 tools and behaviours to link research and policy.

The research fund will consider two types of proposals: Type 1 projects will involve both implementing a specific intervention using Web 2.0 to link policy and analyzing, documentating and evaluating the intervention while Type 2 projects will document and evaluate one or more current or recent projects making use of Web 2.0 to link policy and research. A maximum of US$15,000 is available for Type 1 projects and US$7,500 for Type 2.

Independent researchers and organisations (universities, government agencies, NGOs, research centres) are invited to apply. Applicants must reside in Latin America and the projects to be performed and analyzed must also be located in and relevant to the region.

Full details are available in the attached CFPs or in Spanish at http://impacto2.comunica.org and in English at http://impacto2.comunica.org/?page_id=23

Impacto 2.0 is a project of Fundaci�n Comunica, with the financial support of the International Development Research Centre (IDRC) and the participation of APC, DIRSI, PRODIC of the University of the Republic of Uruguay and CIESPAL.

–
| Bruce Girard | www.comunica.org |
| tel: +598 2 410.2979 | mobile: +598 99 189.652 |
| Dr. Pablo de Mar�a 1036 | Montevideo, Uruguay |

Evaluation of Humanitarian Action with ALNAP (Active Learning Network for Accountability and Performance in Humanitarian Action)

14-16 June, 2010

This course is an introductory-to- intermediate level course and has the overall aim of making evaluations of humanitarian action more effective in contributing to the improved performance of interventions and to improve the quality of the evaluation process. This 3-day training course is based on an update of the ALNAP training modules. The course will also introduce some new material, specifically:

• on joint evaluations: the rationale, experience and learning to date, interwoven throughout the training programme
• on evaluating policy as well as projects and programmes
• on innovative learning processes as part of the evaluation process.

Based on ALNAP and Channel Research’s experience of running training programmes on evaluation, this course will be open to evaluators, managers and users of evaluations. It presents an opportunity for learning and exchange between participants from different backgrounds and who play different roles. The course is facilitated by Channel Research, Margie Buchanan Smith and John Cosgrave. It will be held near Brussels.

Please find attached the course outline and application form, which can also be found on the following link: http://www.channelresearch.com/humanitarian-action/

For any further information, please contact Maria Bak on bak@channelresearch.com.

Advanced Humanitarian Action Evaluation Practice and Ethics with ALNAP (Active Learning Network for Accountability and Performance in Humanitarian Action)

17-18 June, 2010

This advanced training in the Evaluation of Humanitarian Action focuses on dealing with practical challenges in evaluation. This is the second advanced follow up course on the training in Evaluation of Humanitarian Action. This workshop is for those with extensive experience of evaluating humanitarian action, either as an evaluator or an evaluation manager. It will provide an opportunity for participants to reflect on their own
evaluation experience and to use this and the experiences of their peers to improve their own work.

The course will focus on areas which often pose challenges for evaluators and evaluation managers. It will also equip participants with a range of evaluation and evaluation management tools and resources, so that those who wish can deepen their knowledge of Particular aspects of evaluation. The course will touch base on Real-time evaluation, joint evaluation, the question of utilization of evaluation and provide guidance on ethical issues in evaluation.

The course will take place close to Brussels and will be led by Margie Buchanan-Smith and John Cosgrave, two humanitarian evaluators whose broad experience will ground the course.

Please find attached the application form, which can also be found on the following link: http://www.channelresearch.com/humanitarian-action/

For any further information, please contact Maria Bak on bak@channelresearch.com.

Best Regards,

Maria Bak , Knowledge Manager, Channel Research, 45B Route des Marnières, 1380 Lasne, Belgium, Tel:  +32 2 633 6529, Fax: +32 2 633 3092,  Mobile: +32 (0) 473 12 36 23, Skype: maria_bbak , mail: bak@channelresearch.com , www.channelresearch.com

The Paris Declaration (PD), adopted in 2005, lays down principles and procedures for enhancing the effectiveness of aid and specifies them in twelve targets supplied with monitorable indicators and to be achieved by 2010. The quantitative assessment by the OECD of the progress towards 2010 reveals considerable delays of donors and partner countries. With the global economic crisis and growing concern about climate change the targets of the PD have become a great challenge.

EADI in cooperation with the Institute of Development Policy and Management (IOB) will hold a 2 days intensive training workshop in Brussels for development professionals and practitioners. This training workshop is part of the EADI Masterclasses.

Lecturers:

• Robrecht Renard, IOB, Universiteit Antwerpen
• Dr. Nadia Molenaers, IOB, Universiteit Antwerpen
• Nathalie Holvoet, IOB, Universiteit Antwerpen
• Gaelle Doleans, Policy Officer, European Commission DG Development

The training programme consists of the following elements:

15^th April

10:00-13:00 Is the Paris Declaration desirable?

• A short history of the evolution of aid modalities and instruments
• Insights from the aid effectiveness debate
• The proposed new aid architecture as a boost to aid effectiveness (model, rationale, etc.)
• From rhetoric to reality: where do we stand?

(11:30-11:45 Coffee break)

13:00-14:00 Lunch

14:00-17:00 Is the Paris Declaration realistic? The political limits of aid architecture reform

• The nature of the state and quality of institutions in recipient countries
• The political constraints of donors
• The overambitious agenda the civil society is confronted with
• The political effects of aid

(15:30-15:45 Coffee break)

19:00-21:00 Socials

16^th April

9:00-12:00 Monitoring and Evaluation in the context of changing aid modalities: importance, challenges and moving forward

• Importance of M&E
• The Paris Declaration and the M&E reform agenda
• Progress in the realisation of the reform agenda: evidence for recipient countries and donors
• Suggestions to move forward: a two-track approach

(10:30-10:45 Coffee break)

12:00-13:00 Lunch

13:00-14:30 Debt Relief Issues

14:30-17:00 Hands-on exercises, group work or simulations

The language of the workshop is English.

Participants will receive:

• An USB Stick with relevant publications and documents
• Access to wiki with publications and training session documentation
• Certificate of participation provided by EADI

Register online (Number of participants is limited)

Participation fee is 650 EUR. There is a special discount of 100 EUR for EADI members and partners and also for students with valid student ID. Lunch and refreshments are included. Please be aware that EADI does not offer any funding.

Terms and conditions:We reserve the right to cancel any class with insufficient registration, in which case, the tuition fee will be refunded.