Evaluating the Evaluators: Some Lessons from a Recent World Bank Self-Evaluation

February 21, 2012 blog posting by Johannes Linn, at Brookings
Found via @WorldBank_IEG tweet

“The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work…. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

Key lessons as seen by Linn

  • An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
  • An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
  • An evaluation of evaluations should assess:
    o How effectively impact evaluations are used;
    o How scaling up of successful interventions is treated;
    o How the experience of other comparable institutions is utilized;
    o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
    o Whether and how the governance of the institution is evaluated; and
    o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed

Read the complete posting, with arguments behind each of the above points, here

The Big Push forward: The Australian Debate (Oct 2011)

October 26, 2011 by Chris Roche.

On 19 October 2011, Oxfam Australia hosted a ‘Big Push Forward‘ event in Melbourne with the co-conveners of this initiative – Rosalind Eyben and Irene Guijt. Sixty development practitioners, including AusAid staff and academics came together to discuss whether the concerns voiced by the Big Push Forward project are relevant in Australia.

 HOW RELEVANT ARE THE ISSUES TO AUSTRALIA?

Following an introduction from Rosalind and Irene, we  had short inputs from three speakers on how these issues resonated in our part of the world.  Dennis Altman, from the Institute of Human Security, at La Trobe University suggested that the neo-liberal language which permeates Western society has been recast in the development world. into an auditing culture, focusing on evaluation, monitoring, and counting beans.  Marc Purcell the CEO of Australia’s International NGO umbrella group ACFID noted that the commitment to international aid in Australia is extremely brittle, and that the public debate about aid in Australia has led to a deep anxiety in government about how the aid programme is being perceived. But he argued that maybe it’s no bad thing for economists to look at the work of ‘pampered NGOs’. Jess Dart, the Managing Director of consulting company Clear Horizon, felt that whilst Australian NGOs do more internal evaluation than most there was a view expressed at this year’s Australasian Evaluation Conference that ‘development is the cowboy of evaluation’.  If we can’t tell the story of what we’ve done, people will ask for results. There are lots of really good methods out there and we can use these to offer solid alternatives to tell more complex stories of transformation.

OZIFYING THE THEMES
Continue reading “The Big Push forward: The Australian Debate (Oct 2011)”

Horizontal Evaluation: Fostering Knowledge Sharing and Program Improvement within a Network

Authors: Thiele, Graham; Devaux, Andre; Velasco, Claudio; Horton, Douglas
American Journal of Evaluation, v28 n4 p493-508 2007

Abstract: Horizontal evaluation combines self-assessment and external evaluation by peers. Papa Andina, a regional network that works to reduce rural poverty in the Andean region by fostering innovation in potato production and marketing, has used horizontal evaluations to improve the work of local project teams and to share knowledge within the network. In a horizontal evaluation workshop, a project team and peers from other organizations independently assess the strengths and weaknesses of a research and development (R&D) approach being developed and then compare the assessments. Project team members formulate recommendations for improving the R&D approach, and peers consider ways to apply it back home. Practical results of horizontal evaluation have included strengthening the R&D approaches being developed, experimenting with their use at new sites, improvements in other areas of work, and strengthened interpersonal relations among network members. (Contains 2 tables.)”

Also available as ILAC Brief: http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief13_Horizontal_Evaluation.pdf

And a  Spanish version of the same Brief

Peer Review: A Tool for Co-operation and Change

This paper, which has been prepared by Fabrizio Pagani with the assistance of colleagues in the Legal Directorate and contributions from other Services and Directorates, examines the practice of peer review and peer pressure in the context of international organisations. In particular, the paper studies the experience of the Organisation for Economic Co-operation and Development, which conducts dozens of county and thematic examinations using the peer review methodology every year.

The study starts by defining peer review as the systematic examination and assessment of the performance of a State by other States, with the ultimate goal of helping the reviewed State improve its policy making, adopt best practices, and comply with established standards and principles. The paper then clarifies the related concept of peer pressure, and proceeds to examine the main component of a peer review exercise: the institutional basis, the principles and criteria according to which the review is conducted, the actors, and the procedures. The last section outlines the role that peer reviews can play and sets out the conditions under which they can strengthen co-operation and bring change. Finally, the Annexes present the different peer review studies undertaken in the Organisation.”

%d bloggers like this: