August 2010 – Monitoring and Evaluation NEWS

Measuring Up: HIV-related advocacy evaluation training pack (draft)

HIV-related advocacy evaluation training for civil society organisations.

Produced by the International HIV/AIDS Alliance (Secretariat), International Council of AIDS Service Organizations (ICASO), July 2010, 38 pages. Available as .pdf

“This training pack is published by the Alliance and the International Council of AIDS Service Organizations (ICASO) and consists of two guides designed for advocacy, monitoring and evaluation staff of civil society organisations (including networks) who are involved in designing, implementing and assessing advocacy projects at different levels. The purpose of these guides is to increase users’ capacity to evaluate the progress and results of their advocacy work. The guides aim to:

1. help users to identify and confront the challenges faced by community-based organisations evaluating HIV-related advocacy
2. introduce new thinking for designing advocacy evaluations
3. give users the opportunity to apply some aspects of the evaluation design process to their specific contexts
4. make users aware that advocacy evaluation is a fast-growing and evolving field, with a large number of publications on advocacy evaluation design, approaches and methods available via the Internet and summarised in the resources section of the learner’s guide.”

Updated MSC bibliography

PLEASE NOTE. The bibliography below has now been superseded by a more comprehensive bibliography here. This now includes pdf copies of many of the papers plus a search facility. It will continue to be updated

This (now older) page is intended to provide an update of the bibliography in the 2005 Most Significant Change technique (MSC) Users Guide

Please feel free to suggest additions to this list, through the Comment facility below, or by emailing the editor (Rick Davies)

Papers

Using Most Significant Change Stories to document the impact of the Teaching Teachers for the Future Project: An Australian Teacher Education Story (2012) Deborah Heck, Trudy Sweeney. Abstract here
The Ambassadors’ Stories- a case study on using the Most Significant Change method with the Whitehorse Sustainable Ambassadors (undated). Available as pdf.
Learning to inform action: Most Significant Change technique as a tool to build evaluative capacity at Tostan ,(2012), Harris, Sarah, . Capstone Collection. Paper 2548.
Assessing changes in social capacity: experience with the ‘Most Significant Change’ technique (2009) Richard Moore and Leonie Offer. Extension Farming Systems Journal volume 5 number 1
MSC: Misconceptions, strengths and challenges (2009) Fiona Kotvojs and Carolina Lasambouw. Presented at the Australasian Evaluation Conference September 2009 Available as pdf
Most Significant Change Stories from the Challenge Program on Water and Food (CPWF) (2009) Compiled and edited by Cristina de Leon, Boru Douthwaite and Sophie Alvarez, CPWF Impact Project. 92 pages. Available as .pdf
Most significant change technique and the national training system: Core Ideas (2009) Dr John Mitchell & Suzy McKenna
M&E Waardewerken. Monitoring en Evaluatie van het innovatienetwerk Waardewerken met de ‘Most Significant Change’ methode. Jong, Daniël de, Sandra Bos, Marlèn Arkesteijn and Jorieke Potters (2008), Wageningen UR ASG. (M&E of Waardewerken. Monitoring and Evaluation of the Innovation Network ‘Waardewerken'(Multifunctional Agriculture) using the ‘Most Significant Change’ method)
Cambio más significativo (CMS): evaluando sin indicadores (2008) 2 pages
Measuring social impacts in conservation: experience in using the most significant change method (2008) Lizzie Wilder, Matt Walpole, Fauna & Flora International,
The Most Significant Change: using participatory video for monitoring and evaluation (2007) by CHRIS LUNCH. 5 pages. Available as .pdf
The most significant lessons about the Most Significant Change technique (2007) Juliet Willetts, Paul Crawford Development in Practice, Volume 17, Issue 3 June 2007 , pages 367 – 379
Stories of Significance: Redefining Change. An assortment of community voices and articulations. A report based on an evaluation of a programme on “Community Driven Approaches to Address the Feminisation of HIV/AIDS in India” by means of the ‘Most Significant Change’ Technique. (July 2007)
Methoden voor monitoring en evaluatie van innovatieprojecten. (Monitoring and Evaluation Methods for Innovation Projects), Arkesteijn, Marlèn, Jorieke Potters and Barbara van Mierlo (2007), Wageningen UR PPO, Lelystad.
This report examines the usefulness/ applicability of various M&E methods (MSC, Learning Histories, result orientend M&E etc) for innovation networks.
Evaluation: LandLearn’s most significant change (2006) Lydia Fehring, Jenny Pettenon, Ann Fagan, Kathryn Goyen, Jessica Connor. Paper given at the APEN International Conference 6-8 March 2006 at Beechworth, Victoria, Australia
Monitoreo con el Cambio Mas Significativo (CMS): La experienca de Ibis en Nicaragua (2006) Ibis, Denmark.
“Learning from Capacity Building Practice:Adapting the ‘Most Significant Change’ Approach to Evaluate Capacity Building Provision by CABUNGO in Malawi“ by Rebecca Wrigley: – Praxix Paper No. 12, INTRAC, Oxford, December (2006)
‘Most Significant Change’ Pilot Project Evaluation Report (2004) Prepared by Institute for Sustainable Futures For ADRA Laos
Exploring Perceptions of ‘Significant Change’ in Reforming Schools (2003) Dr Rosie LeCornu, Dr Judy Peters. Dept of Education and Child Development, Government of South Australia
Doing Away With Predetermined Indicators: Monitoring using the Most Significant Changes Approach (2002)Peter Sigsgaard – Mellemfolkeligt Samvirke, Denmark

Powerpoints

Seven sets of slides, used for 2 day MSC training in Delhi, 2008 by Rick Davies . Available on request , on condition of willingness to share any adaptations made

Undated Most Significant Change, by Tracey Delaney. 17 slides.

YouTube video

2007 Jess Dart on MSC: Part 1, Part 2, Part 3, Part 4, Part 5, Q&A.

Other

The introductory page on MSC, also on this website
Clear Horizon’s webpage on MSC
Database for Most Significant Change type data

Designing Initiative Evaluation A Systems-oriented Framework for Evaluating Social Change Efforts

W. K. Kellogg Foundation, 2007. 48 pages. Available as pdf.

“Purpose”

“This document is designed for use by external evaluators who conduct initiative evaluations for theW.K. Kellogg Foundation (WKKF) – and, hopefully, other foundations and government agencies. It presents a systems-oriented framework and four general designs for initiative and cluster evaluation. The designs are based on systems concepts related to change and the dynamics of systems. The focus is not on considering all ideas about systems that could be applied to initiative evaluation, rather on how different dynamics within systems can serve as the basis for initiative evaluation designs.” Continue reading “Designing Initiative Evaluation A Systems-oriented Framework for Evaluating Social Change Efforts”

UK Evaluation Society 2010 Annual Evaluation Conference

Evaluation in a turbulent world: Challenges, opportunities and innovation in evaluation practice
Date: 22-23 November 2010
Venue: Macdonald Burlington Hotel, Birmingham

Abstracts are now invited for this year’s UKES Annual Evaluation Conference. The on-line submission form is available via the conference website www.profbriefings.co.uk/ukes2010 The closing date for receipt of submissions is 13 August 2010.

With the effects of the financial crisis still being felt, and with a new coalition government in Number 10, many evaluators find themselves operating in a very different policy environment. In particular the rhetoric has changed from tackling the crisis (a central theme of last year’s conference looking at impact) to talk of austerity and cutting back the public sector, a major source of sponsorship for evaluation. While this environment offers tough challenges, it also presents opportunities – in particular in the development and promotion of new evaluation methodologies, relationships and approaches. More than ever there will be a need to assess what is of value, what has quality and in what circumstances evaluation can contribute to informed policy-making and debate. Continue reading “UK Evaluation Society 2010 Annual Evaluation Conference”

The American Evaluation Association annual conference: Evaluation Quality

Date: November 10-13, 2010
Venue: San Antonio, Texas

The American Evaluation Association invites evaluators from around the world to attend its annual conference to be held Wednesday, November 10, through Saturday, November 13, 2010 in San Antonio, Texas. We’ll be convening at the lovely Grand Hyatt San Antonio, right in the heart of the vibrant city and adjacent to the Riverwalk’s nightlife, restaurants, and strolling grounds. Discounted hotel reservations will be available in March.

AEA’s annual meeting is expected to bring together approximately 2500 evaluation practitioners, academics, and students, and represents a unique opportunity to gather with professional colleagues in a collaborative, thought-provoking, and fun atmosphere.

The conference is broken down into 44 Topical Strands that examine the field from the vantage point of a particular methodology, context, or issue of interest to the field as well as the Presidential Strand highlighting this year’s Presidential Theme of Evaluation Quality. Presentations may explore the conference theme or any aspect of the full breadth and depth of evaluation theory and practice.

Proposals are due by midnight in the Eastern time zone, on Friday, March 19, 2010.
For more information: http://www.eval.org/eval2010/10cfp.htm

RESOURCE PACK ON SYSTEMATIZATION OF EXPERIENCES

ActionAid International, 2009, 104 pages. Available as pdf 3.39Mb

See also the associated AAI website on systematization

“Systematization is a methodology that offers a way to do all of the above. It allows us to:

Organise and document what we have learnt through our work
Better understand the impact of our work and the ways in which change happens
Develop deeper understanding about our work and the challenges we face to inform new ways of working
Capture and communicate the complexity and richness of our work

Systematization “helps people involved in different kinds of practice to organize and communicate what they have learned. We are talking about …so called …. lessons learned, about which everybody talks nowadays, but are not so easy to produce.” (AAI systematization resource pack, pg. 1, 2009)”

Critique of Governance Assessment Applications

GRDC Helpdesk Research Report by Sumedh Rao, Governance and Social Development Resource Centre, July 2010. 16 pages. Available as pdf

Query: Identify the key literature that critiques the use and application of governance assessments. Enquirer: DFID

Contents
1. Overview
2. General critiques
3. Critiques of measurement
4. Worldwide Governance Indicators (WGI)
5. African Peer Review Mechanism (APRM)
6. Other assessments
7. Donor Guidance
8. Initiatives for improving assessments

Including a bibliography of 39 annotated references Continue reading “Critique of Governance Assessment Applications”

Narrative Research

David Snowden, 2010. 21 pages. Available as pdf, from the Cognitive Edge site

“Narrative Research, … lays the foundation for the use of narrative research and inquiry methods not only in the project but broadly in the field of research and consultancy…. Elements of it together with general material on Complexity Theory will be published as a chapter in a book on Naturalising Decision Making in the Fall of 2010.”

Listen First: a pilot system for managing downward accountability in NGOs

Alex Jacobs and Robyn Wilford. Development in Practice, Volume 20, Number 7, September 2010 Available as pdf

“Abstract: This article reports on a research project intended to develop systematic ways of managing downward accountability in an international NGO. Innovative tools were developed and trialled in six countries. The tools comprised a framework, de?ning downward accountability in practical terms, and three management processes. They were successfully used to
(a) encourage staff to improve downward accountability in ways relevant to their context;
(b) hear bene?ciaries’ assessments of the level of accountability achieved and the value of the NGO’s work; and (c) generate quanti?ed performance summaries for managers. Taken together, they form a coherent draft management system. Areas for further research are identied.”

There’s more related material at www.listenfirst.org,

The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance

Alnoor Ebrahim, V. Kasturi Rangan, Social Enterprise Initiative, Harvard Business School (2010) Working Paper 10-099 Available as pdf

ABSTRACT

“Leaders of organizations in the social sector are under growing pressure to demonstrate their impacts on pressing societal problems such as global poverty. We review the debates around performance and impact, drawing on three literatures: strategic philanthropy, nonprofit management, and international development. We then develop a contingency framework for measuring results, suggesting that some organizations should measure long-term impacts, while others should focus on shorter-term outputs and outcomes. In closing, we discuss the implications of our analysis for future research on performance management.”