2008 Reader on Measuring and Reporting Results

(From the Donor Committee for Enterprise Development website)

Description

In the absence of much discussion on the subject, it remains rather sensitive, and one that people therefore try to avoid. Meanwhile, external pressures are growing, for more information; they are coming from donors (e.g. through the Paris Declaration, the MDG deadline), new players and aid models (e.g. social investors) and increased visibility (e.g. Live8). This Reader argues that practitioners need to seize the initiative and to develop answers, before someone else does it for them. In the absence of good data, critics will always be able to say: ‘if you cannot measure it, maybe it is not there’.

A brief overview is therefore given of current understanding in the field, including particularly the terms, indicators and methodologies in use. It is argued that multi-agency agreement in these areas would yield very important benefits, in addition to an approximate comparison of performance; for example:

– agencies could add impacts achieved across all of their country programmes, enabling them to report results for the agency as a whole;
– agencies would also be able to make informed choices about which intervention strategies to fund

Examples are given of impacts measured in a standard format, including for example cost per job created; since the resulting numbers are very different in magnitude, they make a rational conversation about strategy choice possible – even if they are only correct to within +/- 50%. Agreement now needs to be built around the key parameters for formulating these numbers, including for example the multipliers to use for indirect impacts.

Approximate measures do not replace the need for rigorous impact assessments. But agreement between agencies on a small number of indicators, and their application across a wide range of interventions, would win recognition for the achievements of the PSD community. Affordable mechanisms are needed, to ensure that the numbers produced are credible – for example through certification of the methodologies used.

Associated documents
» 2008 PSD Reader on Measuring and Reporting Results, by Jim Tanburn (650 kB)
» Espagnol: Documento de trabajo 2008 sobre le desarrollo del sector privado: Medicion e informe de resultados (1 Mb)
» Francais: Document de base 2008 sur le developpement du secteur prive: Quantifier et rapporter les resultats (902 kB)
Jim Tanburn is Coordinator of the Donor Committee for Enterprise Development ( www.enterprise-development.org) and also run a couple of inter-agency databases (www.Value-Chains.org and www.BusinessEnvironment.org).

Quality COMPAS and Dynamic COMPAS course organized by Groupe URD at its head office

(from Alnap email list)

Training Course (in English): The Dynamic COMPAS and the Quality COMPAS

(quality assurance method and software for humanitarian projects)

23 – 27 June 2008 in Plaisians (Provence)

The Quality COMPAS method and Dynamic COMPAS software are project and information management tools for humanitarian projects. They will be the subject of a short training course to be held in Plaisians (Groupe URD’s head office) from 23rd to 27th June 2008.

The course will be conducted in English.

Project and information management is essential to ensure the quality of humanitarian projects. Many of the weaknesses which have been identified in projects over the last decade do not come from a lack of technical knowledge on the part of humanitarian actors but rather because qualitative factors have not been properly taken into account.

Drawing its content from the COMPAS method, the course will cover subjects such as (1) conducting a situation analysis which goes further than a simple needs analysis, (2) designing a project beyond the logical framework, (3) defining objectives and indicators in keeping with all the quality criteria, (4) developing and implementing a monitoring system, (5) understanding the difference between monitoring and evaluation, etc.

The course is organised around a case study which gives participants practical experience of quality management and using the COMPAS method and the Dynamic COMPAS software (to download it and/or to know more about the COMPAS: http://www.compasqualite.org/en/index/index.php)

This course has been designed for national and international aid workers involved in project management activities like needs assessment, design, monitoring, self-evaluation and evaluation.

Do not hesitate to contact us for further information.

Pierre Brunet

Training Unit Coordinator

Improving poverty measurement in Sri Lanka

(From ELDIS Poverty Reporter)

Authors: Gunewardena,D.
Produced by: Centre for Poverty Analysis, Sri Lanka (2005)

Recently, conceptual advances in poverty measurement have been made:

  • acceptance of the multidimensionality of poverty
  • parallel use of monetary, capability, social exclusion and participatory approaches
  • better measurement of the dynamics of poverty and vulnerability
  • a rudimentary but growing agenda for the measurement of empowerment
  • empirical work comparing the results of different approaches
  • availability of non-traditional instruments of data collection

Reviewing studies that measure poverty in Sri Lanka, this paper finds that poverty measurement in Sri Lanka has also evolved considerably:

  • establishment of an official poverty line
  • adoption of the cost of basic needs (CBN) poverty measurement methodology by the Department of Census and Statistics
  • operationalising the multidimensionality of poverty via the human poverty index and multidimensional composite index
  • ne w survey instruments and methodologies
  • considerable data generation “capacity” of the Department of Census and Statistics (DCS) and the Statistics Department of the Central Bank of Sri Lanka (CBSL)
  • availability of a large amount of administrative data

However, there has not been much progress in measuring the dynamics of poverty, mainly because of a lack of panel data. Measurement of vulnerability and empirical work comparing the results of quantitative and qualitative approaches are also scarce. The existing data still has to be made into an information system and there is a lack of highly disaggregated data as well as data for the North and the East.

Given those weaknesses and strengths, the following steps toimprove poverty measurement, data generation and dissemination in Sri Lanka are proposed:

  • identifying user needs
  • developing appropriate equivalence scales to assign household expenditure to individuals
  • deriving relative and subjective poverty lines
  • constructing baseline datasets for the North and East
  • combining qualitative and quantitative methods of poverty analysis and data collection
  • making surveys consistent and comparable across time and instruments
  • introducing multi-topic surveys
  • generating panel data
  • publication of consistent and comparable poverty statistics
  • maintenance of an internet site with information on data for monitoring poverty
  • construction of public use data files from the Census and relevant household surveys

Available online at: http://www.eldis.org/go/about-eldis&id=36895&type=Document

Symposium: Policy and Programme Evaluation in Europe : cultures and prospects

Date: July 3 and 4, 2008
Venue: European Parliament in Strasbourg, France

Dear Sir or Madam,

It is our pleasure to send you the enclosed programme of the Société Française de l’Évaluation’s next symposium, entitled “The evaluation of public policies in Europe: cultures and futures”. It is available here : http://www.sfe.asso.fr/fr/strasbourg2008/programme.html

The symposium shall take place on July 3 and 4, 2008 at the European Parliament in Strasbourg, France. It is an official event of the French Presidency of the European Union 2008.

The event has been planned for all actors involved in the evaluation of public policies, and more generally for all those engaged in improving the effectiveness of public sector activities.

Its ambition is to share good practices, methods and cultures in evaluation on a European scale, as is borne out by our partnership with DeGEval – Gesellschaft für Evaluation (http://www.degeval.de), as well as the contribution of around ten other national societies for evaluation. We aim to use the diversity of European practices as a collective strength and a resource for everyone.

We are thus looking forward to welcoming a large audience, from six hundred to eight hundred participants, representing a vast diversity of the European spectrum, consisting of experts, representatives of international organisations, national, regional and local civil services and authorities, as well as academics and representatives of civil society. The exchanges of views throughout the symposium shall be facilitated through simultaneous translation between French and English.

We would be very honoured if you could come and contribute to the exchanges of those assembled with your own experience and thoughts.

For practical information and registration :
http://www.sfe.asso.fr/fr/strasbourg2008/registration-practical-information.html

It is our hope that we will have the pleasure of welcoming you at the European Parliament next July 3 and 4.

Yours sincerely,

President of the SFE,
Guy CAUQUIL
http://www.sfe.asso.fr

Net-Map Training in Washington

Net-Map Tool- Training for Research and Impact Management

Net-Map is a visualization approach to Social Network Analysis that was developed by IFPRI post-doc Eva Schiffer in her research on multi-stakeholder water governance in northern Ghana. Net-Map is a tool for understanding and improving complex social situations with multiple actors, diverse goals and unclear distribution of influence.

“You want to learn how to use Net-Map in your own specific context, get a first idea of the social network measures we use to understand our data, be guided through the process of data entry and ask me all your remaining questions?

Why not participate in our IFPRI seminar:

Net-Map Tool-Pool Training for Research and Impact Management

on the 14th May in Washington DC?

The seminar has two parts:

Part 1: An Introduction to Net-Map (12:30 PM to 2:00 PM)

Part 2: Practical Training on Net-Map (2:30 PM to 4:30 PM)
(GMT-05:00) Eastern Time (US & Canada)

This seminar is aimed both at researchers and practitioners who want to use influence network visualisation in their work. It does not require previous knowledge in Social Network Analysis.

Go here (236 K) to find out more, and contact me at ifpri-netmap@cgiar.org if you want to either participate in person or via the internet.”

Posted on May 1, 2008 by Eva Schiffer, (on her website, Net-Map Toolbox: Influence mapping of social networks)

Invitation to join a dedicated discussion forum on reconstructing baseline data

From: XCeval@yahoogroups.com On Behalf Of Jim Rugh
Sent: April 28, 2008 5:47 PM
To: XCEval listserv; MandE NEWS
Subject: [XCeval] Invitation to join a dedicated discussion forum on reconstructing baseline data

We realize that any evaluation that purports to be an “impact evaluation” needs to compare “before-and-after” (pre-test + post-test data) and “with-and-without” (the counterfactual – what would have happened without the intervention being evaluated). Yet in our experience the majority of evaluations conducted of development projects and programs do not have comparable baseline data, nor appropriate comparison (much less “control”) groups. Although the discussion of counterfactuals and pre-test + post-test comparisons frequently focuses on quantitative evaluations designs, the need to understand baseline conditions is equally important for qualitative evaluations. What can be done to strengthen evaluations in such cases? In other words, what can be done to reconstruct baseline and counterfactual data?

We (Jim Rugh and Michael Bamberger) are planning a follow-up volume to “RealWorld Evaluation: Working under budget, time, data and political constraints” (Sage Publications 2006). (More information can be found at www.RealWorldEvaluation.org.)
Continue reading “Invitation to join a dedicated discussion forum on reconstructing baseline data”

Impact Assessment: Training to be provided by INTRAC

Date: 07 May 2008 – 09 May 2008
Venue: London, UK

Training event organised by INTRAC

Course fee: £475.00
Number of days: 3
Description:
With increased pressures on delivery and accountability, the need has never been greater for Civil Society and other development organisations to assess the long-term impact of their work. In three fruitful days you will explore the current state of the debate about impact assessment as well as reviewing current practice and methodologies. Learn to assess the effectiveness of your work.Course objectives
• Understand what is meant by impact assessment and how the concept has emerged
• Explore the relationship between impact assessment and other forms of evaluative activity
• Explore different approaches and alternative methodologies in conducting impact assessment
• Identify ways of getting a representative picture e.g. case studies, sampling methods, and triangulation between quantitative and qualitative data
• Consider impact assessment in different contexts e.g. in programmes and projects, organisationally, and in advocacy work

Link to application form

“Is Your Campaign Making A Difference”

…has just been published by the Campaigning Effectiveness Programme at NCVO

The book, written by Jim Coe and Ruth Mayne, seeks to advance the case that simple and effective monitoring and evaluation can, if implemented well, become a powerful tool for social and political change. It is aimed at campaigners and managers who are thinking of introducing monitoring and evaluation approaches within their campaign, or who are looking at ways of improving an existing approach.

Part 1 (available as a free download) provides an overview of the key elements of monitoring and evaluating campaigning, outlining some simple approaches. Part 2 looks in more detail at approaches to tracking outcomes, impact and ways of working. The final sections of the book consider how to design a coherent approach, with commentary on appropriate frameworks, tool and techniques.

Information available at

www.ncvo-vol.org.uk/publications/publication.asp?id=8612

3rd annual Measuring Effectiveness conference:’Participation, Empowerment and Downward Accountability’.

Greetings,

The 3rd annual Measuring Effectiveness conference will be held in Melbourne, on Thursday 25th & Friday 26th September, 2008.

We sincerely hope that you will again be inspired to attend this important event. This year sees World Vision Australia and The Australian National University partnering to bring you a conference themed around ‘Participation, Empowerment and Downward Accountability’.

Attached is the call for papers, requested for submission by Friday 20th June, 2008. That gives you 8 weeks to submit your paper. Competition is increasing each year, so please ensure that you meet this deadline to ensure your paper is given full consideration. For all further details please refer to the attached. Please also distribute this amongst your colleagues and networks who may also be interested.

Conference updates will be posted regularly on the World Vision website, and registrations will again be managed online.We will endeavour to have the conference brochure available online in late May 08, and the final draft conference program, outlining the speakers/presenters and session outlines, available online by late August 08. Further email correspondence will also be sent out in the coming months, however the best source of informatoin will be the website so please check this regularly. You will also find information and papers from previosu ME conferences, as well as other development conferences.

http://www.worldvision.com.au/learn/conferences/index.asp

Please distribute this email amongst your colleagues and networks who may also be interested.

Regards,

Melissa Cadwell | Program Coordinator |
Program Effectiveness | World Vision Australia
phone / fax: +61 3 9287 2769
Email : measuringeffectiveness@worldvision.com.au
Website : http://www.worldvision.com.au/learn/conferences/index.asp

M&E training to be provided by Mosaic

Dear Colleague,
Time to register for Mosaic’s capacity building workshops planned for this July 2008: 1) Stakeholder Participation in: Planning, Needs Assessment, Monitoring and Evaluation; 2) Results-based Management, Appreciative Inquiry and Open Space Technology; and 3) Participatory Monitoring and Evaluation.

Please jot these dates down on your calendar and forward this message to any interested colleagues. We look forward to your participation.

Stakeholder Participation in Planning, Needs Assessment, Monitoring and Evaluation using PRA/PLA Tools
Ottawa, Canada
July 7-12, 2008

The Stakeholder Participation workshop focuses on core participatory concepts, tools and their application. This is an intensive six-day workshop set in the community to maximize learning, group interaction and networking. Topics include The Origins of Participatory Development, Learning and Application of PRA/PLA tools, the Application of Participation to Project Design, Monitoring and Evaluation, Developing Effective Facilitation Skills, Building Action Plans and Team-Building. Two-day community assignments proposed by community-based organizations in the Ottawa region will allow participants to apply tools learned in the workshop to real-life situations. This is also a great opportunity to network with other practitioners, NGOs, donors, and action researchers from all over the world.

For further information, please refer to the web site at http//www.mosaic-net-intl.ca or by email at wkshop05@mosaic-net-intl.ca.

Results-based Management, Appreciative Inquiry and Open Space Technology
Ottawa, Canada
July 14-18, 2008

This new workshop introduces participants to Results-based Management, Appreciative Inquiry and Open Space Technology. Demonstrate the effectiveness of your programmes with Results-based Management. Master what we mean by results, develop programme/organizational plans which are results-based and design performance monitoring systems based on indicators and participatory methods. You will also expand your repertoire of tools to also learn about Appreciative Inquiry and Open Space and how they can be applied to your organization, programme and/or project. These approaches are increasingly being used around the world to tap into new ways to do our work in ways that are more results-oriented, more appreciative and less problem-focused and more self-organized vs top down.

For further information, please refer to the web site at http//www.mosaic-net-intl.ca or by email at wkshop05@mosaic-net-intl.ca.

Participatory Monitoring and Evaluation
Ottawa, Canada
July 21-26,2008

Participatory Monitoring and Evaluation (PM & E) involves a different approach to project monitoring and evaluation by involving local people, project stakeholders, and development agencies deciding together about how to measure results and what actions should follow once this information has been collected and analyzed. This intensive six day experiential workshop is practically focused with daily excursions into the community and a three-day community assignment. Topics covered at the workshop include Origins of PM & E, Skills and Attributes of a PM & E facilitator, Learning PM & E Tools, Designing a Monitoring and Evaluation Framework, Quantitiative and Qualitative Indicators and Building Actions Plan and much more.

All workshops organized by Mosaic are sensitive to issues of gender, ethnicity, race, and class and how these can influence outcomes and how we see the world if they are absent from our assumptions, direct participation, our analysis and conclusions.

Can’t attend the workshops? Contact us to custom design a workshop to suit the specific needs of your organization.

For further information, please refer to the web site at http//:www.mosaic-net-intl.ca or by email Rayna at wkshop05@mosaic-net-intl.ca. Send us your full mailing address and we will send you a brochure.

Francoise Coupal
Founder of Mosaic.net International, Inc.