Monitoring and Evaluation NEWS

Randomised controlled trial testing the effects of transparency on health care in Uganda

(from the great AidInfo website)

“At aidinfo we conduct research and liaise with aid donors and recipients to build up a case for aid transparency. We want to show that improving and increasing the amount that donors report on their aid contributions can help communities to track aid spending. In turn, donors and governments will be more accountable for their aid spending. It is expected that in this way aid will reach more people on the ground, helping to contribute more in the fight against poverty.

This is all well and good, but it is difficult to prove. Svensson’s work, then, is of great importance to us here.

This Study by Reinikka and Svensson (2005) found that in 1995 only 20 percent of a primary education grant program to rural Uganda actually reached its intended target. This figure rose by a striking 60 percent in 2001 when information was published detailing where this money was going; a full 80 percent of funds reached their intended destination, greatly improving education services in the area.

Björkman and Svensson (2009) followed up on this study with a compelling randomised controlled trial testing the effects of transparency on health care in Uganda. The experiment randomly assigned community health clinics to receive published ‘report cards’ and NGO-organised public meetings on the quality of the clinics’ health care.

The results of this transparency ‘treatment’ rivalled the effects of the best health interventions involving expensive new medicines, equipment, and procedures. Waiting time for care decreased, absenteeism among doctors and nurses plummeted, clinics got cleaner, fewer drugs were stolen, 40-50 percent more children received dietary supplements and vaccines, health services got used more, and, powerfully, 33 percent fewer children died under the age of five. This amounted to 550 saved lives in a small area of Uganda encompassing merely 55,000 households.

This is strong evidence that access to information about services empowers citizens to get better services and saves lives.”

Tags: aidinfo, corruption, transparency, uganda

Posted in Uncategorized | No Comments » |  Print This Post

Social Psychology and Evaluation

Tags: bias, psychology

Posted in Books | No Comments » |  Print This Post

Conference about “The Future of Evaluation in Modern Societies”, Germany.

“The Center for Evaluation (CEval) of Saarland University, Germany, is a globally active research institute for applied social science in the field of evaluation and member of the DeGEval (German Evaluation Society). On this occasion, we organize an international conference about “The Future of Evaluation in Modern Societies” on 14th and 15th June 2012 in Saarbruecken, Germany.

The objective of this event is to discuss the role of evaluation in societies comprehensively and on an international comparison for bringing different discussion strands together into a joint debate. For keynote speeches and lectures, we could already win numerous renowned scientists from the USA, Latin America, Africa and Europe.

Please find the detailed program and registration form on our homepage: http://futureofevaluation.ceval.de

You also find a review about our recent book “A Practioner Handbook on Evaluation” which will appeal to evaluation practitioners, policy-makers who conduct evaluations in their daily work, students training in applied research and organizations which are implementing projects and programs that could be the subject of an evaluation.

—————————————

Maria Albrecht,  M.A., Center for Evaluation (CEval), Saarland University, P.O. Box 15 11 50, 66041 Saarbrücken – Germany, Fon: +49 (0)681 302-3561, Fax: +49 (0)681 302-3899, www.ceval.de

Tags: germany

Posted in Conferences, Types of Events | No Comments » |  Print This Post

UKES CONFERENCE 2012 Evaluation for results: What counts? Who stands to gain? How is it done?

16 March 2012
The Macdonald Hotel, Birmingham

[from UKES website] UKES conferences address leading issues of the day in programme and policy evaluation. The 2012 Annual Conference will address the current drive towards evaluation focused on results – frequently linked to ‘Payment by Results’ and what, in international development and elsewhere, is familiar as ‘Results-Based Management’.

Evaluators and those who commission evaluation who advocate a focus on results reflect a legitimate concern with the productivity and efficiency of programmes and the capacity of interventions to secure gains and improvements in practice and provision. They point out that programmes should be held to account to accomplish what they were designed to do and paid for, often out of public funds. A primary focus on results seeks to emphasise main effects and outcomes that have been valued and agreed. In times of austerity and unusually scarce resources, proponents of a strong focus on results argue that emphasising value for money is socially responsible.

Others argue that an over-emphasis on measuring a programme’s results neglects important questions of how results are generated in a context, whether results capture the real quality and accomplishments of a programme, and how those results may reflect the values and ambitions of all programme stakeholders. They remind us of secondary effects and ‘unintended beneficiaries’ of programmes that may not be readily captured by results. Some also raise questions about the source of criteria over what counts as a worthwhile result given that not all programme achievements can be measured, and stakeholders may differ over a programme’s objectives. 

Against this background conference participants are invited to contribute their own perspectives on the dominant issues they consider relevant to the theory and practice of evaluation in the public interest. We anticipate a lively and informative debate to stimulate professional learning and to contribute to the improvement of evaluation practice and commissioning.

Potential contributors are invited to propose discussions, seminar presentations, lectures or poster sessions which explore issues around this theme. Those issues may fall within one of the following categories – though you are invited to propose your own theme:?

• How do we define a valid ‘result’ and whose results get counted?
• How do we best measure a result – including taking account of counterfactuals?
• How do we understand where results came from, what significance they have and whether they can be replicated – i.e. what is the relation between a result and context?
• Where do benchmarks come from to measure results achievement?
• If a result is, say, a 4% improvement – how do we know whether that is a lot or a little under the circumstances?
• How do we represent the circumstances and mechanisms that give rise to a result?
• How do we account for programme accomplishments that are not represented in results?
• Is results-measurement a robust foundation for replication/extension of a programme?

A formal call for papers and proposals for sessions will be circulated shortly.  The conference will be preceded on 15 March 2012 with a choice of training workshops on specialist topics.

Posted in Conferences, Types of Events | No Comments » |  Print This Post

3ie’s proposes a Commitment to Evaluation Indicator (c2e)

International Initiative for Impact Evaluation (3ie) -Terms of Reference for a Research Consultancy– White paper for the Commitment to Evaluation Indicator

“Background: Experience to date shows that the use of evidence by donors and governments when designing and adopting development programmes remains sporadic. There are many examples where a programme was shown to have no impact but was expanded, as well as examples of programmes with positive impact being terminated. To promote better use of evaluation evidence in policy making and programme design, 3ie is launching a Commitment to Evaluation (c2e) indicator. The indicator will provide a measurement of government and donor agency use of evaluation evidence allowing for recognition and reward for progress and good practice. The indicator will be developed and piloted in 2012 for donor agencies with the intent to recognize donors that make systematic use of evidence and thus motivate others to do the same.

3ie’s initiative follows the example of other successful efforts to use awards or indexes to focus the attention of policymakers. Indexes such as the UN Development Programme’s Human Development index, Transparency International’s Corruption Perception index, and the Centre for Global Development’s Quality of ODA (QuODA) index have raised awareness on key issues and influenced practice of governments and development agencies. The Mexican National Council for the Evaluation of Social Development Policy (CONEVAL) annual award for good practices in social evaluation has strengthened political buy-in and commitment to evaluation in Mexico. In developing this c2e indicator, 3ie will draw from the lessons learned by similar initiatives on how best to motivate and award evaluation practices and build and run an effective cross-agency and cross-country indicator. More detailed background information on the rationale and theory of change behind the project is available in the discussion note in the annex.” See ToRs for rest of the text including annex.

Tags: 3ie, index, indicator

Posted in Uncategorized | 1 Comment » |  Print This Post

Diversity and Complexity

by Scott Page, 2011. Available on Google Books Princeton University Press, 14/07/2011 – 296 pages

Abstract: This book provides an introduction to the role of diversity in complex adaptive systems. A complex system–such as an economy or a tropical ecosystem–consists of interacting adaptive entities that produce dynamic patterns and structures. Diversity plays a different role in a complex system than it does in an equilibrium system, where it often merely produces variation around the mean for performance measures. In complex adaptive systems, diversity makes fundamental contributions to system performance. Scott Page gives a concise primer on how diversity happens, how it is maintained, and how it affects complex systems. He explains how diversity underpins system level robustness, allowing for multiple responses to external shocks and internal adaptations; how it provides the seeds for large events by creating outliers that fuel tipping points; and how it drives novelty and innovation. Page looks at the different kinds of diversity–variations within and across types, and distinct community compositions and interaction structures–and covers the evolution of diversity within complex systems and the factors that determine the amount of maintained diversity within a system.Provides a concise and accessible introduction. Shows how diversity underpins robustness and fuels tipping points .Covers all types of diversity. The essential primer on diversity in complex adaptive systems.

RD Comment: This book is very useful for thinking about the measurement of diversity. In 2000 I wrote a paper “Does Empowerment Start At Home? And If So, How Will We Recognise It?” in which I argued that…

“At the population level, diversity of behaviour can be seen as a gross indicator of agency (of the ability to make choices), relative to homogenous behaviour by the same set of people. Diversity of behaviour suggests there is a range of possibilities which individuals can pursue. At the other extreme is standardisation of behaviour, which we often associate with limited choice. The most notable example being perhaps that of an army. An army is a highly organised structure where individuality is not encouraged, and where standardised and predictable behaviour is very important. Like the term “NGO” or “non-profit”, diversity is defined by something that it is not -  a condition where there is no common constraint, which would otherwise lead to a homogeneity of response. Homogeneity of behaviour may arise from various sources of constraint. A flood may force all farmers in a large area to move their animals to the high ground. Everybody’s responses are the same, when compared to what they would be doing on normal day. At a certain time of the year all farmers may be planting the same crop. Here homogeneity of practice may reflect common constraints arising from a combination of sources: the nature of the physical environment, and the nature of particular local economies. Constraints on diversity can also arise within the assisting organisation. Credit programs can impose rules on loan use, specific repayment schedules and loan terms, as well as limiting when access to credit is available, or how quickly approval will be give.”

See also…

• Page’s 2006 book “The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies“
• Scott Page on Leveraging Diversity on YouTube, in 2010
• Scott Page on Resilience (part 1), (part 2) (part 3) (part 4) (part 5) on YouTube, in 2009

Tags: complexity, diversity, measurement

Posted in Books, Media | No Comments » |  Print This Post

3ie and the Funding of Impact Evaluations

A DISCUSSION PAPER FOR 3IE’S MEMBERS. by Rick Davies, July 2011. Commissioned by the Office of Development Effectiveness, AusAI. Available as pdf.

The purpose of this discussion paper is to inform AusAID’s and other 3ie members’ engagement with 3ie (the International Initiative for Impact Evaluation). It precedes the forthcoming evaluation of 3ie, and is more limited in scope. It is expected to be complementary and useful to the larger Department for International Development (DFID) study now underway, Developing a broader range of rigorous designs and methods for impact evaluations, as the author of this report is also a member of that study team.

AusAID is a member of 3ie and provides core funding to 3ie to contribute to the global public good of policy-relevant evidence on what works in development. Direct benefit to AusAID is not the purpose of the membership. However, it is important to AusAID that 3ie’s work is relevant to AusAID’s partners, particularly partners with low income and/or in fragile countries. AusAID’s Office of Development Effectiveness (ODE) manages AusAID’s membership of 3ie and has commissioned this discussion paper.

The focus of this discussion paper is on 3ie methodological approach, used in both the funded impact evaluations and systematic reviews, and how this has changed over time. Read the rest of this entry »

Posted in Uncategorized | 1 Comment » |  Print This Post

RealWorld Evaluation Working Under Budget, Time, Data, and Political Constraints

Second Edition, by Michael Bamberger, Jim Rugh, Linda Mabry. Sage Publications,  Nov 2011,

“This book addresses the challenges of conducting program evaluations in real-world contexts where evaluators and their clients face budget and time constraints and where critical data may be missing. The book is organized around a seven-step model developed by the authors, which has been tested and refined in workshops and in practice. Vignettes and case studies—representing evaluations from a variety of geographic regions and sectors—demonstrate adaptive possibilities for small projects with budgets of a few thousand dollars to large-scale, long-term evaluations of complex programs. The text incorporates quantitative, qualitative, and mixed-method designs, and this Second Edition reflects important developments in the field since the publication of the First Edition. ”

See also the associated website: http://www.realworldevaluation.org/ Bamberger and Rugh have presented many workshops on RealWorld Evaluation in many countries. A copy of various versions and translations of the PowerPoint presentations and other materials are accessible on the next pages of this website. Read the rest of this entry »

Tags: bamburger, constraints, mabry, real world, rugh

Posted in Uncategorized | No Comments » |  Print This Post

First reports published by UK Independent Commsssion for Aid Impact

…on 22nd November, 2011.

Two cover general areas of the programme:

• ICAI’s Approach to Effectiveness and Value for Money; and

• The Department for International Development’s (DFID) Approach to Anti-Corruption;

Two cover specific programmes in DFID’s country offices:

• DFID’s Climate Change Programme in Bangladesh; and

• DFID’s Support to the Health Sector in Zimbabwe.

See the ICAI website for further details

RD Comment: re ICAI’s Approach to Effectiveness and Value for Money paper, see my Comments here. In summary:

• This paper is confusingly titled. It is really about the ICAIs approach to evaluation, and covers more than “value for money and effectiveness”

• The 4e’s analysis of the concepts of “value for money and effectiveness” has potential, but seems to be taken nowhere thereafter.

• The proposed workings of the “traffic lights” system are opaque, it is not clear how these judgements will be built up from subsidiary judgements.

•  The worst possible performance judgement (i.e. the red light) is  that “The programme meets few of the criteria for effectiveness and value for money. It is performing poorly. Immediate and major changes need to be made” Is that all? Don’t some programmes deserve to be called failures? Don’t some programmes need to be closed down rather than “fixed?  If these possibilities are meant to be considered then they need to be clearly stated. For more on the importance of recognising failure, see “Do we need a Minimum Level of Failure (MLF)?“

Posted in Uncategorized | No Comments » |  Print This Post

M&E Software: A List

Well, the beginnings of a list…

Please note: No guarantee can be given about the accuracy of information provided on the linked websites about the M&E software concerned, and its providers

Stand alone systems

• PacPlan: “Results-Based Planning, Monitoring and Evaluation Software and Process Solution”
• Newdea – Monitoring and Evaluation Software, on YouTube
• AidProject M+E for Donor-funded aid projects
• e-Thamini Monitoring and Evaluation Software (Kenya)
•  HIV/AIDS  Data Capturing And Reporting Platform[Monitoring and Evaluation System] (Kenya)
• Prome Web: A project management, monitoring and evaluation software. Adapted for aid projects in developing countries

Online systems

• Monitoring and Evaluation Online: Online Monitoring and Evaluation Software Tool
• Views  online monitoring, evaluation and reporting system
• M&E Online

Survey supporting software

• Online Survey Comparison Chart, comparing six different services
• EpiSurveyor lets anyone create an account, design forms, download them to phones, and start collecting data in minutes, for free. (Kenya)
• Open Data Kit (ODK) is a free and open-source set of tools which help organizations author, field, and manage mobile data collection solution
• EthnoCorder is mobile multimedia survey software for your iPhone
• Mobile data collection tools – Comparison matrix – 13 tools including above

Program Logic Modeling

• DoView – Visual outcomes and results planning
• IdeaTree – Simultaneous Collaboration & Brainstorming Using Mind Maps
• yEd – diagram editor that can be used to  generate drawings of diagrams.  FREE

Other lists

• A software list compiled by CES UK: A review of 14 generic monitoring and evaluation systems, plus a 1 page framework for deciding on an M&E system. The 14 systems are show on the left sidebar, click on each to see a matrix of attiributes. (Found courtesy of Orla Cronin)
• Social Network Analysis software: A list

Other other

• Monitoring & Evaluation Software open group on LinkedIn (i.e. you dont need to sign in)

If you have software, or lists of software, which you would like to see added here, please use the Comment facility below

Tags: Software, systems

Posted in Software | 8 Comments » |  Print This Post