(from the great AidInfo website)

“At aidinfo we conduct research and liaise with aid donors and recipients to build up a case for aid transparency. We want to show that improving and increasing the amount that donors report on their aid contributions can help communities to track aid spending. In turn, donors and governments will be more accountable for their aid spending. It is expected that in this way aid will reach more people on the ground, helping to contribute more in the fight against poverty.

This is all well and good, but it is difficult to prove. Svensson’s work, then, is of great importance to us here.

This Study by Reinikka and Svensson (2005) found that in 1995 only 20 percent of a primary education grant program to rural Uganda actually reached its intended target. This figure rose by a striking 60 percent in 2001 when information was published detailing where this money was going; a full 80 percent of funds reached their intended destination, greatly improving education services in the area.

Björkman and Svensson (2009) followed up on this study with a compelling randomised controlled trial testing the effects of transparency on health care in Uganda. The experiment randomly assigned community health clinics to receive published ‘report cards’ and NGO-organised public meetings on the quality of the clinics’ health care.

The results of this transparency ‘treatment’ rivalled the effects of the best health interventions involving expensive new medicines, equipment, and procedures. Waiting time for care decreased, absenteeism among doctors and nurses plummeted, clinics got cleaner, fewer drugs were stolen, 40-50 percent more children received dietary supplements and vaccines, health services got used more, and, powerfully, 33 percent fewer children died under the age of five. This amounted to 550 saved lives in a small area of Uganda encompassing merely 55,000 households.

This is strong evidence that access to information about services empowers citizens to get better services and saves lives.”

“The Center for Evaluation (CEval) of Saarland University, Germany, is a globally active research institute for applied social science in the field of evaluation and member of the DeGEval (German Evaluation Society). On this occasion, we organize an international conference about “The Future of Evaluation in Modern Societies” on 14th and 15th June 2012 in Saarbruecken, Germany.

The objective of this event is to discuss the role of evaluation in societies comprehensively and on an international comparison for bringing different discussion strands together into a joint debate. For keynote speeches and lectures, we could already win numerous renowned scientists from the USA, Latin America, Africa and Europe.

Please find the detailed program and registration form on our homepage: http://futureofevaluation.ceval.de

You also find a review about our recent book “A Practioner Handbook on Evaluation” which will appeal to evaluation practitioners, policy-makers who conduct evaluations in their daily work, students training in applied research and organizations which are implementing projects and programs that could be the subject of an evaluation.

—————————————

Maria Albrecht,  M.A., Center for Evaluation (CEval), Saarland University, P.O. Box 15 11 50, 66041 Saarbrücken – Germany, Fon: +49 (0)681 302-3561, Fax: +49 (0)681 302-3899, www.ceval.de

16 March 2012
The Macdonald Hotel, Birmingham

[from UKES website] UKES conferences address leading issues of the day in programme and policy evaluation. The 2012 Annual Conference will address the current drive towards evaluation focused on results – frequently linked to ‘Payment by Results’ and what, in international development and elsewhere, is familiar as ‘Results-Based Management’.

Evaluators and those who commission evaluation who advocate a focus on results reflect a legitimate concern with the productivity and efficiency of programmes and the capacity of interventions to secure gains and improvements in practice and provision. They point out that programmes should be held to account to accomplish what they were designed to do and paid for, often out of public funds. A primary focus on results seeks to emphasise main effects and outcomes that have been valued and agreed. In times of austerity and unusually scarce resources, proponents of a strong focus on results argue that emphasising value for money is socially responsible.

Others argue that an over-emphasis on measuring a programme’s results neglects important questions of how results are generated in a context, whether results capture the real quality and accomplishments of a programme, and how those results may reflect the values and ambitions of all programme stakeholders. They remind us of secondary effects and ‘unintended beneficiaries’ of programmes that may not be readily captured by results. Some also raise questions about the source of criteria over what counts as a worthwhile result given that not all programme achievements can be measured, and stakeholders may differ over a programme’s objectives. 

Against this background conference participants are invited to contribute their own perspectives on the dominant issues they consider relevant to the theory and practice of evaluation in the public interest. We anticipate a lively and informative debate to stimulate professional learning and to contribute to the improvement of evaluation practice and commissioning.

Potential contributors are invited to propose discussions, seminar presentations, lectures or poster sessions which explore issues around this theme. Those issues may fall within one of the following categories – though you are invited to propose your own theme:?

• How do we define a valid ‘result’ and whose results get counted?
• How do we best measure a result – including taking account of counterfactuals?
• How do we understand where results came from, what significance they have and whether they can be replicated – i.e. what is the relation between a result and context?
• Where do benchmarks come from to measure results achievement?
• If a result is, say, a 4% improvement – how do we know whether that is a lot or a little under the circumstances?
• How do we represent the circumstances and mechanisms that give rise to a result?
• How do we account for programme accomplishments that are not represented in results?
• Is results-measurement a robust foundation for replication/extension of a programme?

A formal call for papers and proposals for sessions will be circulated shortly.  The conference will be preceded on 15 March 2012 with a choice of training workshops on specialist topics.

International Initiative for Impact Evaluation (3ie) -Terms of Reference for a Research Consultancy– White paper for the Commitment to Evaluation Indicator

“Background: Experience to date shows that the use of evidence by donors and governments when designing and adopting development programmes remains sporadic. There are many examples where a programme was shown to have no impact but was expanded, as well as examples of programmes with positive impact being terminated. To promote better use of evaluation evidence in policy making and programme design, 3ie is launching a Commitment to Evaluation (c2e) indicator. The indicator will provide a measurement of government and donor agency use of evaluation evidence allowing for recognition and reward for progress and good practice. The indicator will be developed and piloted in 2012 for donor agencies with the intent to recognize donors that make systematic use of evidence and thus motivate others to do the same.

3ie’s initiative follows the example of other successful efforts to use awards or indexes to focus the attention of policymakers. Indexes such as the UN Development Programme’s Human Development index, Transparency International’s Corruption Perception index, and the Centre for Global Development’s Quality of ODA (QuODA) index have raised awareness on key issues and influenced practice of governments and development agencies. The Mexican National Council for the Evaluation of Social Development Policy (CONEVAL) annual award for good practices in social evaluation has strengthened political buy-in and commitment to evaluation in Mexico. In developing this c2e indicator, 3ie will draw from the lessons learned by similar initiatives on how best to motivate and award evaluation practices and build and run an effective cross-agency and cross-country indicator. More detailed background information on the rationale and theory of change behind the project is available in the discussion note in the annex.” See ToRs for rest of the text including annex.

by Scott Page, 2011. Available on Google Books Princeton University Press, 14/07/2011 – 296 pages

Abstract: This book provides an introduction to the role of diversity in complex adaptive systems. A complex system–such as an economy or a tropical ecosystem–consists of interacting adaptive entities that produce dynamic patterns and structures. Diversity plays a different role in a complex system than it does in an equilibrium system, where it often merely produces variation around the mean for performance measures. In complex adaptive systems, diversity makes fundamental contributions to system performance. Scott Page gives a concise primer on how diversity happens, how it is maintained, and how it affects complex systems. He explains how diversity underpins system level robustness, allowing for multiple responses to external shocks and internal adaptations; how it provides the seeds for large events by creating outliers that fuel tipping points; and how it drives novelty and innovation. Page looks at the different kinds of diversity–variations within and across types, and distinct community compositions and interaction structures–and covers the evolution of diversity within complex systems and the factors that determine the amount of maintained diversity within a system.Provides a concise and accessible introduction. Shows how diversity underpins robustness and fuels tipping points .Covers all types of diversity. The essential primer on diversity in complex adaptive systems.

RD Comment: This book is very useful for thinking about the measurement of diversity. In 2000 I wrote a paper “Does Empowerment Start At Home? And If So, How Will We Recognise It?” in which I argued that…

“At the population level, diversity of behaviour can be seen as a gross indicator of agency (of the ability to make choices), relative to homogenous behaviour by the same set of people. Diversity of behaviour suggests there is a range of possibilities which individuals can pursue. At the other extreme is standardisation of behaviour, which we often associate with limited choice. The most notable example being perhaps that of an army. An army is a highly organised structure where individuality is not encouraged, and where standardised and predictable behaviour is very important. Like the term “NGO” or “non-profit”, diversity is defined by something that it is not -  a condition where there is no common constraint, which would otherwise lead to a homogeneity of response. Homogeneity of behaviour may arise from various sources of constraint. A flood may force all farmers in a large area to move their animals to the high ground. Everybody’s responses are the same, when compared to what they would be doing on normal day. At a certain time of the year all farmers may be planting the same crop. Here homogeneity of practice may reflect common constraints arising from a combination of sources: the nature of the physical environment, and the nature of particular local economies. Constraints on diversity can also arise within the assisting organisation. Credit programs can impose rules on loan use, specific repayment schedules and loan terms, as well as limiting when access to credit is available, or how quickly approval will be give.”

See also…

• Page’s 2006 book “The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies“
• Scott Page on Leveraging Diversity on YouTube, in 2010
• Scott Page on Resilience (part 1), (part 2) (part 3) (part 4) (part 5) on YouTube, in 2009

A DISCUSSION PAPER FOR 3IE’S MEMBERS. by Rick Davies, July 2011. Commissioned by the Office of Development Effectiveness, AusAI. Available as pdf.

The purpose of this discussion paper is to inform AusAID’s and other 3ie members’ engagement with 3ie (the International Initiative for Impact Evaluation). It precedes the forthcoming evaluation of 3ie, and is more limited in scope. It is expected to be complementary and useful to the larger Department for International Development (DFID) study now underway, Developing a broader range of rigorous designs and methods for impact evaluations, as the author of this report is also a member of that study team.

AusAID is a member of 3ie and provides core funding to 3ie to contribute to the global public good of policy-relevant evidence on what works in development. Direct benefit to AusAID is not the purpose of the membership. However, it is important to AusAID that 3ie’s work is relevant to AusAID’s partners, particularly partners with low income and/or in fragile countries. AusAID’s Office of Development Effectiveness (ODE) manages AusAID’s membership of 3ie and has commissioned this discussion paper.

The focus of this discussion paper is on 3ie methodological approach, used in both the funded impact evaluations and systematic reviews, and how this has changed over time.

Contents
1. Introduction
1.1 Purpose and approach
1.2 A summary of 3ie’s stated methodological approach
1.3 Debates on impact evaluation
1.4 What we don’t know
2. The Issues
2.1 Are RCTs being promoted to the exclusion of other approaches?
2.2 Is attention being given to how it works, as well as what works?
2.3 Are RCTs focusing on the simple questions and ignoring the more difficult?
2.4 Do the specific requirements of RCTs limit their range of application?
2.5 Who can afford RCTs?
2.6 How valid are the results?
2.7 How generalisable are the results?
2.8 Risks of unintended effects
2.9 Ethical issues
2.10 A reminder of the value of experimental approaches
3. A Wider View: Methodology in the Context of Strategy
4. Future Directions
4.1 Obtaining more value from existing investments
4.2 Development of new methodologies
4.3 Focusing on the strategic objectives

Second Edition, by Michael Bamberger, Jim Rugh, Linda Mabry. Sage Publications,  Nov 2011,

“This book addresses the challenges of conducting program evaluations in real-world contexts where evaluators and their clients face budget and time constraints and where critical data may be missing. The book is organized around a seven-step model developed by the authors, which has been tested and refined in workshops and in practice. Vignettes and case studies—representing evaluations from a variety of geographic regions and sectors—demonstrate adaptive possibilities for small projects with budgets of a few thousand dollars to large-scale, long-term evaluations of complex programs. The text incorporates quantitative, qualitative, and mixed-method designs, and this Second Edition reflects important developments in the field since the publication of the First Edition. ”

See also the associated website: http://www.realworldevaluation.org/ Bamberger and Rugh have presented many workshops on RealWorld Evaluation in many countries. A copy of various versions and translations of the PowerPoint presentations and other materials are accessible on the next pages of this website.

What’s New in the Second Edition of Real World Evaluation?

• A greater focus on responsible professional practice, codes of conduct, and the importance of ethical standards for all evaluations.
• Some new perspectives on the debate over the “best” evaluation designs. While experimental designs can address the important issues of selection bias, such statistical designs are potentially vulnerable to a number of important threats to validity. These include process and contextual analysis, collecting information on sensitive topics and from difficult-to-reach groups, difficulties in adapting to changes in the evaluation design, and implementation strategies. Experience also suggests that strong statistical designs can be applied only in a very small proportion of evaluations.
• There are many instances in which well-designed nonexperimental designs will be the best option for assessing outcomes of many programs, particularly for evaluating complex programs and even “simple” programs that involve complex processes of behavioral change.
• The importance of understanding the setting within which the evaluation is designed, implemented, and used.
• Program theory as a central building block of most evaluation designs. The expanded discussion incorporates theory of change, contextual and process analysis, multilevel logic models, using competing theories, and trajectory analysis.
• The range of evaluation design options has been considerably expanded, and case studies are included to illustrate how each of the 19 designs has been applied in the field.
• Greater emphasis is given to the benefits of mixed-method evaluation designs.
• A new chapter has been added on the evaluation of complicated and complex development interventions. Conventional pretest-posttest comparison group designs can rarely be applied to the increasing proportion of development assistance channeled through complex interventions, and a range of promising new approaches—still very much “work in progress”—is presented.
• Two new chapters on organizing and managing evaluations and strengthening evaluation capacity. This includes a discussion of strategies for promoting the institutionalization of evaluation systems at the sector and national levels.
• The discussion of quality assurance and threats to validity has been expanded, and checklists and worksheets are included on how to assess the validity of quantitative, qualitative, and mixed-method designs.

…on 22nd November, 2011.

Two cover general areas of the programme:

• ICAI’s Approach to Effectiveness and Value for Money; and

• The Department for International Development’s (DFID) Approach to Anti-Corruption;

Two cover specific programmes in DFID’s country offices:

• DFID’s Climate Change Programme in Bangladesh; and

• DFID’s Support to the Health Sector in Zimbabwe.

See the ICAI website for further details

RD Comment: re ICAI’s Approach to Effectiveness and Value for Money paper, see my Comments here. In summary:

• This paper is confusingly titled. It is really about the ICAIs approach to evaluation, and covers more than “value for money and effectiveness”

• The 4e’s analysis of the concepts of “value for money and effectiveness” has potential, but seems to be taken nowhere thereafter.

• The proposed workings of the “traffic lights” system are opaque, it is not clear how these judgements will be built up from subsidiary judgements.

•  The worst possible performance judgement (i.e. the red light) is  that “The programme meets few of the criteria for effectiveness and value for money. It is performing poorly. Immediate and major changes need to be made” Is that all? Don’t some programmes deserve to be called failures? Don’t some programmes need to be closed down rather than “fixed?  If these possibilities are meant to be considered then they need to be clearly stated. For more on the importance of recognising failure, see “Do we need a Minimum Level of Failure (MLF)?“

Well, the beginnings of a list…

Please note: No guarantee can be given about the accuracy of information provided on the linked websites about the M&E software concerned, and its providers

Stand alone systems

• PacPlan: “Results-Based Planning, Monitoring and Evaluation Software and Process Solution”
• Newdea – Monitoring and Evaluation Software, on YouTube
• AidProject M+E for Donor-funded aid projects
• e-Thamini Monitoring and Evaluation Software (Kenya)
•  HIV/AIDS  Data Capturing And Reporting Platform[Monitoring and Evaluation System] (Kenya)
• Prome Web: A project management, monitoring and evaluation software. Adapted for aid projects in developing countries

Online systems

• Monitoring and Evaluation Online: Online Monitoring and Evaluation Software Tool
• Views  online monitoring, evaluation and reporting system
• M&E Online

Survey supporting software

• Online Survey Comparison Chart, comparing six different services
• EpiSurveyor lets anyone create an account, design forms, download them to phones, and start collecting data in minutes, for free. (Kenya)
• Open Data Kit (ODK) is a free and open-source set of tools which help organizations author, field, and manage mobile data collection solution
• EthnoCorder is mobile multimedia survey software for your iPhone
• Mobile data collection tools – Comparison matrix – 13 tools including above

Program Logic Modeling

• DoView – Visual outcomes and results planning
• IdeaTree – Simultaneous Collaboration & Brainstorming Using Mind Maps
• yEd – diagram editor that can be used to  generate drawings of diagrams.  FREE

Other lists

• A software list compiled by CES UK: A review of 14 generic monitoring and evaluation systems, plus a 1 page framework for deciding on an M&E system. The 14 systems are show on the left sidebar, click on each to see a matrix of attiributes. (Found courtesy of Orla Cronin)
• Social Network Analysis software: A list

Other other

• Monitoring & Evaluation Software open group on LinkedIn (i.e. you dont need to sign in)

If you have software, or lists of software, which you would like to see added here, please use the Comment facility below