Monitoring and Evaluation NEWS » Uncategorized

Randomised controlled trial testing the effects of transparency on health care in Uganda

rick davies — Wed, 18 Jan 2012 18:11:11 +0000

“At aidinfo we conduct research and liaise with aid donors and recipients to build up a case for aid transparency. We want to show that improving and increasing the amount that donors report on their aid contributions can help communities to track aid spending. In turn, donors and governments will be more accountable for their aid spending. It is expected that in this way aid will reach more people on the ground, helping to contribute more in the fight against poverty.

This is all well and good, but it is difficult to prove. Svensson’s work, then, is of great importance to us here.

This Study by Reinikka and Svensson (2005) found that in 1995 only 20 percent of a primary education grant program to rural Uganda actually reached its intended target. This figure rose by a striking 60 percent in 2001 when information was published detailing where this money was going; a full 80 percent of funds reached their intended destination, greatly improving education services in the area.

Björkman and Svensson (2009) followed up on this study with a compelling randomised controlled trial testing the effects of transparency on health care in Uganda. The experiment randomly assigned community health clinics to receive published ‘report cards’ and NGO-organised public meetings on the quality of the clinics’ health care.

The results of this transparency ‘treatment’ rivalled the effects of the best health interventions involving expensive new medicines, equipment, and procedures. Waiting time for care decreased, absenteeism among doctors and nurses plummeted, clinics got cleaner, fewer drugs were stolen, 40-50 percent more children received dietary supplements and vaccines, health services got used more, and, powerfully, 33 percent fewer children died under the age of five. This amounted to 550 saved lives in a small area of Uganda encompassing merely 55,000 households.

This is strong evidence that access to information about services empowers citizens to get better services and saves lives.”

3ie’s proposes a Commitment to Evaluation Indicator (c2e)

rick davies — Tue, 10 Jan 2012 23:27:51 +0000

International Initiative for Impact Evaluation (3ie) -Terms of Reference for a Research Consultancy– White paper for the Commitment to Evaluation Indicator

“Background: Experience to date shows that the use of evidence by donors and governments when designing and adopting development programmes remains sporadic. There are many examples where a programme was shown to have no impact but was expanded, as well as examples of programmes with positive impact being terminated. To promote better use of evaluation evidence in policy making and programme design, 3ie is launching a Commitment to Evaluation (c2e) indicator. The indicator will provide a measurement of government and donor agency use of evaluation evidence allowing for recognition and reward for progress and good practice. The indicator will be developed and piloted in 2012 for donor agencies with the intent to recognize donors that make systematic use of evidence and thus motivate others to do the same.

3ie’s initiative follows the example of other successful efforts to use awards or indexes to focus the attention of policymakers. Indexes such as the UN Development Programme’s Human Development index, Transparency International’s Corruption Perception index, and the Centre for Global Development’s Quality of ODA (QuODA) index have raised awareness on key issues and influenced practice of governments and development agencies. The Mexican National Council for the Evaluation of Social Development Policy (CONEVAL) annual award for good practices in social evaluation has strengthened political buy-in and commitment to evaluation in Mexico. In developing this c2e indicator, 3ie will draw from the lessons learned by similar initiatives on how best to motivate and award evaluation practices and build and run an effective cross-agency and cross-country indicator. More detailed background information on the rationale and theory of change behind the project is available in the discussion note in the annex.” See ToRs for rest of the text including annex.

3ie and the Funding of Impact Evaluations

rick davies — Sun, 08 Jan 2012 04:56:39 +0000

A DISCUSSION PAPER FOR 3IE’S MEMBERS. by Rick Davies, July 2011. Commissioned by the Office of Development Effectiveness, AusAI. Available as pdf.

The purpose of this discussion paper is to inform AusAID’s and other 3ie members’ engagement with 3ie (the International Initiative for Impact Evaluation). It precedes the forthcoming evaluation of 3ie, and is more limited in scope. It is expected to be complementary and useful to the larger Department for International Development (DFID) study now underway, Developing a broader range of rigorous designs and methods for impact evaluations, as the author of this report is also a member of that study team.

AusAID is a member of 3ie and provides core funding to 3ie to contribute to the global public good of policy-relevant evidence on what works in development. Direct benefit to AusAID is not the purpose of the membership. However, it is important to AusAID that 3ie’s work is relevant to AusAID’s partners, particularly partners with low income and/or in fragile countries. AusAID’s Office of Development Effectiveness (ODE) manages AusAID’s membership of 3ie and has commissioned this discussion paper.

The focus of this discussion paper is on 3ie methodological approach, used in both the funded impact evaluations and systematic reviews, and how this has changed over time.

Contents
1. Introduction
1.1 Purpose and approach
1.2 A summary of 3ie’s stated methodological approach
1.3 Debates on impact evaluation
1.4 What we don’t know
2. The Issues
2.1 Are RCTs being promoted to the exclusion of other approaches?
2.2 Is attention being given to how it works, as well as what works?
2.3 Are RCTs focusing on the simple questions and ignoring the more difficult?
2.4 Do the specific requirements of RCTs limit their range of application?
2.5 Who can afford RCTs?
2.6 How valid are the results?
2.7 How generalisable are the results?
2.8 Risks of unintended effects
2.9 Ethical issues
2.10 A reminder of the value of experimental approaches
3. A Wider View: Methodology in the Context of Strategy
4. Future Directions
4.1 Obtaining more value from existing investments
4.2 Development of new methodologies
4.3 Focusing on the strategic objectives

RealWorld Evaluation Working Under Budget, Time, Data, and Political Constraints

rick davies — Mon, 26 Dec 2011 21:41:31 +0000

Second Edition, by Michael Bamberger, Jim Rugh, Linda Mabry. Sage Publications, Nov 2011,

“This book addresses the challenges of conducting program evaluations in real-world contexts where evaluators and their clients face budget and time constraints and where critical data may be missing. The book is organized around a seven-step model developed by the authors, which has been tested and refined in workshops and in practice. Vignettes and case studies—representing evaluations from a variety of geographic regions and sectors—demonstrate adaptive possibilities for small projects with budgets of a few thousand dollars to large-scale, long-term evaluations of complex programs. The text incorporates quantitative, qualitative, and mixed-method designs, and this Second Edition reflects important developments in the field since the publication of the First Edition. ”

See also the associated website: http://www.realworldevaluation.org/ Bamberger and Rugh have presented many workshops on RealWorld Evaluation in many countries. A copy of various versions and translations of the PowerPoint presentations and other materials are accessible on the next pages of this website.

What’s New in the Second Edition of Real World Evaluation?

A greater focus on responsible professional practice, codes of conduct, and the importance of ethical standards for all evaluations.
Some new perspectives on the debate over the “best” evaluation designs. While experimental designs can address the important issues of selection bias, such statistical designs are potentially vulnerable to a number of important threats to validity. These include process and contextual analysis, collecting information on sensitive topics and from difficult-to-reach groups, difficulties in adapting to changes in the evaluation design, and implementation strategies. Experience also suggests that strong statistical designs can be applied only in a very small proportion of evaluations.
There are many instances in which well-designed nonexperimental designs will be the best option for assessing outcomes of many programs, particularly for evaluating complex programs and even “simple” programs that involve complex processes of behavioral change.
The importance of understanding the setting within which the evaluation is designed, implemented, and used.
Program theory as a central building block of most evaluation designs. The expanded discussion incorporates theory of change, contextual and process analysis, multilevel logic models, using competing theories, and trajectory analysis.
The range of evaluation design options has been considerably expanded, and case studies are included to illustrate how each of the 19 designs has been applied in the field.
Greater emphasis is given to the benefits of mixed-method evaluation designs.
A new chapter has been added on the evaluation of complicated and complex development interventions. Conventional pretest-posttest comparison group designs can rarely be applied to the increasing proportion of development assistance channeled through complex interventions, and a range of promising new approaches—still very much “work in progress”—is presented.
Two new chapters on organizing and managing evaluations and strengthening evaluation capacity. This includes a discussion of strategies for promoting the institutionalization of evaluation systems at the sector and national levels.
The discussion of quality assurance and threats to validity has been expanded, and checklists and worksheets are included on how to assess the validity of quantitative, qualitative, and mixed-method designs.

First reports published by UK Independent Commsssion for Aid Impact

rick davies — Wed, 21 Dec 2011 01:53:22 +0000

…on 22nd November, 2011.

Two cover general areas of the programme:

• ICAI’s Approach to Effectiveness and Value for Money; and

• The Department for International Development’s (DFID) Approach to Anti-Corruption;

Two cover specific programmes in DFID’s country offices:

• DFID’s Climate Change Programme in Bangladesh; and

• DFID’s Support to the Health Sector in Zimbabwe.

See the ICAI website for further details

RD Comment: re ICAI’s Approach to Effectiveness and Value for Money paper, see my Comments here. In summary:

• This paper is confusingly titled. It is really about the ICAIs approach to evaluation, and covers more than “value for money and effectiveness”

• The 4e’s analysis of the concepts of “value for money and effectiveness” has potential, but seems to be taken nowhere thereafter.

• The proposed workings of the “traffic lights” system are opaque, it is not clear how these judgements will be built up from subsidiary judgements.

• The worst possible performance judgement (i.e. the red light) is that “The programme meets few of the criteria for effectiveness and value for money. It is performing poorly. Immediate and major changes need to be made” Is that all? Don’t some programmes deserve to be called failures? Don’t some programmes need to be closed down rather than “fixed? If these possibilities are meant to be considered then they need to be clearly stated. For more on the importance of recognising failure, see “Do we need a Minimum Level of Failure (MLF)?“

The Impact of Economics Blogs

rick davies — Mon, 31 Oct 2011 08:17:22 +0000

David McKenzie (World Bank, BREAD, CEPR and IZA) and Berk Özler (World Bank). Policy Research Working Paper 5783. August 2011. Available as pdf. See also the authors’ blog about this paper.

Introduction: Practically nonexistent a decade ago, blogs by economic scholars have become commonplace. Economics blogs, such as Freakonomics, Marginal Revolution, Paul Krugman and Greg Mankiw, have built large followings – whether measured by subscriptions in Google Reader or by average daily page views (1). Cowen (2008) argues that blogs are the main way that the general public consumes economics in a given day and guesstimates that “…about 400,000 people are reading economics blogs and digesting them” on a daily basis.

These blogs not only give their creators an outlet to disseminate their ideas and work immediately in a format that is more accessible, but also enable instant feedback, are easy to share on the open web, and allow the bloggers a personal style rather than the inaccessible format of academic journals (Glenn, 2003; Dunleavy and Gilson 2011).

Our motivation in examining the impact of economics blogs stems from two observations about blogs and questions that arise from these. First, it seems fair to state that “…informing is the core business of blogging.” (McKenna and Pole 2008, p. 102) This leads to the question of whether blogs improve the dissemination of research findings and whether their readers are indeed more informed (2). On the one hand, coupling the large readership of blogs with the argument of Cowen (2008) that the best ones are written at a level far higher than that of any major newspapers offers the promise that economics blogs may have sizeable effects on the dissemination of economic research and on the knowledge and attitudes of their readers.

On the other hand, Sunstein (2008) argues that the blogosphere might be causing “group polarization” and creating “information cocoons” – making it unlikely that blogs would cause a significant change in the knowledge and attitudes of their readers (3).

Bell (2006, p.75) summarizes another common perception of blogs, as “…a largely harmless outlet for extroverted cranks and cheap entertainment for procrastinating office workers.” 4 Combined with the possibility that blogging gives scholars the freedom to write about topics outside their area of expertise (what Jacob T. Levy called “public-intellectualitis? in his blog) this would suggest that impacts of blogs are likely to be negligible.

Second, it also seems plausible that blogging might enhance the reputation of the blogger. Kahn (2011) states: “The shrewd academic uses his blog to market his ideas and to “amplify? his new academic results. This is a type of branding.”

Reading bloggers talk about the reasons they blog produces a list of benefits to themselves: blogging is an outlet for ideas and observations not suitable for other media; blogging takes the blogger, at least the successful ones, from relative obscurity to being well-known; instant feedback allows the scholar blogger to sharpen her arguments by having to defend them publicly; it’s fun to interact with other economists; blogging may lead to regular writing gigs or other professional opportunities, etc. Furthermore, blogging by individual scholars may have positive spillover benefits for their institutions. Conversely, almost every scholar blogger seems to have questioned at some point on his blog about why he is spending so much time blogging instead of revising that manuscript.

While revealed preference suggests there is value in blog posts to both the scholars who maintain them and to the large number of individuals who read them, there is, to date, no quantitative evidence of their impacts (5).

This paper aims to answer three questions regarding the impacts of economics blogs.

First, do blogs improve dissemination of working papers or journal articles?

Second, do they raise the profile of their creators?

Third, do they cause changes in attitudes among their readers or lead to increased knowledge?

We conduct event study analysis using download data from RePEc to answer the first question, and combine RePEc data with recent survey data on most admired economists to answer the second. To answer the third set of questions, we use evidence from a recent survey we conducted on the role of blogs in economics, and take advantage of a randomized experiment in which a random sample of the respondents of this survey were encouraged to read a new economics blog. The novel survey and the experiment are also used to shed light on the question of whether blogs improve the public awareness and the reputation of their creators, and of their affiliated institutions.

We find that links to scholarly articles (either publications or working papers) in blogs lead to substantial jumps in their likelihood of being downloaded, with the impact increasing with the popularity of the blog providing the link. We also find evidence that blogs enhance the recognition of their creators, so that bloggers are more likely to be admired or respected among their peers than other scholars of similar publication records. Using the experiment that provided random encouragement to read a new blog produced by four researchers from the World Bank, we find an improvement in the perceived quality of research produced at the World Bank and an increased interest by the survey respondents in working there as a researcher (6).

However, the same experiment shows that the short-run impacts on attitudes and knowledge are relatively weak. While we detect some changes in attitude among predetermined sub-groups of interest, we find no evidence of impact on the full sample. On knowledge, reading the new blog seems to improve awareness of findings from recent studies for the average reader, but not for the marginal reader who only reads because of encouragement. To our knowledge, this paper provides the first quantitative evidence on the impacts of economics blogs and covers a number of the more important likely channels of influence.

The rest of the paper is as follows. Section 2 presents the impacts of blogs on dissemination of economics research using event study analysis, and Section 3 provides evidence on the effect of blogs on the reputation of their creators. Section 4 uses our new survey data to describe how blogs are used by graduate students in economics, junior faculty, and field workers in development. Section 5 presents impacts of reading a new blog, exploiting the randomized encouragement design. Section 6 provides a discussion of whether economics blogs influence policy, while Section 7 concludes.

THE EVALUATION OF ORGANIZATION PERFORMANCE: NORMATIVE PRESCRIPTIONS VS. EMPIRICAL RESULTS

rick davies — Sun, 30 Oct 2011 14:46:45 +0000

Vic Murray, University of Victoria, 2004. Available as pdf

Abstract: This paper reviews the underlying theoretical bases for the evaluation of organizational performance. It then examines representative samples of empirical research into actual evaluation practices in a variety of nonprofits in Canada, the U.S. and Britain. Some of the most popular tools and systems for evaluation currently recommended by consultants and others are then reviewed. Looking at this prescriptive literature, it is shown that, by and large, it takes little account of the findings of empirical research and, as a result, its approaches may often prove ineffective. An alternative that attempts to integrate the research findings with practical tools that has value for practitioners is then be suggested.

Introduction

It is a perplexing, but not uncommon, phenomenon in the world of nonprofit organization studies how little connection there is between the work of those who offer advice on how organizations in this sector might become more effective and that of those who carry out formally designed empirical research into how these organizations actually behave. Nowhere is this gap between “how to” and “what is” more apparent than in the field of performance assessment and evaluation.

Commons Select Committee to Scrutinise the DFID’s Annual Report & Resource Accounts

rick davies — Wed, 14 Sep 2011 17:59:43 +0000

13 September 2011

“The International Development Committee is to conduct an inquiry into the Department for International Development’s Annual Report and Accounts 2010-11 and the Department’s Business Plan 2011-15.

Invitation to submit Written Evidence

The Committee will be considering

Changes since the election to DFID’s, role, policies, priorities and procedures;
The implications of changes for management styles, structures, staffing competences and capacity to deliver; and
The overall impact on the efficiency, effectiveness and cost-effectiveness of DFID’s activities.

The Committee invites short written submissions from interested organisations and individuals, especially on the following areas: the implementation of the structural reform plan: the bilateral, multilateral and humanitarian reviews; DFID administration costs; expenditure on, and dissemination of research; and the use of technical assistance and consultants.

The deadline for submitting written evidence is Monday 10 October 2011. A guide for written submissions to Select Committees may be found on the parliamentary website at: http://www.parliament.uk/commons/selcom/witguide.htm

FURTHER INFORMATION:
Committee Membership is as follows: Malcolm Bruce MP, Chair (Lib Dem, Gordon), Hugh Bayley MP (Lab, City of York), Richard Burden MP (Lab, Birmingham, Northfield), Sam Gyimah MP (Con, East Surrey), Richard Harrington MP (Con, Watford), Pauline Latham MP (Con, Mid Derbyshire), Jeremy Lefroy MP (Con, Stafford), Michael McCann MP (Lab, East Kilbride, Strathaven and Lesmahagow), Alison McGovern MP (Lab, Wirral South), Anas Sarwar MP (Lab, Glasgow Central), Chris White MP (Con, Warwick and Leamington).
Specific Committee Information: indcom@parliament.uk / 020 7219 1223/ 020 7219 1221
Media Information: daviesnick@parliament.uk / 020 7219 3297 Committee Website: www.parliament.uk/indcom

Measuring Impact on the Immeasurable? Methodological Challenges in Evaluating Democracy and Governance Aid

rick davies — Wed, 07 Sep 2011 21:44:34 +0000

by Jennifer Gauck, University of Kent, Canterbury – Department of Politics, 2011. APSA 2011 Annual Meeting Paper. Available as pdf

Abstract:

“Recent debates over the quality, quantity and purpose of development aid has led to a renewed emphasis on whether, and in what circumstances, aid is effective in achieving development outcomes. A central component of determining aid effectiveness is the conduct of impact evaluations, which assess the changes that can be attributed to a particular project or program. While many impact evaluations use a mixed-methods design, there is a perception that randomized control trials (RCTs) are promoted as the “gold standard” in impact evaluation. This is because the randomization process minimizes selection bias, allowing for the key causal variables leading to the outcome to be more clearly identified. However, many development interventions cannot be evaluated via RCTs because the nature of the intervention does not allow for randomization with a control group or groups.”

“This paper will analyze the methodological challenges posed by aid projects whose impacts cannot be evaluated using randomized control trials, such as certain democracy and governance (D&G) interventions. It will begin with a discussion of the merits and drawbacks of cross-sectoral methods and techniques commonly used to assess impact across a variety of aid interventions, including RCTs, and how these methods typically combine in an evaluation to tell a persuasive causal story. This paper will then survey the methods different aid donors are using to evaluate the impact of projects that cannot be randomized, such as governance-strengthening programs aimed at a centralized public-sector institution. Case studies will be drawn from examples in Peru and Indonesia, among others. This paper will conclude by analyzing how current methodological emphases in political science can be applied to impact evaluation processes generally, and to D&G evaluations specifically.”

RD Comment: See also the 3ie webpage on Useful resources for impact evaluations in governance which includes a list of relevant books, reports, papers, impact evaluations, systematic reviews, survey modules/tools and website

Developing a Monitoring and Evaluation Framework: A list

rick davies — Wed, 31 Aug 2011 21:14:56 +0000

[Aplogies: This page is still at the draft stage, there are some formating and other problems]

A suggested definition of a M&E Framework:

A document that tells you who is expected to know what, as well as when and how they are expected to know.

The list (under development, suggestions welcomed):

Clear Horizon: Developing Frameworks Overview (Undated!) Includes seven steps:

CES UK (Undated!) A framework for monitoring and evaluation

What should be found within an M&E framework / plan? Rick Davies, 2008, one pager note on suggested contents. Soon to be revised