The Big Push forward: The Australian Debate (Oct 2011)

October 26, 2011 by Chris Roche.

On 19 October 2011, Oxfam Australia hosted a ‘Big Push Forward‘ event in Melbourne with the co-conveners of this initiative – Rosalind Eyben and Irene Guijt. Sixty development practitioners, including AusAid staff and academics came together to discuss whether the concerns voiced by the Big Push Forward project are relevant in Australia.

 HOW RELEVANT ARE THE ISSUES TO AUSTRALIA?

Following an introduction from Rosalind and Irene, we  had short inputs from three speakers on how these issues resonated in our part of the world.  Dennis Altman, from the Institute of Human Security, at La Trobe University suggested that the neo-liberal language which permeates Western society has been recast in the development world. into an auditing culture, focusing on evaluation, monitoring, and counting beans.  Marc Purcell the CEO of Australia’s International NGO umbrella group ACFID noted that the commitment to international aid in Australia is extremely brittle, and that the public debate about aid in Australia has led to a deep anxiety in government about how the aid programme is being perceived. But he argued that maybe it’s no bad thing for economists to look at the work of ‘pampered NGOs’. Jess Dart, the Managing Director of consulting company Clear Horizon, felt that whilst Australian NGOs do more internal evaluation than most there was a view expressed at this year’s Australasian Evaluation Conference that ‘development is the cowboy of evaluation’.  If we can’t tell the story of what we’ve done, people will ask for results. There are lots of really good methods out there and we can use these to offer solid alternatives to tell more complex stories of transformation.

OZIFYING THE THEMES
Continue reading “The Big Push forward: The Australian Debate (Oct 2011)”

MPs report on Department for International Development Financial Management

The Commons Public Accounts Committee publishes its 52nd report of Session 2010-12, on the basis of evidence from the Department for International Development (DfID).

“The Rt Hon Margaret Hodge MP, Chair of the Committee of Public Accounts, said:

“The amount DfID spends on aid will rise by 35% by 2013, but at the same time the Department has to cut its overall running costs by a third.
Achieving this level of savings at a time of rapid expansion in frontline services involve a substantial challenge if taxpayers’ money is to be properly protected and value for money secured. [emphasis added]

The Department is going to be spending more in fragile and conflict-affected countries and the danger to the taxpayer is that there could be an increase in fraud and corruption. However, the Department could not even give us information as to the expected levels of fraud and corruption and the action they were taking to mitigate it.

Unfortunately, the Department has not always kept its eye on the financial ball, and in 2010 stopped monitoring its finance plan. That must not happen again and DFID should report publicly on its financial management.

The Department’s ability to make informed spending decisions is undermined by its poor understanding of levels of fraud and corruption. Its current approach is too reactive and it needs to develop a sound framework for making sure funds are spent properly on the ground. This will be even more important as the Department channels more of its funding into fragile and conflict-affected states.

The Department’s current plan is to spend more via multilateral organizations and less through bilateral programmes. This poses a risk to value for money because the Department will have less oversight than it does over country-to-country programmes. Indeed, we are concerned that the strategy has more to do with the fact that it is easier to spend through multilaterals than go through the process of assessing value for money of bilateral programmes. [emphasis added]

To maximise the amount of aid that gets through to the frontline, DfID should have clear plans for how it is going to reduce or control running costs – particularly when channelling funding through partner and multilateral organizations with a management overhead at every stage.”[emphasis added]

Margaret Hodge was speaking as the committee published its 52nd Report of this Session which, on the basis of evidence from the Department for International Development, examined its financial management capability, its increasing focus on value for money, and the challenges it faces in managing its increasing programme budget while reducing its overall running costs.”

RD Comment: See Rick on the Road blog posting Thursday, July 24, 2008: An aid bubble? – Interpreting aid trends which raises the same issues as highlighted in bold above.

See also HoC International Development Committee, Committee Room 15 Working effectively in fragile and conflict-affected states: DRC, Rwanda and Burundi

How to monitor and evaluate anti-corruption agencies

Guidelines for agencies, donors and evaluators

By Jesper Johnsøn, Hannes Hechler, Luís De Sousa, Harald Mathisen (2011)  Bergen: Chr. Michelsen Institute (U4 Issue 2011:8) 84 p. Available as pdf

“The number of Anti-corruption agencies (ACAs) around the world has increased dramatically over the past decades. Nevertheless, the value of ACAs is increasingly being questioned by international donors and national governments. Frequently, ACAs are not considered to deliver on the high expectations bestowed upon then.

Evaluations of individual agencies were collected and analysed to assess the evidence underlying the assumptions about the effectiveness of ACAs. Surprisingly, few evaluations had actually been done, and even fewer measured the actual outcomes and impacts of the ACA. Thus, whilst opinions about ACAs are many, the actual evidence about their performance is scarce. To develop this body of evidence, ACAs need to do a better job at establishing results-based indicators for their work, showing how activities lead to impact, and collecting data.

To which extent the perceived failure of ACAs is an issue of measurement or design can therefore not be answered with any certainty. The value of ACAs can only be determined once evidence-based evaluations are conducted.

To this end, the report provides technical, methodological, and practical guidance to assist staff of ACAs in undertaking monitoring and evaluation and shows how the outcomes and impact of the work of ACAs can be evaluated in an objective, evidence-based manner.”

 

27 Oct Symposium: NGO-IDEAs – grassroots based impact monitoring

Date: 27 October 2011
Venue: Bonn, Germany

Dear all,
For the last three years, 14 German NGOs with 40 NGOs from Asia and Africa have been working to develop tools for assessing change and its causes, in short: impact monitoring that were to be specifically relevant for NGOs. Under the name of NGO-IDEAs, tools were developed through which target groups set their own goals and monitor their achievements. NGOs  and in some cases government agencies use the data generated for quantitative and qualitative assessment and reporting on outcome/impact.
Filtering according to poverty category is possible. The tools have been successfully applied in various sectors (from Saving and Credit through lactating mothers and primary schools to the inclusion of Persons with Disabilities). NGOs in East Africa, South Asia and the Philippines are now sustaining the application of the tools and spread them spontaneously to other projects and organisations. Programs have been improved through the tools and the application itself leads to more ownership and autonomy of grassroot organisations.
On Oct 27, the results of the 3-year phase will be discussed publicly. Two partners Mary Mate, Catholic Diocese of Embu; Kenya and Alma de la Paz, Kapwa Upliftment Foundation, Philippines) will present their experiences and international experts will discuss the approach. Speakers and Panel: Christiane Bögemann-Hagedorn, Head of Civil Society and Economy Department and Michaela Zintl, Head of Evaluation Department (BMZ/German Ministry for Development), Robert Chambers IDS Sussex), Cecile Kusters (Centre of Development Innovation,Wageningen), Susanne Neubert (National Opinion Research Centre at the University of Chicago, Dörte Segebart, Freie Universität, Berlin).
Some places are still available. More information and the registration form: http://www.ngo-ideas.net/news/, items of 10/10/2011 and 06/30/2011.
Publications can be downloaded from www.ngo-ideas.net/publications. The symposium focuses on the Impact Toolbox – one of three main products of NGO-IDEAs.

Check the Impact Toolbox: http://www.ngo-ideas.net/impact_toolbox/ and
the Field Experience: http://www.ngo-ideas.net/field_experiences/

NGO-IDEAs also has an advisory role to VENRO, the German development NGO’s umbrella body. VENRO recently published a position paper “Quality before Proof” that sets “empowerment” as one of four purposes of impact observation. http://www.venro.de/evaluation.html. The symposium takes place in the wider context of this paper that has generated some discussion in Germany.

Kind regards

Bernward Causemann/NGO-IDEAs

—
Bernward Causemann, www.causemann.org

Monitoring Policy Dialogue: Lessons From A Pilot Study

By Sadie Watson And Juliet Pierce. September 2008. DEPARTMENT FOR INTERNATIONAL DEVELOPMENT. Evaluation Report WP27

Executive Summary

In 2007, a tool and process was developed for improving the recording and impact of policy dialogue initiatives across DFID. It was based on an adaptation of current project cycle management (PCM) requirements for programme spending. A pilot was devised to test the proposed tool and process in terms of:

• Assessing the value in recording and monitoring policy related activities in a similar way to that of spend activities;

• Finding the most effective and useful approach in terms of process;

• Identifying succinct ways to capture intentions and to measuring performance;

• Clarifying the type and level of support and guidance required to roll the process out across DFID.

The ten participating pilot teams represented different aspects of DFID’s policy work, conducting different types of policy dialogue activities. The consultants were asked to monitor and evaluate the six month pilot. They were also asked to review approaches to managing and monitoring policy dialogue and influencing activities in other organisations. This report highlights some lessons and observations from the pilot. It outlines some emerging issues and provides some pointers for DFID to consider as it continues to develop into an organisation where policy dialogue and influencing are increasingly important aid tools.
Continue reading “Monitoring Policy Dialogue: Lessons From A Pilot Study”

On evaluation quality standards: A List

 

The beginnings of a list. Please suggest others by using the Comment facility below

Normative statements:

Standards for specific methods (and fields):

Meta-evaluations:

  • Are Sida Evaluations Good Enough?An Assessment of 34 Evaluation Reports” by Kim Forss, Evert Vedung, Stein Erik Kruse,Agnes Mwaiselage, Anna Nilsdotter, Sida Studies in Evaluation 2008:1  See especially Section 6: Conclusion, 6.1 Revisiting the Quality Questions, 6.2 Why are there Quality Problems with Evaluations?, 6.3 How can the Quality of Evaluations be Improved?, 6.4 Direction of Future Studies. RD Comment:  This study has annexes with empirical data on the quality attributes of  34 evaluation reports published in the Sida Evaluations series between 2003 and 2005. It BEGS a follow up study to see if/how these various quality ratings correlate in any way with the subsequent use of the evaluation reports. Could Sida pursuaded to do something like this?

Ethics focused

  • Australasian Evaluation Society

Journal articles

Checklists:

  • Evaluation checklists prepared by the Western Michegan University ,covering Evaluation Management, Evaluation Models, Evaluation Values and Criteria, Metaevaluation, Evaluation Capacity Building / Institutionalization, and Checklist Creation

Other lists:

Workshop: Case Studies in Development Evaluation: Validity, Generalisation and Learning

Venue: University of Copenhagen
Date: May 21-23, 2012
Invitation and Call for Papers to International Workshop
Centre for Social Science Development Research

The Evaluation Department of the Danish Ministry of Foreign Affairs and the Centre for Social Science Development Research at Institute of Food and Resource Economics at the University of Copenhagen are pleased to invite you to submit an abstract (preparatory to a full paper) to an International Workshop, which will focus on methodological and practical considerations when using case studies in evaluations of development. The workshop will be organised in collaboration with the journal Evaluation:the international journal of theory, research and practice.

A large number of development evaluations are broader, learning-oriented evaluations based on cases at country, sector or project level. A key challenge in these evaluations is how to deal with the question of external validity. Within the field of development evaluation the methodological and practical debate on how to address this issue has been relatively limited.

The Organising Committee of the International Workshop is seeking abstracts that address theoretical/methodological challenges as well as more practical experiences when using case studies in learning-oriented development evaluations. Abstracts could e.g. focus on: Continue reading “Workshop: Case Studies in Development Evaluation: Validity, Generalisation and Learning”

Asian Development Bank: 2011 Annual Evaluation Review

Available at ADB website

Background

This report summarizes the key findings and lessons of evaluation studies carried out in 2010, and provides trends in the success rates of ADB operations. It also reviews the recommendations from evaluation reports and the status of actions taken by ADB Management in response to these recommendations. The report also reviews the work program accomplishments of IED in 2010.

Key Findings and Issues

Declining performance in terms of success rates. According to the past data, success rates have not reached 80%, although this is ADB’s corporate target for 2012. Performance began to decline in approval year 2000 after peaking at over 70%. Although the declining trend is supported by a limited sample size, the project and program performance report (PPR) system indicates that this trend will continue unless significant corrective measures are taken. Based on the new PPR system, about 25% of ongoing projects are facing implementation challenges and are at risk of not meeting their objectives (which confirms IED’s previous findings that portfolio performance ratings were overrated in PPRs)-an important issue that needs to be addressed.
Continue reading “Asian Development Bank: 2011 Annual Evaluation Review”

Essentials of Utilization-Focused Evaluation

Michael Quinn Patton, August 2011. Sage publications

Publisher’s description:

“Based on Michael Quinn Patton’s best-selling Utilization-Focused Evaluation, this briefer book provides an overall framework and essential checklist steps for designing and conducting evaluations that actually get used. The new material and innovative graphics present the utilization-focused evaluation process as a complex adaptive system, incorporating current understandings about systems thinking and complexity concepts. The book integrates theory and practice, is based on both research and professional experience, and offers new case examples and cartoons with Patton’s signature humor. ”
Continue reading “Essentials of Utilization-Focused Evaluation”

Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,

Date: 17-18 October 2011
Venue: ANU, Canberra, Australia

Concept Note, Chris Roche & Linda Kelly, 4 August 2011

The Developmental Leadership Program (DLP)[1] addresses an important gap in international thinking and policy about the critical role played by leaders, elites and coalitions in the politics of development. At the core of DLP thinking is the proposition that political processes shape developmental outcomes at all levels and in all aspects of society: at national and sub-national levels and in all sectors and issue areas.

Initial findings of the DLP research program confirm that development is a political process and that leadership and agency matter. This is of course not new, but the DLP research provides important insights into how, in particular, leadership, elites and formal and informal coalitions can play a particularly important and under-recognized role in institutional formation (or establishing the ‘rules of the game’), policy reform and development processes[2].

International aid therefore needs to engage effectively with political processes. It needs to be flexible and be able to respond when opportunities open up. It needs to avoid the danger of bolstering harmful political settlements.

Furthermore Monitoring & Evaluation (M&E) mechanisms need to be improved and made compatible with flexible programming and recognize the importance of ‘process’ as well as outcomes. Donors should invest in a range of monitoring and evaluation techniques and approaches which are more aligned with the kinds of non-linear and unpredictable processes which characterise the kinds of political processes which drive positive developmental outcomes. This is important because it can be argued that, at best, current approaches are simply not appropriate to monitor the kinds of processes DLP research indicates are important; or, at worst, they offer few incentives to international assistance agencies to support the processes that actually lead to developmental outcomes Continue reading “Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,”

%d bloggers like this: