Tag: Theory of Change

Where there is no single Theory of Change: The uses of Decision Tree models

Eliciting tacit and multiple Theories of Change

Rick Davies, November 2012. Unpublished paper. Available as pdf version available hereand a 4 page summary version

This paper begins by identifying situations where a theory-of-change led approach to evaluation can be difficult, if not impossible. It then introduces the idea of systematic rather than ad hoc data mining and the types of data mining approaches that exist. The rest of the paper then focuses on one data mining method known as Decision Trees, also known as Classification Trees.  The merits of Decision Tree models are spelled out and then the processes of constructing Decision Trees are explained. These include the use of computerised algorithms and ethnographic methods, using expert inquiry and more participatory processes. The relationships of Decision Tree analyses to related methods are then explored, specifically Qualitative Comparative Analysis (QCA) and Network Analysis. The final section of the paper identifies potential applications of Decision Tree analyses, covering the elicitation of tacit and multiple Theories of Change, the analysis of project generated data and the meta-analysis of data from multiple evaluations. Readers are encouraged to explore these usages.

Included in the list of merits of Decision Tree models is the possibility of differentiating what are necessary and/or sufficient causal conditions and the extent to which a cause is a contributory cause (a la Mayne)

Comments on this paper are being sought. Please post them below or email Rick Davies at rick@mande.co.uk

Separate but related:

See also: An example application of Decision Tree (predictive) models (10th April 2013)

Postscript 2013 03 20: Probably the best book on Decision Tree algorithms is:

Rokach, Lior, and Oded Z. Maimon. Data Mining with Decision Trees: Theory and Applications. World Scientific, 2008. A pdf copy is available

Guidance for designing, monitoring and evaluating peacebuilding projects: using theories of change

CARE, June 2012. Available as pdf

“To advance the use of theory-based inquiry within the field of peacebuilding, CARE International and International Alert undertook a two and a half year research project to develop light touch methods to monitor and evaluate peacebuilding projects, and pilot these in Democratic Republic of Congo (DRC), Nepal and Uganda. This document, Guidance for designing, monitoring and evaluating peacebuilding project: using theories of change emerges from the efforts of peacebuilders who field tested the processes to define and assess the changes to which they hoped to contribute.

The main audiences for this guide are conflict transformation and peacebuilding practitioners, non-governmental organisations (NGOs) and donor agencies. Other actors in the conflict transformation and peacebuilding field may also find it useful.”

Contents page

Acknowledgements
1. Overview
1.1 The problem we seek to address
1.2 The research that developed the guidance
1.3 Definitions
2. Theories of change
2.1 What is a theory of change?
2.2 Why is it important to explicitly state theories of change?
3. Using theories of change for project or programme design
3.1 Carry out a conflict analysis
3.2 Design an intervention
3.3 Develop a results hierarchy
3.4 Articulate the theories of change
4.  Monitoring and evaluating of a project or programme based on  its theories of change
4.1 Identify / refine the theories of change
4.2 Assess a project or programme’s relevance
4.3 Decide what you want to learn: choose which theory of change
4.4 Undertake outcome evaluation
4.5  Design a research plan using the monitoring and evaluation grid to assess  whether the theory of change is functioning as expected, and collect data according to the plan
4.6 Data collection methods
4.7 Helpful hints to manage data collection and analysis
4.8 Analysis of data
5.  Present your findings and ensure their use
Annex 1: Questions to ask to review a conflict analysis
Annex 2: A selection of conflict analysis tools and frameworks
Annex 3: Additional resources
Notes

Review of the use of ‘Theory of Change’ in International Development

By Isabel Vogel. Funded by DFID, 2012

Review of the use of ‘Theory of Change’ in international development (full report)
Review of the use of ‘Theory of Change’ in international development (summary)
Appendix 3: Examples of Theories of Change

1. Executive Summary
‘Theory of change’ is an outcomes-based approach which applies critical thinking to the design, implementation and evaluation of initiatives and programmes intended to support change in their contexts. It is being increasingly used in international development by a wide range of governmental, bilateral and multi-lateral development agencies, civil society organisations, international non-governmental organisations and research programmes intended to support development outcomes. The UK’s Department for International Development (DFID) commissioned this review of how theory of change is being used in order to learn from this growing area of practice. DFID has been working formally with theory of change in in its programming since 2010. The purpose was to identify areas of consensus, debate and innovation in order to inform a more consistent approach within DFID.
Continue reading “Review of the use of ‘Theory of Change’ in International Development”

UNDERSTANDING ‘THEORY OF CHANGE IN INTERNATIONAL DEVELOPMENT: A REVIEW OF EXISTING KNOWLEDGE

DANIELLE STEIN AND CRAIG VALTERS, JULY 2012. Available as pdf.

THIS PUBLICATION IS AN OUTPUT FROM A COLLABORATION BETWEEN THE ASIA FOUNDATION AND THE [LSE] JUSTICE AND SECURITY RESEARCH PROGRAMME

Summary

This is a review of the concepts and common debates within ‘Theory of Change’ (ToC) material, resulting from a search and detailed analysis of available donor, agency and expert guidance documents. The review was undertaken as part of a Justice and Security Research Program

(TAF) collaborative project, and focuses on the field of international development. The project will explore the use of Theories of Change (ToCs) in international development programming, with field research commencing in August 2012. While this document will specifically underpin the research of this collaboration, we also hope it will be of interest to a wider audience of those attempting to come to grips with ToC and its associated literature.

From the literature, we find that there is no consensus on how to define ToC, although it is commonly understood as an articulation of how and why a given intervention will lead to specific change. We identify four main purposes of ToC – strategic planning, description, monitoring and evaluation and learning – although these inevitably overlap. For this reason, we have adopted the term ‘ToC approaches’ to identify the range of applications associated with this term. Additionally, we identify some confusion in the terminology associated with ToC. Of particular note is the lack of clarity surrounding the use of the terms ‘assumption’ and ‘evidence’. Finally, we have also drawn out information on what authors feel makes for ToC ‘best practice’ in terms of both content and process, alongside an exploration of the remaining gaps where more clarity is needed.

A number of ‘key issues’ are highlighted throughout this review. These points are an attempt to frame the literature reviewed analytically, as informed by the specific focus of the JSRP-TAF collaboration. These issues are varied and include the confusion surrounding ToC definitions and use, the need to ‘sell’ a ToC to a funder, how one can know which ‘level’ a ToC should operate on, the relationship between ToC and evidence-based policy, and the potential for accuracy, honesty and transparency in the use of ToC approaches.

This paper does not aim to give definitive answers on ToC; indeed there are many remaining important issues that lie beyond the scope of this review. However, in highlighting a number of key issues surrounding current understandings of ToC approaches, this review hopes to pave the way for more constructive and critical discussion of both the concept and practical application of ToCs.

THEORY OF CHANGE REVIEW – A report commissioned by Comic Relief

Cathy James, September 2011. 33 pages. Available as pdf.

The review approach Comic Relief’s international grants team commissioned this review to capture staff and partners’ experiences in using theory of change; to identify others in development that are using theory of change and analyse their different approaches; and to draw together learning from everyone to inform what Comic Relief does next.

The review combined analysis of literature with 32 short interviews of people with experience and knowledge of theory of change. The literature included reports, guidelines, study notes, theory of change examples and other relevant documents. The review included interviews with members of Comic Relief’s international grants team; Comic Relief grant partners (both UK and southern organisations); freelance consultants; UK organisation development consultants and researchers; North American research organisations, consultancy groups and foundations; International Nongovernmental organisations (INGOs); and academics.
This report was commissioned by Comic Relief and written by Cathy James, an independent consultant. The views expressed in this report are those of the author and do not necessarily represent the views of Comic Relief

Contents

A. INTRODUCTION

A1. Why do this review?

A2. How was the review approached?

A3. What does the review cover?

B. WHAT IS THEORY OF CHANGE?

B1. What are the origins of theory of change?

B2. Who is interested in theory of change?

B3. What do people mean by theory of change?

B4. What approaches are people taking to theory of change?

B5. How is theory of change different and how does it fit with other processes?

C. HOW IS COMIC RELIEF USING THEORY OF CHANGE?

C1. How has Comic Relief’s international grants team used theory of change?

C2. How have Comic Relief partners used theory of change?

D. WHAT DIFFERENCE HAS THEORY OF CHANGE MADE?

DI. What difference has theory of change made to Comic Relief partners?

D2. What do others say about the benefits of using theory of change?

E. WHAT HAS BEEN LEARNED ABOUT USING THEORY OF CHANGE?

E1. Who is theory of change most useful for?

E2. What kind of approach has been most helpful?

E3. What have been the main challenges?

F. CONCLUSIONS AND RECOMMENDATIONS

F1. Hot topics

F2. Conclusions

F3. Some suggestions for those using or advocating theory of change to think about

 

 

Peacebuilding with impact: Defining Theories of Change

Care International UK, January 2012. 12 pages. Available as pdf

Executive Summary: “Focusing on theories of change can improve the effectiveness of peacebuilding interventions. A review of 19 peacebuilding projects in three confict-affected countries found that the process of articulating and reviewing theories of change adds rigour and transparency, clarifes project logic, highlights assumptions that need to be tested, and helps identify appropriate participants and partners. However, the approach has limitations, including the diffculty of gathering theory validating evidence.

While they are not a panacea, devoting greater attention to theories of change is a simple and relatively inexpensive means of increasing the quality of peacebuilding interventions. Donors and peacebuilding agencies should review their procedures to encourage and accommodate more widespread focus on theories of change, and ensure adequate resources are set aside to allow appropriate monitoring of these theories throughout the life of an intervention.

A focus on theories of change led to the following key fndings:
• Clarifying project logic helps highlight tenuous assumptions;
• Clearly identifying the aims of activities and measures of success strengthens project design;
• Determining the appropriate actors to work with, and not just the easy-to-reach, enables better programme focus;
• More explicit links need to be made between local level activities and national peace processes for desired changes to occur;
• Confict analysis is critical for determining the relevance of activities but is rarely done;
• Staff often require support in ensuring their theories of change are suffciently explicit;
• Current project planning tools do not help practitioners articulate their theories of change;
• Gathering evidence to validate a theory of change is challenging, particularly in conditions of conflict and fragility;
• Critical review of theories of change needs to be undertaken in conjunction with other forms of evaluation to have maximum value;
• Theories of change can encourage an overly linear approach, when change in con?ict contexts can be more organic or systemic.

Recommendations:
1 Donors should revise their logical frameworks guidance to encourage the use of theories of change, notably to include them within the ‘assumptions and risks’ column of existing logical frameworks or by adding an additional column.
2 Theories of change need to be as precise, nuanced and contextually specific as possible and be based on broad conflict analysis.
3 Practitioners need to articulate theories of change within a hierarchy of results and to review these periodically throughout the implementation of a project, particularly if conflict dynamics change.
4 Donors should encourage funded agencies to review their theories of change throughout the project cycle and make resources available for this.”

Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,

Date: 17-18 October 2011
Venue: ANU, Canberra, Australia

Concept Note, Chris Roche & Linda Kelly, 4 August 2011

The Developmental Leadership Program (DLP)[1] addresses an important gap in international thinking and policy about the critical role played by leaders, elites and coalitions in the politics of development. At the core of DLP thinking is the proposition that political processes shape developmental outcomes at all levels and in all aspects of society: at national and sub-national levels and in all sectors and issue areas.

Initial findings of the DLP research program confirm that development is a political process and that leadership and agency matter. This is of course not new, but the DLP research provides important insights into how, in particular, leadership, elites and formal and informal coalitions can play a particularly important and under-recognized role in institutional formation (or establishing the ‘rules of the game’), policy reform and development processes[2].

International aid therefore needs to engage effectively with political processes. It needs to be flexible and be able to respond when opportunities open up. It needs to avoid the danger of bolstering harmful political settlements.

Furthermore Monitoring & Evaluation (M&E) mechanisms need to be improved and made compatible with flexible programming and recognize the importance of ‘process’ as well as outcomes. Donors should invest in a range of monitoring and evaluation techniques and approaches which are more aligned with the kinds of non-linear and unpredictable processes which characterise the kinds of political processes which drive positive developmental outcomes. This is important because it can be argued that, at best, current approaches are simply not appropriate to monitor the kinds of processes DLP research indicates are important; or, at worst, they offer few incentives to international assistance agencies to support the processes that actually lead to developmental outcomes Continue reading “Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,”

Evaluation of Governance – A Study of the Government of India’s Outcome Budget

by Anand P. Gupta,  Economic Management Institute, New Delhi, India
in Journal of Development Effectiveness, 2:4, 566-573, December 2010.

[Found courtesy of Public Financial Management Blog]

“In 2005, the Government of India launched an apparently excellent initiative – the Outcome Budget – with the objective of changing the culture of measuring performance in terms of the amount of money spent against the budgeted allocations, to one of measuring performance in terms of the delivery of the outcomes that people are concerned with. This paper describes how the Outcome Budget was launched, articulates the theory of change underlying the Outcome Budget, presents a case study of the Outcome Budget of the Government of India’s Accelerated Power Development and Reforms Programme, and discusses the lessons that the Government of India may learn from its experience with the Outcome Budget.

The paper argues that the Outcome Budget has failed. This has happened because the assumptions of the theory of change underlying the Outcome Budget have not been satisfied. The failure of the Outcome Budget has extremely important lessons for the Independent Evaluation Office, which the Government of India has decided to set up. The paper articulates the theory of change underlying the Independent Evaluation Office. This theory assumes that policymakers in India currently demand rigorous impact evaluations of public interventions and will continue to demand such evaluations in future, not because they have to comply with any requirement but because they really want to know the answers to the impact evaluation questions of ‘what works, under what conditions does it work, for whom, what part of a given intervention works, and for how much?’, so that they may draw appropriate lessons from these answers and use these lessons while designing and implementing public interventions in future.  However, given Indian public officials’ current culture, the Independent Evaluation Office may not make any visible difference in development effectiveness in India.

The paper, published in Journal of Development Effectiveness, Volume 2, Number 4 (December 2010), is amongst the Journal’s “most read” (downloaded) papers, and is currently on the free download list of most read papers.”

DPC Policy Discussion Paper: Evaluating Influencing Strategies and Interventions

A paper to the DFID Development Policy Committee. Available as pdf  June 2011

Introduction
“1 The Strategy Unit brief of April 2008 envisaged that DFID should become more systematic in planning and implementing influencing efforts. Since then, procedures and guidance have been developed and there is an increasingly explicit use of influencing objectives in project log frames and more projectisation of influencing efforts. Evaluation studies and reports have illustrated the wide variety of DFID influencing efforts and the range of ambition and resources involved in trying to generate positive changes in the aid system or in partner countries. These suggest that being clear and realistic about DFID’s influencing objectives, the stakeholders involved and the specific changes being sought, is the fundamental requirement for an effective intervention. It is also the basis for sound monitoring and evaluation.
2 To support this initiative, the Evaluation Department organised a series of workshops in 2009 and 2010 to further develop the measurement and evaluation of influencing interventions producing a draft How to Note with reference to multilateral organisations in September 2010. However, with the changes to DFID’s corporate landscape in 2010 and early 2011 this work was put on hold pending the conclusion of some key corporate pieces of work .
3. An increase in demand for guidance is also noted given the changing external environment. DFID is now positioning itself to address the demands of the changing global aid landscape with new initiatives, such as the Global Development Partnerships programme. This has a relatively small spend, however its success will be measured largely by the depth and reach of its influence.
4. The Evaluation Department is now seeking guidance on how important the Development Policy Committee considers the evaluation of influencing interventions, and the direction in which it would like this developed.
5. This Paper sets out why evaluation of influencing interventions is important, why now, key theories of change and an influencing typology, value for money of an influencing intervention and metrics, and finally , the challenges of measuring influence.”

See also the associated “Proposed Influencing Typology”

The paper also refers to “Appraising, Measuring and Monitoring Influencing: How Can DFID Improve?” by the DFID Strategy Unit April 2008, which does not seem to be available on the web.

RD Comment: I understand that this is considered as a draft document and that comments on it would be welcomed. Please feel free to make your comments below

Theory of Change: A thinking and action approach to navigate in the complexity of social change processes

Iñigo Retolaza Eguren, HIVOS/DD/UNDP, May 2011 Available as pdf.

“This guide has been jointly published by Hivos and UNDP, and is aimed at the rich constellation of actors linked to processes of social development and change: bilateral donors, community leaders, political and social leaders, NGO’s representatives, community-base organizations, social movements, public decision makers, and other actors related to social change processes.

The Theory of Change approach applied to social change processes represents a thinking-action alternative to other more rigid planning approaches and logics. When living in complex and conflictive times, we need to count with more flexible instruments that allow us to plan and monitor our actions in uncertain, emergent, and complex contexts from a flexible and non-rigid logic. As known, this thinking-action approach is also applied to institutional coaching processes and to the design of social development and change programs.

In general terms, the Guide synthesizes the core of the methodological contents and steps that are developed in a Theory of Change design workshop. The first part of the Guide describes some theoretical elements to consider when designing a Theory of Change applied to social change processes. The second part describes the basic methodological steps to develop in every design of a Theory of Change. For reinforcing this practical part, a workshop route is included, illustrating the dynamics in a workshop of this kind.

The approach and contents of the guide emerge from the learning synthesis of the author, Iñigo Retolaza, as facilitator of Theory of Change design processes where social change actors from several Latin American countries have been involved. His two main bodies of experience and knowledge are: (i) the learning space offered by Hivos, where he could facilitate several Theory of Change workshops with Hivos partner organisations in South and Central America, and (ii) his professional relation with the Democratic Dialogue Regional Project of UNDP, from a research-action approach around dialogic processes applied to various areas of the socio-political field: national dialogues on public policy making and adjusting and legislative proposals, facilitation of national and regional dialogue spaces on several issues, capacity building on dialogue for social and political leaders from several countries in the region”

 

 

%d bloggers like this: