A CEDIL Inception Paper, by Rick Davies. August 2018. A pdf copy is available here
Abstract: This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation.
The paper then outlines a range of different ways of addressing these problems which could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.
Table of Contents
What is a theory of change?
What is the problem?
A summary of the problems….
And a word in defence….
Six possible ways forward
Why so little progress?
Implications for CEDIL and DFID
References
Postscript: Michael Bamberger’s 2018 07 13 comments on this paper
- A criticism of many TOCs is that they only describe how a program will achieve its intended objectives and they do not address th challenges of identifying and monitoring potential unintended and often undesired, outcomes (UOs) While some UOs could not have been anticipated, many others could, and these should perhaps be built into the model. For example, there is an extensive literature documenting negative consequences for women of political and economic empowerment, often including increased domestic violence. So these could be built into the TOC, but in many cases they are not.
- Many, but certainly not all, TOCs do not adequately address the challenges of emergence the fact that the environment in which the program operates; the political and organizational arrangements; and the characteristics of the target population and how they respond to the program are all likely to change significantly during the life of the project. Many TOCs implicitly assume that the project and its environment remain relatively stable throughout the project lifetime. Of course, many of the models you describe do not assume a stable environment, but it might be useful to flag the challenges of emergence. Many agencies are starting to become interested in agile project management to address the emergence challenge.
- Given the increasing recognition that most evaluation approaches do not adequately address complexity, and the interest in complexity-responsive evaluation approaches, you might like to focus more directly on how TOCs can address complexity. Complexity is, of course, implicit in much of your discussion, but it might b useful to highlight the term.
- Do you think it would be useful to include a section on how big data and data analytics can strengthen the ability to develop more sophisticated TOCs. Many agencies may feel that many of the techniques you mention would not be feasible with the kinds of data they collect and their current analytical tools.
- Related to the previous point, it might be useful to include a brief discussion of how accessible the quite sophisticated methods that you discuss would be to many evaluation offices. What kinds of expertise would be required? where would the data come from? how much would it cost. You don’t ned to go into too much detail but many readers would like guidance on which approaches are likely to be accessible to which kinds of agency.
- Your discussion of “Why so little progress?” is critical. It is my impression that among the agencies with whom I have worked, while many evaluations pay lip-service to TOC, the full potential of the approach is very often not utilized. Often the TOC is constructed at the start of a project with major inputs from an external consultant. The framework is then rarely consulted again until the final evaluation report is being written, and there are even fewer instances where it is regularly tested, updated and revised. There are of course many exceptions, and I am sure experience may be different with other kinds of agencies. However, I think that many implementing agencies (and many donors) have very limited expectations concerning what they hope TOC will contribute. There is probably very little appetite among many implementing agencies (as opposed to a few funding agencies such as DFID) for more refined models.
- Among agencies where this is the case, it will be necessary to demonstrate the value-added of investing time and resources in more refined TOCs. So it might be useful to expand the discussion of the very practical, as opposed to the broader theoretical, justifications for investing in the existing TOC.
- In addition to the above considerations, many evaluators tend to be quite conservative in their choice of methodologies and they are often reluctant to adopt new methodologies – particularly if these use approaches with which they are not familiar. New approaches, such as some of those you describe can also be seen as threatening if they might undermine the status of the evaluation professional as expert in his/her field.
A very nice piece of paper, especially in defining the problem, much less in offering a solution – the mind that created the problem cannot solve it. The author needs to ask his own theory of change in writing a paper about the theory of change. Much simpler solutions are feasible if one shifts the approach from simplistic-positivistic TOC to complex one.