Befani, B., & Stedman-Bryce, G. (2016). Process Tracing and Bayesian updating for impact evaluation. Evaluation, 1356389016654584. http://doi.org/10.1177/1356389016654584
.
Abstract: Commissioners of impact evaluation often place great emphasis on assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. Current theory-based approaches fail to provide evaluators with guidance on how to collect data and assess how strongly or weakly such data support contribution claims. This article presents a rigorous quali-quantitative approach to establish the validity of contribution claims in impact evaluation, with explicit criteria to guide evaluators in data collection and in measuring confidence in their findings. Coined as ‘Contribution Tracing’, the approach is inspired by the principles of Process Tracing and Bayesian Updating, and attempts to make these accessible, relevant and applicable by evaluators. The Contribution Tracing approach, aided by a symbolic ‘contribution trial’, adds value to impact evaluation theory-based approaches by: reducing confirmation bias; improving the conceptual clarity and precision of theories of change; providing more transparency and predictability to data-collection efforts; and ultimately increasing the internal validity and credibility of evaluation findings, namely of qualitative statements. The approach is demonstrated in the impact evaluation of the Universal Health Care campaign, an advocacy campaign aimed at influencing health policy in Ghana.
.
Rick Davies comment: Unfortunately this paper is behind a paywall, but it may become more accessible in the future. If so, I recommend reading it, along with some related papers. These include a recent IIED paper on process tracing: Clearing the fog: new tools for improving the credibility of impact claims, by Barbara Befani, Stefano D’Errico, Francesca Booker, and Alessandra Giuliani. This paper is also about combining process tracing with Bayesian updating. The other is Azad, K. (n.d.). An Intuitive (and Short) Explanation of Bayes’ Theorem, which helped me a lot. Also worth watching out for are future courses on contribution tracing run by Pamoja. I attended their first three-day training event on contribution tracing this week. It was hard going but by the third day I felt I was getting on top of the subject matter. It was run by Befani and Stedman-Bryce, the authors of the main paper above. Why am I recommending this reading? Because the combination of process tracing and Bayesian probability calculation strikes me as a systemic and transparent way of assessing evidence for and against a causal claim. The downside is the initial difficulty of understanding the concepts involved. Like some other impact assessment tools and methods what you gain in rigor seems to then be put at risk by the fact that it is difficult to communicate how the method works, leaving non-specialist audiences having to trust your judgement, which is what the use of such methods tries to avoid in the first place. The other issue which I think needs more attention is how you aggregate or synthesize multiple contribution claims that are found to have substantial posterior probability. And niggling in the background is a thought: what about all the contribution claims that are found not to be supported, what happens to these?
