Rick Davies comment: The text below is an excerpt from a longer blog posting found here: Impact as narrative, by Bruce Wydick
“I want to suggest one particular tool that I will call the “median impact narrative,” which (though not precisely the average–because the average typically does not factually exist) recounts the narrative of the one or a few of the middle-impact subjects in a study. So instead of highlighting the outlier, Juana, who has built a small textile empire from a few microloans, we conclude with a paragraph describing Eduardo, who after two years of microfinance borrowing, has dedicated more hours to growing his carpentry business and used microloans to weather two modest-size economic shocks to his household, an illness to his wife and the theft of some tools. If one were to choose the subject for the median impact narrative rigorously it could involve choosing the treated subject whose realized impacts represent the closest Euclidean distance (through a weighting of impact variables via the inverse of the variance-covariance matrix) to the estimated ATTs.
Consider, for example, the “median impact narrative” of the outstanding 2013 Haushofer and Shapiro study of GiveDirectly, a study finding an array of substantial impacts from unconditional cash transfers in Kenya. The median impact narrative might recount the experience of Joseph, a goat herder with a family of six who received $1100 in five electronic cash transfers. Joseph and his wife both have only two years of formal schooling and have always struggled to make ends meet with their four children. At baseline, Joseph’s children went to bed hungry an average of three days a week. Eighteen months after receiving the transfers, his goat herd increased by 51%, bringing added economic stability to his household. He also reported a 30% reduction in his children going to bed hungry in the period before the follow-up survey, and a 42% reduction in number of days his children went completely without food. Tests of his cortisol indicated that Joseph experienced a reduction in stress, about 0.14 standard deviations relative to same difference in the control group. This kind of narrative on the median subject from this particular study cements a truthful image of impact into the mind of a reader.
A false dichotomy has emerged between the use of narrative and data analysis; either can be equally misleading or helpful in conveying truth about causal effects. As researchers begin to incorporate narrative into their scientific work, it will begin to create a standard for the appropriate use of narrative by non-profits, making it easier to insist that narratives present an unbiased picture that represents a truthful image of average impacts.”
Some of the attached readers’ Comments are also of interest e.g.
“The basic point is a solid and important one: sampling strategy matters to qualitative work and for understanding what really happened for a range of people.
One consideration for sampling is that the same observables (independent vars) that drive sub-group analyses can also be used to help determine a qualitative sub-sample (capturing medians, outliers in both directions, etc).
A second consideration, in the spirit of lieberman’s call for nested analyses (or other forms of linked and sequential qual-quant work), the results of quantitative work can be used to inform sampling of later qualitative work, targeting those representing the range of outcomes values.”
Read more on this topic from this reader here http://blogs.worldbank.org/publicsphere/1-2014
Rick Davies comment: If the argument for using median impact narratives is accepted the interesting question for me is then how do we identify median cases? Bruce Wydick seems to suggest above that this would be done by looking at impact measures and finding a median case among those (Confession: I don’t fully understand his reference to Euclidean distance and ATTs). I would argue that we need to look at median-ness not only in impacts, but also in other attributes of the cases, including the context and interventions experienced by each case. One way of doing this is to measure and use Hamming distance as a measure of similarity between cases, an idea I have discussed elsewhere. This can be done with very basic categorical data, as well as variable data
Postscript: Some readers might ask “Why not simply choose sources of impact narratives from a randomised sample of cases, as you might do with quantitative data? Well, with a random sample of quantitative data you can average the responses. But you just cannot do that with a random sample of narrative data, there is no way of “averaging” the content of a set of texts. But you would end up with a set of stories that readers might then themselves “average out” into one overall impression in their own minds. But that will not be a very transparent or consistent process.
