Tag: stories

Connecting communities? A review of World Vision’s use of MSC

A report for World Vision, by Rick Davies and Tracey Delaney, Cambridge and Melbourne, March 2011. Available as pdf

Background to this review

“This review was undertaken by two monitoring and evaluation consultants, both with prior experience in the use of the Most Significant Change (MSC) technique. The review was commissioned by World Vision UK, with funding support from World Vision Canada. The consultants have been asked to “focus on what has and has not worked relating to the implementation and piloting of MSC and why; establish if the MSC tools were helpful to communities that used them; will suggest ideas for consideration on how MSC could be implemented in an integrated way given WV’s structure, systems and sponsorship approach; and what the structural, systems and staffing implications of those suggestions might be”. The review was undertaken in February-March 2011 using a mix of field visits (WV India and Cambodia), online surveys, Skype interviews, and document reviews.

MSC is now being used, in one form or another, in many WV National Offices (NOs). Fifteen countries using MSC were identified through document searches, interviews and an online survey, and other users may exist that did not come to our attention. Three of these countries have participated in a planned and systematic introduction of MSC as part of WV’s Transformational Development Communications (TDC) project; namely Cambodia, India and the Philippines.  Almost all of this use has emerged in the last four years, which is a very brief period of time. The ways in which MSC has been used varies widely, some of which we would call MSC in name only. Most notably, where the MSC question is being used, but where there is no subsequent selection process of MSC stories. Across almost all the users of MSC that we made contact with there was a positive view of the value of the MSC process and the stories can produce. There is clearly a basis here for improving the way MSC is used within WV, and possibly widening the scale of its use. However, it is important to bear in mind that our views are based on a largely self-selected sample of respondents, from 18 of the 45 countries we sought to engage.”

Contents

Glossary. 4
1.      Executive Summary. 5

1.1 Background to this review.. 5

1.2 Overview of how MSC is being used in WV. 5

1.3 The findings: perceptions and outcomes of using MSC. 6

1.4 Recommendations emerging from this review.. 7

1.5 Concluding comment about the use of MSC within WV. 12

2.      Review purpose and methods. 13

2.1 World Vision expectations. 13

2.2 Review approach and methods. 13

2.3 The limitations of this review.. 14

3.      A quick summary of the use of MSC by World Vision.. 15

4.      How MSC has been used in World Vision.. 17

4.1 Objectives: Why MSC was being used. 17

4.2 Processes: How MSC was being used. 18

Management 18

Training. 19

Domains of change. 19

Story collection. 20

A review of some stories documented in WV reports. 22

Story selection. 24

Verification. 26

Feedback. 26

Quantification. 27

Secondary analysis. 27

Use of MSC stories. 28

Integration with other WV NO and SO functions. 29

4.3 Outcomes: Experiences and Impacts. 30

Evaluations of the use of MSC. 30

Experiences of MSC stories. 30

Who benefits. 31

Impacts on policies and practices. 31

Summary assessments of the strengths and weaknesses of using MSC. 32

5.      How MSC has been introduced and used in TDC countries. 36

5.1 Objectives: Why MSC was being used. 36

5.2 Process in TDC: a comparison across countries. 36

Management and coordination of MSC process. 36

Training and support 37

Use of domains. 39

Story collection. 39

Story Selection. 43

Feedback on MSC stories. 46

Use of MSC stories. 47

Role out of TDC pilot – extending the use of MSC to all ADPs. 49

Integration and/or adoption of MSC into other sections of the NO.. 50

5.3 The outcomes of using MSC in the TDC. 51

Experiences and reactions to MSC. 51

Who has benefited and how.. 52

5.4 Conclusions about the TDC pilot. 55

 

 

“Intelligence is about creating and adjusting stories”

…says Gregory Treverton, in his Prospect article “What should we expect of our spies?” , June 2011

RD comment: How do you assess the performance of intelligence agencies, in the way they collect and make sense of the world around them? How do you explain their failure to predict some of the biggest developments in the last thirty years, including the collapse of the Soviet Union, the failure to find weapons of mass destruction (WMD) in Iraq, and  the contagion effects in the more recent Arab Spring?

The American intelligence agencies described by Treverton struggle to make sense of vast masses of information, much of which is incomplete and ambiguous. Storylines emerge and become dominant, which have some degree of fit with the sorrounding political context. “Questions not asked or stories not imagined by policy are not likely to be developed by intelligence”. Referring to the end of the Soviet Union Treverton identifies two possible counter-measures: “What we could have expected of intelligence was not better prediction but earlier and better monitoring  of internal shortcomings. We could also have expected competing stories to challenge the prevailing one. Very late, in 1990, an NIE, “The deepening crisis in the USSR”, did just that laying our four different scenarious, or stories for the coming year”. ”

Discussing the WMD story, he remarks “the most significant part of the WMD story was what intelligence and policy shared: a deeply held mindset that Saddam must have WMD…In the end if most people believe one thing, arguing for another is hard. There is little pressure to rethink the issue and the few dissenters in intelligence are lost in the wilderness. What should have been expected from intelligence in this case was a section of the assessments asking what was the best case that could be made that Iraq did not have WMD.”

Both sets of suggestions seem to have some relevance to the production of evaluations. Should alternate interpretations be more visible? Should evaluations reports contain their own best counter-arguments (as a a free standing section, not simply as straw men to be dutifuly propped up then knocked down)?

There are also other echoes in Treverton’s paper with the practice and problems of monitoring and evaluating aid interventions. The pressing demand for immediate information, at the expense  of a long term perspective: “We used to do analysis, now we do reporting” says one American analyst. Some  aid agency staff have reported similar problems. Impact evaluations? Yes, that would be good, but in reality we are busy meeting the demand for information about more immediate aspects of performance.

Interesting conclusions as well: “At the NIC, I came to think that, for all the technology, strategic analysis was best done in person. I came to think that our real products weren’t those papers, the NIEs. Rather they were the NIOs, the National Intelligence Officers—the experts, not papers. We all think we can absorb information more efficiently by reading, but my advice to my policy colleagues was to give intelligence officers some face time… In 20 minutes, though, the intelligence officers can sharpen the question, and the policy official can calibrate the expertise of the analyst. In that conversation, intelligence analysts can offer advice; they don’t need to be as tightly restricted as they are on paper by the “thou shalt not traffic in policy” edict. Expectations can be calibrated on both sides of the conversation. And the result might even be better policy.”

Narrative Research

David Snowden, 2010.  21 pages. Available as pdf, from the Cognitive Edge site

“Narrative Research, … lays the foundation for the use of narrative research and inquiry methods not only in the project but broadly in the field of research and consultancy…. Elements of it together with general material on Complexity Theory will be published as a chapter in a book on Naturalising Decision Making in the Fall of 2010.”

%d bloggers like this: