Month: March 2012

Analytic Rigour in Information Analysis – Lessons from the intelligence community?

This post was prompted by a blog posting by Irene Guijt about a presentation by Michael Patton at a workshop in Wageningen last week (which I also attended). The quotes below come from a webpage about Zelik, Patterson and Woods’ Rigour Attribute Model , which outlines eight attributes of a rigorous process of information analysis, along with guidance on recognising the extent to which each criteria has been met.

The model is summarised in this Analytical Rigor Poster (PDF)

Quotes from the website

“The proliferation of data accessibility has exacerbated the risk of shallowness in information analysis, making it increasingly difficult to tell when analysis is sufficient for making decisions or changing plans, even as it becomes increasingly easy to find seemingly relevant data. In addressing the risk of shallow analysis, the assessment of rigor emerges as an approach for coping with this fundamental uncertainty, motivating the need to better define the concept of analytical rigor.”

“Across information analysis domains, it is often difficult to recognize when analysis is inadequate for a given context. A better understanding of rigor is an analytic broadening check to be leveraged against this uncertainty. The purpose of this research is to refine the understanding of rigor, exploring the concept within the domain of intelligence analysis. Nine professional intelligence analysts participated in a study of how analytic rigor is judged. The results suggest a revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation. Based on this insight, a model for assessing rigor was developed, identifying eight attributes of rigorous analysis. Finally, an alternative model of briefing interactions is proposed that integrates this framing of rigor into an applied context. This research, although specific in focus to intel analysis, shows the potential to generalize across forms of information analysis.

The references  provided include:

Zelik, D. J., Patterson, E. S., & Woods, D. D. (2010). Measuring attributes of rigor in information analysis. In E. S. Patterson & J. E. Miller (Eds.), Macrocognition metrics and scenarios: Design and evaluation for real-world teams. Aldershot, UK: Ashgate. (ISBN: 978-0-7546-7578-5) Currently, the best source for a detailed discussion of our ongoing research on analytical rigor is this forthcoming book chapter which proposes rigor as a macrocognitive measure of expert performance.

Zelik, D., Patterson, E. S., & Woods, D. D. (2007, June). Understanding rigor in information analysis. Paper presented at the 8th International Conference on Naturalistic Decision Making, Pacific Grove, CA. (PDF) (VIDEO) This paper, presented at the Eighth International Naturalistic Decision Making Conference, provides a more formal overview of our current research.

Modeling Rigor in Information Analysis: A Metric for Rigor Poster (PDF) This poster provides an overview of the rigor model, identifying the aspects of the attributes that contribute to low, moderate, and high rigor analysis processes. It also overviews the rigor metric as applied to the LNG Scenario study.

Reducing the Risk of Shallow Information Analysis Google TechTalk  David D. Woods’ discussion of our analytical rigor research at a Google TechTalk provides a dynamic presentation of the material. Google TechTalks are designed to disseminate a wide spectrum of views on topics including Current Affairs, Science, Medicine, Engineering, Business, Humanities, Law, Entertainment, and the Arts. This talk was originally recorded on on April 10, 2007.

THEORY OF CHANGE REVIEW – A report commissioned by Comic Relief

Cathy James, September 2011. 33 pages. Available as pdf.

The review approach Comic Relief’s international grants team commissioned this review to capture staff and partners’ experiences in using theory of change; to identify others in development that are using theory of change and analyse their different approaches; and to draw together learning from everyone to inform what Comic Relief does next.

The review combined analysis of literature with 32 short interviews of people with experience and knowledge of theory of change. The literature included reports, guidelines, study notes, theory of change examples and other relevant documents. The review included interviews with members of Comic Relief’s international grants team; Comic Relief grant partners (both UK and southern organisations); freelance consultants; UK organisation development consultants and researchers; North American research organisations, consultancy groups and foundations; International Nongovernmental organisations (INGOs); and academics.
This report was commissioned by Comic Relief and written by Cathy James, an independent consultant. The views expressed in this report are those of the author and do not necessarily represent the views of Comic Relief

Contents

A. INTRODUCTION

A1. Why do this review?

A2. How was the review approached?

A3. What does the review cover?

B. WHAT IS THEORY OF CHANGE?

B1. What are the origins of theory of change?

B2. Who is interested in theory of change?

B3. What do people mean by theory of change?

B4. What approaches are people taking to theory of change?

B5. How is theory of change different and how does it fit with other processes?

C. HOW IS COMIC RELIEF USING THEORY OF CHANGE?

C1. How has Comic Relief’s international grants team used theory of change?

C2. How have Comic Relief partners used theory of change?

D. WHAT DIFFERENCE HAS THEORY OF CHANGE MADE?

DI. What difference has theory of change made to Comic Relief partners?

D2. What do others say about the benefits of using theory of change?

E. WHAT HAS BEEN LEARNED ABOUT USING THEORY OF CHANGE?

E1. Who is theory of change most useful for?

E2. What kind of approach has been most helpful?

E3. What have been the main challenges?

F. CONCLUSIONS AND RECOMMENDATIONS

F1. Hot topics

F2. Conclusions

F3. Some suggestions for those using or advocating theory of change to think about

 

 

Peacebuilding with impact: Defining Theories of Change

Care International UK, January 2012. 12 pages. Available as pdf

Executive Summary: “Focusing on theories of change can improve the effectiveness of peacebuilding interventions. A review of 19 peacebuilding projects in three confict-affected countries found that the process of articulating and reviewing theories of change adds rigour and transparency, clarifes project logic, highlights assumptions that need to be tested, and helps identify appropriate participants and partners. However, the approach has limitations, including the diffculty of gathering theory validating evidence.

While they are not a panacea, devoting greater attention to theories of change is a simple and relatively inexpensive means of increasing the quality of peacebuilding interventions. Donors and peacebuilding agencies should review their procedures to encourage and accommodate more widespread focus on theories of change, and ensure adequate resources are set aside to allow appropriate monitoring of these theories throughout the life of an intervention.

A focus on theories of change led to the following key fndings:
• Clarifying project logic helps highlight tenuous assumptions;
• Clearly identifying the aims of activities and measures of success strengthens project design;
• Determining the appropriate actors to work with, and not just the easy-to-reach, enables better programme focus;
• More explicit links need to be made between local level activities and national peace processes for desired changes to occur;
• Confict analysis is critical for determining the relevance of activities but is rarely done;
• Staff often require support in ensuring their theories of change are suffciently explicit;
• Current project planning tools do not help practitioners articulate their theories of change;
• Gathering evidence to validate a theory of change is challenging, particularly in conditions of conflict and fragility;
• Critical review of theories of change needs to be undertaken in conjunction with other forms of evaluation to have maximum value;
• Theories of change can encourage an overly linear approach, when change in con?ict contexts can be more organic or systemic.

Recommendations:
1 Donors should revise their logical frameworks guidance to encourage the use of theories of change, notably to include them within the ‘assumptions and risks’ column of existing logical frameworks or by adding an additional column.
2 Theories of change need to be as precise, nuanced and contextually specific as possible and be based on broad conflict analysis.
3 Practitioners need to articulate theories of change within a hierarchy of results and to review these periodically throughout the implementation of a project, particularly if conflict dynamics change.
4 Donors should encourage funded agencies to review their theories of change throughout the project cycle and make resources available for this.”