Analytic Rigour in Information Analysis – Lessons from the intelligence community?

This post was prompted by a blog posting by Irene Guijt about a presentation by Michael Patton at a workshop in Wageningen last week (which I also attended). The quotes below come from a webpage about Zelik, Patterson and Woods’ Rigour Attribute Model , which outlines eight attributes of a rigorous process of information analysis, along with guidance on recognising the extent to which each criteria has been met.

The model is summarised in this Analytical Rigor Poster (PDF)

Quotes from the website

“The proliferation of data accessibility has exacerbated the risk of shallowness in information analysis, making it increasingly difficult to tell when analysis is sufficient for making decisions or changing plans, even as it becomes increasingly easy to find seemingly relevant data. In addressing the risk of shallow analysis, the assessment of rigor emerges as an approach for coping with this fundamental uncertainty, motivating the need to better define the concept of analytical rigor.”

“Across information analysis domains, it is often difficult to recognize when analysis is inadequate for a given context. A better understanding of rigor is an analytic broadening check to be leveraged against this uncertainty. The purpose of this research is to refine the understanding of rigor, exploring the concept within the domain of intelligence analysis. Nine professional intelligence analysts participated in a study of how analytic rigor is judged. The results suggest a revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation. Based on this insight, a model for assessing rigor was developed, identifying eight attributes of rigorous analysis. Finally, an alternative model of briefing interactions is proposed that integrates this framing of rigor into an applied context. This research, although specific in focus to intel analysis, shows the potential to generalize across forms of information analysis.

The references  provided include:

Zelik, D. J., Patterson, E. S., & Woods, D. D. (2010). Measuring attributes of rigor in information analysis. In E. S. Patterson & J. E. Miller (Eds.), Macrocognition metrics and scenarios: Design and evaluation for real-world teams. Aldershot, UK: Ashgate. (ISBN: 978-0-7546-7578-5) Currently, the best source for a detailed discussion of our ongoing research on analytical rigor is this forthcoming book chapter which proposes rigor as a macrocognitive measure of expert performance.

Zelik, D., Patterson, E. S., & Woods, D. D. (2007, June). Understanding rigor in information analysis. Paper presented at the 8th International Conference on Naturalistic Decision Making, Pacific Grove, CA. (PDF) (VIDEO) This paper, presented at the Eighth International Naturalistic Decision Making Conference, provides a more formal overview of our current research.

Modeling Rigor in Information Analysis: A Metric for Rigor Poster (PDF) This poster provides an overview of the rigor model, identifying the aspects of the attributes that contribute to low, moderate, and high rigor analysis processes. It also overviews the rigor metric as applied to the LNG Scenario study.

Reducing the Risk of Shallow Information Analysis Google TechTalk  David D. Woods’ discussion of our analytical rigor research at a Google TechTalk provides a dynamic presentation of the material. Google TechTalks are designed to disseminate a wide spectrum of views on topics including Current Affairs, Science, Medicine, Engineering, Business, Humanities, Law, Entertainment, and the Arts. This talk was originally recorded on on April 10, 2007.

One thought on “Analytic Rigour in Information Analysis – Lessons from the intelligence community?”

Comments?

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: