I have written a detailed response to your post, in this attached Word doc.

I cant copy the text here without losing important formating

regards, rick davies

We’ve developed a tool for PAQI that focuses less on developing visualizations of data, and more on interaction with the data. The tool is an iPhone app called EthnoCorder (http://www.ethnocorder.com). It basically lets you code statements of actors WHILE recording those statements using digital video. So you get essentially the same type of data you would from sensemaker (i.e. it could be plugged into the analytical/visualization software discussed above, as it gets exported in standard SQL format), but you also have the video to drill down into as primary data later (sensemaker basically uses secondary data, the actual statement of the actor is lost). EthnoCorder pulls Participatory Video work into this conversation. Our surveys don’t simply collect data and stories, and allow for real-time indexing, but through iterations of the process, we also use EC as a platform for screening stories, collecting further reactions, and coding those statements (the iterations of the MSC process). Its all closely related conceptually, but with more emphasis on interacting with the data in a participatory way. This makes a big difference when we start recognizing Patrick’s point that all the analytics really need to result in a sense-making process that people can meaningfully engage with.

I can see spending time mastering these data visualization suites, tinkering with them, and being amazed by the complex images they can produce, but still only having secondary information. With EthnoCorder I can deal with the data using easily accessible spreadsheet software like Excel, and directly relate that back to the primary qualitative data (we embed demonstrative video clips in our reports to allow the reader to interact with the primary qualitative data that the quantitative analysis draws on). You get a thorough, rich understanding of the data without getting tangled in a web of artificial network graphs. This lets you spend more time engaging the actual problem.

Even using basic descriptive statistics we already get blow-back from donors complaining that our reports read more like ‘research’ than ‘development’. I think this goes to show how starved the industry still is for really solid analysis of qualitative data that forces us to confront the complexity of the contexts we work in. I find the most helpful advice from Snowden’s (2007) writings to be that leaders responding to complex problems need to “increase levels of interaction and communication”. I’m not convinced that a focus on data visualization really does that. EthnoCorder is much better suited for this task, and I’m looking forward to seeing all these tools and ideas converge.

I’m always interested in looking at the data in different ways, and I bristle when people assert dismissively that I’m using ‘fancy tools’, so I am not saying that these data visualization packages are worthless. However, I think we have to be careful of being mystified by technology and math, lest we lose touch with the reality of issues at hand.

Cheers,
Nathan

I recently had the occasion to use Rick’s PAQI technique in an assignment for DG Translation of the European Commission about the “contribution of translation to the multilingual society in the EU” (see the final report “here”:http://ec.europa.eu/dgs/translation/publications/studies/multilingual_society_study_fr.pdf. It’s in French but an English summary is available).
At first we had not planned to do so: we expected to perform an extended literature review to answer the very vague yet simple question “What are the effects of translation?”. But after days identifying and reading books and articles, and doing interviews, we started to understand that there was no simple answer to that question, and we found out that we simply did not have a way to make sense of so much fragmented and conflicting information with the planned tools.

So we decided to answer differently. It had to be cheap, it had to be methodologically sound, and it had to be quick enough to keep more or less on schedule, and of course it had to be able to answer in a detailed way to the question asked. My first idea was to rely on concept mapping as developed by William Trochim (see “here”:http://www.socialresearchmethods.net/mapping/mapping.htm), but the statistical method used to show the clusters of answers (multidimensional scaling) is out of my reach, when network analysis as used in Rick’s PAQI is a lot easier to me.

The method used:
* First, based on the work done, we listed 50 assumptions detailing effects of translation (e.g. “translation gives access to ancient cultures”).
* Then we set up an expert panel of 9 members + 1 member of the Steering Committee (the others would not want to participate because they felt they had not enough expertise) and asked them to comment, revise and complete the list (which they did: the final list was composed of 83 different items). Because we were dealing with experts from 7 countries, we asked a professional translater to edit the list, in order to avoid misunderstandings;
* Then we asked the experts to sort and rank the items. We used websort.net (free for 10 participants or less) for the sorting and a googledocs questionaire for the ranking.
* After that we analysed the results using the free network analysis software “Cytoscape”:http://www.socialresearchmethods.net/mapping/mapping.htm, which I have been using since two years for that kind of purpose.
* We obtained a weighed cluster map and various statistical information which was then discussed twice: with the members of the expert panel during a phone meeting in which they were asked to discuss what they learnt (they also sent a 500-word note afterwards) and with the Steering Committee.
The method proved tremendously useful. The clusters turned into sections of the report, and the relations between items within those clusters often provided a “story” on how translation was having effects on society. The mix of statistics and expertise in the method was also highly credible for the Steering Committee and also for the experts of the field, and for a reduced cost. It is also a scalable method: we used it with 10 experts, but it is definitely possible to make the same with 100 or more participants.

Now an advice on the conditions for success: it is crucial to spend as much time as possible on the first phase (definition of the assumptions that will be ranked and sorted). Had we had enough time, I would have organised at least another round dedicated to improving the wording and make sure that there are no ambiguity left. This is all the more important that the assumptions we had were simple sentences, to make sorting and ranking easier, but complex causal relations can only difficultly be reduced to simple assumptions.

Thomas Delahais (“euréval”:www.eureval.fr)

Here tow more references you might be interested in:

* Brandes, U., Kenis, P.N., Raab, J., Schneider, V., & Wagner, D. (1999). Explorations into the Visualization of Policy Networks. Journal of Theoretical Politics, 11(1), 75-106.
* Brandes, U., Kenis, P.N., & Raab, J. (2006). Explanation through network visualization. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2(1), 16-23.

Papers can be freely accessed through:
http://publications.uvt.nl/repository/p.kenis/publications.html

Hi Rick

Thanks for this – I’m intrigued by this, but I’m missing the last link… which is how insight is derived, and what kinds of insights are derived. This is something common to all network visualisation techniques I find… OK so there’s a map, so what? In social network analysis, for example, the map typically helps analysts/participants identify places in the network that look like they warrant further investigation – eg bottlenecks, disconnects, cliques. These can be either positive or negative forces, depending on the drivers behind that structure, and the overall context. There are different ways of undertaking that investigation to find the “story” behind the structure. I’m curious about where the sensemaking portion lies in the PAQI model – do you have examples of insights and how they are derived and then actioned? I understand this is a work in progress!

My response was as follows:

Hi Patrick

Good questions

I dont think I can promise anyone automatic “insight” as a result of usig the PAQI or any other method. And I doubt if David would do so either re his use of Sensemaker.

Both are tools for providing better/different forms of description of large amounts of qualitative data (and much larger amounts in the case of Sensemaker)

With many forms of measurement and description it is useful to ask people what they expect to be found and then show them what is actually found, and to then discuss and learn from the difference. An NGO in India that I have been working with carried out a large baseline survey of capacities of CBOs in a HIV/AIDS program, that generated a set of performance scores. In response to my suggestion they asked the grantees supporting the CBOs what their expectations were about the CBOs’ scores, then they shared the actual scores, then discussed then differences. In a number of instances this led to agreement on how the survey instrument needed to be changed. I hope in other cases it led to agreement on how the CBOs needed to change!

On the PAQI web page I started with this assumption: If we are able to develop better representations of complex bodies of information then this will provide us with more informed choices about how to respond to the content of that information.

That is what David is doing, providing better representations of large amounts of qual data, less tainted by researcher’s existing beliefs. I think he would then argue the need to explore the outliers, as as well as any central tendency or “averages” The ability to explore data, that is both aggregated but has some structure, is common to both David’s Sensemaker and PAQI

If we look at the one example I presented, which was the result of a very quick inquiry of 5 staff members at the end of a workshop: their classification of 24 Indonesian districts where their project was working. If I had a chance to continue talking to them today I would start by asking them how many clusters do they think might emerge from this analysis, and what would they be.After sharing the aggregated results and discussing any differences between expected and actual results, there could be two ways forward. Ask what are the implications for (a) project design and activities, (b) improved use of this PAQI method). On the former, I would love to know whether given the existence of at least two main clusters of districts (two more can also be identified less distinctly) whether there is any difference in the project strategies being pursued in those areas. And if there is none, should there be a difference?

On the significance of measurement on its own, I often use an imagined situation where two people walk into a room each holding a tape measure. They both proceed to measure the dimensions of a large hole in the wall. One walks away happy, the other walks away unhappy. The first is an air conditioning installer, the other is a security expert. So what use is a tape measure (/network diagram)? Its just numbers on arbitrary scale (/a set of dots connected by lines). Well it turns out it is important if we have prior expectations about what we want to see, but probably meaningless if we dont. Theory and measurement are both needed.

Does this help?

PS: There are other ways of eliciting expectations prior to sharing results, with network diagrams. One easy method is to show network diagrams without visible labels on the nodes and ask participants to identify who is where. Then to make the labels visible.