Monitoring and Evaluation NEWS

Participatory aggregation of qualitative information (PAQI)

[or, Network visualisations of qualitative data]

This page is a work in progress, so patience please…

An overview

Problem: How do you aggregate large amounts of qualitative data, in a way that does not destroy the interesting details, and/or impose your own interpretations on the data ? E.g counting frequencies of references to things or events of specific interest to the researcher.

Assumption: If we are able to develop better representations of complex bodies of information then this will provide us with more informed choices about how to respond to the content of that information.

The core idea: A combination of two methods can help us aggregate and analyse qualitative information in a way that is participatory, transparent, and systematic.

The two methods are:

1. Pile sorting / card sorting: A simple participatory method of eliciting people’s tacit knowledge, especially the way they categorise people, objects, events etc

2. Social Network Analysis (software): A systematic means of aggregating, visualising and then exploring relationships between people, objects, events

Linking concept: When people categorise people, objects, events, etc, they create relationships beween those events. Two or more entities in the same category can be seen to be related to each other, by that joint membership. And when they categorise objects they also add information to them, in the form of category labels or descriptions (What Dave Snowden calls self-indexing).

How it works in detail

1. Pile sorting

Pile or card sorting is a very simple exercise, where participants are asked to sort a set of objects into groups, on the basis of their similarity (i.e. the attributes that they share), as seen by the participant. Having done so, participants are then asked to explain what the objects in each group have in common, and a label is developed for that group, on the basis of that description.

The kind of sorting proposed here is “open sorting” by multiple participants, who are give a common set of objects to sort into categories of their choice. Open sorting means participants are allowed to sort the set of objects into any number of categories, as they see fit.

Within this specific application (PAQI) the objects themselves are generated by the participants, prior to their briefing on the sorting exercise. Participants representing different stakeholders are asked to brainstorm a set of ideas, each of which is written on a filing card, or Post-It note. These cards could describe their views on:

1. Possible objectives for a project (if the focus is on planning),
or
2. Impacts of the project that have been noticed so far (if the focus is on evaluation)

With small groups sorting could be done by individuals. With larger groups, it may be more appropriate to have sub-groups (representing different interests) do their own joint sorting exercise.

Sorting exercises can be done in workshop settings, or online, using services such as WebSort.net (my preference) or OptimalSort. Online sorting can be efficient in terms of use of time, but opportunities are lost to discuss with the participant their experience of the sorting exercise and their rationale for the completed sorting.

PS1: I have set up a seperate post on references and resources on card sorting

PS2: How is pile sorting different from tagging? (a) Tags are usually only one or two words long, whereas descriptions given in pile sort exercises can be whole sentences or longer. The qualitative data is richer; (b) The same tag may be applied to various items at different points in time, and as a result it’s meaning may vary each time. Descriptions given during pile sorts are given to a set of objects at the same time. There is likely to be more consistency of meaning.

2. Network analysis of card sorting results

Once you have results from a set of card sorting exercises there three kinds of network visualisations that can be produced, showing three kinds of relationships:

Between the sorted items
- Example: A network diagram showing similarities between 24 districts in Indonesia as seperately pile sorted by 5 staff members of a project working in all those districts.
- - Items that have been categorised in the same way by different respondents are shown with strong links.
  - Groups of items with similar characteristics are visible as cliques or clusters of items. For example, Alore, Sumba Barat and Sumba Timur
  - Items that were categorised differently by different respondents have weak links and are more likely to be on the periphery of the network.

Between the categories used to describe them,
- Example: A network diagram showing the similarities in the categories used by the 5 staff members, to classify the 24 districts.
  - Categories that have many of the same items as members are shown to be strongly linked. For example, in the Indonesian project example, the A4 category label was “These are remote areas” and the C9 label was ” Islands, you need boats to get there. Small populations, different coping mechanisms” Frequently shared categories tell us about common concerns
  - Categories with few shared items as members are shown as having weak or non-existent links. For example, those on the top left of the network diagram. These may stil be of value, in that they add value by telling us something that other categories dont.
Between the participants who sorted them
- Example: A network diagram showing the connections between these 5 participants, arising from similarities in the way they categorised the items
  - Participants who have categorised many of the items in the same groups are shown as having strong links. PS: In the example above, there seem to be more similarities between gender than across gender of participants. There are two clusters, of men and women.

Data processing

These diagrams were produced using UCINET & NetDraw (a package). Very briefly, this involved producing the following files:

For relationships between sorted items:
- Create a .txt file in a specific Dl file format, known as PARTITION, as shown in this example
  - This shows five sets of sort results, seperated by a # marker. With each set, each row shows a set of items put into one group, by the participant
- Convert this to the first Ucinet file, using these commands: Data>Inputs text file>Input text file in DL format:
- Aggregate the five sets of data into one items x items matrix, by using these commands: Transform>Matrix Operations>Within Dataset>Aggregations>Input dataset: [the new file you created], Sum, Break-out resultsby: rows and columns
- Then view the saved file in Netdraw. View with link strength >1, because you want to see the connections created by multiple participants, not one.
For relationships between categories used
- Take the original .txt file in PARTITION format and re-structure it as a .txt file in another Dl file format known as EDGELIST2, as shown in this example.
  - N=66 because there are 24 items and 42 categories. A1-4 are categories used by the first respondent, B1-6, by the second etc. Each row lists items put in that category
- Convert this to the second Ucinet file, using these commands: Data>Inputs text file>Input text file in DL format:
- This shows a categories x items matrix
- This needs to be converted to a one mode matrix, of categories x categories. Use these commands: Data>Affiliations (2-mode to 1-mode)>Input data set:
- Then view the saved file in Netdraw. View with link strength >1, because all categories will have at least one shared item with others.
  - PS: You can also use Netdraw to visualise the two-mode categories x items matrix (See Necheles reference below)
For relationships between the respondents
- Use these commands: Tools>Similarities (e.g.correlations,)> Input Datset: name of first Ucinet file above, Measure of profile similarity: Correlation, Compute similarities amongsts: Columns.
- You then have a matrix of correlation values, ranging from 0 to 1. To make these easier to discriminate, when using NetDraw, it is best to multiple them by 100. Use these commands: Transform>Matrix Operations>Within Dataset>Cellwise Transformations>Multiply by constant
- Then view the saved file in Netdraw. Focus on relationships with above average strength (because all participants will have some similarities in their classifications)

Adding qualitative “flesh” to the quantitative “bones”

The network diagrams are the structure. They are the results of all the sorting activities by all the participants. But in the process of sorting the items each participants also added qualitative information, in the form of descriptions of the categories they created. In the Indonesian example 33 category desciptions were provided by the 5 participants. This next section [when completed] will describe how that qualitative information can be made accessable, as people explore the individual nodes and links in the network diagrams. This information will be in the form of node and link attributes.

PS: I have set up a seperate posting on the merits of different kinds of social network analysis software, including UCINET and NetDraw.

References

Visualizing Proximity Data by Rich DeJordy, Stephen P. Borgatti, Chris Roussin, Daniel S. Halgin, on the merits of network models versus multi-dimensional scaling (MDS) for analysing the results of pile sorts (described in title as proximity data). They identified the potential well before I did. I have been more focused on its application.

My earlier explanations of this type of analysis can be found here:

PS: I have just discovered this paper which describes a network visualisation of pile sorting of photographs taken by participants: The “Teen Photovoice Project: A Pilot Study to Promote Health Through Advocacy” (2007) by Jonathan W. Necheles, MD, MPH, Emily Q. Chung, MPH, Jennifer Hawes-Dawson, BA, Gery W. Ryan, PhD, La’Shield B. Williams, Heidi N. Holmes, Kenneth B. Wells, MD, MPH, Mary E. Vaiana, PhD, Mark A. Schuster, MD, PhD. Two pile sorts were carried out. The first was an unconstrained pile sort, generating 41 categories (described as themes). The second was a constrained pile sort where the researchers seem to have predefined the common categories to be used by all participants, based on the results of the first sorting. The results of these piles sorts were then visualised as a two- mode network (group labels x items), and then shared and discussed with the participants. “Participants were asked to interpret the relationships between piles and pictures to foster a better understanding of how they perceived the pictures and themes.”