Hierarchical Card Sorting (HCS)

A simple tool for qualitative research and inquiry,  also useful for planning and evaluation

Contents of this page

Background

This is a tool I developed in 1993, while doing PhD field work with the Christian Development Commission of Bangladesh (CCDB), in Dhaka, Bangladesh. I subsequently wrote two how-to-do-it papers describing the method (both of which have been on the web for some time):

This page now integrates the contents of those two papers, and adds further content developed since then. An initial clarification may help. Hierarchical Card Sorting refers to the process. Treemaps refer to the product of that process.

What is Hierarchical Card Sorting (HCS)?

HCS is one of many types of card sorting methods (also known as pile sorting). Card sorting has been used in many contexts, from traditional ethnography to the modern day business of designing usable websites (See references below). In these contexts card sorting is typically used to elicit people’s mental models: the categories they use, what belongs to these categories, and how the categories relate to each other.

What use is it?

In many organisations people accumulate a lot of knowledge, but often it is tacit and informal in nature. As such. it is not so easily shared. Yet sharing that knowledge can make a difference, other people can make use of it, and they can help correct it and improve it. A HCS can help make people’s knowledge more explicit and publicly available, contestable and usable.

How do you do it?

The HCS method asks people about significant differences. About differences which are important to them and which have (or had) consequences. It has similarities in origin and approach with the Most Significant Change (MSC) technique. Central to the HCS is a question about the “most significant [static] difference”, whereas MSC asks about the “most significant change”. Both ask respondents to make observations and interpretations. The design of both tools was influenced by Gregory Bateson, especially his book “Mind and Nature: A Necessary Unity” (1979), in which he argues that information is “a difference that makes a difference”. In turn, many people would argue that knowledge is structured information. The HCS is about eliciting and representing people’s knowledge (i.e as a structured set of differences that make a difference).

Normally the HCS is used with one respondent. However, the process outlined below can easily be used with a small group. The steps:

    1. Identify the respondent’s area of expertise or knowledge that you want to explore. For example, the knowledge of animal diseases held by a paravet, or knowledge of local NGOs held by a INGO staff member working on NGO capacity building.
    2. Generate a list of actual cases which will be sorted. If possible, try to select cases that represent a wide variety of examples. In many cases you will want to select cases that the respondent is responsible for in some way, and thus should be expected to know about them. For example, a list of cases treated by the paravet in the last month, or a list of NGO grantees managed by an INGO desk officer. Don’t be too ambitious, especially to start with. Large numbers of cases (25+) will make the process more time consuming and will run the risk of boring the respondent and interviewer. Write the name of each case down on a separate card. Cases might be events (treatment provided) or entities (clients).
    3. Place all the cards in one pile (see this as the trunk of a tree) and begin by asking the respondent to tell you about some of the differences between all these cases. The purpose of this initial question is simply to generate a shared awareness of the large number of differences that (inevitably) exist. It is a warm up exercise. If the respondent finds this difficult, randomly select two cards at a time, and ask them to compare them, with a view to finding interesting differences.
    4. Ask the respondent to sort all the cards into two piles of any size (see these as the first two branches above the trunk of a tree), representing what they think is the most significant difference between all the cases represented on the cards. Emphasise that it is their opinion of “significant” which is important. If you want to direct their attention in a particular direction then use a prefix to the question, such as “In your roles as…what do you think is the most significant difference between…?. Or “Considering the objectives of this organisation…what do you think is the most significant difference between…?
    5. Emphasise that a distinction is significant if it makes a difference. Because respondents may casually offer a difference simply to oblige the interviewer it is important to check its significance by asking “What difference does this difference make ?” If one can’t be identified then suggest to the respondent that they consider if there are other differences which might be more significant.
    6. Keep a record of which cards are placed in which pile. This is easier if all the cards have all been numbered beforehand. And then write down a description of the reported difference between the two piles. And what difference that difference makes.
    7. Take one of the two piles at at time and repeat stages 4 to 6 above. Then repeat this process with second pile. There should now be four piles. Repeat the same process with these piles until there is only one card left in each pile. One way of keeping track of this process is to use a large piece of paper, to draw a tree whose branches spilt into smaller and smaller branches.
      • You can develop the tree structure in  various ways: (a) breadth-first, or (b) or depth first, or (c) a mix of both. Your choice might best be guided by where the respondent seems to be most at ease.
    8. In some cases there may be more than one example left in a pile but the respondent may not be able to identify an important difference between them. Don’t force them to do so, but simply note that no further difference could be identified.
    9. Document the results of the exercise in a form that shows how the different groups and sub-groups or cards are nested, along with their perceived differences and the differences these make. See the next section here for examples.
    10. Other steps can then be taken to make use of the knowledge structure elicited by the HCS. See Making use of HCS results below.

Examples of HCS results

The HCS process generated tree structures, which can be represented in a table or diagram form. Figure 1 is a tree in table form (using Excel), produced in 1993. Click on the image to read the text, and click again to enlarge. Read the results from left to right. Green shaded sections each refer to one of two binary types that the respondent would like to see funded more in the future. These preferences have then been converted to an overall ranking, as seen on the right. This example is incomplete because while it shows differences, it does not yet show the differences they make i.e. the expected or realised consequences.

F==
Figure 1: A classification of local organisations funded by donor NGO in Bangladesh in 1993

The same results can be shown as a more explicit tree structure (shown below without the text notes). Red lines = preferred types of partnerships in future (discussed below). Any diagramming software can be used to do this. My favorite is the free version of yEd.

Figure 2: Pile sort results as a network structure

Another example is shown in Figure 3 below. This is the result of an interview with a rural community development worker, where I asked about the most significant differences between the villages they were working with. The wealth ranking on the right is based on binary comparisons made at each junction of the tree, by the respondent. After the tree had been constructed, I had asked them which of the two sub-groups of villages at each junction they thought were more versus less poor.

Figure 3: Pile sort results that include “the difference the difference makes” – based on an interview with a rural community development worker in west Africa
Figure 4: A donor NGO’s view of the most significant difference between its grantees, in one thematic portfolio (click to enlarge)

Figure 5 was developed by a groups of 4 people, rather than by one single respondent, This was within a planning session, concerned with the design of a new rural livelihoods programme.

Figure 5

Two user accounts of the use the HCS

Bangladesh

In 1994 staff of the Research Unit of the Christian Commission for Development in Bangladesh (CCDB) used the cardless version of the same method in the process of exploring poor peoples conceptions of health, disease and medicine. The following quote describes the results of two applications of the method:

“In two somities [beneficiary groups] we also asked the members to come up with different medicaments they use when they are ill. We asked them to divide these medicaments into two groups. We asked for an explanation after each division. After dividing the group into two other groups we asked them to do the same for the two other sub-groups. In the first somity they divided the medicaments into allopathic and herbal medicines. The allopathic medicines were divided into tablets and syrup. The herbal medicine into medicines used for cough and influenza, and medicines for influenza.

In the second somity they divided the medicaments into medicines given by the kobiraj and medicines given by the allopathic doctor. The medicines given by the kobiraj were divided into medicines for weakness and medicines for influenza and headache. The medicines of the allopathic doctor were divided into medicines for pain and medicines for gastric burns.

We used this exercise to get more information about people’s concepts of medicines. When we asked about [the consequences of] the difference between herbal and allopathic medicines it was mentioned that allopathic medicines cure better and sooner but they are costly and can give weakness. Herbs are “softer” to the body, they don’t harm the body, they are cheap , and although they work slowly they keep the body healthy.” (People’s Health and Nutrition, January 1995, CCDB)

Afghanistan

I thought I would let you know that I have tried your treemap method for the first time. Despite thinking it sounded rather unlikely to be useful when I first read it, it has provided a very nice alternative to classic wealth ranking for investigating the structure of villages. The people I am working with had the same unease as I do about launching into wealth analysis. We therefore tried your method as it seemed to offer a way round the problem. We ask the number of families in the village, then ask about differences between them with respect to livestock and well-being. Following the process as you describe, we rapidly get a good idea of the wealth structure of villages, which is far more disparate than I had imagined. From this exercise it is possible to find people to interview in more detail from each category. Repeating the exercise with paravets, we then asked them to indicate what proportion of their work was with which group. This worked well and provided an indication of who was benefiting most from the services of the programme.

Problems and limitations

1. Some caution may be appropriate. Like all participatory methods it requires some trust and confidence in the relationship between yourself and the person whose views you are seeking. Secondly, there is no guarantee that the views that are expressed will be stable over time. Peoples views of the world change, and the expression of their views is often very context dependent.

2. Some people react at some stage to the exercise by saying “There is no difference between these“. Here I have cautiously tried to give many examples of possible differences, while being careful not to lead in any particular direction. I have emphasised that differences can be found even between objects that look identical, the question is which of these is most important from their point of view. I also emphasise that we are looking for relative rather than absolute significance. But if the person is really struggling, especially after having already identified previous differences between the earlier bigger piles, I do not push them.

3. Another problem is almost the opposite in nature. People can approach the task in what appears to be an un-engaged manner, blithely tossing off distinctions which don’t seem too significant. When this happens I have tried asking “In what way is that significant, what difference does that make ?, checking to see that the respondent can articulate the significance, and if not checking to see if they really understand the exercise.

4. Another problem relates to respondents who are almost too helpful. As can be seen from the tables below it is common for some respondents to report more than one difference. When well organised I have dutifully noted these down and then asked the respondent, after reading them back, “and which of these……are the most significant ?” Failing to do this has meant I have been the one that ends up speculating on their relative importance to the respondent.

5. Many respondents find it easier to identify differences, than it is to identify the differences these differences will make. Often both respondent and interviewer assume that this is self-evident and fail to document this part of the exercise. Yet these beliefs can contain important assumptions or hypotheses about the way things work, which can benefit by being openly described, scrutinised and tested.

Making use of HCS results

Value can be obtained from tree diagrams at two stages:(a) during the creation of the tree diagrams, and (b) through comparisons made between parts of the structure once it has been created.

Ethnographic use

During creation of a tree diagrams the main use is as a ethnographic tool: a means of understanding people’s view of the world. There are three types of use:

    • Identifying the distinctions that people see as important. This is evident in the contents of the differences reported. It is also evident in how early in the exercise they are reported, and how often they are reported (on different branches).
    • Identifying the limits to people’s knowledge: When respondents cannot identify differences between two or more entities the limits to their knowledge seem to have been reached. Knowing what people do not know about can be important, especially when they might be expected to, or claim to have, expertise in that area.
    • Identifying the direction of learning:  It is also worth noting where there is more versus less differentiation of knowledge, visible respectively where where branches end up with a single case rather than multiple cases, which have not yet been differentiated. See Figure 2 above for an example.
Planning and evaluation use
Exploring

After creation, the tree structure can be used to ask a series of comparison questions. At each junction respondents can be asked questions about the available binary choices i.e. between the two alterative branches. The default process for soliciting and documenting these judgements is to proceed from the trunk to the leaves. At least four different kinds of questions can be asked:

Questions about: The past The future
Differences in degree (more vs less) 1 2
Differences in kind (categorical) 3 4

Some examples of each of these types of questions are:

    • Which of these two groups do you think has been most successful, so far?” [1]
    • What is the most significant difference in the nature of the achievements of these two groups, so far” [3]
    • Which of these two groups do you think will be most successful over the next x period of time?“[2]
    • In the next six months, how will your work with this group be different, compared to this group?” [4]
    • Which of these two groups do you expect will present the most problems in the next six months?” [2]
    • What do you think will be the most significant difference in the problems face by these two groups, in the next six months?” [4]
    • Which of these two groups do you plan to be spending more time with in the next six months ?” [2]

The answers, and associated rationales, can be added as a further annotation to the tree diagram, at the relevant junction.

Caveat: Sometimes people may not be able to make a make a choice (of degree) or provide a choice (of kind). In which case, simply note this fact, and proceed down the tree structure to the next binary distinction.

Ranking: In addition, the answers to all of the binary “more versus less” questions can be used to generate an overall ranking of the items in the set that has been sorted. See Figures 1 and 3 for examples. This can be done by examining each branching point, and re-sorting – where necessary – the two sub groups such that the one identified as “more …” is at the top, and the one identified as “less…” is at the bottom. When this is done with all branching points, then the aggregate result will be a completely ranked set of entities, still within an intact HCS tree structure.

Testing

Reiteration

A HCS can be carried out at multiple points in time, prior to, during and after an intervention. In order to: (a)  identify current status of the people/organisations being compared, (b) to compare these with past assessments, and (c) to detail plans for the upcoming future. This kind of application would be most useful in situations where there was a diversity of contexts and needs, and the need for variations in the specifications of interventions.

Scatter plots

Once you have two or more sets of ranking for the items that have been sorted the relationship between these can be visualised in a scatter plot. Here is a good argument for more use of such simple devices

https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-02133-w#Sec5

Creating a testable Theory of Change

Each branch of a tree diagram can potentially be seen as a causal configuration, i.e.a set of conditions associated with an outcome seen in the entities that have been sorted . A complete set of branches can then be seen as a particular kind of Theory of Change, one that is notably different in at least two ways.

    • Firstly, because there will be multiple branches – it will be capable to representing equifinality i.e the reality that there are often multiple alternate means of reaching the same outcome. It will also be capable of representing asymmetrical causal processes i.e. where an outcome can be absent not because of the absence of its usual causes, but become of the presence of other influences.
    • Secondly, contra to more conventional representations, each segment in a branch is not a consecutive event, forming a sequence of events. Rather, each segment is an additional kind of difference that is expected to make a difference to the outcome observable in the entity at the end of each branch.

PS: This type of tree structure can be seen as a type of “Ethnographic Decision Tree”, a variant of a method developed by Christina Gladwin, in the 1980s. The main difference is that the ethnography is done with someone who is familiar with all the N entities, not a set of individuals representing each of those entities.

As can be seen in Figures 1 and 3 above (and explained here) it is possible to convert  a set of binary predictions about expected relative success into a ranking. This ranking data can then be compared to independent measures of  success, also converted to rank value, to test the validity of the elicited predictive model. Figure 6 below shows the results of one such comparison, summarised in the form of a Confusion Matrix.  The 27 items were projects within one of a grantee’s portfolios.  The card sort described significant difference between the projects and the measure of project success was obtained from a sperate monitoring system.

In this analysis each set of ranking values has been dichotomised into higher versus lower rank values (initial at the median value). The distribution of values in the table can then be evaluated using range of different measure.  One common measure is Accuracy (= True Positives + True Negatives/All cases).  In this instance the HCS predictions had an overall Accuracy of 59%

Figure 6: Confusion Matrix summary of HCS predictions of project success versus independent measures of project success

Experiments with different cut-off points can often identify a better performing prediction model. In this example, lowering the cut-off point for the predicted success status to less than 8 (the lowest ranking available) increased the performance to 74%.

The False Negative (FN) cases are where one or more other configurations (within the “less successful” ranked cases) might be responsible for the “more successful” status. The False Positive (FP) cases are where other features outside the configurations (within the “more successful” ranked cases) must have some yet-to-be-identified role, and be responsible for their “less successful” status.

Other Applications
  1. Assessing capacity building activities: When supporting capacity building work with individuals or whole organisations, we might expect that this assistance, either in the short or long term, would make a difference to the assisting person or organisations relationships with their clients. For example, the service provider might be more sensitive to the differences between client’s needs. Or, they may also be more up to date in their knowledge about their various clients’ needs. Or, the differences they see between their clients (that they think are significant) may be more reflective of their clients concerns, and not just their own. That knowledge can be elicited during a HCS process, where the clients names are the items being sorted.
  2. Doing a stakeholder analysis. It is possible to use tree diagram as a means of doing a stakeholder analysis in a development project. This can initially be from the perspective from one observer, possibly an individual stakeholder. Firstly, a list of cases reflecting the maximum possible variety of stakeholders are identified. The process of inquiry then starts at the trunk, with the respondent being asked to identify “the most significant difference between all the stakeholders in the project”.Optionally, after a prefix saying “Bearing in mind the objectives of this project…” .  This is then followed by a question about the difference that difference makes i.e. the actual or expected consequences. Then each of the two initial categories of stakeholder are progressively differentiated until all cases are located as a leaf of their own.The net result is a nested classification of all the listed stakeholders. Information is generated not only about the differences between types of stakeholders, but also about the consequences, past, present or future of those differences.
  3. Portfolio analysis: “Most measurement and adaptive management approaches were developed for and from individual projects (Sweetman and Conboy, 2018). Senior managers and public officials, however, are often interested in results at more aggregate levels, looking across multiple projects at wider portfolios of work” (Buffardi et al, 2019). Because a HCS is focused on a collection of entities, rather than one entity, it has potential as a useful tool for portfolio type analyses.  The following purposes and questions associated with portfolio analyses, identified in the above ODI paper, could be addressed by a HCS exercise:
    • Overview/health check: What areas need more attention?
    • Hypothesis testing: Which approach works better? What should be scaled up, down or discontinued?
    • Balancing/hedging: How can the portfolio maintain a pipeline of outcomes over different time frames, which range in their likelihood they will be achieved and level of risk
    • Comparative advantage/future positioning:  In the next five years, how can we maximise the value of our investment and unique contribution relative to others? What should we move out of and expand into? How should future resources be allocated.
  4. Intersectionality: “…is an analytical framework for understanding how aspects of a person’s social and political identities combine to create different modes of discrimination and privilege“… An intersectional analysis considers all the factors that apply to an individual in combination, rather than considering each factor in isolation” (Wikipedia) HCS could be easily used for these purposes. Firstly, by re-iteratively eliciting respondent’s views of the “most significant difference” between different people (groups or individuals), such each has their own composite identity, but which is shared with others when viewed at different levels of aggregation. Secondly, by then posting exploratory binary choice questions about past or proposed differences in interventions affecting different groups and sub-groups.
  5. Analysis of Most Significant Change (MSC) stories: MSC stories can be treated as items to be sorted, using the HCS process. The resulting tree structure will provide an overview of all the MSC stories, at multiple levels of detail. Binary exploration questions can then be asked, that make use of the same tree structure. Asking about, for example, how the respondent thinks an organisation should respond differently to each type of change described in the MSC stories
Aggregation of multiple HCS results

If multiple individuals are asked the the same HCS question then their results can be aggregated, and then analysed, with at least two different purposes in mind:

    1. Identification of similarities and differences between different people’s views of the world
    2. Identification of a synthesised model,  one that best predicts an outcome of interest, based on the views of all the participants in the HCS exercises
Qualitative methods

The first method is qualitative, and involves what could be called a macro-HCS.  Each HCS sort result is in effect treated as a new type of “card” or case. Participants are then asked to identify the differences between these different HCS results. Firstly to generate a list of such differences, in order to reflect on them. Then to select “the most significant”, on a reiterated basis, generating  a tree diagram, of the kind already seen above.  Figure 8 is an example of a Macro-HCS, using each of the previously described HCS results as “cards”

Figure 8

A variant of this approach could be called a meta-HCS. In Figure 8 the “cards” (i.e. HCS exercises)  have been differentiated on the basis of the different types of cases they each analyse (i.e. NGOs and villages). But if all the HCS exercises were using the same set of cards (e.g., same set of NGOs), then the Figure 8 Meta-HCS would have to focus on differences between the kinds of differences documented in each HCS.

I think the following Figure 8b example qualifies as a meta-HCS. In 1992 I interviewed the chief executives of 32 largest NGOs in Bangladesh. In that interview they were asked “What is the most significant difference between this and other NGOs [in Bangladesh]?”. I summarised their answers by asking myself the same question, but with the CEO’s distinctions between the NGOs being the objects of concern.  Figure 8b is the HCS tree tree structure that I developed using the reiteration of that question as built into the HCS method.

Figure 8b

For reflections on the significance of the contents of this structure see pages 161-163 of my 1998 PhD thesis on organisational learning in NGOs

Quantitative methods

Aggregation, using social network analysis software

Step 1: The raw material for this kind of analyses are data matrices, one per respondent. In each matrix the rows represent the names of the cards sorted, the columns represent the differences identified between the cards, and the cell values of 1 mean the column description applies to the row card, and 0 that it does not. So, the data matrix for Figure 1 would look that shown in Figure 8 below.

Figure 9: Initial data set generated by a HCS exercise

Note that there are many gaps in the matrix, where the respondent has not yet told us about the presence or absence of an attribute that describes a difference, in relation to some of the cases. This is because the information that has been provided during the HCS exercise about each difference has always only been about a sub-set of cases, starting from the second-from-left column in Figure 3 and moving rightwards. This missing data, from each participant’s matrix,  could subsequently gathered using a standardised survey instrument, because the differences with missing data points have already been identified.

Step 2: Figure 9 is called an affiliation matrix, Social Network Analysis (SNA) terminology. This now needs to be converted into an adjacency matrix, where both the rows and columns represent the cases (i.e. the entities sorted), and the cell values indicate the number of attributes (differences) shared by the particular combination of row and column cases. This conversion, shown below, has been done using Ucinet.

Figure 10

Step3: Steps 1 and 2 can be repeated for each HCS generated by each respondent, using the same set of sorted entities. The contents of the adjacency matrices can then be added up, into a summary or aggregated adjacency matrix. The larger the set of respondents the better, because the individual cell values in the summary matrix will be subject to less and less influence by each additional respondent

Using the associated network visualisation software NetDraw this summarised data can then be visualised as a network structure, where links between cards e.g. #1-12 above, reflect how often those cards were co-located in the same group. Ucinet can also be used to calculate the correlation between the different respondents’ adjacency matrices. That correlation matrix can then be visualised as a network structure, where the strength of links between each respondent shows how correlated their matrix cell values are, as a whole.

Synthesising, using prediction modelling

Another available option is to use supervised machine learning algorithms, to search and find the combination of attributes that have been identified by all participants, which is the most accurate predictor of the presence of the outcome of interest, or its absence. This can be done using Rapid Miner Studio, or EvalC3QCA software can also be for the same purpose.

To do this it is first necessary to include information  about the outcome status of each case, as an extra column on the right side of the data set, as shown in Figure 11, already available in Figure 1. According to whatever outcome was of interest. And to ensure inclusion of missing values, as already noted above.

Figure 11: Additional development of a HCS data set, prior to a QCA or predictive modelling analysis

Future directions

Incremental HCS

Sometimes the contents of the items to be sorted can be quite complex. Such as paragraphs of text, rather than simply names of people, places, events, activities or things. In these circumstances being able to read all of them then split them into the first two HCS groups can be quite cognitively demanding. An alternative is to take a more incremental approach, involving a series of pair comparisons. This involves the following steps::

    1. Take two items at random. Nest these two items under a common category e.g. rocks– that best describes what they have in common. Then create two subcategories, one for each item. Label each category with a description of how you think it is most significantly different from each other. E.g. rocks, small and big
  1. Introduce another randomly selected item.
      1. If it belongs to one of the two existing categories just created (e.g. small rocks) then create two sub-categories within  that existing category, describing the most significant difference between pre-existing item already there, and the new item. E.g. small round rocks and small square rocks
      2. If it does not belong to either existing categories, create a new common category that describes what all the items (including the new one) have in common (e.g. In my garden). Then place the new items in a new subcategory (e.g plants) adjacent to the old common category, which is now a sub-category (e.g. rocks).
  2. Introduce a fourth randomly selected item.
    1. See if it fits any existing category, proceeding from the largest existing category, down to the smallest category that it fits, then follow step 2.1 above. E.g rocks with moss and bare rocks
    2. If not, follow step 2.2 above the highest set of two categories that it does not fit. E.g In my street and my neighbours garden
  3. Reiterate step 3..
Animated GIF version of the construction process (5 second per slide, 4 slides). Click to view, if it does not move

Objections: Why not put the bare and moss covered rocks distinction under the category of big rocks or small rocks? Answer: In many situations, this choice is a reflection of the respondent’s priorities. For example, they might answer: “My garden is influenced by a Japanese gardening aesthetic, where rocks are as important as plants, and moss covered rocks are seen as having more interest than bare rocks

Caveats: The tree structure shown above is incomplete, not all items (not shown here) have yet been introduced and placed within the tree. So it would premature to think that the respondent saw more difference between rocks than between plants, at this stage

Question: Can you use the same distinction in different parts of the tree? For example, big versus small bare rocks. or bare round rocks and bare square rocks. Answer: Yes. Even “small round rocks which are very small, versus small round rocks which are just small”

Here is a tree structure I developed following this approach. Numbers on the left at the ID numbers of storylines generated during a ParEvo exercise. Alternatively, the text of MSC stories could be the subject of the same kind of sorting exercise.

Software

Online accessible software will be developed that enables individuals and groups to carry out multiple HCS exercises and to explore the knowledge structures generated by those exercises. Figure 15 outlines the proposed overall workflow. If you are interested to be part of the development and early testing of this software, then email: rick.davies@gmail.com

Figure 15: Workflow for HCS online

PS 2021 08 13: it looks like there may be an easy shortcut to achieve a workable online collaborative platform for HCS exercises. At least the sorting stage in the above flow chart. I am currently exploring the features available in Miro: An Online Whiteboard & Visual Collaboration Platform.  Steps involved:

      1. Lists all items (N)
      2. Create number of columns needed to fully sort all of these items (Square root of N)
      3. Place all items to be sorted in column A
      4. Participant sort, and move items into two groups in column C
        1. Insert text describing each group, in column A, adjacent to respective piles in column B
      5. Repeat step 4, until all items are fully sorted
Figure 16: Example of a HCS structure on an online whiteboard. View online here
Validating HCS results

Is it possible to validate a HCS analysis using one-to-one interviews of people representing the individual entities within the sorted set? What sort of interview questions would be useful to ask individuals? Two possibilities could be tested:

Unstructured sorting

      • What is the MSD between you and many others in this set?
        • Set sorted into 2 piles, on of which includes respondent, and 2 piles named and described
          • Focus in on the pile the respondent is a member of , ask  same question again:
            • Reiterate as afar as possible: Developing a more and more specific identity
  1.  Structured sorting
      • Which of these differences (  from previous HCS) represent the most significant difference between yourself and many others in this group?
        • Set sorted into 2 piles, on of which includes respondent, and 2 piles named and described
          • Focus in on the pile the respondent is a member of , ask  same question again:
            • Reiterate as afar as possible: Developing a more and more specific identity.

Similar methods 

Divisive hierarchical card sorting

Courtesy of Joachim Harloff, I have come across Sorting Data: Collection and Analysis by A.P.M. Coxon, published by Sage in 1999.  In this useful book Coxon (page 26) refers to “divisive hierarchical card sorting” and the fact that it was first documented by James S Boster, a cognitive anthropologist,  in his 1986 journal article ” Can Individuals Recapitulate the Evolutionary Development of Color Lexicons? Ethnology 25(1):61-74. There he was looking for lack of difference in color perception across disparate communities as possible evidence of non-cultural determinants of that perception. The question Boster asked respondents was slightly different to the HCS question described above, it focused on similarities rather than differences: “I would like you to do is sort these colors into two groups on the basis of which colors you think are most similar to each other” (page 64). Another more important difference was that there was no follow up question asking “why do you think that difference was most significant”. Perhaps for two reasons. Firstly,  because there was no need, the respondents were being asked about differences in color, independent of any context that could give one color more significance than another. Secondly, there was no theory of information (such as Bateson’s) imbedded in the method.  Boster subsequently wrote a paper on “the successive pile sorts” in 1994, as did others in the same period (Wong, 1991), but the method described there appears to be a more complex process of both agglomeration and differentiation of a variable number of starter piles.

There are many different ways of doing sorting exercises. To find out more see also the more recent “How to Sort” by Harloff and Coxon, 2007, and their Method of Sorting website

Q-sort

in 2021 Bob Williams asked me ” … how does it differ from Q Sort?”

Q-sort, or Q-methodology is described by Wikipedia as… a research method used in psychology and in social sciences to study people’s “subjectivity”—that is, their viewpoint. Q was developed by psychologist William Stephenson. It has been used both in clinical settings for assessing a patient’s progress over time (intra-rater comparison), as well as in research settings to examine how people think about a topic (inter-rater comparisons)”

The differences as I see them:

1. Q-sort works with statements, whereas HCS can work with just about any entity. I have asked people to do HCS with names of people, organisations, geographic locations, villages, project activities, etc. I even did one years ago in Burkina Faso, where I asked a small group of vegetable growers about the most significant differences between the different parts of a large vegetable garden they collectively managed. One thing I remember was that they distinguished between old and new areas of ground in the garden, because the older areas had been cultivated for longer and were now easier to dig compared to the newer areas. Being a desk-wallah, the relevance of that distinction would never have occurred to me

2. Q-sort asks people to rank items i.e. statements, whereas HCS just ask people to sort things into 2 piles. I think the latter is less cognitively demanding (but can still be very informative).

3. Q-sort asks for rankings that adhere to a particular type of distribution (normal, I think), whereas with HCS, the two piles can be of any size.

4. It appears that in a Q-sort ranking the facilitator determines the broad criteria for ranking. Wikipedia: “a subject might be given statements like “He is a deeply religious man” and “He is a liar,” and asked to sort them from “most like how I think about this celebrity” to “least like how I think about this celebrity.” whereas with HCS no guidance is given on the type of differences that people should focus on. Though they could be, I suppose.

5. Q-sort involves (but may not require?) factor analysis of the sort results, says Wikipedia. But there is no statistical analysis associated with HCS. (Though you can do simple analysis of the tree structures that are developed and where items are within that structure)

6. The tree structure generated by a HCS can be used by participants for simple planning and evaluation purposes (as described above, on this web page) , whereas I don’t think the Q-sort results are so readily usable by the participants

Treejack

Treejack is the name of an online tool developed by Optimal Sort, which allows web designers to test the usability of website navigation structures, which are usually in the form of a tree structure. Respondents are asked to find different kinds of items by navigating their way through a labelled tree structure. The data generated by multiple respondents’s search behavior is then made visible, using various data visualisation methods.

Why mention it here? Because it could be a useful way of testing how people make use of a HCS tree structure once developed. The only difference between website and HCS tree structures is that junctions in the former often offer more than two alternative branches.

The Treejack  metrics include:

    • First click: The first branch they followed
    • Last click: Their chosen locations
    • Path descriptions: Exactly as followed by each respondent
    • Success: Did the respondent correctly identify the location of the mentioned item?
    • Direct: Did the respondent need to backtrack at any stage in order to identify the correct location
    • Time taken: To find the correct location

extra references

Some extra resources on card/pile sorting and related methods:

Susan Lowes on Pile Sorts
Susan Lowes on Pile Sorts

 

2 thoughts on “Hierarchical Card Sorting (HCS)”

Comments?

This site uses Akismet to reduce spam. Learn how your comment data is processed.