The Ethnographic Explorer: Help Pages
Contents > More specific uses
| CONTENTS |
| 1. Configurational analyses Qualitative Comparative Analysis (QCA) is a method of causal analysis that seeks explanations in terms of the presence of configurations of multiple causal conditions (as distinct from isolated variables). Decision Tree classifiers, which are a form of supervised machine learning, also represent predictive models in the form of configurations of conditions. Each branch of Ethnographic Explorer sort structure can be considered as a particular configuration (i.e. combination of conditions), that describe the cases that fit within that branch, and which might have some predictive or causal significance. These models can be developed using the Sort and Compare stages, and then tested using the Contrast stage. In the Contrast stage the X axis can represent the predictive models, developed during the Sort and Compare stages. The Y axis values can represent the observed outcome status of each case. This data can come from a second Compare stage, which asked for assessments of outcomes, or from an imported set of independent measures (converted to rank values). The performance of these models can be viewed in multiple ways: (a) eyeballing the scatter plot distribution, (b) examining the rank correlation, (c) examining the Confusion Matrix and associated Classification Accuracy measure. The best model can be summarised in this form: "Cases which rank more than ...in terms of their expected performance can be expected to have an observed performance of more than ...rank". The best available model can be found by eyeballing the scatter plot and then adjusting the cut-off points of both axes, until the best available Classification Accuracy measure is found. |
| 2. Measurement rubrics A rubric is an annotated scale, designed to assess performance. The annotations are attached to each point in the scale, and describe what criteria must be met for that point in the scale to be applicable. Often a given point on a scale will require the meeting of multiple criteria e.g. Outstanding (4): Presentation was on topic, fulfilled time requirement, and had impeccable grammar. The presenter was engaging and familiar with the material. This has been described as holistic rubric. An analytic rubric will have separate but aggregatable scores for each of these criteria. The comparison of sorted cases on the basis of their observed performance can be used as a measurement rubric when applicable and applied to a comparable set of cases seen elsewhere. The criteria of performance take the form of a configuration, i.e a combination of the descriptions of differences seen along each branch of the tree. It could probably be described as a type of analytic rubric. Attributes near the root of the tree strcture have a higher weight or influence, than those near the branch ends, on the final ranking of any category of cases The advantage of a rubric constructed this way is that each point in the scale has an example case. It is not just an abstract and ungrounded definition. Niki Wood, of Chemonics UK, has pointed out that the same process could also be away of breaking out of the "OECD-DAC evaluation criteria" box, to discover other context-relevant evaluation criteria. |
| 3. Analysis of Value for Money Please see Julian King's postings on the subject of Value for Money criteria, to start with, here: https://www.julianking.co.nz/vfi/criteria/ The diagram below is especially useful for thinking about the different forms of Value for Money, as relationships between two measures. There are five different types of relationships visible in this diagram.  How could the Ethnographic Explorer be used in this context? At the Comparison stage two types of comparison questions can be asked, representing the two parts of any one of the five type of Value for Money relationship. For example: 1. Which of these two groups of cases involves the greater use of resources than the other? 2. Which of these two groups of cases has had a more successful outcome? Then aggregated rankings could then be used at the Contrast stage to generate a scatter plot showing their relationship Follow up questions at the Comparison stage would be obviously be useful. These could include more qualitative comparisons. For example: 1. What is the most significant difference in the way the resources have been used in this group, versus the other? 2. In what way has this group been more successful than the other? Comparisons can also be made about expected futures, not only past performance. For example: 1. In the next year, which of these two groups of cases do you think will need more resources than the other? 2. In the next year, which of these two groups of cases do you think will be more successful in achieving its expected outcomes? Along with supporting qualitative comparisons, such as: 1. In the next year, why do you think will this group of cases need more resources than the other? 2. In the next year, why do you think will this group of cases will be more successful in achieving its expected outcomes? |
| 4. Stakeholder analysis It is possible to use Hierarchical Card Sort as a means of doing a stakeholder analysis in a development project. This can initially be from the perspective from one observer, possibly an individual stakeholder, such as project manager. Firstly, a list of actors reflecting the maximum possible variety of stakeholders are identified. (Ethnographers refer to this as Free Listing) The process of inquiry then starts at the trunk, with the respondent being asked to identify “the most significant difference between all the stakeholders in the project”. Optionally, after a prefix saying “Bearing in mind the objectives of this project…” . This is then followed by a question about the difference that difference makes i.e. the actual or expected consequences. Then each of the two initial categories of stakeholder are progressively differentiated until all cases are located in a group of their own.The net result is a nested classification of all the listed stakeholders. Information is generated not only about the differences between types of stakeholders, but also about the consequences, past, present or future of those differences. |
| 5. Portfolio reviews Portfolios of projects are natural sets of cases amenable to a sorting, comparison and contrast form of analysis. They are entities having a common feature (e.g. all funded by X for purpose y) but also having some diversity, sometimes owing to the different time period and associated conditions) when they were funded. Whie there may be a theory of change for each project the theory of change for the portfolio as a whole will be a highly level summarising type document, which typically will not give much attention to differentiating the function of different types of projects within the portfolio. There will be identifiable staff responsible for the portfolio, who are expected to know about the contents of the portfolio. That knowledge will be a mix of formal, informal, tacit and explicit knowledge. The sorting stage is an opportunity to bring much of the more informal and tacit knowledge to the foreground, and to enable it to be shared between new and old staff, and to be questioned. The comparison stage is an opportunity to spell out the details of part and future strategies, by asking qualitative and quantitative comparison questions, about differences between groups of cases in terms of inputs, activities, outputs, outcomes and impacts. The contrast stage is an opportunity to evaluate plans relative to performance, by contrasting aggregated ranking judgements of expectations against observations. The Value for Money discussion above spells out some comparisons that can be made of this kind. A wider categorisation and listing of the range of possible questions can be found here. |
| 6. Analysis of Most Significant Change (MSC)
stories MSC stories can be treated as items to be sorted. The resulting tree structure will provide an overview of all the MSC stories, at multiple levels of detail. Binary exploration questions can then be asked, that make use of the same tree structure. Asking about, for example, how the respondent thinks an organisation should respond differently to each type of change described in the MSC stories. |