## Prediction Matrices

Update December 2014: This page is now a subsidiary section of the page on Predictive Models

Purpose: Prediction Matrices are for building and testing complex predictions

Suitability: Where interventions take place in multiple settings in parallel; where there is some variation in the ways those interventions are implemented across the different settings; and where there is some variation in the nature of local settings where the interventions take place. For example, a maternal health improvement project implemented by District Health Offices in different districts across Indonesia

The core idea: A useful prediction about large scale changes can be built up out of many small micro-judgements, using relative rather than absolute judgements

Caveat Emptor: This method was developed for use in Indonesia a few years ago, but never tested out in practice

### Introduction

The Prediction Matrix is a relatively simple tool for developing and testing predictions of how different events are expected to lead to a particular outcome. This can be useful at two points in time

• When retrospectively trying to assess how much various project activities have already contributed to a known outcome
• When planning a package of activities that are expected to contribute to a desired outcome (which has a specific target)

The Prediction Matrix does this by developing a model of how a project works. This can then be used to generate a predicted outcome, which can then be compared to a known (or planned) outcome. If the predicted outcome fits the known outcome, the model can be said to be working, because it fits well with reality. If it does not fit well, then this signals to us that we have to revise our thinking about what causes the outcomes. This will have implications for evaluation findings (if used retrospectively) and the contents of plans for future activities (if used prospectively)

The Prediction Matrix has its limitations, which will be described. But it does have some advantages over some other simpler alternatives, such as a one-to-one cross-tabulation between an expected cause (present and absent) and a known outcome (present and absent). The problem with simple cross tabulations is that they leave out the possibility that the outcome may be the result of multiple causes, including causes a project has no control over.

The Prediction Matrix can be produced using Excel, projected onto a screen in a workshop setting. An example matrix format is shown below. It should be referred to, step by step, when reading instructions below on how to construct a model, and its predictions about outcomes.

### Process for constructing the model

1. Identify the outcome that is of interest. For example, in the IMHEI project in Indonesa, this was the percentage of deliveries assisted by trained health staff. This was recognised as a proxy measure of improved maternal health.

2. Identify the locations where data is available on this outcome. Many locations are better than few. So if data is available at district as well as province level, break it down to district level. In the IMHEI project this data was available for four provinces.

• List the locations in row 10 (insert more if needed)

• Insert the data on known outcomes, for each of these locations in row 28

NB: If the Prediction Matrix is being used for planning purposes, the outcome data could be the  levels of coverage (in % terms)  expected by the end of the plan period

3. Identify the potential causes of differences these outcomes across the districts, including those causes the project can influence and others it cant influence

• List these in column labelled “Expected causes of the outcome“, in rows 11 to 20 (and add more rows if needed)

• Convert any negatively stated causes into positives E.g. from “geographic isolation”, to “proximity to regional capital”. So that all causes have the same direction of influence (i.e. helping to improve maternal health)

4. You may recognise that not all causes are equally important. Some might be expected to have a much bigger overall effect than others. You can build this view in to the model by allocating 100 “cause” points down the column on the left of the list of causes. Place many points in row for the cause you think will have a big overall effect, across the project as a whole. Place few points in the row for the cause you will have a relatively small overall impact, across the project as a whole. Make sure all listed causes have some points, or remove the cause that you don’t want to give any points to. Make sure all the points allocated add up to 100 (look at the Check Sum row at the bottom, row 22)

5. Now look at each cause, in turn. Look across the locations in the same row and identify where it is expected to have a big effect, a small effect, or no effect at all. Use 100 “cause” points to indicate where the effects are expected to be. Place many points in the location cell where a big effect is expected. Place few points where a little effect is expected; place no points where no effect is expected. Place equal points in all cells, if the effect is expected to be equal in all locations. But make sure the total number of points in the row = 100 (see Check Sum column on the far right).

6. When assigning points to each cause in each location, make use of all available and relevant information (statistics, reports, staff observations) that has any merit. It may be useful to make a note of which of these sources were used. Use the Right Click>Inert Comment function in Excel to record these for any cell

7. Go through the same process again, with each of the other each expected causes, working your way down each of the rows of causes.

8. As you do this, a cumulative point score will appear in row 24, for each location. The cells values will signify the predicted relative impact of all the causes on each location. Each cell value here = the sum of  (the value in each “district” cell above (created in step 5), multiplied by the % “cause” points you have already given to the cause in that row (created in step 4)). You can see the exact formula in this Excel file, by placing the cursor on one of the row 24 cells

9. Look at the graph that is shown below the matrix. The graph shows the relationship between two sets of figures in the model

• The predicted impact scores in row 25

• The known outcomes, in row 28

10. Also shown below in row 31, is a correlation figure, showing how well these two sets of figure correlate with each other 0.99 is a very high correlation, 0.11 is a very low correlation. I should state here that the example shown here is an imagined one. In practice a correlation of 0.94 is probably very unlikely.

11. If the two sets of figures are highly correlated, the model is fitting well with reality. If there is a weak or non-existent correlation, it has a poor fit

12. If the model does not fit with reality, then the cell values and the weightings of each cause can be changed, to produce a better fit. BUT this should be done carefully. In principle the choice of all cell values (which are the participant’s judgements) need to be accountable. That is, it should be possible to explain to non-participants why those values have been chosen, when compared to others in the same row. This is where the inserted evidence comments, mentioned above, will be useful.

Suggestion: When collecting the micro-judgements on the “cause” points to be allocated across the causes (step 4) and across the locations (step 5 – 7)  it would be best to obscure rows 24 and below, to prevent any emerging macro level trends from influencing the micro-judgements. Rows 24 and below could be revealed when all micro-judgements have been completed.

### Commentary on the method

The Prediction Matrix is making use of subjective judgements and interpretations, but at the price of requiring those judgements to be transparent and accountable. So, if cell values are changed, to improve the fit of the model with reality, then the reasons for those changes need to be clearly explained.

Behind the design of the Prediction Matrix are some important assumptions:

Two assumptions are related to large scale programs:

1. In large scale programs most outcomes of concern have multiple causes

2. The combination of causes that leads to a specific outcome is often context specific. They vary location to location

Two assumptions relate to how to build good models:

1. The more detailed a model is, the more vulnerable it is to disproof. Vulnerability to disproof is desirable, over time it should lead to improvement in the model. The models produced by the Prediction Matrix has two dimensions of detail:

• The number of causes (more are better)

• The number of locations where those causes may be present (more are better)

2. The more transparent a model is, the more vulnerable it is to disproof. Two aspects of the Prediction Matrix are transparent:

• The importance weightings given to each cause

• The relative impact weightings given in each location

What is not yet transparent in the Excel version of the Prediction Matrix, but which could be (via inserted Comments) are:

• The reasons given for different weightings to the causes

• The reasons given for different impact weightings

The limitations of the Prediction Matrix are:

• Explanations given for different cell values may not be based on very clear or substantial arguments and evidence

o This means that choices of cell values should be discussed and debated as much as possible, and well documented. And then exposed to external scrutiny. This is why it best to develop the Prediction Matrix in a workshop setting.

• A good fit between predicted and actual outcomes could be achieved by more than one set of cell values in the matrix. There may be more than one “solution”

o If this is found to be the case, in a particular real life application, then the important question is which of these sets of cell values can be best explained by the available evidence and argument.

When assessing the value of the Prediction Matrix it should be compared to other tools available or usable in the same context for the same purpose, not against an ideal standard that no one can meet.

## Relationship to other methods

### 1. Realist Evaluation

The structure of a Prediction Matrix can be related to Pawson and Tilley’s concept of Context-Mechanism-Outcome configurations, in their school of Realist Evaluation. The Contexts are the district locations and values given in their cells to what could be called the mediating variables listed in rows 16 to 20. The Mechanism are the interventions (independent variables) listed in rows 11 to 15, and the values given to their cells in each location. The expected Outcome is in row 24.

When I shared  a description of the Prediction Matrix with Nick Tilley in 2006 he commented: “ I enjoyed this. It looks a useful tool. I like the corrigibility [i.e ability to be adjusted and improved]. I can see the fit with what david and I were saying.On a realist front I guess what might emerge are not underlying causal mechanisms but flags for them.

### 2. Qualitative Comparative Analysis (QCA)

This family of methods was developed by Charles Ragin. This method also involves looking a relatively small number of cases and how differences in their attributes relate to differences in observed outcomes. In contrast to the Prediction Matrix QCA matrices simply indicate the presence or absence of an attribute (via a 0 or 1), not its relative importance (via the ranking value). And instead of showing all locations as seperate entries, locations or incidences which have the same attributes are collapsed into one entry, with an additional attribute describing its frequency of occurence.  The process of then identifying the relationship between these different configurations and the presence/absence of the observed outcomes also differs. Through a process of comparison, facilitated by software, one or more combinations of attributes are found which can predict the observed outcomes.

PS: In Using Qualitative Comparative Analysis – (QCA) and Fuzzy Sets. Peer C. Fiss says “QCA is not useful in very small-N situations (e.g. less than 12 cases)” These are the circumstances where ranking is possible. Wendy Olsen says QCA is best for cases between 9 and 200

PS: Fuzzy Set QCA allows cases to have a degree of an attribute, not just an attribute or not.

## Consultation Draft: “Better information: better aid” Accra, August 2008

Produced by aidinfo. aidinfo is an initiative to contribute to faster poverty reduction by making aid more
transparent.

This is a draft for consultation that summarises the evidence we have gathered so far. We welcome suggestions, additions, comments and corrections.
Continue reading “Consultation Draft: “Better information: better aid” Accra, August 2008 “

## INVITATION: Building the Evidence to Reduce Poverty – launch of the public consultation on DFID’s new Evaluation Policy

Date: Tuesday 9th December, 2.30pm
Venue: Department for International Development, Room 3W11, 1 Palace Street, London, SW1E 5HE

Chair: Sue Owen, Director General of Corporate Performance, DFID. With presentations from David Peretz, Chair of the Independent Advisory Committee on Development Impact; Nick York, Head of Evaluation Department, DFID

RSVP Kirsty Burns, Evaluation Department, Venue: kirsty-burns@dfid.gov.uk, 01355 84 3602, by Friday 5th December 2008

Background notes

Development is about achieving results that make a difference for the poor in their daily lives.
Evaluation is a key instrument both to inform decision makers and to hold DFID to account for its choices and actions.

The Independent Advisory Committee on Development Impact (IACDI) was established in December 2007, with members selected for their international development and evaluation expertise. Its formation was an important step forward towards strengthening evaluation for DFID. It demonstrated that the UK Government is committed to independent, open, and transparent scrutiny of its development assistance.

The new policy comes at the end of the first year of IACDI’s oversight of DFID’s evaluation work.
It is vital that we also draw on the views of our delivery partners across the world, and this is why the draft policy, along with a proposed list of topics to focus evaluation on over the next three years, is being put out for public consultation.

This event marks the launch of the external consultation process, which will be open for 12 weeks. DFID will launch its final policy in March.

You and your organisation are invited to take part in the consultation process, beginning with this event. There you will have an opportunity to put questions to David Peretz, the Chair of the Independent Advisory Committee on Development Impact, as well as Sue Owen, DFID’s Director General for Corporate Performance and Nick York, DFID’s Head of Evaluation.

Please let us know promptly if you plan to attend or if a colleague will attend in your place. Names need to be provided to DFID security staff to ensure admission.

Further details will then be sent to those joining the event closer to the time.

## Participatory Impact Assessment: a Guide for Practitioners

The Feinstein International Center has been developing and adapting participatory approaches to measure the impact of livelihoods based interventions since the early nineties. Drawing upon this experience, the guide aims to provide practitioners with a broad framework for carrying out project level Participatory Impact Assessments (PIA) of livelihoods interventions in the humanitarian sector. Other than in some health, nutrition, and water interventions in which indicators of project performance should relate to international standards, for many interventions there are no ‘gold standards’ for measuring project impact. This guide aims to bridge this gap by outlining a tried and tested approach to measuring the impact of livelihoods projects. The tools in the guide have been field tested over the past two years in a major research effort, funded by the Bill & Melinda Gates Foundation and involving five major humanitarian NGOs working across Africa.

## Impact assessment: Drivers, dilemmas and deliberations

Prepared for Sightsavers International by Jennifer Chapman & Antonella Mancini
jenny.chapman@tiscali.co.uk antonella.mancini@blueyonder.co.uk 9 pages 10th April 2008

“This paper investigates key debates and issues around impact assessment and performance measurement for UK development NGOs. It was originally written for Sightsavers to stimulate debate and thinking among staff, Board and senior management team. This version has been amended to be relevant for a wider NGO audience. It is based on the authors’ many years experience, reading of key documents and 11 interviews with informants selected because they are inluential in these debates and/or they have first hand experience of trying to implement impact assessment or performance measurement systems within NGOs. The paper has been put together in a relatively short period of time and does not claim to be based on rigorous research.”

## Glossary of Key Terms in Evaluation and Results Based Management

(via Xceval email list)

We are pleased to inform you that the Arabic version of the DAC Evaluation Network’s “Glossary of Key Terms in Evaluation and Results Based Management,” has been released. The glossary is now available in thirteen languages! The multilingual glossary serves to promote shared understandings and facilitate joint work in evaluation. The strong demand for new versions of the Glossary is an indication of its relevance for DAC members and other development partners around the world. The Arabic Glossary was produced in collaboration with the Islamic Development Bank and the African Development Bank.

You can find a video link in English and Arabic presenting the new glossary on our website. The interviews were held at the recent launch event at the African Development Bank. The Islamic Development Bank will make an official launch with the Arab co-ordination group later in the month of June.

The Secretariat

## Monday Developments issue on NGO accountability

(via Niels Keijzer on the Pelikan email list)

The December 2007 issue of Monday Developments, a monthly magazine
States), explores key accountability issues for NGOs. Through various
angles, the issue looks into “(…) the conflicts organizations face with
scarce resources, demanding missions and the need to evaluate progress and
effectiveness.”

The articles include views on the topic from development donors, the
Humanitarian Accountability Project, the importance of listening for
accountability, implications for evaluation standards and practice,
downward accountability, …

http://www.interaction.org/files.cgi/6117_MDDec2007.pdf

## The Social Framework as an alternative to the Logical Framework

Caveat: This post describes a proposal by Rick Davies that is still a work in progress, being tested to see how well it works and where it works. Your comments and suggestions are welcome. Please use the Comment box at the end of this posting.

### A Social Framework…

• is a format for describing an expected pathway of influence through a wider network of people, groups or organisations.
• is a Logical Framework re-designed as if people and their relationships mattered
• is a way of summarizing the theory-of-change within a development project, in a form that can be monitored and evaluated. And which can be easily explained to others.

The Social Framework uses the idea of pathways as a bridging concept, which can connect up two very different ways of thinking about development projects. One is the Logical Framework, which provides a very linear view of development, where events happen in a sequence of  steps, in one direction. The other is a network view of development, where change can be taking place simultaneously, in many different locations, in the relationships between many different actors.

### The basic idea

The diagram below shows a number of actors, connected by relationships. It is a simple network diagram, that can be drawn using Excel or social network analysis software. The thick blue line shows a particular pathway through that network that is of interest. What is expected to happen along that pathway can be described in detail using a Social Framework format, which itself is an adaptation of the Logical Framework.

In the table below each of the actors that are on the blue pathway above have been given a row, where the expected changes in that actor’s behavior can be described in detail. The rows of actors are in the same sequence as the chain of actors in the diagram above.

Unlike the Logical Framework, there can be as many rows in the tables as needed, depending on how long the chain of actors is along the pathway. The concept of a chain of relationships has some similarity with the concepts of  Value Chains and Supply Chains

### Other differences between the Social and Logical Framework

The two frameworks appear very similar in that both describe an intended process of change as a series of events taking place across a sequence of rows. Starting at the bottom and going upwards. But there are important differences…

#### 1. Time versus people

In the Logical Framework this vertical dimension represents the flow of time, starting from the present at the bottom and moving to the future at the top. This flow is broken into different stages, represented by each row. The types of events taking place in each of these rows are given different names, typically: Activities, Outputs, Purpose (or outcome) and Goal (or impact). One of the challenges facing users of the Logical Framework is agreeing on where events should be placed within which categories. For example, as activities or outputs, or as outcomes or impacts. Communicating the difference between these categories to non-specialists is even more of a challenge.

In the Social Framework the vertical dimension represents a chain of actors connected by their relationships. Actors can be individuals, organisations or groups, or larger categories of organisations or groups. This choice depends on the scale of the event that needs to be described by the Social Framework. In the Social Framework, the relationships between actors are the means by which change happens. In the Logical Framework change is often described in more abstract terms.

#### 2. Length and direction of influence

Unlike the four rows in the Logical Framework, this chain can be as long or short as is needed. Unlike the Logical Framework causation is likely to work in both directions, up and down the chain of relationships. Actors influence others, and they are also influenced by those others.

#### 3. Using the traditional four columns

Both the Social and Logical Framework involve the use of four columns: a narrative description of the expected change, observable indicators of those changes (OVIs), sources of information on those indicators (MoVs), and assumptions about those changes’ relationships to wider events. The Social Framework design has deliberately kept, but adapted, these elements of the Logical Framework.

The narrative column describes the expected changes in the actors (and their relationships with each other). In the Logical Framework the narrative description is expected to be written in a depersonalised passive voice. In contrast,  the actor-centred description in the Social Framework will make it much easier to understand and communicate, the “storyline”.

The MoV column does not simply say where the necessary information (about the expected changes) can be found, but also who will know about these changes. Information needs to be known about by someone to be of any use. Information that exists but is not known to anyone is in effect useless.

The assumptions column describes what other relationships will also be important, because their actions (or inaction) may affect what happens to the actor in each row of the Social Framework. It is important to remember that most Social Frameworks will describe a chain of actors forming a pathway through a wider and more complex network of relationships. For example, in the table above the Assumptions section of the row describing the National NGO should say something about what is expected of their relationships with bilateral agencies and INGOs, which they also have relationships with (shown by the orange links in the diagram).

In a Social Framework there is still a connecting logic between the different rows, as there is in the Logical Framework. However, it is a social logic, with this type of form:

If the National NGO is able to provide technical advice on advocacy strategies to the Local CSOs…

And INGOs continue to fund those Local CSOs for at least three years…

Then the Local CSOs will be able to engage more effectively with the National Government

#### 4. Distributed accountability

One of the potential benefits of the Social Framework is that because there are change objectives for each actor in a pathway, responsibility for the whole chain functioning as intended is distributed amongst all the actors in that pathway. In Logical Framework descriptions of projects, responsibility for success often seems to lay almost solely with one organisation, usually that one closest to the intended beneficiaries. For more on this idea, see my blog posting on distributed accountability in the Katine project in Uganda

### Potential complications

#### 1. Multiple pathways

In a given project setting there may be more than one pathway. In the diagram below it is quite likely that the National NGO is communicating with the INGOs and with some Bilateral Agencies. The INGOs might be interacting with the Companies as well as Local CSOs. Where these parallel strategies are an important part of the overall project design these auxiliary pathways could be documented in supporting Social Frameworks. Their existence could be referred to in the appropriate Assumptions cell of the central Social Framework.

#### 2. Multiple views of how someone should change

For any given actor in a chain of relationships there may be different views about how they should change (e.g. they will have their own view, and so will others in immediate relationships with them). How do you reconcile these different views?

If the Social Framework was developed through a participatory process then these differences should be expected to arise during that process and may be resolved. It should be relatively easy to design a Social Framework by participatory means because each stakeholder should be able to see where they fit into the picture, either directly as an actor in the pathway, or indirectly via an Assumptions statement in one or more of the rows.

If the Social Framework was developed to reflect the views of one stakeholder, then their conflicting view of how another actor needs to change may limit their ability to affect change down a given pathway.  Or, on discovering that there is a difference in views, they may then try to persuade the other to change in the way they think is needed, and end up being successful. This might be the case with the Local CSOs relationship with National Government, in the simple diagrammatic view shown above.

#### 3. How do you fit short term, medium term and long term changes in Social Framework?

For a given actor there may be different objectives (expected changes) for different time periods. In the short term they may be to be able to do x, in the medium term they may hope to be to do y and in the long term they may want to be to do z.  Multiple objectives can be listed, in time order, within each actor’s own row. Similarly with the indicators for each of these in the next column.

Changes in the short versus long term can also be captured by describing multiple pathways, some of which are important in the early stages of a project and some which are more important later on, during and after the project ends.

#### How does all this relate to Outcome Mapping?

I am not an advocate of Outcome Mapping, but there is an overlap in approach with the actor focused structure of the Social Framework. Elsewhere, I have written a one-pager looking at the similarities and differences between Outcome Mapping and Network Models (which Social Frameworks relate to).

If people are using Outcome Mapping but also want something like the Logical Framework to summarise the project intentions (and theory of change) then the following interpretations might be useful:

• In a Social Framework, adjacent actors are each others’ Boundary Partners. Other actors mentioned in the Assumptions column of a given actor might also become their Boundary Partners.
• Outcome Challenges are the expected changes to be described in the first column, for each actor in the Social Framework
• Progress markers could be listed in the Indicators column, for the respective Boundary Partner
• Strategy maps could be described using a network diagram similar to the one used immediately above. Each pathway would need to be highlighted, including their relative importance.

### Postscript

1. I have just been re-reading the new DFID “Guidance on using the revised Logical Framework”  On page 9 there is a graphic illustrating an “example of a Results Chain and how it aligns with the logframe format” It interested me because it is a good example of yet another disembodied theory of change, where changes happen but there are no identifiable actors present (except children at Purpose level). This must make the process of monitoring and evaluation more difficult (even in this simple example) and make communication of the project design to others more difficult also.

2. 18th May 2011: Recently in the course of other work I came across this OECD DAC definition of impact: “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.” Interpreted literally this focus on the long-term could mean that any changes during a project implementation period, commonly three years, would not be considered to be a kind of impact. This could be a bit of a problem in projects aimed at reducing infant or child mortality by improving the quality of health service provision. In the case of maternal and infant health projects, it would be commonly expected that some reductions in the numbers of cases of child and maternal deaths would occur within a project’s lifespan. In fact, the sooner these kinds of changes happened the better.

However, if impact was defined in a different way then kind of this perverse anomaly would not arise. Impact could be defined in terms of social rather than temporal distance, as changes in the lives of people of ultimate concern. In this case, it is mothers and infants, who are at the end of one or more chains of actors, through which aid resources and their effects flow. The Social Framework uses this concept of social distance, whereas the vertical structure of the Logical Framework uses either temporal distance or a conflation of social and temporal distance.

3. 28th October 2011. In February this year Louise Shaxson & Ben Clench, of the Delta Partnership, published a 10 page Working Paper titled “Outcome mapping and social frameworks: tools for designing, delivering and monitoring policy via distributed partnerships” “Partnership working is becoming increasingly important in the policymaking process: no more so than in the UK with the coalition government’s ‘Big Society’ agenda. But the problem is not limited to the UK: the international search for hybrid forms of governance takes on a new urgency as we move towards an era of light touch regulation, small government, and localism. This paper describes two tools which will help policymakers take a rigorous approach to designing, delivering and monitoring policymaking in the face of these complex issues.”

4. 14th April 2016. Since writing the above postings I have had two related ideas. The first is that I have for a long time been ambivalent about what should be described at each level in the Social Framework (as described in the tabular format above) – should it be an actor or a relationship between two actors, or both (i.e. row 1 describes actor A, row 2 describes relationship between actors A & B, row 3 describes actor B, etc). Events described in rows that refer to specific actors (if they are organisations)  are essentially happening in a more micro-level set of relationships i.e. within the organisation concerned)

The other idea relates to how you would map the network structure that contains the pathways and surrounding relationships we are interested in. I now think this could be done using something called a greedy algorithm, whereby the exploration of one actor’s ego network would lead on to the exploration of the ego network of the most important actor in the first ego network, and so on. The most important relationship in each ego network (and thus connecting ego networks) would be the causal pathway we would be interested in. In network analysis terms this would be known as a “maximal spanning tree”. Of course, what constituted “most important actor” would have been to be defined in advance of what would in effect be a participatory network mapping exercise.

.

## ParEvo – a web assisted participatory scenario planning process

…is to record some ongoing reflections on my experience of running two pre-tests of ParEvo carried out in late 2018 and early 2019.

### Two pre-tests are underway

• One involves 11 participants developing a scenario involving the establishment of an MSC (Most Significant Change) process in a development programme in Nigeria. These volunteers were found via the MSC email list. They came from 7 countries and 64% were women.
• The other involves 11 participants developing a Brexit scenario following Britain failing to reach an agreement with the EU by March 2019. These participants were found via the MandE NEWS email list. They came from 9 countries and 46% were women.

For more background (especially if you have not been participating) see this 2008 post on the process design and this 2019 Conference abstract talking about these pre-tests

### Reflections so far

Issues arising…

1. How many participants should there be?
• In the current pre-tests, I have limited the number to around 10. My concern is that with larger numbers there will be too many story segments (and their storylines) for people to scan and make a single preferred selection. But improved methods of visualising the text contributions may help overcome this limitation. Another option is to allow/encourage individual participants to represent teams of people, e.g. different stakeholder groups. I have not yet tried this out.
2. Do the same participants need to be involved in each iteration of the process?
1. My initial concern is that not doing so would make some of the follow up quantitative analysis more difficult, but I am not so concerned about that now, its a manageable problem. On the other hand, it is likely that some people will have to drop out mid-process, and ideally, they could be replaced by others, thus maintaining the diversity of storylines.
3. How do you select an appropriate topic for a scenario planning exercise?
1. Ideally, it would be a topic that was of interest to all the participants and one which they felt some confidence in talking about, even if only in terms of imagined futures. One pre-test topic, the use of MSC in Nigeria, was within these bounds. But the other was more debatable: the fate of the UK after no resolution of BREXIT terms by 29th March 2019
4. How should you solicit responses from participants?
1. I started by sending a standard email to all the (MSC scenario) participants, but this has been cumbersome and has risks. It is too easy to lose track of who contributed what text, to add to what existing storyline. I am now using two-part single question survey via SurveyMonkey. This enables me to keep a mistake-free record of who contributed what to what, and who has responded and who has not. But this still involves sending multiple communications, including reminders, and I have sometimes confused what I am sending to whom.  A more automated systems is definitely needed.
5. How should you represent and share participants responses?
1. This has been done in two forms. One is a tree diagram, showing all storylines, where participants can mouseover nodes to immediately see each text segment. Or they can click on each node to go to a separate web page and see complete storylines. These are both laborious to construct, but hopefully will soon be simplified and automated via some tech support which is now under discussion. PS: I have now resorted to only using the tree diagram with mouseover.
6. Should all contributions be anonymous?
1. There are two types of contributions: (a) the storyline segments contributed during each iteration of the process, (b) Comments made on these contributions, that can be enabled on the blog page that hosts each full storyline to date. This second type was an afterthought, whereas the first is central to the process.
2. The first process of contributing to storylines designed to make authorship anonymous, so people would focus on the contents.  I think this remains a good feature.
3. The second process of allowing people to comment has pros and cons. The advantage is that it can enrich the discussion process, providing a meta-level to the main discussion which is the storyline development. The risk, however, is that if the comments are not enabled to be anonymous then a careful reader of the comments can sometimes work out who made which storyline contributions. I have tried to make comments anonymous but they still seem to reveal the identity of the person making the comment. This may be resolvable. PS: This option is now not available, while I am only using the tree diagram to show storylines. This may need to be changed.
7. How many iterations should be completed?
1. It has been suggested that participants should know this in advance, so that their story segments don’t leap in the future too quickly, or the reverse, progress the story too slowly. With the Brexit scenario pre-test I am inclined to agree. It might help to saying at the beginning that there will be 5 iterations, ending in the year 2025. With the MSC scenario pre-test I am less certain, it seems to be moving on at a pace I would not have predicted
2. I am now thinking it may also be useful to spell out in advance the number of iterations that will take place. And perhaps even suggest each one will represent a given increment in time, say a month or a year, or…
8. What limits should there be on the length of the text that participants submit?
1. I have really wobbled on this issue, ranging from 100-word limits to 50-word limits to no voiced limits at all. Perhaps when people select which storyline to continue the length of the previous contributions will be something they take into account? I would like to hear participants views on this issue. Should there be word limits, and if so, what sort of limit?
9. What sort of editorial intervention should there be by the facilitator, if any?
1. I have been tempted, more than once, to ask some participants to reword and revise their contribution. I now limit myself to very basic spelling corrections, checked with the participant, if necessary. I was worried that some participants have a limited grasp of the scenario topic, but now think that just has to be part of the reality, some people have little to go on when anticipating specific the future, and others may have “completely the wrong idea”, according to others. As the facilitator, I now think I need to stand back and let things run.
2. Another thought I had some time ago is that the facilitator could act as the spokesperson for “the wider context”, including any actors not represented by any of the participant’s contributions so far. At the beginning of a new iteration, they could provide some contextual text that participants are encouraged to bear in mind when designing their next contribution. If so, how / where should this context information be presented?
10. How long should a complete exercise take?
1. The current pre-tests are stretching out over a number of weeks. But I think this will be an exception. In a workshop setting where all participants (or teams of) have access to a laptop and internet, it should be possible to move through a quite a few iterations within a couple of hours. In other non-workshop settings perhaps a week will be long enough, if all participants have a stake in the process. Compacting the available time might generate more concentration and focus. The web app now under development should also radically reduce the turnaround time between iterations because manual work done by the facilitator will be automated.
11. Is my aim to have participants evaluate the completed storylines realistic?
1. After the last iteration, I plan to ask each participant, probably via an online survey page, to identify: (a) the most desirable storyline, (b) the most likely to happen storyline. But I am not sure if this will work. Will participants be willing to read every storyline from beginning to end? Or will they make judgments on the basis of the last addition to each storyline, which they will be more familiar with? And how much will this bias their judgments (and how could I identify if it does)?