Known unknowns: How to communicate certainly in an uncertain world

“From the speed of global warming to the likelihood of developing cancer, we must grasp uncertainty to understand the world. Here’s how to know your unknowns” By Anne Marthe van der Bles, New Scientist, 3rd July 2019

https://www.newscientist.com/article/mg24332372-600-known-unknowns-how-to-communicate-certainly-in-an-uncertain-world/

The above reminds me of the philosophers’ demands in The Hitchhikers Guide to the Galaxy: “We demand rigidly defined areas of doubt and uncertainty!” The philosophers were representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons and they wanted the universe’s second-best computer (Deep Thought) turned off, because of a demarcation dispute/.  It turns out, according to the above paper, that their demands were not so unreasonable after all :-)

Navigation by Judgment: Why and When Top Down Management of Foreign Aid Doesn’t Work

Errors arising from too much or too little control can be seen or unseen. When control is too little, errors are more likely to be seen. People do things they should not have done. When control is too much, errors are likely to be unseen, people don’t do things they should have done. Given this asymmetry, and other things being equal, there is a bias towards too much control

Honig, Dan. 2018. Navigation by Judgment: Why and When Top Down Management of Foreign Aid Doesn’t Work. Oxford, New York: Oxford University Press.

Contents

Preface
Acknowledgments
Part I: The What, Why, and When of Navigation by Judgment
Chapter 1. Introduction – The Management of Foreign Aid
Chapter 2. When to Let Go: The Costs and Benefits of Navigation by Judgment
Chapter 3. Agents – Who Does the Judging?
Chapter 4. Authorizing Environments & the Perils of Legitimacy Seeking
Part II: How Does Navigation by Judgment Fare in Practice?
Chapter 5. How to Know What Works Better, When: Data, Methods, and Empirical Operationalization
Chapter 6. Journey Without Maps – Environmental Unpredictability and Navigation Strategy
Chapter 7. Tailoring Management to Suit the Task – Project Verifiability and Navigation Strategy
Part III: Implications
Chapter 8. Delegation and Control Revisited
Chapter 9. Conclusion – Implications for the Aid Industry & Beyond
Appendices
Appendix I: Data Collection
Appendix II: Additional Econometric Analysis
Bibliography

YouTube presentation by the author: https://www.youtube.com/watch?reload=9&v=bdjeoBFY9Ss

Snippet from video: Errors arising from too much or too little control can be seen or unseen. When control is too little, errors are more likely to be seen. People do things they should not have done. When control is too much, errors are likely to be unseen, people don’t do things they should have done. Given this asymmetry, and other things being equal, there is a bias towards too much control

Book review: By Duncan Green in his 2018 From Poverty to Power blog

Publishers blurb:

Foreign aid organizations collectively spend hundreds of billions of dollars annually, with mixed results. Part of the problem in these endeavors lies in their execution. When should foreign aid organizations empower actors on the front lines of delivery to guide aid interventions, and when should distant headquarters lead?

In Navigation by Judgment, Dan Honig argues that high-quality implementation of foreign aid programs often requires contextual information that cannot be seen by those in distant headquarters. Tight controls and a focus on reaching pre-set measurable targets often prevent front-line workers from using skill, local knowledge, and creativity to solve problems in ways that maximize the impact of foreign aid. Drawing on a novel database of over 14,000 discrete development projects across nine aid agencies and eight paired case studies of development projects, Honig concludes that aid agencies will often benefit from giving field agents the authority to use their own judgments to guide aid delivery. This “navigation by judgment” is particularly valuable when environments are unpredictable and when accomplishing an aid program’s goals is hard to accurately measure.

Highlighting a crucial obstacle for effective global aid, Navigation by Judgment shows that the management of aid projects matters for aid effectiveness

Reflecting the Past, Shaping the Future: Making AI Work for International Development

USAID, September 2018. 98 pages. Available as PDF

Rick Davies comment: A very good overview, balanced, informative, with examples. Worth reading from beginning to end.

Contents

Introduction
Roadmap: How to use this document
Machine learning: Where we are and where we might be going
• ML and AI: What are they?
• How ML works: The basics
• Applications in development
• Case study: Data-driven agronomy and machine learning
at the International Center for Tropical Agriculture
• Case study: Harambee Youth Employment Accelerator
Machine learning: What can go wrong?
• Invisible minorities
• Predicting the wrong thing
• Bundling assistance and surveillance
• Malicious use
• Uneven failures and why they matter
How people influence the design and use of ML tools
• Reviewing data: How it can make all the difference
• Model-building: Why the details matter
• Integrating into practice: It’s not just “Plug and Play”
Action suggestions: What development practitioners can do today
• Advocate for your problem
• Bring context to the fore
• Invest in relationships
• Critically assess ML tools
Looking forward: How to cultivate fair & inclusive ML for the future
Quick reference: Guiding questions
Appendix: Peering under the hood [ gives more details on specific machine learning algorithms]

See also the associated USAID blog posting and maybe also  How can machine learning and artificial intelligence be used in development interventions and impact evaluations?

 

 

Bayesian belief networks – Their use in humanitarian scenarios An invitation to explorers

By Aldo Benini. July 2018. Available here as a pdf

Summary

This is an invitation for humanitarian data analysts and others –  assessment, policy and advocacy specialists, response planners and grant writers – to enhance the reach and quality of scenarios by means of so-called Bayesian belief networks. Belief networks are a powerful technique for structuring scenarios in a qualitative as well as quantitative approach. Modern software, with elegant graphical user interfaces, makes for rapid learning, convenient drafting, effortless calculation and compelling presentation in workshops, reports and Web pages.

In recent years, scenario development in humanitarian analysis has grown. Until now, however, the community has hardly ever tried out belief networks, in contrast to the natural disaster and ecological communities. This note offers a small demonstration. We build a simple belief network using information currently (mid-July 2018) available on a recent violent crisis in Nigeria. We produce and discuss several possible scenarios for the next three months, computing probabilities of two humanitarian outcomes.

Figure 1: Belief network with probability bar charts (segment)

We conclude with reflections on the contributions of belief networks to humanitarian scenario building and elsewhere. While much speaks for this technique, the growth of competence, the uses in workshops and the interpretation of graphs and statistics need to be fostered cautiously, with consideration for the real-world complexity and for the doubts that stakeholders may harbor about quantitative approaches. This note is in its first draft. It needs to be revised, possibly by several authors, in order to connect to progress in humanitarian scenario methodologies, expert judgment and workshop didactics

RD Comment: See also the comment and links provided below by Simon Henderson on his experience (with IOD/PARC) of trialing the use of Bayesian belief networks

Representing Theories of Change: Technical Challenges with Evaluation Consequences

A CEDIL Inception Paper, by Rick Davies. August 2018.  A pdf copy is available here 

 

Abstract: This paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events rather than the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from Internet sources six structural problems are described along with their consequences for evaluation.

The paper then outlines a range of different ways of addressing these problems which could be used by programme designers, implementers and evaluators. The paper concludes with some caution speculating on why the design problems are so endemic but also pointing a way forward. Four strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.

Table of Contents

What is a theory of change?
What is the problem?
A summary of the problems….
And a word in defence….
Six possible ways forward
Why so little progress?
Implications for CEDIL and DFID
References

Postscript: Michael Bamberger’s 2018 07 13 comments on this paper

I think this is an extremely useful and well-documented paper.  Framing the discussion around the 6 problems, and the possible ways forward is a good way to organize the presentation.  The documentation and links that you present will be greatly appreciated, as well as the graphical illustrations of the different approaches.
Without getting into too much detail, the following are a few general thoughts on this very useful paper:
  1. A criticism of many TOCs is that they only describe how a program will achieve its intended objectives and they do not address th challenges of identifying and monitoring potential unintended and often undesired, outcomes (UOs)  While some UOs could not have been anticipated, many others could, and these should perhaps be built into the model.  For example, there is an extensive literature documenting negative consequences for women of political and economic empowerment, often including increased domestic violence.  So these could be built into the TOC, but in many cases they are not.
  2. Many, but certainly not all, TOCs do not adequately address the challenges of emergence the fact that the environment in which the program operates; the political and organizational arrangements; and the characteristics of the target population and how they respond to the program are all likely to change significantly during the life of the project.  Many TOCs implicitly assume that the project and its environment remain relatively stable throughout the project lifetime.  Of course, many of the models you describe do not assume a stable environment, but it might be useful to flag the challenges of emergence. Many agencies are starting to become interested in agile project management to address the emergence challenge.
  3. Given the increasing recognition that most evaluation approaches do not adequately address complexity, and the interest in complexity-responsive evaluation approaches, you might like to focus more directly on how TOCs can address complexity.  Complexity is, of course, implicit in much of your discussion, but it might b useful to highlight the term.
  4. Do you think it would be useful to include a section on how big data and data analytics can strengthen the ability to develop more sophisticated TOCs.  Many agencies may feel that many of the techniques you mention would not be feasible with the kinds of data they collect and their current analytical tools.
  5. Related to the previous point, it might be useful to include a brief discussion of how accessible the quite sophisticated methods that you discuss would be to many evaluation offices.  What kinds of expertise would be required?  where would the data come from? how much would it cost.  You don’t ned to go into too much detail but many readers would like guidance on which approaches are likely to be accessible to which kinds of agency.
  6. Your discussion of “Why so little progress?” is critical.  It is my impression that among the agencies with whom I have worked,  while many evaluations pay lip-service to TOC, the full potential of the approach is very often not utilized.  Often the TOC is constructed at the start of a project with major inputs from an external consultant.  The framework is then rarely consulted again until the final evaluation report is being written, and there are even fewer instances where it is regularly tested, updated and revised.  There are of course many exceptions, and I am sure experience may be different with other kinds of agencies.  However, I think that many implementing agencies (and many donors) have very limited expectations concerning what they hope TOC will contribute.  There is probably very little appetite among many implementing agencies (as opposed to a few funding agencies such as DFID) for more refined models.
  7. Among agencies where this is the case, it will be necessary to demonstrate the value-added of investing time and resources in more refined TOCs.  So it might be useful to expand the discussion of the very practical, as opposed to the broader theoretical, justifications for investing in the existing TOC.
  8. In addition to the above considerations, many evaluators tend to be quite conservative in their choice of methodologies and they are often reluctant to adopt new methodologies – particularly if these use approaches with which they are not familiar.  New approaches, such as some of those you describe can also be seen as threatening if they might undermine the status of the evaluation professional as expert in his/her field.

Computational Modelling of Public Policy: Reflections on Practice

Gilbert G, Ahrweiler P, Barbrook-Johnson P, et al. (2018) Computational Modelling of Public Policy: Reflections on Practice. Journal of Artificial Societies and Social Simulation 21: 1–14. pdf copy available

Abstract: Computational models are increasingly being used to assist in developing, implementing and evaluating public policy. This paper reports on the experience of the authors in designing and using computational models of public policy (‘policy models’, for short). The paper considers the role of computational models in policy making, and some of the challenges that need to be overcome if policy models are to make an effective contribution. It suggests that policy models can have an important place in the policy process because they could allow policy makers to experiment in a virtual world, and have many advantages compared with randomised control trials and policy pilots. The paper then summarises some general lessons that can be extracted from the authors’ experience with policy modelling. These general lessons include the observation that ofen the main benefit of designing and using a model is that it provides an understanding of the policy domain, rather than the numbers it generates; that care needs to be taken that models are designed at an appropriate level of abstraction; that although appropriate data for calibration and validation may sometimes be in short supply, modelling is ofen still valuable; that modelling collaboratively and involving a range of stakeholders from the outset increases the likelihood that the model will be used and will be fit for purpose; that attention needs to be paid to effective communication between modellers and stakeholders; and that modelling for public policy involves ethical issues that need careful consideration. The paper concludes that policy modelling will continue to grow in importance as a component of public policy making processes, but if its potential is to be fully realised, there will need to be a melding of the cultures of computational modelling and policy making.

Selected quotes: For these reasons, the ability to make ‘point predictions’, i.e. forecasts of specific values at a specific time in the future, is rarely possible. More possible is a prediction that some event will or will not take place, or qualitative statements about the type or direction of change of values. Understanding what sort of unexpected outcomes
can emerge and something of the nature of how these arise also helps design policies that can be responsive to unexpected outcomes when they do arise. It can be particularly helpful in changing environments to use the model to explore what might happen under a range of possible, but dfferent, potential futures – without any commitment about which of these may eventually transpire. Even more valuable is a finding that the model shows that certain outcomes could not be achieved given the assumptions of the model. An example of this is the use of a whole system energy model to develop scenarios that meet the decarbonisation goals set by the EU for 2050 (see, for example, RAENG 2015.)

Rick Davies comment: A concise and very informative summary with many useful references. Definitely worth reading! I like the big emphasis on the need for ongoing collaboration and communication between model developers and their clients and other model stakeholders However, I would have liked to see some discussion of the pros and cons of different approaches to modeling e.g. agent-based models vs Fuzzy Cognitive Mapping and other approaches. Not just examples of different modelling applications, useful as they were.

See also: Uprichard, E and Penn, A (2016) Dependency Models: A CECAN Evaluation and Policy Practice Note for policy analysts and evaluators. CECAN. Available at: https://www.cecan.ac.uk/sites/default/files/2018-01/EMMA%20PPN%20v1.0.pdf (accessed 6 June 2018).

Representing Theories of Change: Technical Challenges and Evaluation Consequences

 

CEDIL – Centre for Evaluation Lecture Series
The Centre of Excellence for Development Impact and Learning (CEDIL) and the Centre for Evaluation host a lecture series addressing methods and innovation in primary studies.

Watch the live-streamed lecture here

London School of Hygiene and Tropical Medicine. Lecture Two – Wednesday 30th May 2018 – Dr Rick Davies 12:45-14:00  Jerry Morris B, LSHTM 15-17 Tavistock Place, London, WC1H 9SH

“This lecture will summarise the main points of a paper of the same name. That paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events, rather the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from a range of Internet sources six structural problems are described, along with their consequences for evaluation. The paper then outlines six different ways of addressing these problems, which could be used by programme designers and by evaluators. These solutions range from simple to follow advice on designing more adequate diagrams, to the use of specialist software for the manipulation of much more complex static and dynamic network models. The paper concludes with some caution, speculating on why the design problems are so endemic but also pointing a way forward. Three strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper.”

The paper referred to in the lecture was commissioned by CEDIL and is now pending publication in a special issue of an evaluation journal

The Book of Why: The New Science of Cause and Effect

by Judea Pearl, Allen Lane, May 2018

Publisher blurb: “‘Correlation does not imply causation.’ This mantra was invoked by scientists for decades in order to avoid taking positions as to whether one thing caused another, such as smoking and cancer and carbon dioxide and global warming. But today, that taboo is dead. The causal revolution, sparked by world-renowned computer scientist Judea Pearl and his colleagues, has cut through a century of confusion and placed cause and effect on a firm scientific basis. Now, Pearl and science journalist Dana Mackenzie explain causal thinking to general readers for the first time, showing how it allows us to explore the world that is and the worlds that could have been. It is the essence of human and artificial intelligence. And just as Pearl’s discoveries have enabled machines to think better, The Book of Why explains how we can think better.

Introduction: Mind over data (pdf copy)

Chapter 1: The Ladder of Causation (pdf copy)

Reviews: None found yet, but they will be listed here when found

 

The surprising usefulness of simple measures: HappyOrNot terminals

as described in this very readable article by David Owen:
Customer Satisfaction at the Push of  a Button – HappyOrNot terminals look simple, but the information they gather is revelatory. New Yorker, 2 February 2018, pages 26-29

Read the full article here

Points of interest covered by the article include:

  1. What is so good about them
  2. Why they work so well
  3. Can people “game” the data that is collected
  4. The value of immediacy of data collection
  5. How value is added to data points by information about location and time
  6. Example of real life large scale applications
  7. What is the worse thing that could happen

Other articles on the same subject:

Rick Davies comment: I like the design of the simple experiment described in the first para of this article. Because the locations of the petrol stations were different, and thus not comparable, the managers swapped the “treatment” given to each station i.e the staff they thought were making a difference to the performance of these stations.

Searching for Success: A Mixed Methods Approach to Identifying and Examining Positive Outliers in Development Outcomes

by Caryn Peiffer and Rosita Armytage, April 2018, Development Leadership Program Research Paper 52. Available as pdf

Summary: Increasingly, development scholars and practitioners are reaching for exceptional examples of positive change to better understand how developmental progress occurs. These are often referred to as ‘positive outliers’, but also ‘positive deviants’ and ‘pockets of effectiveness’.
Studies in this literature promise to identify and examine positive developmental change occurring in otherwise poorly governed states. However, to identify success stories, such research largely relies on cases’ reputations, and, by doing so, overlooks cases that have not yet garnered a reputation for their developmental progress.

This paper presents a novel three-stage methodology for identifying and examining positive outlier cases that does not rely solely on reputations. It therefore promises to uncover ‘hidden’ cases of developmental progress as
well as those that have been recognised.

The utility of the methodology is demonstrated through its use in uncovering two country case studies in which surprising rates of bribery reduction occurred, though the methodology has much broader applicability. The advantage of the methodology is validated by the fact that, in both of the cases identified, the reductions in bribery that occurred were largely previously unrecognised.

Contents: 
Summary
Introduction 1
Literature review: How positive outliers are selected 2
Stage 1: Statistically identifying potential positive outliers in bribery reduction 3
Stage 2: Triangulating statistical data 6
Stage 3: In-country case study fieldwork 7
Promise realised: Uncovering hidden ‘positive outliers’ 8
Conclusion 9
References 11
Appendix: Excluded samples from pooled GCB dataset 13

Rick Davies comment: This is a paper that has been waiting to be published, one that unites a qual and quant approach to identifying AND understanding positive deviance / positive outliers [I do prefer the latter term, promoted by the authors of this paper]

The authors use regression analysis to identify statistical outliers, which is appropriate where numerical data is available.. Where the data is binary/categorical is possible to use other methods to identify such outliers. See this page on the use of the EvaLC3 Excel app to find positive outliers in binary data sets.