Evidence of the Hawthorne effect – worth knowing about and watching out for

Posted on 16 October, 2014 – 2:30 PM

(copied from the World Bank “Development Impact” blog)

Quantifying the Hawthorne Effect

Submitted by Jed Friedman on 2014/10/16 This post is co-authored with Brinda Gokul

Many who work on impact evaluation are familiar with the concept of the Hawthorne effect and its potential risk to the accurate inference of causal impact. But if this is a new concept, let’s quickly review the definition and history of the Hawthorne effect:

  • The Hawthorne effect refers to study participants’ alteration of behavior solely as a result of being observed (rather than as a result of the intervention). Hence for the effect to exist it is necessary for the subjects to realize they are under observation. The term originates from the Western Electric Company’s Hawthorne Works Plant in Chicago where, in the late 1920s and early 1930s, researchers tried to study the effects of altered workplace lighting on worker productivity. It turned out that worker productivity improved when the lighting was increased, but also improved when the lighting was dimmed. Indeed it became apparent that whenever a change was implemented, such as a change in work hours, productivity improved for a period of time. The conclusion: productivity was not being affected by the changes in workplace conditions but instead by the self-knowledge of workers that they were under observation.

So the Hawthorne effect may present a challenge to the validity of causal inference (when agents respond to the knowledge they are being studied rather than respond to the changed environment as a result of the intervention) or may present a challenge to the accuracy of measurement (when the fact of observation alters the behavior measured). Clearly any effect magnitude, and indeed whether the effect arises at all, depends on the study context including the type of behavior observed. Yet only a handful of studies have attempted to identify and measure the Hawthorne effect.

My colleague Brinda Gokul and I recently reviewed the health economics and public health literature that explicitly study the effect in the general field of health. This is a hard question to get at, but some inventive studies, at times utilizing new technology, have given us some fascinating results. (It’s a fairly nascent literature, and at the bottom of this post we list the papers that we have found.)

With respect to behavior of health providers in developing countries, one of the more extensive studies in understanding the Hawthorne effect was conducted in the Arusha region of Tanzania and resulted in a series of papers by Kenneth Leonard and Melkiory Masatu. The challenge in measuring the Hawthorne effect is that we need to also have “stealth” data on subject behaviors when they are not aware of observation. The Tanzania’s study trick was to use patient recall interviews, conducted soon after the clinic visit, to reconstruct the actions of the clinicians and specifically their adherence to proper medical protocols. This stealth data is then compared with what is recorded by trained enumerators when they observe the clinician treating patients.

Of course the first step is to validate the accuracy of the “stealth” patient recall data, which the investigators do by comparing the enumerator observation record with patient recall data for the patient visits that were explicitly observed. It turns out there is a high degree of concurrence, with agreement between observer and patient on approximately 70% of the items measured.

Prior to the arrival of the research team, patient recall measured an average 53% baseline adherence to medical protocol by health providers, after the arrival of the research team, this adherence increased by almost 10 percentage points (or 20%). And it appears that the adherence to protocol actually matters, as cases that have higher adherence also have higher rates of accurate diagnosis and higher patient satisfaction. Of note is that the Hawthorne effect was rather short lived – providers returned to baseline level of adherence after 10-15 observed patient visits. I’ve been told of this same effect by education researchers who say that teachers habituate quite rapidly to video observation and return to “normal” levels of practice in within a day or so after the introduction of the camera.

Another example is from the Indian state of Orissa where investigators evaluating sanitation efforts to increase latrine use and reduce open defecation took a technological approach to the collection of stealth data: they developed a small hidden device to monitor latrine activity that recorded the times when the latrine door was opened. They called this device a PLUM – passive latrine use monitor. Amongst households with a latrine, the PLUM recorded members of the household using the latrine on average 2.11 times in the morning and 0.53 times in the afternoon.

Once this baseline data was collected, the study induced the Hawthorne effect by sending an observer to stand outside selected latrines for a five hour period in order to validate the passive monitor. It turned out that when a known observer was in place, household latrine compliance increased in the morning from 2.11 times to 2.71 and in the afternoon from 0.53 to 1.03 times. (Clasen, 2012).

For several healthcare studies in Europe, Canada and the United States, results also identified the Hawthorne effect. Here is a selection of findings:

  • By contrasting hand sanitation practices recorded discretely through a real-time location system (RLTS) – which recorded all dispensations from hand sanitizer stations as well as the movement of explicit hygiene auditors in an acute care hospital – researchers identified the Hawthorne effect in an acute care research hospital. The researchers recorded a rate of 3.75 dispensations per hour when the sanitizer station was visible to the auditor compared to 1.48 dispensations from a station not currently under observation. In contrast, there were only 1.07 dispensations in the week before observer arrival.
  • Another study of hand sanitation practices contrasted antiseptic hand rub (AHR) use among medical personal in five intensive care units both in the presence and absence of an announced observer of sanitary practice. When the official observer was absent, a trained covert observer – a research nurse part of the medical team – also recorded AHR use. The Hawthorne effect had a marked impact on AHR compliance: 29% baseline compliance and 45% compliance in the overt period. (Ruden)
  • The Hawthorne effect was induced in paramedic reporting practices solely by written notification of an upcoming quality-improvement audit, thus indicating that the effect does not always require direct observation or feedback. In this case, post audit announcement, the rate of incomplete paramedic reports declined by 20%. This finding also suggests, of course, a behavioral mechanism behind many Hawthorne effects – the perceived demand for performance. (Campbell)

Many of these reviewed studies look at small samples and are relatively short-term. So persistence of the observed effect is an important open question as well as the interaction between observation and the complexity of the behavior studied.

Here is the list of work attempting to quantify the Hawthorne effect that we have found for the health related field – please add to it if you know of others (in any field) – we’d be very grateful.

Some References

Campbell, JP, VA Maxey, WA Watson. “Hawthorne Effect: Implications for Pre-hospital Research.” Annals of Emergency Medicine, 26.5 (1995): 590-94.

Clasen T, Fabini D, Boisson S, Taneja J, Song J, Aichinger E, Bui A, Dadashi S, Schmidt W, Burt Z, Nelson K. “Making Sanitation Count: Developing and Testing a Device for Assessing Latrine Use in Low-Income Settings.” Environmental Science & Technology 46.6 (2012): 3295-3303.

De Amici, D, C Klersy, F Ramajoli, L Brustia, and P Politi. “Impact of the Hawthorne Effect in a Longitudinal Clinical Study: The Case of Anesthesia.” Controlled Clinical Trials 21 (2000): 103-14.

Eckmanns T, Bessert J, Behnke M, Gastmeier P, Ruden H. “Compliance With Antiseptic Hand Rub Use In Intensive Care Units: The Hawthorne Effect. Infection Control and Hospital Epidemiology”, 27 (2006): 931-934.

PH Feil, JS Grauer, CC Gadbury-Amyot, K Kula, MD McCunniff, “Intentional use of the Hawthorne effect to improve oral hygiene compliance in orthodontic patients,” Journal of Dental Education, 66 (2002): 1129-1135.

Grol, RP, WH Verstappen, T van der Weijden, G Riet. “Block Design Allowed For Control Of The Hawthorne Effect In A Randomized Controlled Trial Of Test Ordering.” Journal of Clinical Epidemiology, 57 (2004): 1119-1123.

Kohli E, Ptak J, Smith R, et al. “Variability in the Hawthorne effect with regard to hand hygiene practices: independent advantages of overt and covert observers.” PloS ONE, 8 (2013):353746

Leonard, KL. “Is patient satisfaction sensitive to changes in the quality of care? An exploitation of the Hawthorne effect.” Journal of Health Economics, 27 (2008): 444-459.

Leonard, KL, and MC Masatu. “Outpatient Process Quality Evaluation and the Hawthorne Effect.” Social Science & Medicine 63 (2006): 2330-340.

Leonard, KL, and MC Masatu. “Using the Hawthorne Effect to Examine the Gap between a Doctor’s Best Possible Practice and Actual Performance.” Journal of Development Economics 93.2 (2010): 226-34.

McCarney, R, J Warner, S Iliffe, R van Haselen, M Griffin, P Fisher, “The Hawthorne Effect: a Randomised Controlled Trial.” BMC Medical Research Methodology 7 (2007):  30

McGlynn, EA, R Mangione-Smith, M Elliott, & L McDonald. “An Observational Study of Antibiotic Prescribing Behavior and the Hawthorne Effect.” Health Services Research, 37 (2002), 1603-1623.

Fernald, DH, L Coombs, L DeAlleaume, D West, B Parnes. “An Assessment of the Hawthorne Effect in Practice-based Research.” The Journal of the American Board of Family Medicine, 25 (2012): 83-86.

Srigley, J, C Furness, G. Baker, M Gardam. “Quantification of the Hawthorne effect in hand hygiene compliance monitoring using an electronic monitoring system: A retrospective cohort study.” The International Journal of Healthcare Improvement, 10 (2014): 1-7.

VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Why evaluations fail: The importance of good monitoring (DCED, 2014)

Posted on 17 September, 2014 – 5:04 PM

Adam Kessler and Jim Tanburn, August 2014, Donor Committee for Enterprise Development (DCED). 9 pages. Available as pdf.

Introduction:  A development programme without a strong internal monitoring system often cannot be effectively evaluated. The DCED Standard for Results Measurement is a widely-used monitoring framework, and this document discusses how it relates to external evaluations. Why should evaluators be interested in monitoring systems? How can the DCED Standard support evaluations, and vice versa? Who is responsible for what, and what are the expectations of each? This document expands previous work by the UK Department for International Development (DFID).

This document is relevant for evaluators, those commissioning evaluations, and practitioners in  programmes using the DCED Standard and undergoing an evaluation. It provides a basis for dialogue with the evaluation community; the aims of that dialogue are to identify sources of evaluation expertise
available to support programmes using the DCED Standard, and to promote the Standard to programmes needing to improve their monitoring system. We would welcome further discussions on the topic, and invite you to contact us at Results@Enterprise-Development.org with any questions or comments.

Contents
1 Introduction
2 Why should evaluators be interested in monitoring?
2.1 Good monitoring is essential for effective management
2.2 Good monitoring is essential for effective evaluation
2.3 Some evaluation methodologies incorporate monitoring
3 What is the DCED Standard for Results Measurement?
4 How does the DCED Standard support evaluation?
4.1 The DCED Standard promotes clear theories of change
4.2 The DCED Standard provides additional data to test the theory of change
5 How do evaluations supplement the DCED Standard?
5.1 Evaluations are independent
5.2 Evaluations have more expertise and larger budgets
5.3 Evaluations can examine broader effects
5.4 Evaluations and the DCED Standard are for different audiences
6 Division of responsibilities between evaluator and programme team
7 Key References and further reading

VN:F [1.9.22_1171]
Rating: 0 (from 2 votes)

DCED publications on M&E, M&E audits and the effectiveness of M&E standards

Posted on 17 September, 2014 – 10:47 AM

This posting is overdue. The Donor Committee for Enterprise Development (DCED) has been producing a lot of material on results management this year. Here are some of the items I have seen.

Of particular interest to me is the DCED Standard for Results Measurement. According to Jim Tanburn, there are about 60-70 programmes now using the standard. Associated with this is an auditing service offered by DCED. From what I can see nine programmes have been audited so far. Given the scale and complexity of the standards, the question in my mind, and probably that of others, is whether their use makes a significant difference to the performance of the programmes that have implemented the standards.Are they cost-effective?

This would not be an easy question to answer in any rigorous fashion, I suspect. There are likely to be many case-specific accounts of where and how the standards have helped improve performance, and perhaps some of where they have have not helped or even hindered. Some accounts are already available via the Voices from the Practitioners part of the DCED website.

The challenge would be how to aggregate judgements about impacts on a diverse range of programmes in a variety of settings. This is the sort of situation where one is looking for the “effects of a cause”, rather than “the causes of an effect”, because there is a standard intervention (adoption of the standards) but it is one which may have many different effects. A three step process might be feasible, or at least worth exploring:

1. Rank programmes in terms of the degree to which they have successfully adopted the standards. This should be relatively easy, given that there is a standard auditing process

2. Rank programmes in terms of the relative observed/reported effects of the standards. This will be much more difficult because of the apple and pears  nature of the impacts. But I have been exploring a way of doing so here: Pair comparisons: For where there is no common outcome measure? Another difficulty, which may be surmountable, is that “all the audit reports remain confidential and DCED will not share the contents of the audit report without seeking permission from the audited programmes”.

3. Look for the strength and direction of the correlation between the two measures, and for outliers (poor adoption/big effects, good adoption/few effects) where lessons could be learned.

 

VN:F [1.9.22_1171]
Rating: +1 (from 3 votes)

Reframing the evidence debates: a view from the media for development sector

Posted on 9 August, 2014 – 5:48 PM

Abraham-Dowsing, Kavita, Anna Godfrey, and Zoe Khor. 2014. “Reframing the Evidence Debates: A View from the Media for Development Sector”. BBC Media Action. Available as pdf. This is part of BBC Media Action’s Bridging Theory and Practice series. An accompanying appendices document is available here. It includes priority research questions, and more detail on the evidence examples cited in the paper. The report was prepared with funding from the UK Department for International Development.

Introduction : “Donors, policy-makers and practitioners need evidence  to inform their policy and programming choices, resource allocation and spending decisions, yet producing and making use of high-quality research and evidence is not straightforward. This is particularly the case in sectors that do not have a long history of research or evaluation, that are operating in fragile states with low research capacity and that are trying to bring about complex change. The media for development sector (see Box 1) is one such example. Nonetheless, donors, governments and private foundations working in international development have long recognised the importance of independent media and information sources in their work and the role that communication can play in bringing about change. Despite this recognition, however, in debates around evidence on the role of media and communication in achieving development communication in achieving development outcomes, assertions of “no evidence” or “not enough evidence” are commonplace. With the evidence agenda gaining more prominence in the development sector, there is a risk for any sector that finds it difficult to have a clear, concise and cohesive narrative around its evidence of impact.

This paper is based on a series of interviews with practitioners, evaluators and donors working in the media for development sector, and looks at their understanding of what counts as evidence and their views on the existing evidence base. It argues that compelling evidence of impact does exist and is being used – although this varies by thematic area. For example, it highlights that evidence in the area of health communication is stronger and more integrated into practice compared with other thematic areas such as media and governance or  humanitarian response outcomes. The paper also contends that, alongside evidencing development outcomes (for example, media’s impact on knowledge, attitudes, efficacy, norms and behaviours), more evidence is needed to answer specific questions about how, why and in what ways media and communication affect people and societies – and how this varies by local context.

The paper argues that the lack of clear evidential standards for reporting evidence from media for development programmes, the limited efforts to date to collate and systematically review the evidence that does exist, and the lack of relevant fora in which to critique and understand evaluation findings, are significant barriers to evidence generation. The paper calls for an “evidence agenda”, which creates shared evidential standards to systematically map the existing evidence, establishes fora to discuss and share existing evidence, and uses strategic, longer-term collaborative investment in evaluation to highlight where evidence gaps need to be filled in order to build the evidence base. Without such an agenda, as a field, we risk evidence producers, assessors and funders talking at cross purposes. ”

As the paper’s conclusion states, we actively welcome conversations with you and we expect that these will affect and change the focus of the evidence agenda. We also expect to be challenged! What we have tried to do here is articulate a clear starting point, highlighting the risk of not taking this conversation forward. We actively welcome your feedback during our consultation on the paper which runs from August until the end of October 2014, and invite you to share the paper and appendices widely with any colleagues and networks who you think appropriate.

Contents page
Introduction
1. What is evidence? An expert view
2. Evidence – what counts and where are there gaps?
3. Building an evidence base – points for consideration
4. The challenges of building an evidence base
5. An evidence agenda – next steps in taking this conversation forward
Conclusion
Appendix 1: Methodology and contributors
Appendix 2: Examples of compelling evidence
Appendix 3: Priority research questions for the evidence agenda
Appendix 4: A note on M&E, R & L and DME
Appendix 5: Mixed methods evaluation evidence – farmer field schools
Appendix 6: Methodological challenges

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

CDI conference proceedings: Improving the use of M&E processes and findings

Posted on 17 July, 2014 – 12:12 PM

“On the 20th and 21st of March 2014 CDI organized her annual ‘M&E on the cutting edge’ conference on the topic: ‘Improving the Use of M&E Processes and Findings’.

This conference is part of our series of yearly ‘M&E on the cutting edge’ events. The conference was held on the 20th and 21st of March 2014 in Wageningen, the Netherlands. This conference particularly looked at under what conditions the use of M&E processes and findings can be improved. The conference report can now be accessed here in pdf format

Conference participants had the opportunity to learn about:

  • frameworks to understand utilisation of monitoring and evaluation findings and process;
  • different types of utilisation of monitoring and evaluation process and findings, when and for whom these are relevant;
  • conditions that improve utilisation of monitoring and evaluation processes and findings.

Conference presentations can be found online here: http://www.managingforimpact.org/event/cdi-conference-improving-use-me-processes-and-findings

VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Gender, Monitoring, Evaluation and Learning – 9 new articles in pdfs

Posted on 10 July, 2014 – 10:27 AM
…in Gender & Development Volume 22, Issue 2, July 2014   Gender, Monitoring, Evaluation and Learning
“In this issue of G&D, we examine the topic of Gender, Monitoring, Evaluation and Learning (MEL) from a gender equality and women’s rights perspective, and hope to prove that a good MEL system is an activist’s best friend! This unique collection of articles captures the knowledge of a range of development practitioners and women’s rights activists, who write about a variety of organisational approaches to MEL. Contributors come from both the global South and the global North and have tried to share their experience accessibly, making what is often very complex and technical material as clear as possible to non-MEL specialists.”

Contents

The links below will take you to the article abstract on the Oxfam Policy & Practice website, from where you can download the article for free.

Editorial

Introduction to Gender, Monitoring, Evaluation and Learning
Kimberly Bowman and Caroline Sweetman

Articles

Women’s Empowerment Impact Measurement Initiative
Nidal Karim, Mary Picard, Sarah Gillingham and Leah Berkowitz
A review of approaches and methods to measure economic empowerment of women
and girls
Paola Pereznieto and Georgia Taylor

Helen Lindley

Capturing changes in women’s lives: the experiences of Oxfam Canada in applying
feminist evaluation principles to monitoring and evaluation practice
Carol Miller and Laura Haylock
A survivor behind every number: using programme data on violence against women
and girls in the Democratic Republic of Congo to influence policy and practice
Marie-France Guimond and Katie Robinette

Learning about women’s empowerment in the context of development projects: do the figures tell us enough?

Jane Carter, Sarah Byrne, Kai Schrader, Humayun Kabir, Zenebe Bashaw
Uraguchi, Bhanu Pandit, Badri Manandhar, Merita Barileva, Norbert Pijls & Pascal Fendrich

Resources

Compiled by Liz Cooke

Resources List – Gender, Monitoring, Evaluation and Learning

 

VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)

Review of evaluation approaches and methods for interventions related to violence against women and girls (VAWG)

Posted on 9 July, 2014 – 3:57 PM

[From the R4D website] Available as pdf

Raab, M.; Stuppert, W. Review of evaluation approaches and methods for interventions related to violence against women and girls (VAWG). (2014) 123 pp.

Summary:

The purpose of this review is to generate a robust understanding of the strengths, weaknesses and appropriateness of evaluation approaches and methods in the field of development and humanitarian interventions on violence against women and girls (VAWG). It was commissioned by the Evaluation Department of the UK Department for International Development (DFID), with the goal of engaging policy makers, programme staff, evaluators, evaluation commissioners and other evaluation users in reflecting on ways to improve evaluations of VAWG programming. Better evaluations are expected to contribute to more successful programme design and implementation.

The review examines evaluations of interventions to prevent or reduce violence against women and girls within the contexts of development and humanitarian aid.

Rick Davies comment: This paper is of interest for two reasons: (a) The review process was the subject of a blog that documented its progress, from beginning to end. A limited number of comments were posted on the blog by interested observers (including myself) and these were responded to by the reviewers; (b) The review used Qualitative Comparative Analysis (QCA) as its means of understanding the relationship between attributes of evaluations in this area and their results. QCA is an interesting but demanding method even when applied on a modest scale.

I will post more comments here after taking the opportunity to read the review with some care.

The authors have also invited comments from anyone else who is interested, via their email address available in their report

Postscript 2014 10 08: At the EES 2014 at Dublin I made a presentation of the Triangulation of QCA findings, which included some of their data and analysis. You can see the presentation here on Youtube (it has attached audio). Michaela and Wolfgang have subsequently commented on that presentation and in turn I have responded to their comments

 

VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Incorporating people’s values in development: weighting alternatives

Posted on 7 July, 2014 – 10:44 PM

Laura Rodriguez Takeuchi, ODI Project Note 04, June 2014. Available as pdf

“Key messages:

  • In the measurement of multidimensional well-being, weights aim to capture the relative importance of each componentto a person’s overall  well-being. The choice of weights needsto be explicit and could be used to incorporate people’sperspectives into a final metric.
  • Stated preferences approaches aim to obtain weights from individuals’ responses to hypothetical scenarios. We outline six of these approaches. Understanding their design and limitations is vital to make sense of potentially dissimilar result.
  • It is important to select and test an appropriate method for specific contexts, considering the challenges of relying on people’s answers. Two methodologies, DCE and PTO, are put forward for testing in a pilot project.”

See also:Laura Rodriguez Takeuchi blog posting on Better Evaluation: Week 26: Weighing people’s values in evaluation

Rick Davies comment: Although this was a very interesting and useful paper overall, I was fascinated by this part of Laura’s paper ”

Reflecting on the psychology literature, Kahneman and Krueger (2006) argue that it is difficult to deduce preferences from people’s actual choices because of limited rationality:

“[People] make inconsistent choices, fail to learn from experience, exhibit reluctance to trade, base their own satisfaction on how their situation compares with the satisfaction of others and depart from the standard  model of the rational economic agent in other ways.”   (Kahneman and Krueger 2006: 3)

Rather than using these ‘real’ choices, stated preferences approaches rely on surveys to obtain weights from individuals’ responses to hypothetical scenarios.

This seems totally bizarre. What would happen if we insisted on all respondents’ survey responses being rational, and applied various other remedial measures to make them so!  Would we end up with a perfectely rational set of responses that have no actual fit with how people or behave in the world? How useful would that be? Perhaps this is what happens when you spend too much time in the company of economists? :-))

On another matter…. Table 1 usefully lists eight different weighting methods, which are explained in the text. However this list does not include one of the simplest methods that exists, and which is touched upon tangentially in the reference to the South African study on social perceptions of material needs (Wright, 2008). This is the use of weighted checklists, where respondents choose both items on a checklist and the weights to be given to each item, in a series of binary (yes/no) choices. This method was used in a series of household poverty surveys in Vietnam in 1997 and 2006 using an instrument called a Basic Necessities Survey. The wider potential uses of this participatory and democratic method are are discussed in a related blog on weighted checklists.

Postscript: Julie Newton has point out this useful related website:

  • Measuring National Well-being.  “ONS [UK Office for National Statistics] is developing new measures of national well-being. The aim is to provide a fuller picture of how society is doing by supplementing existing economic, social and environmental measures. Developing better measures of well-being is a long term programme. ONS  are committed to sharing ideas and proposals widely to ensure that the measures are relevant and founded on what matters to people.” Their home page lists a number of new publications on this subject
VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

Composite measures of local disaster impact – Lessons from Typhoon Yolanda, Philippines

Posted on 2 June, 2014 – 3:03 PM

by Aldo Benini, Patrice Chataigner, 2014. Available as pdf

Purpose:”When disaster strikes, determining affected areas and populations with the greatest unmet  needs is a key objective of rapid assessments. This note is concerned with the logic and scope for improvement in a particular tool, the so-called “prioritization matrix”, that has increasingly been employed in such assessments. We compare, and expand on, some variants that sprang up in the same environment of a large natural disaster. The fixed context lets us attribute differences to the creativity of the users grappling with the intrinsic nature of the tool, rather than to fleeting local circumstances. Our recommendations may thus be translated more easily to future assessments elsewhere.

The typhoon that struck the central Philippines in November 2013 – known as “Typhoon Yolanda” and also as “Typhoon Haiyan” – triggered a significant national and international relief response. Its information managers imported the practice, tried and tested in other disasters, of ranking affected communities by the degree of impact and need. Several lists, known as prioritization matrices, of ranked municipalities were produced in the first weeks of the response. Four of them, by different individuals and organizations, were shared with us. The largest in coverage ranked 497 municipalities.

The matrices are based on indicators, which they aggregate into an index that determines the ranks. Thus they come under the rubric of composite measures. They are managed in spreadsheets. We review the four for their particular emphases, the mechanics of combining indicators, and the statistical distributions of the final impact scores. Two major questions concern the use of rankings (as opposed to other transformations) and the condensation of all indicators in one combined index. We propose alternative formulations, in part borrowing from recent advances in social indicator research. We make recommendations on how to improve the process in future rapid assessments.”

Rick Davies comment: Well worth reading!

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)

How Shortcuts Cut Us Short: Cognitive Traps in Philanthropic Decision Making

Posted on 30 May, 2014 – 11:48 AM

Beer, Tanya, and Julia Coffman. 2014. “How Shortcuts Cut Us Short: Cognitive Traps in Philanthropic Decision Making”. Centre for Evaluation Innovation. Available as pdf

Found courtesy of “people-centered development” blog (michaela raab)

Introduction: “Anyone who tracks the popular business literature has come across at least one article or book, if not a half dozen, that applies the insights of cognitive science and behavioral economics to individual and organizational decision making.   These authors apply social science research to the question of why so many strategic decisions yield disappointing results, despite extensive research and planning and the availability of data about how strategies are (or are not) performing.  The diagnosis is that many of our decisions rely on mental shortcuts or “cognitive traps,” which can lead us to make uninformed or even bad decisions.   Shortcuts provide time-pressured staff with simple ways of making decisions and managing complex strategies that play  out an uncertain world. These shortcuts affect how we access information, what information  we pay attention to, what we learn, and whether and how we apply what we learn. Like all  organizations, foundations and the people who work in them are subject to these same traps.  Many foundations are attempting to make better decisions by investing in evaluation and other data collection efforts that support their strategic learning. The desire is to generate more timely and actionable data, and some foundations have even created staff positions dedicated entirely to supporting learning and the ongoing application of data for purposes of continuous improvement.  While this is a useful and positive trend, decades of research have shown that despite the best of intentions, and even when actionable data is presented at the right time, people do not automatically make good and rational decisions. Instead, we are hard-wired to fall into cognitive traps  that affect how we process (or ignore) information that could help us to make better judgments.”

Rick Davies comment: Recommended, along with the videosong by Mr Wray on cognitive bias, also available via Michaela’s blog

VN:F [1.9.22_1171]
Rating: +1 (from 1 vote)