Monitoring and Evaluation NEWS

These are the topics covered by two papers I have come across today, courtesy of Peter Barbrook-Johnson, of Surrey University. Both papers provide good overviews of their respective fields.

Moon, K., Adams, V. M., Dickinson, H., Guerrero, A. M., Biggs, D., Craven, L., … Ross, H. (2019). Mental models for conservation research and practice. Conservation Letters, 1–11.

Abstract: Conservation practice requires an understanding of complex social-ecological processes of a system and the different meanings and values that people attach to them. Mental models research offers a suite of methods that can be used to reveal these understandings and how they might affect conservation outcomes. Mental models are representations in people’s minds of how parts of the world work. We seek to demonstrate their value to conservation and assist practitioners and researchers in navigating the choices of methods available to elicit them. We begin by explaining some of the dominant applications of mental models in conservation: revealing individual assumptions about a system, developing a stakeholder-based model of the system, and creating a shared pathway to conservation. We then provide a framework to “walk through” the stepwise decisions in mental models research, with a focus on diagram based methods. Finally, we discuss some of the limitations of mental models research and application that are important to consider. This work extends the use of mental models research in improving our ability to understand social-ecological systems, creating a powerful set of tools to inform and shape conservation initiatives.

PDF copy here

Voinov, A. (2018). Tools and methods in participatory modeling: Selecting the right tool for the job. Environmental Modelling and Software, 109, 232–255.

Abstract: Various tools and methods are used in participatory modelling, at di?erent stages of the process and for di?erent purposes. The diversity of tools and methods can create challenges for stakeholders and modelers when selecting the ones most appropriate for their projects. We o?er a systematic overview, assessment, and categorization of methods to assist modelers and stakeholders with their choices and decisions. Most available literature provides little justi?cation or information on the reasons for the use of particular methods or tools in a given study. In most of the cases, it seems that the prior experience and skills of the modelers had a dominant e?ect on the selection of the methods used. While we have not found any real evidence of this approach being wrong, we do think that putting more thought into the method selection process and choosing the most appropriate method for the project can produce better results. Based
on expert opinion and a survey of modelers engaged in participatory processes, we o?er practical guidelines to improve decisions about method selection at di?erent stages of the participatory modeling process

PDF copy here

A note for ACAPS, by Aldo Benini, A. (2018). PDF available at https://www.acaps.org/sites/acaps/files/resources/files/20180115_acaps_technical_note_subjective_measures_full_report.pdf

Purpose and motivation

This note seeks to sensitize analysts to the growing momentum of subjective methods and measures around, and eventually inside, the humanitarian field. It clarifies the nature of subjective measures and their place in humanitarian needs assessments. It weighs their strengths and challenges. It discusses, in considerable depth, a small number of instruments and methods that are ready, or have good potential, for humanitarian analysis.

Post World War II culture and society have seen an acceleration of subjectivity in all institutional realms, although at variable paces. The sciences responded with considerable lag. They have created new methodologies – “mixed methods” (quantitative and qualitative), “subjective measures”, self-assessments of all kinds – that claim an equal playing field with distant, mechanical objectivity. For the period 2000-2012, using the search term “subjective measure”, Google Scholar returns around 600 references per year; for the period 2013 – fall 2017, the figure quintuples to 3,000. Since 2012, the United Nations has been publishing the annual World Happiness Report; its first edition discusses validity and reliability of subjective measures at length.

Closer to the humanitarian domain, poverty measurement has increasingly appreciated subjective data. Humanitarian analysis is at the initial stages of feeling the change. Adding “AND humanitarian” to the above search term produces 8 references per year for the first period, and 40 for the second – a trickle, but undeniably an increase. Other searches confirm the intuition that something is happening below the surface; for instance, “mixed method  AND humanitarian” returns 110 per year in the first, and 640 in the second period – a growth similar to that of “subjective measures”.

Still in some quarters subjectivity remains suspect. Language matters. Some collaborations on subjective measures have preferred billing them as “experience-based measures”. Who doubts experience? It is good salesmanship, but we stay with “subjective” unless the official name of the measure contains “experience”.

What follows 

We proceed as follows: In the foundational part, we discuss the nature of, motivation for, and reservations against, subjective measures. We provide illustrations from poverty measurement and from food insecurity studies. In the second part, we present three tools – scales, vignettes and hypothetical questions – with generic pointers as well as with specific case studies. We conclude with recommendations and by noting instruments that we have not covered, but which are likely to grow more important in years to come

Rick Davies comment: High recommended!