From: XCeval@yahoogroups.com On Behalf Of Jim Rugh
Sent: April 28, 2008 5:47 PM
To: XCEval listserv; MandE NEWS
Subject: [XCeval] Invitation to join a dedicated discussion forum on reconstructing baseline data
We realize that any evaluation that purports to be an “impact evaluation” needs to compare “before-and-after” (pre-test + post-test data) and “with-and-without” (the counterfactual – what would have happened without the intervention being evaluated). Yet in our experience the majority of evaluations conducted of development projects and programs do not have comparable baseline data, nor appropriate comparison (much less “control”) groups. Although the discussion of counterfactuals and pre-test + post-test comparisons frequently focuses on quantitative evaluations designs, the need to understand baseline conditions is equally important for qualitative evaluations. What can be done to strengthen evaluations in such cases? In other words, what can be done to reconstruct baseline and counterfactual data?
We (Jim Rugh and Michael Bamberger) are planning a follow-up volume to “RealWorld Evaluation: Working under budget, time, data and political constraints” (Sage Publications 2006). (More information can be found at www.RealWorldEvaluation.org.)
The provisional title of the follow-up workbook is “Capacity Development for RealWorld Evaluation” and this will provide practical tools, guidelines and learning exercises (both for self-instruction and trainers). The hard-copy book will be accompanied by a website with case studies, more complete examples of the techniques discussed in the book, and additional material for trainers.
For this purpose a new discussion forum is being launched within the framework of the proposed book and training materials. We are sending this invitation to persons who would like to be contributors to this discussion and collection of examples, to suggest/ provide material that could be used on the website and/or cited in the book. The source of all material would of course be fully acknowledged. We would also hope that the material would be helpful to trainers (including ourselves) who organize courses and workshops on evaluation.
If you would like to participate in this discussion and have experiences you are willing to share, you are invited to join this new forum by sending a blank e-message to RealWorldEval-subscribe@yahoogroups.com (RealWorldEval-subscribe@yahoogroups.com).
Below we provide more details about this initiative:
The discussion topic: Reconstructing Baseline and Counterfactual Data to Strengthen Impact Evaluations. What are the solutions? Are there practical examples from evaluations that have dealt with these constraints?
The very extensive and lively debates on the merits (and drawbacks) of randomized control trials and “rigorous” quasi-experimental designs diverts attention from the fact that few evaluators are ever in a position to use any of these “state-of-the-art” designs.
Although to the best of our knowledge no statistics are available, we would estimate that at least 75 percent of “impact evaluations” of development projects are conducted without access to any systematic information on the conditions of the project population before the start of the project or other intervention, and even fewer evaluations have access to baseline data on relevant comparison groups. In fact many bilateral and multi-lateral development agencies, UN organizations and NGOs have accepted, explicitly or implicitly, that their impact evaluations are not commissioned until late in the project cycle and do not have access to baseline data. When more rigorous evaluations are conducted this is often a special initiative, often in cooperation with other agencies, and largely in isolation from the large number of standard impact evaluations that continue to rely exclusively on end-of-project data without any way to directly measure change in outcome conditions within the target population, nor any comparative data on “normal” conditions, i.e. where this project was not operating.
The purpose of this discussion is to invite interested persons, especially those who have conducted evaluations in developing countries, to share the many different strategies, tools and techniques that you and your agencies have developed to “reconstruct” baseline data and to estimate the conditions of the project and comparison groups prior to the start of the project or program being evaluated. The questions that we would like to discuss include:
a. What are the consequences of assessing impacts without access to baseline data? Are more rigorous impact evaluations a luxury that only a few well-funded agencies can afford, or are there serious consequences when future operational and policy decisions are based exclusively on ex-post data?
b. What tools and techniques are available for reconstructing information on baseline conditions? We are especially interested in case studies where these techniques have been applied.
c. Are there cost-effective and operationally feasible ways for agencies to reconstruct baseline data, either as a standard operating procedure or for selected priority evaluations?
Some of the many approaches that participants in this discussion forum might like to discuss include (but are not limited to):
1. the creative identification and use of secondary data (including qualitative document analysis);
2. using monitoring data and the many types of administrative information that implementing agencies generate (this includes the important question of designing “evaluation-ready” programs);
3. recall by key informants, focus groups and other community and group interview techniques;
4. PRA and other participatory evaluation methods;
5. a wide range of other techniques used by qualitative researchers to reconstruct respondent’s memories and perceptions of the past.
Again, if you are interested in participating in this discussion, you are invited to join this new forum by sending a blank e-message to RealWorldEval-subscribe@yahoogroups.com . Alternatively, you could write directly to JimRugh@mindspring.com .
Jim Rugh (and Michael Bamberger)
Independent International Program Evaluator
JimRugh@MindSpring.com
Home/Office phone: +1-865-908-3133
Mobile phone: +1-865-696-0401
451 Rugh Ridge Way #1
Sevierville, TN 37876-1393 USA
Skype ID= JimRugh
www.RealWorldEvaluation.org
AEA (American Evaluation Association) representative to the
IOCE (www.InternationalEvaluation.com)
Remember: Think evaluatively!
