On evaluation quality standards: A List

Posted on 26 September, 2011 – 12:42 PM

 

The beginnings of a list. Please suggest others by using the Comment facility below

Normative statements:

Standards for specific methods (and fields):

Meta-evaluations:

  • Are Sida Evaluations Good Enough?An Assessment of 34 Evaluation Reports” by Kim Forss, Evert Vedung, Stein Erik Kruse,Agnes Mwaiselage, Anna Nilsdotter, Sida Studies in Evaluation 2008:1  See especially Section 6: Conclusion, 6.1 Revisiting the Quality Questions, 6.2 Why are there Quality Problems with Evaluations?, 6.3 How can the Quality of Evaluations be Improved?, 6.4 Direction of Future Studies. RD Comment:  This study has annexes with empirical data on the quality attributes of  34 evaluation reports published in the Sida Evaluations series between 2003 and 2005. It BEGS a follow up study to see if/how these various quality ratings correlate in any way with the subsequent use of the evaluation reports. Could Sida pursuaded to do something like this?

Ethics focused

  • Australasian Evaluation Society

Journal articles

Checklists:

  • Evaluation checklists prepared by the Western Michegan University ,covering Evaluation Management, Evaluation Models, Evaluation Values and Criteria, Metaevaluation, Evaluation Capacity Building / Institutionalization, and Checklist Creation

Other lists:

VN:F [1.9.22_1171]
Rating: +2 (from 2 votes)

Tags: , , , Print This Post Print This Post

  1. 6 Responses to “On evaluation quality standards: A List”

  2. Very useful, but could I also suggest to include the Evaluation Norms and Standards prepared by the UN Evaluation Group and that guide the work of the various UN agencies? The can be downloaded at http://www.unevaluation.org/normsandstandards/index.jsp?doc_cat_source_id=4

    Juha

    VA:F [1.9.22_1171]
    Rating: 3.0/5 (1 vote cast)
    VA:F [1.9.22_1171]
    Rating: +2 (from 2 votes)

    By Juha Uitto on Sep 27, 2011

  3. USAID’s Evaluation Policy 2011 could be added to this list, as an example of donor-specific evaluation standards/practices.

    It is available at the following link:
    http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

    By Mehr on Sep 28, 2011

  4. Hello Rick, a fine resource thank you.

    1. Do Codes of Ethics fit here? I suspect so. The Australian Evaluation Society’s are here: http://www.aes.asn.au/about/ .

    2. Altman, DG, Schulz, KF, Moher, D, Egger, M, Davidoff, F, Elbourne, D, Gotzsche, PC & Lang, T 2001, ‘The revised CONSORT statement for reporting randomized trials: explanation and elaboration’, Annals of Internal Medicine, vol. 134, no. 8, pp. 663-94.
    Overwhelming evidence now indicates that the quality of reporting of randomized, controlled trials (RCTs) is less than optimal. Recent methodologic analyses indicate that inadequate reporting and design are associated with biased estimates of treatment effects. Such systematic error is seriously damaging to RCTs, which boast the elimination of systematic error as their primary hallmark. Systematic error in RCTs reflects poor science, and poor science threatens proper ethical standards. A group of scientists and editors developed the CONSORT (Con solidated S tandards o f R eporting T rials) statement to improve the quality of reporting of RCTs. The statement consists of a checklist and flow diagram that authors can use for reporting an RCT. Many leading medical journals and major international editorial groups have adopted the CONSORT statement. The CONSORT statement facilitates critical appraisal and interpretation of RCTs by providing guidance to authors about how to improve the reporting of their trials. This explanatory and elaboration document is intended to enhance the use, understanding, and dissemination of the CONSORT statement. The meaning and rationale for each checklist item are presented. For most items, at least one published example of good reporting and, where possible, references to relevant empirical studies are provided. Several examples of flow diagrams are included. The CONSORT statement, this explanatory and elaboration document, and the associated Web site ( http://www.consort-statement.org ) should be helpful resources to improve reporting of randomized trials. Throughout the text, terms marked with an asterisk are defined at end of text.

    3. Des Jarlais, DC, Lyles, C & Crepaz, N 2004, ‘Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement’, American Journal of Public Health, vol. 94, no. 3, pp. 361-6.
    Developing an evidence base for making public health decisions will require using data from evaluation studies with randomized and nonrandomized designs. Assessing individual studies and using studies in quantitative research syntheses require transparent reporting of the study, with sufficient detail and clarity to readily see differences and similarities among studies in the same area. The Consolidated Standards of Reporting Trials (CONSORT) statement provides guidelines for transparent reporting of randomized clinical trials. We present the initial version of the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement. These guidelines emphasize the reporting of theories used and descriptions of intervention and comparison conditions, research design, and methods of adjusting for possible biases in evaluation studies that use nonrandomized designs.

    Regards – David

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)

    By David McDonald on Sep 29, 2011

  5. This list is a very useful compendium of sources for evaluators and their stakeholders.

    However, users should be very careful to match the standards and criteria they emply with the life cycle stage of the program they are evaluating. The evaluative data needs are very different for programs in their early stages of development, or when the evaluation is to support the improvement of program delivery at a local level. I am drafting a publication on this topic, but it is not yet ready for distribution.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)

    By Mary Ann Scheirer, PhD on Oct 4, 2011

  6. Both a checklist and a large database, the Impact Reporting and Investment Standards for monitoring the impact of corporate social responsibility in rich and poor countries alike has grown a lot in importance for impact investors:

    http://iris.thegiin.org/

    The American Evaluation Association also has their own guidelines:

    http://www.eval.org/Publications/GuidingPrinciples.asp

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

    By Elizabeth on Oct 6, 2011

  7. I am happy to learn it but I don’t know any such list, sorry.

    VA:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

    By mindxstudio on Nov 23, 2011

Post a Comment