Findings and recommendations from the Evaluation Methods Advisory Panel at WFP

Introduction……………………………………………………………………………………………1
1. Approaches and methods……………………………………………………………………3
2. Evaluation guidance …………………………………………………………………………..5
3. Use of theory-based evaluation …………………………………………………………7
4. Evaluability assessments and linkages with evaluation design ……….9
5. Triangulation, clarity, and transparency…………………………………………..11
6. Lessons to strengthen WFP’s evaluation function …………………………..13
Annex 1: Short biographies of members of the EMAP………………………….16
Annex 2: Evaluation documents reviewed by the EMAP………………………17
Annex 3: Selection of evaluations for review by the EMAP………………….19

The Evaluation Methods Advisory Panel Given the increase in the number of
evaluations and the complex and diverse contexts in which the World Food Programme (WFP) operates, the WFP Office of Evaluation (OEV) has created an Evaluation Methods Advisory Panel (EMAP) to support improving
evaluation methodology, approaches, and methods, and to reflect on international best practice and innovations in these areas. The Panel was launched in January 2022. Currently composed of six members (listed in annex 1), it complements provisions in the WFP evaluation quality assurance system (EQAS).

Purpose and Scope

The aims of the Annual Review are to:

• Reflect on evaluation approaches and methods used in evaluations,
  and progress towards improving and broadening the range of
  methodologies
• Identify systemic and structural challenges
• Derive lessons to increase quality and utility in future evaluations

The EMAP Annual Report covers most evaluations conducted by WFP’s evaluation function – Policy Evaluations (PEs), Complex Emergency Evaluations (CEEs), Strategic Evaluations (SEs), Decentralized Evaluations
(DEs), and Country Strategic Plan Evaluations (CSPEs) – in 2022-2023 (see Annex 3). It is based on reviews undertaken by EMAP members (“the reviewers”), and discussions and workshops between the reviewers and
WFP. EMAP has not examined system-wide and impact evaluations.

Process

Two approaches to the EMAP reviews were undertaken. In one strand of activities, EMAP members received a selection of completed CSPE and DE evaluation reports (ERs), and the related terms of reference (ToR) and inception reports (IRs), for their review. The other strand of EMAP activities was giving feedback on draft outputs for Policy Evaluations (PEs),
Complex Emergency Evaluations (CEEs) and Strategic Evaluations (SEs).

Two EMAP advisers wrote this Annual Report; the process of preparing it entailed:

• Review of the advice provided by EMAP on WFP evaluations during 2023.
• Discussion of the draft annual report with OEV, Regional Evaluation Officers (REOs) and other EMAP advisors in a two-day workshop at WFP. This report  incorporates key elements from these discussions.

As in 2022, the 2023 review faced the following limitations:

• The review included 14 DEs, 10 CSPEs, 5 PEs, 3 CEEs and 3 SEs, but analysed outputs were at different stages of development. EMAP reviewers
  prepared review reports for DEs and CSPEs based on finalised ToRs,
  inception and evaluation reports. Conversely, for SEs, PEs and CEEs, the
  reviews examined draft concept notes, ToRs, IRs, ERs, and two literature
  reviews.
• Not all EMAP reviews undertaken in 2023 were finalised in time for the
  synthesis process undertaken to prepare the Annual Report.
• Most reviews followed a structure provided by WFP which varied by
  evaluation type. For instance, the DE review template included a section on
  overall evaluation approaches and methods which was not included in the
  CSPE review template. Some reviews did not use the templates provided but
  added comments directly to the draft reports.
• Finally, reviewing written evaluation outputs presented challenges to
  explaining why something did or did not happen in an evaluation process
• Unlike in 2022, there was no opportunity for the EMAP to discuss the draft annual report as a panel before sharing it with OEV. The 2023 Annual Report was, however, discussed in a workshop with EMAP members and OEVstaff, including regional evaluation officers, to validate the results and discuss potential ways forward across the different types of evaluations in WFP