Monitoring and Evaluation NEWS

8th March 2008 Minutes of the New Independent Advisory Committee on Development Impact (Contactable via ev-dept@dfid.gov.uk. PS: The committee will have its own website and contact email address by mid-2008)

On independence of evaluation at DFID

(excerpt) ….Comments made by committee members in the ensuing discussion were
as follows:

• There was a case for DFID’s Evaluation Department (EvD) taking on responsibility for oversight, quality assurance and guidance on self-evaluations in DFID. At present this function seemed fragmented and largely left to the judgement of line managers. This would require extra resources for EvD. In any event, self-evaluation in the department needed to be greatly strengthened.

• Independent evaluation was dependent upon good quality information on the ground, and this also would be helped with a stronger culture of monitoring and self-evaluation throughout the department.

• The committee needed to appreciate the realities of a government department in considering independence issues though that did not mean that IACDI could not play a very significant role in protecting and enhancing evaluation independence in DFID

• An increase in DFID budgets meant an increasing need for information on effectiveness. There seemed a contradiction between the need for more, in-depth evaluation and a declining administrative budget for evaluations. In any event protecting independence suggested a need to explore ways to protect the budget for evaluation.

• There were concerns about the current reporting arrangements for the Head of EvD which do not conform to internationally recognised criteria. A number of options existed including a direct reporting line to the PS or a DG or the creation of a new post of DG for Audit and Evaluation to which the head of EvD would report. Independence could be further buttressed by further developing the relationship between the Head of EvD and IACDI (e.g. with respect to employment, removal and performance assessment of the EvD Head) and displaying this reporting relationship as a ‘dotted line’ on the organization chart.

• There were also concerns around the status and grade of the Head of EvD, given the need for the need for the post to have greater visibility and carry greater clout.

• Some felt that, ideally, a head of evaluation should have a contract precluding employment elsewhere in DFID, and that an advantage of upgrading the post was that it could attract good candidates towards the end of their careers for whom this would not be an issue. Any contract of this kind should be for a fixed term, (either of 10 years or more or renewable on the advice of IACDI). Not all however took the view that future employment within DFID should be precluded and all stressed that the rights of the current incumbent should be protected.
• Whatever option were chosen, it was felt that written job descriptions, protocols and arrangements for performance review, with a role for IACDI or its chair, could also usefully buttress the independence of the Head of EVD.

• There was a need to explore further the modalities for (and control of the head of EVD over) staffing: over time EVD may need to change the balance towards an increased role for EVD staff and a lesser role for external consultants.

• There was also a need for clear written protocols for unimpeded access to information in DFID; for rules of engagement with DFID staff in discussing draft reports; for avoiding staff conflicts of interest; and a written policy on disclosure of reports.

• The need for a clear and agreed departmental policy on evaluation, to meet internationally recognised criteria and be reviewed by IACDI, was highlighted by committee members, recognising that work on this was already underway.

On Country Program Evaluations.

(excerpt) … Views expressed by the committee were as follows:

• It was recognised that different types of CPEs are required in different contexts (eg.,fragile states, smaller country programmes).

• CPEs should aim to provide reliable evidence of impact, particularly on poverty reduction. It was pointed out, however, that it would take a much greater effort and better monitoring and baseline data to get at impact.

• Some questioned the value of CPEs in relation to the amount spent on them. Others recognised that they held country Directors to account and that they were valued by DFID senior management. The annual CPE synthesis by EvD generated useful lessons and identified themes.

• Concerns were raised about the apparently low priority given to CPEs by DFID country teams. It was important that they recognised the importance of evaluation and that data collection and monitoring arrangements should be built in from the outset of programmes.

• Over time there would be advantage in country teams being made responsible for most CPEs as a regular component of country policy management so long as there were adequate incentives set by senior management and adequate quality assurance and oversight, probably best provided by EvD. A transitional arrangement might entail an approach which includes part funding of CPEs by country offices.

• EvD might then consider complementing this self evaluation effort by carrying out more in-depth independent evaluations in a few countries; and continuing to produce annual syntheses of all CPEs carried out in DFID, with the possibility of including the results of similar work carried out other donors.

• Such changes would probably take time, however, suggesting that at least for the next year, EvD should be prepared to proceed with its currently planned programme of CPEs.

• DFID should explore amending ToRs and reporting arrangements for CPEs to make it clear that partner governments and in some cases other donors will also benefit from the evaluations.

• EvD should in any event carry out a methodology review for CPEs.

On the Evaluation Work Programme

(excerpt) … The views of committee members were as follows:

• They were concerned at the declining administration budget for evaluation at a time when the overall programme spend was rising fast and use of the programme budget for evaluation was constrained.

• A surprisingly large sum of programme funds was being earmarked for support of impact evaluations including capacity building and international systems. This was questioned by some. Others were supportive of the approach recognising that the work entailed partner governments leading the processes.

• It was noted that nothing was being done by EvD on project evaluation though the Committee acknowledged that other parts of DFID were continuing to commission and undertake these. In future, it would be of interest to use the flexible funding line to find out what was going on at the project level. It would also be useful to evaluate smaller items with potentially greater impact.