“DFID takes very seriously the responsibility to ensure high quality, independent evaluation of its programmes, to provide reliable and robust evidence to improve the value of its global work to reduce poverty.
In December 2007 the Independent Advisory Committee on Development Impact was established to help DFID strengthen its evaluation processes. The Committee is there to work with DFID to:
- Determine which programmes and areas of UK development assistance will be evaluated and when;
- Identify any gaps in the planned programme of evaluations and make proposals for new areas or other priorities as required;
- Determine whether relevant standards (e.g. of the OECD Development Assistance Committee) are being applied; and comment on the overall quality of the programme of evaluation work carried out against these.
DFID and IACDI have therefore been working closely together to define a new policy which will set the course for evaluation in the future. We have also produced a ‘topic list’ of potential areas for evaluation over the coming 3 years. So you will see here two documents on which we would like your feedback, the Draft Evaluation Policy and the Evaluation Topic List.
Central to the policy is the emphasis on greater independence of evaluation, along with stronger partnership working, reflecting global commitments to harmonisation, decentralising evaluation to a greater degree, driving up quality, and ensuring that learning from evaluation contributes to future decision making. We would like you to consider those high level issues when offering your comment and feedback during the time the consultation process is open. This document does not focus on the operational issues; they will be considered in a separate DFID strategy document.
During the consultation period, we would also like to hear your views on which topics you consider to be the greatest priority and why. This will help DFID to make decisions on which are to be given the highest priority.
In summary the issues we are particularly keen for you to focus your feedback on are:
1. The definition of ‘independent evaluation’ – what are your thoughts on the policy approach of DFID, working increasingly with partners, to increase independence in evaluation?
2. What are your views on what’s required to drive up quality across the board in evaluation of international development programmes? What role do you think DFID can most valuably play in this?
3. What are the considerations for DFID strengthening its own evaluation processes, whilst ensuring its commitments to harmonisation remain steadfast?
4. DFID is determined to increase the value of learning from evaluation to inform policy – what are your thoughts on the means to bring this about?
5. DFID is committed to consulting stakeholders during our evaluations, including poor women and men affected by our programmes. Getting representative stakeholders, especially for evaluations which go beyond specific projects and programmes, can often be challenging (for example evaluations of country assistance plans or thematic evaluations). Do you have any ideas on how to improve this?
6. DFID is committed to developing evaluation capacity in partner countries and increasing our use of national systems. What are your thoughts on the challenges and ways forward?
Please send your feedback to evaluationfeedback@dfid.gov.uk . The public consultation will officially close on Tuesday 3rd March but we would appreciate comments as early as possible, so that they can be considered as the operational issues are further thought out.”
In the LIST OF POTENTIAL EVALUATION TOPICS FOR 2009-2012, mentioned above, I was pleased to see the following:
This topic relates directly to a concern I have raised about these divergent aid trends, in the July 2008 posting on my Rick on the Road blog
The need for a meta-evaluation of the results of the decentralised evaluation policy
In the same document (List of Potential Evaluation Topics), readers are invited to comment on “any topics you consider very important that we have not listed here”
One gap which I noted was the lack of any reference to meta-evaluation of the many evaluation activities carried out within the country programmes.
However, the draft Evaluation Policy mentioned above makes eleven references to the role of “decentralised evaluation” DFID’s decentralised evaluations “are those commissioned by our staff responsible for managing DFID’s programmes, policies and partnerships, normally in collaboration with their development partners”
The references to decentralised evaluations covered the following areas:
– increased use of decentralised evaluation as one of the 4 major priorities for developing the evaluation function in DFID. p.11
– sustaining a strong culture of decentralised evaluation across the Department. p.16
– strengthening its advisory and quality support role for decentralised evaluations p.17
– quality assurance of decentralised evaluations. p.4, p.16
– helping to set standards, providing support and advice, and reporting on quality. p4
But there are no references to a systematic or periodic meta-evaluation of decentralised evaluations. This seems like a major omission. Authority for evaluation has been decentralised, and advisory support and guidance will be provided, but there is no evident complementary mechanism for assessing the results.
The need for consultation on evaluation criteria, not just what should be evaluated
There needs to be some debate not just about what is to be evaluated, but on what criteria?
So far, during the present consultation, the question of what to evaluate has been subject of a separate DFID paper, but the question of what criteria has only warranted a short section in an annex. In that annex DFID list “the internationally-agreed evaluation criteria …[that] will be applied to DFID evaluations. They appropriately note that while “It will not be appropriate to investigate every criterion in depth in every evaluation. DFID evaluators will be requested to provide an explanation of the criteria they have chosen (or not) to cover”. The listed criteria are 1. Relevance, 2. Effectiveness, 3. Effeciency, 4. Impact, 5. Sustainability, 6. Coverage, and 7. Coherence.
Elsewhere I have argued for the inclusion of two additional criteria to the traditional DAC 5 (1-5 above).These are equity and transparency
It could be argued that criteria 6 (coverage) already covers equity. However the choice of words can be important. Coverage is an apparently technical term, but equity is explicitly about a value: fairness, of process and outcome. DFID’s desire to eliminate of poverty is a statement about values. Values should be clearly stated, not hidden or assumed.
Transparency is not covered at all. Yet transparency is basic to the whole process of evaluation, especially when viewed in a wider context. Without access to information the ability of stakeholders in development programmes to evaluate performance on any of these criteria will be extremely limited. The importance of access to information was emphasised by the United Nations General Assembly in its first session in 1946, which states: “Freedom of information is a fundamental human right and … the touchstone of all the freedoms to which the UN is consecrated.” (Resolution 59)
More recently DFID was one of the founding signatories to the International Aid Transparency Initiative, publicised at the August 2008 High Level Forum on Aid Effectiveness in Accra, Ghana.
Given this recent statement of position by DFID transparency should clearly be included as an evaluation criteria on the DFID list. If this proposal raises concerns about the list becoming too lengthy, one could argue that it should certainly have higher priority than the newly proposed criteria 7 (coherence). In fact, perhaps it should be criteria number 1, ahead of relevance and all other criteria.