There needs to be some debate not just about what is to be evaluated, but on what criteria?

So far, during the present consultation, the question of what to evaluate has been subject of a separate DFID paper, but the question of what criteria has only warranted a short section in an annex. In that annex DFID list “the internationally-agreed evaluation criteria …[that] will be applied to DFID evaluations. They appropriately note that while “It will not be appropriate to investigate every criterion in depth in every evaluation. DFID evaluators will be requested to provide an explanation of the criteria they have chosen (or not) to cover”. The listed criteria are 1. Relevance, 2. Effectiveness, 3. Effeciency, 4. Impact, 5. Sustainability, 6. Coverage, and 7. Coherence.

Elsewhere I have argued for the inclusion of two additional criteria to the traditional DAC 5 (1-5 above).These are equity and transparency

It could be argued that criteria 6 (coverage) already covers equity. However the choice of words can be important. Coverage is an apparently technical term, but equity is explicitly about a value: fairness, of process and outcome. DFID’s desire to eliminate of poverty is a statement about values. Values should be clearly stated, not hidden or assumed.

Transparency is not covered at all. Yet transparency is basic to the whole process of evaluation, especially when viewed in a wider context. Without access to information the ability of stakeholders in development programmes to evaluate performance on any of these criteria will be extremely limited. The importance of access to information was emphasised by the United Nations General Assembly in its first session in 1946, which states: “Freedom of information is a fundamental human right and … the touchstone of all the freedoms to which the UN is consecrated.” (Resolution 59)

More recently DFID was one of the founding signatories to the International Aid Transparency Initiative, publicised at the August 2008 High Level Forum on Aid Effectiveness in Accra, Ghana.

Given this recent statement of position by DFID transparency should clearly be included as an evaluation criteria on the DFID list. If this proposal raises concerns about the list becoming too lengthy, one could argue that it should certainly have higher priority than the newly proposed criteria 7 (coherence). In fact, perhaps it should be criteria number 1, ahead of relevance and all other criteria.

In the same document (List of Potential Evaluation Topics), readers are invited to comment on “any topics you consider very important that we have not listed here”

One gap which I noted was the lack of any reference to meta-evaluation of the many evaluation activities carried out within the country programmes.

However, the draft Evaluation Policy mentioned above makes eleven references to the role of “decentralised evaluation” DFID’s decentralised evaluations “are those commissioned by our staff responsible for managing DFID’s programmes, policies and partnerships, normally in collaboration with their development partners”

The references to decentralised evaluations covered the following areas:
- increased use of decentralised evaluation as one of the 4 major priorities for developing the evaluation function in DFID. p.11
- sustaining a strong culture of decentralised evaluation across the Department. p.16
- strengthening its advisory and quality support role for decentralised evaluations p.17
- quality assurance of decentralised evaluations. p.4, p.16
- helping to set standards, providing support and advice, and reporting on quality. p4

But there are no references to a systematic or periodic meta-evaluation of decentralised evaluations. This seems like a major omission. Authority for evaluation has been decentralised, and advisory support and guidance will be provided, but there is no evident complementary mechanism for assessing the results.

Evaluation of the effects of a large rise in DFID’s programme budget – and concurrent decrease in administrative budget – on the delivery of the aid programme. Timing: flexible.

This topic relates directly to a concern I have raised about these divergent aid trends, in the July 2008 posting on my Rick on the Road blog