Tag: ICAI

ICAI Year 4 Workplan Consultation opens today

“ICAI is opening a public consultation to help shape our Year 4 Workplan today. We welcome any input that you may have and please feel free to pass on to colleagues. The closing date for responses is Friday 6th September.

The consultation document is available here: http://icai.independent.gov.uk/?p=1637&preview=true  and the text is copied below for your convenience.

ICAI Year 4 Workplan Consultation
The Independent Commission for Aid Impact is the independent body responsible for the scrutiny of UK aid expenditure (Official Development Assistance). We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers.

We are holding a public consultation to inform the development of our workplan for our fourth year of operation (i.e. reports to be published between 12 May 2014 and 11 May 2015). We encourage anyone with an interest in the UK’s aid budget to take this opportunity to have their say and help to shape our plans.

As set out in our 2012-13 Annual Report, we will continue to be guided by our selection criteria of materiality, coverage, interest and risk, while also planning to diversify the types of work that we carry out to make the best use of our increasing body of evidence from our first three years of reports. We will consider different types of reports such as:

  • Programme reviews: this is likely to include scrutinising areas that we believe to be particularly important, given our findings and experience to date, and areas of particular interest to stakeholders;
  • Synthesis reports: this could include drawing together our increasing evidence base from the 35 reports we will have produced by the end of Year 3. This is likely to include some more thematic reviews, for example, synthesising and building upon our findings to date in a particular sector; and
  • More detailed follow-up: by Year 4, changes to impact for intended beneficiaries as a result of our first reports should be more visible. We will develop our approach to follow-up to understand these changes and to encourage further improvement and lesson-learning.

We would welcome your proposals for review topics. Please present these as concisely as possible, giving a summary of what you think ICAI should be reviewing and your reasons for suggesting the topic. Submissions should be no longer than 2,000 words.
It may help to look at our 2012-13 Annual Report (including our Year 3 Workplan in Chapter 5) and our report on ICAI’s Approach to Effectiveness and Value for Money when preparing your submission.

If you would like to submit views to ICAI, please send them to enquires@icai.independent.gov.uk.
The deadline for submissions is Friday 6th September 2013.

Sam Harrison
Communications Manager
Independent Commission for Aid Impact
Dover House | 66 Whitehall | London SW1A 2AW
020 7270 6742 |07500 224642
Due to an ongoing problem the voicemail on my office phone is incorrect. Please email or SMS me if I do not pick up for faster response.
www.independent.gov.uk/icai
Follow us on Twitter: @icai_uk

ICAI Seeks Views on Revised Evaluation Framework

 

Independent Commission for Aid Impact, Wednesday 1 May 2013

 “In our first report, ICAI’s Approach to Effectiveness and Value for Money,we set out an evaluation framework, consisting of 22 questions under 4 guiding criteria (objectives, delivery, impact and learning), to guide our lines of enquiry in reviews. In the light of our experience to date in carrying out our reports, we have reviewed this framework. The revised framework is available at this link: ICAI revised evaluation framework

We are now entering a period of consultation on the revised framework which will run until 24 May 2013. If you have any comments or views, please email enquiries@icai.independent.gov.uk  or post them to: The Secretariat, Independent Commission for Aid Impact, Dover House, 66 Whitehall, London SW1A 2AU”

Open consultation Triennial review of the Independent Commission for Aid Impact (ICAI)

(Website that hosts the text below)

This consultation closes on 26 April 2013

On 21 March the Government announced the triennial review of the Independent Commission for Aid Impact (ICAI) and is seeking views of stakeholders who wish to contribute to the Review. Triennial Reviews of Non-Departmental Public Bodies (NDPBs) are part of the Government’s commitment to review all NDPBs, with the aim of increasing accountability for actions carried out on behalf of the State.

The ICAI’s strategic aim is to provide independent scrutiny of UK aid spending, to promote the delivery of value for money for British taxpayers and to maximise the impact of aid.

The Review will be conducted in line with Cabinet Office principles and guidance, in two stages.

The first stage will:

  • Identify and examine the key functions of the ICAI and assess how these functions contribute to the core business of DFID;
  • Assess the requirement for these functions to continue given other scrutiny processes;
  • If continuing, assess how the key functions might best be delivered; if one of these options is continuing delivery through the ICAI, then make an assessment against the Government’s “three tests”: technical function; political impartiality; and the need for independence from Ministers.

If the outcome of stage one is that delivery should continue through the ICAI, the second stage of the review will:

  • Review whether ICAI is operating in line with the recognised principles of good corporate governance, using the Cabinet Office “comply or explain” standard approach.

In support of these aims we would welcome input and evidence from stakeholders, focused on these main questions:

ICAI’s functions

For the purposes of this review, we have defined ICAI’s key functions as follows:

  • Produce a wide range of independent, high quality/professionally credible and accessible reports (including evaluations, VfM reviews, investigations) setting out evidence of the impact and value for money of UK development efforts;
  • Work with and for Parliament to help hold the UK Government to account for its development programme, and make information on this programme available to the public;
  • Produce appropriately targeted recommendations to be implemented/ followed up by HMG.

Which of these functions do you think are still needed? What would be the impact if ICAI ceased to exist?

Would you define ICAI’s functions differently?

Do you think any of the following delivery mechanisms would be more appropriate or cost effective at delivering these functions: Local government, the voluntary sector, private sector, another existing body or DFID itself?

To date, do you think ICAI has focused on scrutinising UK aid spend or the wider HMG development effort? What do you think it should be doing?

Where do you think ICAI sits on the spectrum between audit and research? Is this where they should be?

How far can and should ICAI have a role in holding HMG to account?

Production of reports

What is the quality of ICAI reports? Is the expertise of those producing the reports appropriate? How does this compare to other scrutiny bodies that you know of?

How far does the methodology used by ICAI add value to other scrutiny of DFID programmes (eg IDC, NAO, DFID internal)?

How far does ICAI involve beneficiaries in its work?

What impact have ICAI reviews had on DFID staff and resources?

How independent do you believe ICAI is? How important do you think this independence is for ICAI’s ability to deliver its functions effectively?

How much of an impact do you think the Commissioners have on ICAI recommendations and reports? What added value do you think they bring? Do they have the right skillset?

Making information available to the IDC and the public

How important do you think ICAI’s role is in providing information about UK development to taxpayers?

What impact has ICAI had on public perceptions of UK development?

Production of targeted recommendations

What has been the added value of ICAI’s recommendations? How do these compare to other scrutiny bodies that you know of?

How far and why have recommendations been followed up?

What impact has ICAI had on DFID’s own approach to monitoring impact and value for money?

How far has ICAI promoted lesson learning in DFID?

General

Do you think ICAI could improve? If so, how do you think ICAI could improve?

Do you have any other comments?

The government is seeking views of stakeholders on the Triennial Review of the Independent Commission for Aid Impact (ICAI).

Contact us by 26 April 2013

Write to us:

email
post
ICAI Review Team KB 2.2
22 Whitehall
London
SW1A 2EG

DFID&UKES Workshop on Development and Evaluation: Practical Ways Forward.

 

Date:  WEDNESDAY 12 OCTOBER 2011
Venue: BIS Conference Centre, Victor ia, London

Objectives:

  • To examine the key contributions of evaluation to international development
  • To provide an update on the accountability framework for evaluation in the UK
  • To explore the role of professional development in building evaluation capacity

THIS ONE DAY EVENT will raise important issues in the world of development and evaluation. The workshop will offer the chance to hear from senior practitioners and will cover the theory and reality as experienced in many contexts. It will update the accountability framework with particular reference to HM Treasury Guidance for Evaluation (the Magenta Book).

A major challenge for organisations is to develop their own staff as evaluation professionals. UKES will offer international insights as well as an update on its own guidance. DFID will report on how it is going about building its own community of evaluators. These will be presented alongside those from the NGO and voluntary sector. The day is relevant to all individuals and organisations with an interest and experience of development and evaluation, including: Donors, Consultants, Public and private sector representatives, Academics, A wide range of professionals

Programme
The workshop will commence at 09.00 and close at 17.30.
Highlights will include:

  • Updates on the Independent Commission for Aid Impact (ICAI),  HM Treasury’s Magenta Book and the Cross Government Evaluation Group (CGEG)
  • How to evaluate in fragile states, conlict environments and other challenging situations
  •  Case studies of evaluation at different levels: national and local,  sector specific
  • How to build professional capacity: use of accreditation and adapting to it a range of organisations at government and civil society level

Registration
The workshop will be held at the BIS Conference Centre, 1 Victoria, Street, London SW1H OET.
The registration fees are as follows:
UKES members  £75.00 + VAT
Non-members  £100.00 + VAT
Registration and the full programme for the workshop are available from the website  www.profbriefings.co.uk/depwf
For any further information, contact the workshop administrators:
Professional Brieings
37 Star Street
Ware
Hertfordshire SG12 7AA
Telephone:
01920 487672
Email:  london@profbrieings.co.uk

Synthesis Study of DFID’s Strategic Evaluations 2005 – 2010

 

A report produced for the Independent Commission for Aid Impact
by Roger Drew, January 2011. Available as pdf.

Summary

S1. This report examined central evaluations of DFID’s work published from 2006 to 2010. This included:
– 41 reports of the International Development Committee (IDC)
– Two Development Assistance Committee (DAC) peer reviews
– 10 National Audit Office (NAO) reports
– 63 reports of evaluations from DFID’s Evaluation Department (EVD)

S2. These evaluations consisted of various types:
– Studies of DFID’s work overall (16%)
– Studies with a geographic focus (46%)
– Studies of themes or sectors (19%)
– Studies of how aid is delivered (19%) (see Figure 1)

S3. During this period, DFID’s business model involved allocating funds through divisional programmes. Analysis of these evaluation studies according to this business model shows that:
– Across regional divisions, the amount of money covered per study varied from £63 million in Europe and Central Asia to £427 million in East and Central Africa.
– Across non-regional divisions, the amount of money covered per study varied from £84 million in Policy Division to £5,305 million in Europe and Donor Relations (see Figure 2).

S4. Part of the explanation of these differences is that the evaluations studied form only part of the overall scrutiny of DFID’s work. In particular, its policy on evaluation commits DFID to rely on the evaluation systems of partner multilateral organisations for assessment of the effectiveness and efficiency of multilateral aid. No central reviews of data generated through those systems were included in the documents reviewed for this study. The impact of DFID’s Bilateral and Multilateral Aid Reviews was not considered, as the Reviews had not been completed by the time this study was undertaken.

S5. The evaluations reviewed had a strong focus on DFID’s bilateral aid programmes at country level. There was a good match overall between the frequency of studying countries and the amount of DFID bilateral aid received (see Table 4). Despite the growing focus on fragile states, such countries were still less likely to be studied than non-fragile countries. Countries that received large amounts of DFID bilateral aid not evaluated in the last five years included Tanzania, Iraq and Somalia (see Table 5). Regional programmes in Africa also received large amounts of DFID bilateral aid but were not centrally evaluated. Country programme evaluations did not consider DFID’s multilateral aid specifically. None of the evaluations reviewed considered why the distribution of DFID’s multilateral aid by country differs so significantly from its bilateral aid. For example, Turkey is the single largest recipient of DFID multilateral aid but receives almost nothing bilaterally (see Table 7).

S6. The evaluations reviewed covered a wide range of thematic, sectoral and policy issues (see Figure 3). These evaluations were, however, largely standalone exercises rather than drawing either retrospectively on data gathered in other evaluations or prospectively including questions into proposed evaluations. More use could have been made of syntheses of country programme evaluations for this purpose.

S7. The evaluations explored in detail the delivery of DFID’s bilateral aid and issues of how aid could be delivered more effectively. The evaluations covered the provision of multilateral aid in much less detail (see paragraph S4). One area not covered in the evaluations is the increasing use of multilateral organisations to deliver bilateral aid programmes. This more than trebled from £389 million in 2005/6 to £1.3 billion in 2009/10 and, by 2009/10, was more than double the amount being provided as financial aid through both general and sectoral budget support combined.

[RD comment:  I had the impression that DFID, like many bilateral donors, does very few ex-post evaluations, so I wanted to find out how correct this view was. I searched for “ex-post” and found nothing. The question then is whether the new Independent Commission for Aid Impact (ICAI) will address this gap – see more on this here]

UK Independent Commission for Aid Impact – Work Plan

Independent Commission for Aid Impact – Work Plan, and the associated Press Release (12 May 2011)

1. This document introduces the Independent Commission for Aid Impact’s first work plan, setting out the reports we envisage initiating over the next three years, from May 2011 to May 2014.

2. Our mandate permits us to examine all UK Government programmes
funded by Official Development Assistance expenditure. In 20091, this
represented £7.4bn, which was spent through bilateral, joint and
multilateral processes by the Department for International Development (DFID) and at least eight other branches of Government. Under the Government’s current plans and guided by its recent reviews of bilateral, multilateral and humanitarian work, this expenditure is due to rise significantly and will change in focus. This range of projects and programmes gives us significant discretion in choosing where to focus the attention of our reports.” ..continues..

See also: FRAMEWORK AGREEMENT BETWEEN THE DEPARTMENT FOR INTERNATIONAL DEVELOPMENT (DFID) AND THE INDEPENDENT COMMISSION FOR AID IMPACT (ICAI) This document sets out the broad framework within which the ICAI will operate as a permanent body (12 May 2011 – 11 May 2015). The Agreement is signed by the Chief Commissioner of the ICAI and DFID. This document, and any future revisions, will be made public on the ICAI website.

[RD Comments: The workplan has three strands of work:

  • Evaluations: are likely to focus on the sustainable development impact achieved by programmes against initial or updated objectives
  • Value for money reviews: will consider whether objectives have been achieved with the optimal use of resources
  • Investigations: could range from general fact-finding in response to external requests, to assessments of compliance with legal and policy responsibilities and examinations of alleged corruption cases.

Regarding the first strand, the OECD DAC definition of impact is “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.” One practical way of defining long term would be as any change observed after the completion of a project (typically 3 years). This would seem to be an appropriate focus for the ICAI because in the past DFID has undertaken very few ex-post evaluations. There is a gap here that needs to be addressed, as there is with quite a few other bilateral agencies.

A further justification lies in the useful connection with value for money reviews. Some organisations, like Global Environmental Facility, define impact as “A fundamental and durable change in the condition of people and their environment brought about by the project”  in other words, a sustained change.  The longer a change is sustained (all other things being equal) the more value for money would seem to have been realised. Assessing impact in the short term (i.e. during the project implementation period) risks understating impact and the associated value for money.

The question that then arises in my mind is to what extent will the ICAI program of evaluations be focused on projects that have been completed, versus those which are still being implemented. I will be asking the ICAI if they could  provide an answer, for example in the form of the percentage of completed versus incompleted projects to be examined in each of the 8 evaluations to be undertaken in year 1]

PS 24 May 2011: See also Howard White’s related question about the ICAI’s use of ex-post and ex-ante evaluations. There he seems to be arguing against ex-post evaluations: “There is a question as to whether the commission restricts itself to ex-post evaluations, done once the intervention is being implemented or completed. Or can it engage in ex-ante designs before the intervention has started? Designing the evaluation prior to the launch of a programme, and collecting baseline data, generally delivers more robust findings.”

This seems like the method tail wagging the programme development dog. Or, looking for a lost wallet under a lamppost.The potential for rigour should not determine what gets evaluated. What gets evaluated should be decided by more strategic considerations. Like the fact that we know very little about the long term effects of most development projects (where long term = after the project interevention ceases).

%d bloggers like this: