Available  on the DFID website and as a pdf.

“Structural Reform Plans are the key tool of the Coalition Government for making departments accountable for the implementation of the reforms set out in the Coalition Agreement. They replace the old, top-down systems of targets and central micromanagement.

The reforms set out in each department’s SRP are designed to turn government on its head, taking power away from Whitehall and putting it into the hands of people and communities. Once these reforms are in place, people themselves will have the power to improve our country and our public services, through the mechanisms of local democratic accountability, competition, choice, and social action.

The reform plans set out in this document are consistent with and form part of the Department’s contribution to the Spending Review. All departmental spending is subject to the Spending Review.

We have adopted a cautious view of the timescales for delivering all legislative measures due to the unpredictability of pressures on Parliamentary time.”

Departmental priorities

1. International commitments
•Honour the UK’s international commitments and support actions to achieve the Millennium Development Goals
2. Value for money
• Make British aid more effective in reducing poverty through improved transparency and value for money
3. Wealth creation
• Make British international development policy more focussed on boosting economic growth and wealth creation
4. Afghanistan, Pakistan, conflict and stabilisation
• Improve the join-up and performance of British development policy in conflict countries, with particular focus on Afghanistan and Pakistan
5. Role of women
• Recognise the role of women in development and promote gender equality
6. Climate change
• Drive urgent action to tackle climate change and support adaptation efforts in developing  countries

(The following six pages detail the plans for each of these priorities)

OECD DAC NETWORK ON DEVELOPMENT EVALUATION, February 2010 Download a pdf copy

“The DAC Network on Development Evaluation is a unique international forum that brings together evaluation managers and specialists from development co-operation agencies in OECD member countries and multilateral development institutions. Its goal is to increase the effectiveness of international development programmes by supporting robust, informed and independent evaluation.

A key component of the Network’s mission is to develop internationally agreed norms and standards to strengthen evaluation policy and practice. Shared standards contribute to harmonised approaches in line with the commitments of the Paris Declaration on Aid Effectiveness. The body of norms and standards is based on experience, and evolves over time to fit the changing aid environment. These principles serve as an international reference point, guiding efforts to improve development results through high quality evaluation.

The norms and standards summarised here should be applied discerningly and adapted carefully to fit the purpose, object and context of each evaluation. This summary document is not an exhaustive evaluation manual. Readers are encouraged to refer to the complete texts available on the DAC Network on Development Evaluation’s website: www.oecd.org/dac/evaluationnetwork. Several of the texts are also available in other languages.”

The DAC Network on Development Evaluation,  OECD, 2010. Download a pdf copy

“Introduction

In June 2009, the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Network on Development Evaluation agreed to undertake a study of its members’ evaluation systems and resources. The study aims to take stock of how the evaluation function is managed and resourced in development agencies and to identify major trends and current challenges in development evaluation. The purpose is to inform efforts to strengthen evaluation systems in order to contribute to improved accountability and better development results. It will be of interest to DAC members and evaluation experts, as well as to development actors in emerging donor and partner countries.

To capture a broad view of how evaluation works in development agencies, core elements of the evaluation function are covered, including: the mandate for central evaluation units, the institutional position of evaluation, evaluation funding and human resources, independence of the evaluation process, quality assurance mechanisms, co-ordination with other donors and partner countries, systems to facilitate the use of evaluation findings and support to partner country capacity development.

This report covers the member agencies of the OECD DAC Network on Development Evaluation.1 See Box 1 for a full list of member agencies and abbreviations. Covering all major bilateral providers of development assistance and seven important multilateral development banks, the present analysis therefore provides a comprehensive view of current policy and practice in the evaluation of development assistance.

The study is split into two sections: section I contains an analysis of overall trends and general practices, drawing on past work of the DAC and its normative work on development evaluation. Section II provides an individual factual profile for each member agency, highlighting its institutional set-up and resources.”

Copied from the DFID website, 3rd June 2010:

[Please post your Comments below and/or on the Guardian Katine website ]

“British taxpayers will see exactly how and where overseas aid money is being spent and a new independent watchdog will help ensure this aid is good value for money, International Development Secretary Andrew Mitchell has announced.

In his first major speech as Development Secretary, Mr Mitchell said he had taken the key steps towards creating an independent aid watchdog to ensure value for money. He also announced a new UKaid Transparency Guarantee to ensure that full information on all DFID’s spending is published on the departmental website.

The information will also be made available to the people who benefit from aid funding: communities and families living in the world’s poorest countries.

These moves come as part of a wider drive to refocus DFID’s work so British taxpayers’ money is spent transparently and on key priority issues such as maternal mortality and disease prevention.”

In Mr Mitchell’s speech, delivered at the Royal Society with Oxfam and Policy Exchange, he argued that overseas aid is both morally right and in Britain’s national interest but that taxpayers need to see more evidence their money is being spent well.

Andrew Mitchell said:

“We need a fundamental change of direction – we need to focus on results and outcomes, not just inputs.  Aid spending decisions should be made on the basis of evidence, not guesswork. That is why we have taken the first steps towards creating a new independent aid watchdog.

“The UK Aid Transparency Guarantee will also help to create a million independent aid watchdogs – people around the world who can see where aid money is supposed to be going – and shout if it doesn’t get there.”

Andrew Mitchell highlighted the results of well-spent aid, saying:

“Development is good for our economy, our safety, our health, our future. It is, quite simply, tremendous value for money: the best return on investment that you’ll find anywhere in government.

“British aid pays for five million children in developing countries to go to primary school every day. That’s roughly the same number as go to primary school in Britain yet it costs only 2.5 per cent of what we spend here. That’s real value for money.”

And he gave this pledge to UK taxpayers:

“To the British taxpayer I say this: our aim is to spend every penny of every pound of your money wisely and well. We want to squeeze every last ounce of value from it. We owe you that.

“And I promise you as well that in future, when it comes to international development, we will want to see hard evidence of the impact your money makes. Not just dense and impenetrable budget lines but clear evidence of real effect.”

Barbara Stocking, Oxfam Chief Executive, said:

“Andrew Mitchell’s announcement of a watchdog to improve the quality of British aid is welcome and something Oxfam has requested for years.

“The new watchdog must be truly independent of government if it is to achieve the best results for the British taxpayer and poor people alike.

“As our report launched today argues, strengthening transparency and accountability will be important in further improving the quality of British aid but they should be pieces in a larger jigsaw. Ensuring aid is poverty focused not politically driven and strengthens the public services poor people rely on are all vital to ensuring that it makes the biggest possible difference to the lives of the world’s poorest. We look forward to working with the new government on these issues.”

Richard Miller, ActionAid UK Director, welcomed today’s announcement on an Aid Transparency guarantee.

“ActionAid has long campaigned for citizens in poor countries to have information about aid provided in their name. We are therefore delighted with the Secretary of State’s announcement of increased transparency in aid spending. It will hopefully establish a new standard of openness and encourage scrutiny by citizens in countries receiving UKaid, as well as assuring UK taxpayers that aid is reaching the poorest.”

Dr Amy Pollard, CAFOD’s aid effectiveness policy analyst said:

“CAFOD are delighted that the Secretary of State today announced that Britain’s aid is to become more transparent.

“Whilst it is not a silver bullet, transparency is a vital condition for improving aid.  Proper information on aid funds is critical in the fight against corruption and essential for promoting accountability in development work.

“Transparency underpins the effort to make aid better – it is impossible to co-ordinate aid, assess results or build real partnerships between donors and developing countries without it.”

Hear the full speach

Read some public comments on the Guardian Katine website

Prepared by Miranda Cahn, Evaluation Advisor, Strategy, Advisory and Evaluation Group, NZAID, Wellington, August 2009. Available online

Executive Summary

Introduction

The New Zealand Agency for International Development (NZAID) is committed to improving evaluative activity1, including evaluations and reviews. Since 2005 NZAID has undertaken annual desk studies of the evaluations and reviews completed by NZAID during the previous calendar year. This 2009 study assesses the quality of 29 NZAID commissioned evaluations and reviews that were submitted to the NZAID Evaluation and Review Committee (ERC) during 2008, and their associated Terms of Reference (TOR). The study identifies areas where quality is of a high standard, and areas where improvement is needed. Recommendations are made on how improvements to NZAID commissioned evaluations and reviews could be facilitated.

The objectives of the study are to:

• assess the quality of the TOR with reference to the NZAID Guidelines on Developing TOR for Reviews and Evaluations

• assess the quality of the NZAID 2008 evaluation and review with reference to the NZAID Evaluation Policy, relevant NZAID Guidelines and Development Assistance Committee of Organisation for Economic Cooperation and Development (DAC) Evaluation Quality Standards

• identify, describe and discuss key quality aspects of the TOR and evaluation and review reports that were of a high standard and those that should be improved in future.

Methodology

A similar methodology was used in this study as for the 2008 study2 in order that a comparison could be made of the quality of NZAID evaluations and reviews (and their associated TOR) in 2007 and 2008. The 2008 TOR were first assessed on individual quality criteria, guided by the NZAID Guideline for Developing Terms of Reference for Reviews and Evaluations. A matrix was developed for this, similar to that used in the similar 2008 study. The evaluation and review reports were then similarly assessed based on the DAC Evaluation Quality standards, NZAID Evaluation Policy Statement and NZAID evaluation guidelines. Overall assessments were assigned to each TOR, and each review or evaluation. In assigning overall assessments, subjective weighting was given to each quality criteria.

1 The term ‘evaluative activity’ is used by NZAID to refer to a range of evaluation processes and includes analyses conducted for planning, monitoring, review and assessment of ongoing or completed development activities

Findings

Terms of Reference (TOR):

There has been an improvement in the quality of the TOR from 2007 to 2008. The area where most improvement was evident in the TOR was in the methodology sections. Improvement was also evident in the 2008 TOR in the:

• description of purpose and rationale

• request for analysis of value for money

• description of the scope of the evaluations/reviews

• outputs and reporting section, and the type of outputs requested being broadened to include feedback workshops and other types of outputs

• instruction for feed back to stakeholders of the findings of the evaluation/review

• description of the management and overseeing of the evaluation.

Despite improvements, the following areas were identified in this 2009 study as key areas (sections in the TOR) where further improvement would improve the quality of the NZAID TOR for evaluations and reviews: objectives, and related evaluation questions; scope; methodology; value for money; outputs and reporting; management and governance. The NZAID Guideline for Developing TOR for Reviews and Evaluations has recently been revised and includes guidance on all of these aspects.

Evaluations and Reviews:

Overall the quality of evaluations and reviews had improved from 2007 to 2008 according to overall assessments allocated for the purpose of this study3. Forty one percent of the 2008 evaluations and reviews were assessed as ‘satisfactory or good in all or many respects’ (highest rank) compared with 22 percent in 2007. Just ten percent of 2008 evaluations and reviews were assessed as ‘not satisfactory’ compared with 28 percent in 2007 (lowest rank). Around half of evaluations and review were assessed as ‘satisfactory or good in some respects’ in both years (medium rank).

The 2008 evaluations and reviews were mostly useful and relevant (as in 2007), their findings meeting the TOR and recommendations flowing logically from the findings. The background and context sections of the reports were generally satisfactory in both years. The review and evaluations had improved considerably between 2007 and 2008 in the following areas:

• reporting

• the way that gender issues had been incorporated into the evaluations and reviews

3 The evaluations and reviews were assessed by the author of the study according to whether they were ‘satisfactory or good in all or many respects’ (highest rank), ‘satisfactory or good in some respects’ (medium rank) or ‘not satisfactory’ (lowest rank). The process for allocating the assessment is described in detail in the report.

• analysis of value for money.

However, as in 2007, confidence in the validity and reliability of the 2008 reports were often compromised by methodologies not being clearly described, and some other issues that are noted below. Furthermore, there was no improvement in the way that objectives of the evaluation or review were described between 2007 and 2008.

The following aspects were identified in this study as key areas where improvement would greatly enhance the quality of the NZAID evaluations and reviews.

• Rationale and purpose of the evaluation or review: reports need to reiterate the rationale and purpose of the evaluation or review in order that sense can be made of the report. This was not done in all the reports.

• Scope of evaluation or review: as with the section on scope in the TOR, so the section on scope in the reports needed to be improved.

• Methodology: better description of information needs (related to evaluation questions); sources of information; data collection methods; data analysis; how NZAID guiding principles have been included in the review or evaluation; and how crosscutting issues have been included. Better description of participatory processes, and moving (where appropriate) from consultative to collaborative and collegiate participatory approaches.

• Description of how ethical issues had been considered.

• Lines of evidence.

• Analysis of value for money.

• Inclusion of crosscutting and mainstreamed issues in a systematic way.

• Reporting: Improvement in areas such as section sequencing; writing style, reducing typological errors; improving executive summaries; better contents pages; and overall, ensuring the reports meet adhere to the NZAID Guideline on the Structure of Evaluation and Review Reports.

Most of these areas of potential improvement have also been identified in previous annual studies of reviews and evaluations, and some aspects have improved over the years.

The study identified a relationship between the quality of TOR and the quality of evaluations and reviews.

Summary of key lessons learned:

1. Despite improvements between 2007 and 2008, there is room for further improvement in TORs, and evaluations and reviews.

2. The quality of the TOR influences the quality of the evaluation or review.

3. The evaluation or review report is the main output of the evaluation that will be available and used by various stakeholders. As such, the reports need to reflect all aspects of the evaluation and review.

4. Contracting evaluators with evaluative experience and skills is important for high quality evaluations and reviews.

5. Submissions to the ERC were useful in confirming the extent to which evaluations and reviews met the TOR objectives and answered the TOR evaluation questions, whether the findings were useful, and whether the recommendations and lessons learned flowed from the findings. However, most submissions were not useful in confirming the extent to which reviews and evaluations met the TOR methodology, or their quality.

6. A participatory approach to this study in the future may provide valuable opportunities for learning on how quality could be improved.

In order that NZAID learns from the evaluations and reviews it commissions, and further improves evaluations and reviews, it is recommended that:

1. NZAID SAEG Evaluation Research and Monitoring (ERM) Team continue to develop processes such as training, advice, guidelines, improved learning from ERC submissions, and support for programme staff, ensuring that areas identified in this report that could be improved continue to be addressed.

2. Evaluations advisors emphasise to NZAID programme and SAEG staff, and contractors, the importance of reports reflecting all aspects of the review or evaluation including, for example, methodology, ethics, principles guiding the review or evaluation, and how these were implemented.

3. An experienced and competent evaluator is included in evaluation and review teams.

4. Peer reviews and appraisals of evaluations and reviews assess the extent to which the reports have described and met ‘process’ aspects of the TOR (eg methodology) as well as the extent to which the purpose and objectives of the review/evaluation have been met (where the emphasis is at present).

5. NZAID Programme staff continue to seek advice from the Evaluation, Research and Monitoring (ERM) Team, identify examples of good evaluative practice in ERC submissions (including where processes have been robust and findings are reliable), and provide feedback on what training and advice provided to them by the ERM team worked and what has not worked.

6. NZAID SAEG ERM Team considers a more participatory and inclusive approach to this study for 2010.

Authors: Espen Villanger ; Alf Morten Jerve
Published in: Journal of Development Effectiveness, Volume 1, Issue 2

Abstract

view references (26)

Publication date: 12 May 2009 . Full report (PDF – 366KB)

Executive Summary

1.  This brief review of the Department for International Development’s (DFID) performance management arrangements during 2008 is a follow-up to our 2002 VFM report on the same topic. It responds to a request from DFID’s Accounting Officer to re-visit the topic periodically, which the C&AG agreed would be valuable. It is based on a desk review of main documents, interviews with DFID staff, and a survey of staff and stakeholder views about evaluation (Appendix 3). We did not undertake a full audit of DFID systems, some of which are in the process of being revised, and we concentrated on those areas of DFID activity most directly related to its performance targets. We drew on recent DFID reviews of monitoring and evaluation, and our findings square well with the results of those reviews.

2.  DFID has responded well to our 2002 recommendations, improving the coverage and specification of its PSA indicators, streamlining the planning and performance monitoring arrangements, instituting better Board review procedures, enhancing the performance review of multilateral funding, and adding to the scale and independence of evaluation work. Perhaps most strikingly, all the staff we spoke to in DFID were fully aware of the Millennium Development Goals (MDGs) and associated Public Service Agreement targets (PSAs), and teams used the associated indicators to structure their debates about priorities and performance – in marked contrast to the findings from our earlier study.

3.  Challenges to effective performance management, however, remain. The prime challenge is in securing sufficient reliable, timely data on poverty reduction outcomes and service delivery outputs to underpin aid targeting and performance analysis. Although DFID is the prime bilateral supporter of developing country statistical capacity, progress in securing better, more frequent poverty-related  data has been slow. DFID has been unable, for example, to set a credible baseline for income poverty in 5 of its 22 PSA countries, and has inadequate trend information in a further 4. An increasing DFID focus on fragile states exacerbates this problem. There are similar gaps in output data – data which are crucial to support performance management and value for money judgements in an area where outcomes can take many years to emerge, and may not be measured even then. Current DFID work to develop standard output indicators is welcome, if limited.

4.  A related issue concerns the challenge of associating inputs with outputs and outcomes. When much bilateral aid was in the form of discrete projects, DFID could associate its own inputs with project outputs and probably intermediate outcomes as well. With increasing use of programmatic aid, those links are problematic in principle, and in practice subject to serious weaknesses in developing country statistical systems. DFID has developed a set of standard indicators and is looking to strengthen economic appraisals to help inform project approval decisions and develop cost-effectiveness benchmarks. It has created a central unit charged with promoting and assessing value for money. And it has revised and strengthened its guidance on the creation of monitoring frameworks, requiring explicit baselines and a “pro-rating” of DFID’s share of benefits according to its share of inputs.  The challenge will be for DFID to maintain the momentum of these initiatives with decreasing staff resources but increasing programme spend.

5.  There are two further areas where, given the significance DFID attaches to them, measurement practice is not sufficiently advanced. First, measurement of the organisational capacity of development partners is insufficiently structured or scaled, given that so much DFID aid incorporates explicit capacity building elements. While there are a number of ratings and assessments that are used in specific sectors, and some overall “government effectiveness” indicators, DFID has no corporate approach to measuring capacity built or sustained. Second, it continues to struggle to measure the impact of its “influencing” work.  Sorting out the critical influences on changes in policies or practices presents technical challenges. Initial DFID measurement trials took place six years ago, in the context of its work with multilateral organisations. And DFID has recently completed further pilots looking to apply approaches designed to measure projects. An evaluation of these pilots confirmed the value of DFID’s ‘logical framework’ approach, while noting some limitations. Performance measurement in this area is hindered by lack of good measurement of inputs: DFID has no staff time booking system that allows it to monitor the scale, cost or nature of resources committed to “influencing”.

6.  DFID has well-established systems for specifying aid objectives, and associated indicators, in a logical hierarchy, and for using those indicators in approving and monitoring its programmes. But its management of these systems has not been sufficiently firm. A number of our recent audit studies have commented on problems with PRISM, the former IT system used to store and collate project and programme data, as well as weaknesses in specific project or programme objectives. In this review the projects we examined had many different indicators of progress, often leading to subjective judgements of project progress. System incentives militate in favour of more, rather than less, favourable judgements. Recently, a DFID-commissioned monitoring and evaluation (M&E) audit found widespread problems with defective logical frameworks and weaknesses in management review. DFID has fixed some of the structural issues and is implementing a new integrated financial and project management system (ARIES). But lack of staff skills in M&E and weak in-the-line reviewremain real issues – staff told us, for example, of a lack of training in M&E. And they alsonoted that potential out-of the-line review bodies, such as the new Investment Committee, would need to address portfolio review and systemic issues by reference to a sample of actual programme proposals, if they were to help secure better compliance with DFID policies on programme appraisal and monitoring.

7.  DFID’s Evaluation Department (EvD) has increased in scale and stature since 2002, and the recent creation of an Independent Advisory Committee on Development Impact (IACDI) provides a further buttress to evaluation independence and quality. EvD remains less structurally independent than many other development agency evaluation units, however, and its recent programme has been slanted towards practice and strategy reviews, rather than focusing on DFID impact, or securing better information on aid cost-effectiveness. Work is in hand to refresh evaluation policy and strategy. Staff and stakeholder perceptions of DFID evaluation are favourable, albeit with clear scope for improvement. Evaluation is not, however, well integrated into general DFID performance management – it is not, for example, tasked with providing a better interpretation of DFID’s performance against PSAs or MDGs, and rarely set to work on cost-effectiveness topics. External stakeholders ranked evaluation’s potential contribution to accountability more highly than did DFID staff.

8.  In an area where outcomes often take many years to emerge, can be difficult to measure, and are subject to many influences other than DFID, programme implementation (and M&E in particular) has attracted less kudos than policy analysis, programme planning and fire fighting. DFID’s 2008 Results Action Plan is designed to address this situation, and is a high corporate priority which will be closely monitored by top management. Stronger management arrangements for performance review have now been put in place to drive the department-wide focus on results with more formal divisional performance review and challenge arrangements, as opposed to structuring scrutiny (and therefore perceptions of credit or recognition) around policy or planning papers.

See also a related report:
Assessing the quality of DFIDs Project Reviews March 2007

The assignment was commissioned to answer;
a. whether the quality of DFID project documentation (particularly reviews) is changing over time, and
b. whether scoring of reviews is consistent across the portfolio.
2. It is primarily a descriptive report of findings. In some cases comments are made, but the principal function is to identify what is taking place.

Produced by the Value for Money Department, FCPD, February 2009.

>>Full text here<<

“The principal changes to the logframe from the earlier (2008) 4×4 matrix are:
•  The Objectively Verifiable Indicator  (OVI) box has been separated into its
component elements (Indicator, Baseline and Target), and Milestones added.
•  Means of Verification has been separated into ‘Source’.
•  Inputs are now quantified  in terms for funds (expressed in Sterling for DFID
and all partners) and in use of DFID staff time  (expressed as Full-Time
Equivalents (FTEs);
•  A Share box now indicates the financial value of DFID’s Inputs as a
percentage of the whole.
•  Assumptions are shown for Goal and Purpose only;
•  Risks are shown at Output and Activities level only;
•  At the Output level,  the Impact Weighting is now shown in the logframe
together with a Risk Rating for individual Outputs
•  Activities are now shown separately (so do not normally appear in the
logframe sent for approval), although  they can be added to the logframe
template if this is more suitable for your purposes.
•  Renewed emphasis on the use of disaggregated beneficiary data within
indicators, baselines and targets.”

EVALUATOR VIEWS OF CURRENT PRACTICE AND RECOMMENDATIONS FOR CHANGE.

This paper was produced independently by Richard Blue, Cynthia Clapp-Wincek, and Holly Benner. MAY 2009

See  Full-Report or Policy_Brief (draft for comment)

“Findings derive from literature review, interviews with senior USG officials and primarily interviews with and survey responses from ‘external evaluators’—individuals who conduct evaluations of U.S. foreign assistance programs, either as part of consulting firms, non-governmental organizations (NGOs), or as individual consultants.   External evaluators were chosen because: 1) the authors are external evaluators themselves with prior USAID and State experience; 2) in recent years, the majority of evaluations completed of USG foreign assistance programs have been contracted out to external evaluation experts; and 3) evaluators are hired to investigate whether foreign assistance efforts worked, or didn’t work, and to ask why results were, or were not, achieved.  This gives them a unique perspective”

Key Findings – Monitoring

The role of monitoring is to determine the extent to which the expected outputs or outcomes of a program or activity are being achieved.  When done well, monitoring can be invaluable to project implementers and managers to make mid-course corrections to maximize project impact.  While monitoring requirements and practice vary across  U.S. agencies and departments, the following broad themes emerged from our research;

•  The role of monitoring in the USG foreign assistance community has changed dramatically in the last 15 years.  The role of USG staff has shifted to primarily monitoring contractors and grantees.  Because this distances USG staff from implementation of programs, it has resulted in the loss of dialogue, debate and learning within agencies.

•  The myriad of foreign assistance objectives requires a multiplicity of indicators. This has led to onerous reporting requirements that try to cover all bases.

•  There is an over reliance on quantitative indicators and outputs of deliverables over which the project implementers have control (such as number of people trained) rather than qualitative indicators and outcomes, expected changes in attitudes, knowledge, andbehaviors.

•  There is no standard guidance for monitoring foreign assistance programs—the requirements at MCC are very different  from those at DOS and USAID.  Some implementing agencies and offices have no guidance or standard procedures.

Key Findings – Evaluation

There is also great diversity in the evaluation policies and practices across USG agencies administering foreign assistance.  MCC has designed a very robust impact evaluation system for its country compacts, but these evaluations have yet to be completed. The Education and Cultural Affairs Bureau at the State Department has well respected evaluation efforts, but there is limited evaluation work in other bureaus and offices in the Department.  USAID has a long and rich evaluation history but neglect and lack of investment, as well as recent foreign assistance reformefforts, have stymied those functions.  The following themes emerged in our study:

The decision to evaluate: when, why and funding:

•  The requirements on the decision to evaluate vary across U.S. agencies. There is no policy or systematic guidance for what should be evaluated and why.  More than three quarters of Survey respondents emphasized the need to make evaluation a requirement and routine part of the foreign assistance programming cycle.

•  Evaluators rarely have the benefit of good baseline data for U.S. foreign assistance projects which makes it difficult to conduct rigorous outcome and impact evaluations that can attribute changes to the project’s investments.

•  While agencies require monitoring and evaluation plans as part of grantee contracts, insufficient funds are set aside for M&E, as partners are pressured to spend limited money on “non-programmatic” costs.

Executing an evaluation:

•  Scopes of work for evaluation often reflect a mismatch between evaluation questions that must be answered and methodology, budget and timeframe given for an evaluation.

•  Because of limited budget and time, the majority of respondents felt  that evaluations were not sufficiently rigorous to provide credible evidence for impact or sustainability.

Impact and utilization of evaluation:

•  Training on M&E is limited across USG agencies.  Program planning, monitoring and evaluation are not included in standard training for State Department Foreign Service Officers or senior managers, a particular challenge when FSOs and Ambassadors become the in-country decision makers on foreign assistance programs.

•  Evaluations do not contribute to agency-wide or interagency knowledge. If “learning” takes place, it is largely confined to the immediate operational unit that commissioned the evaluation rather than contributing to a larger body of knowledge on effective policies and programs.

•  Two thirds of external evaluators polled agreed or strongly agreed that USAID cares more about success stories than careful evaluation.

•  Bureaucratic incentives do not support rigorous evaluation or use of findings – with the possible exception of MCC which supports evaluation but does not yet have a track record on use of findings.

•  Evaluation reports are often too long or technical to be accessible to policymakers and agency leaders with limited time.

Create a Center for Monitoring and Evaluation

A more robust M&E and learning culture for foreign assistance results will not occur without the commitment of USG interagency leadership and authoritative guidance.  Whether or not calls to consolidate agencies and offices disbursing foreign assistance are heeded, the most efficient and effective way to accomplish this learning transformation would be to establish an independent Center for Monitoring and Evaluation (CME), reporting to the Office of the Secretary of State or the Deputy Secretary of State for Management and Resources.  The Center would be placed within the Secretary or Deputy Secretary’s Office to ensure M&E efforts become a central feature of foreign assistance decision-making…”

See the remaining text in the Policy_Brief

Date: 21st-24th June 2009
Venue: Belgium

Channel Research and the Active Learning Network for Accountability and Performance (ALNAP) are inviting participants for Training in Evaluation of Humanitarian Action, Belgium, 21st-24th June 2009 (actual training dates 22nd-24th June 2009).

This course is an introductory-to-intermediate level course and has the overall aim of assisting participants in the design of monitoring systems, and to be able to commission, manage, carry out and use small scale evaluations in humanitarian action. This 3-day training course will use the OECD-DAC evaluation criteria but also introduces new evaluation material specifically on joint evaluations and innovative learning processes as part of an evaluation process.

The course follows on the courses given every year by Channel Research and ALNAP in Belgium. This course has the following aims:

• To make evaluations of humanitarian assistance more effective in contributing to improved performance.
• To improve the quality of evaluation processes.

Based on Channel Research’s experience of running training programmes on evaluation, this course will be open to evaluators, managers and users of evaluations. It presents an opportunity for learning and exchange between participants from different backgrounds and who play different roles in humanitarian assistance and evaluations. The course is facilitated by Margie Buchanan-Smith and John Cosgrave.

Please follow the link here to download the course outline and application form. For any further information, please contact Maria Bak on bak@channelresearch.com

Please do not hesitate to forward these documents to potential participants.

Thank you very much for your attention,

Best Regards
Maria
Channel Research
45 Route des Marnières
1380 Lasne
Belgium
Mobile: +32 (0) 473 12 36 23
www.channelresearch.com