By Sadie Watson And Juliet Pierce. September 2008. DEPARTMENT FOR INTERNATIONAL DEVELOPMENT. Evaluation Report WP27
Executive Summary
In 2007, a tool and process was developed for improving the recording and impact of policy dialogue initiatives across DFID. It was based on an adaptation of current project cycle management (PCM) requirements for programme spending. A pilot was devised to test the proposed tool and process in terms of:
• Assessing the value in recording and monitoring policy related activities in a similar way to that of spend activities;
• Finding the most effective and useful approach in terms of process;
• Identifying succinct ways to capture intentions and to measuring performance;
• Clarifying the type and level of support and guidance required to roll the process out across DFID.
The ten participating pilot teams represented different aspects of DFID’s policy work, conducting different types of policy dialogue activities. The consultants were asked to monitor and evaluate the six month pilot. They were also asked to review approaches to managing and monitoring policy dialogue and influencing activities in other organisations. This report highlights some lessons and observations from the pilot. It outlines some emerging issues and provides some pointers for DFID to consider as it continues to develop into an organisation where policy dialogue and influencing are increasingly important aid tools.
The review highlighted that there are some examples of good practice in policy dialogue planning within DFID, although the systems and tools used vary, and valuable lessons are not being systematically captured. However, there is little evidence that teams are reviewing and evaluating this type of work.
There has been much discussion about the advantages and disadvantages of using logframes for helping to manage policy dialogue initiatives. The rationale for using a logframe approach in the pilot was that it was a recognised tool in the DFID Project Cycle Managmenet approach. It became clear early on in the study that a number of staff had not had experience or training in such approaches and tools, and therefore neither saw the value in them, nor knew how to apply them to managing their policy dialogue work.
Those that were already familiar with the logframe found it to be a very useful tool for strategising, monitoring and sharing their approach with their colleagues. The advantages and disadvantages of using the logframe tool for this type of work can be found in Table 5.
Both staff from some of the pilot teams, and people doing similar work within other organisations have used a number of different tools to help them plan and manage their policy dialogue initiatives. These include scenario planning, outcome mapping, using timelines, After Action Reviews and the use of Microsoft Project Software. These were all considered valuable and useful.
The pilot highlighted the tension between valuing and adopting a Projec Cycle Management approach and attempting to embed the system within ARIES/PRISM (DFID’s central monitoring systems). Completing forms and filling in boxes made people feel as though it was a bureaucratic exercise rather than recognising the value of the process of reviewing, planning and learning. Some of the potential risks and benefits of recording Project Cycle Management information on PRISM have been outlined in section 4.1.a of the report.
All respondents noted that it would be difficult to carry out a rigorous cost-benefit analysis for policy dialogue activities. However, there is a recognition that DFID staff should be able to quantify approximately the staff and other resources that go into this activity and then to relate this to the progress they are achieving. Such an effort would encourage prioritisation and allow managers to justify the use of finite resources.
The review also highlighted the weak stakeholder management systems wthin DFID. In an organisation that is characterised by high staff turnover, this runs the risk of affecting its ability to build relationships and influence effectively.
DFID has several information systems and several reporting systems. It will be important to ensure that any system for monitoring policy dialogue activities is well integrated with these. Since these departmental reporting systems tend to be pitched at a fairly high level or to fall within personal development plans it is important that any new system captures the texture of policy dialogue work at the “middle level”, to ensure that it does not fall off DFID’s radar. The report discusses the issue of types and levels of policy work to include on the new system.
Deliberations around the pilot have raised the question of how far “old” ways of working that are characterised by DFID spend programmes will increasingly give way to “new” ways of working characterised by policy dialogue. If the institutional culture is changing should new ways of working be tied into older systems? This question goes much wider than the pilot and involves issues of staff skills, recruitment priorities, induction and training, management and management information systems. We believe the concepts of project and project cycle management, embedded into team working within DFID, are still the right ones to traverse this institutional cultural change.
Not all policy work lends itself to Project Cycle Managmenet. It would be difficult to apply this approach to some of the reactive work to defend policy stances that is being done by policy teams. However, more proactive policy work implies a desired outcome in the future, such as changed attitudes by key players, or researching and developing policy and getting it implemented. This type of work does lend itself to a Project Cycle Managmenet approach, with a budget for the staff and other inputs, and an estimation of interim outcomes along the way to the intended overall outcomes and impact.
There is therefore a need for a basic core system on PRISM/ARIES and access to a range of Project Cycle Management tools. New staff need routine Project Cycle Management training with an indication of how different planning tools can be varied to accommodate the challenges of their policy work. Leadership will also be ke
