Innovations in Monitoring and Evaluation ‘as if Politics Mattered’,

Date: 17-18 October 2011
Venue: ANU, Canberra, Australia

Concept Note, Chris Roche ^[2] & Linda Kelly, 4 August 2011

The Developmental Leadership Program (DLP)[1] ^[3] addresses an important gap in international thinking and policy about the critical role played by leaders, elites and coalitions in the politics of development. At the core of DLP thinking is the proposition that political processes shape developmental outcomes at all levels and in all aspects of society: at national and sub-national levels and in all sectors and issue areas.

Initial findings of the DLP research program confirm that development is a political process and that leadership and agency matter. This is of course not new, but the DLP research provides important insights into how, in particular, leadership, elites and formal and informal coalitions can play a particularly important and under-recognized role in institutional formation (or establishing the ‘rules of the game’), policy reform and development processes[2] ^[4].

International aid therefore needs to engage effectively with political processes. It needs to be flexible and be able to respond when opportunities open up. It needs to avoid the danger of bolstering harmful political settlements.

Furthermore Monitoring & Evaluation (M&E) mechanisms need to be improved and made compatible with flexible programming and recognize the importance of ‘process’ as well as outcomes. Donors should invest in a range of monitoring and evaluation techniques and approaches which are more aligned with the kinds of non-linear and unpredictable processes which characterise the kinds of political processes which drive positive developmental outcomes. This is important because it can be argued that, at best, current approaches are simply not appropriate to monitor the kinds of processes DLP research indicates are important; or, at worst, they offer few incentives to international assistance agencies to support the processes that actually lead to developmental outcomes

At the same time aid donors are increasingly under pressure to adopt results-based management processes and value for money assessments. There is also growing demand for more rigorous monitoring and evaluation, and greater transparency and public accountability[3] ^[5].

In the light of these issues DLP will host a seminar on the topic of ‘Monitoring and Evaluation as if Politics Matters’.

The purpose of this event is to explore the strengths and weaknesses of current approaches to M&E, and suggest what ‘intelligent mixes’ of different approaches, methods and innovations might be tested by DLP in order to better track and understand how international assistance can best support the emergence and success of developmental leaderships, elites and coalitions.

In particular the workshop will explore what effective mix of different approaches would be required to:

• Capture changes in agency, leadership, elite formation, and coalitions, i.e. the processes that generate developmental leadership
• Evaluate the developmental outcomes (positive and negative) including in particular institutional and policy reform, associated with these changes
• Assess the contribution of international agencies and aid to these processes
• Address the challenges and opportunities that emerge from the results-based, ‘value for money’, and aid-and-development effectiveness agendas which are prominent concerns in donor thinking and policies at present.

The aim of the workshop will be to produce:

• A limited mixture of ‘testable’ methods, which would be trialled by DLP and its partners over the coming period; and
• Some guidance to the international development community about the value of different evaluative approaches for different development processes in the form of a workshop report.

It is our belief that no single method will be suitable to measure the effectiveness of this work.  We will need a range and mix of approaches that draws from the best quantitative, qualitative and participatory approaches, as well as innovations being generated through social network analysis and social media.

We believe that we therefore need a mix of expertise covering a range of M&E approaches.  The workshop will therefore include experts and researchers with different approaches to Monitoring and Evaluation and political analysis, staff responsible for managing quality and M&E processes in DLP partners, including AusAID, and development practitioners working on these issues from developing countries. It would involve between 15 and 20 people.

We envisage a maximum of 4/5 speakers with an appropriate balance between the following areas.

• Theories of Change and Complexity analysis
• Gendered M&E
• Social Network Analysis and M&E of Governance questions
• Quantitative analysis, Experimental and Randomised Control Trials
• Citizen Voice and crowd-sourcing options to generate feedback
• Specialists in assessing institutional and organizational change
• Specialists with expertise in different approaches to analyzing ‘Value for Money’ questions
  • Participatory M&E and Impact Assessment

Currently the confirmed speakers are:

• Patricia Rogers, Professor in Public Sector Evaluation and leader of the research program in Evidence Based Policy and Practice in the Sustainable Health and Well-Being Research Institute at RMIT, Melbourne (www.rmit.edu.au/rd/sustainablehealthandwellbeing ^[6]) and author of ‘Purposeful Program Theory’ with Sue Funnell. Patricia is also a key player in the Impact Evaluation for Development initiative ^[7]. ,
• Rosalind Eyben, Fellow at the Institute of Development Studies, University of Sussex and member of the Participation, Power and Social Change team (http://www.ids.ac.uk/index.cfm?objectid=BB000CE0-5056-8171-7B0079D6144E8C25 ^[8]) an ex Chief Social Development Advisor at DFID. Rosalind is the instigator of the Big Push Forward initiative (http://bigpushforward.wordpress.com/ ^[9]),
• Irene Guijt is an international expert in the application of learning-oriented knowledge processes in international development (http://www.cognitive-edge.com/directoryrecord.php?ID=4621 ^[10]).  She has provided research, advisory, and training services on social and organizational learning, in particular being known for her work on innovative thinking on monitoring, evaluation and learning, most recently engaged in experimenting with ‘SenseMaker’ (http://www.globalgiving.org/stories/ ^[11]))
  • Shawn Powers,  Policy Manager at the Abdul Latif Jameel Poverty Action Lab at MIT a research network specializing in randomized evaluations of social programs (http://www.povertyactionlab.org/powers ^[12])
  • Dr. Maria Melody Garcia ^[13] is a researcher at the German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE) working on impact evaluation . She is the author of Micro-Methods in Evaluating Governance Interventions, a recent overview of  impact evaluation methodologies currently used in the field of governance. focusing specifically on experimental and quasi-experimental designs see http://www.bmz.de/en/zentrales_downloadarchiv/erfolg/BMZ_WP_Micro.pdf ^[14]
  • Esme Gaussen, is an independent monitoring, evaluation and value for money specialist currently based in South Africa.  She has previously worked for the UK National Audit Office on value for money assessments of DFID-supported programmes, and developed M+E tools for the NAO’s International Technical Cooperation Programme.

The format of the workshop will involve an introduction to DLP’s research and findings by Adrian Leftwich, and a short input on the challenges raised by DLP’s partners in the first M&E workshop (by Chris Roche or Linda Kelly). We are then proposing having pairs of experts responding to these challenges suggesting how they might be addressed by DLP, from different perspectives. Speakers would be asked to prepare presnetations in advance. This would be interspersed with Questions and Answers and discussion.

This would lead on to more workshop type sessions exploring testable approaches and methods. A final session might include sharing these findings with a broader range of development practitioners and experts in Canberra.

We envisage a workshop report, that will be part of the DLP research papers series, outlining an overall approach for the M&E of ‘working politically’ as described by DLP.

Chris Roche ^[2] & Linda Kelly on behalf of the Developmental Leadership Program, 31 August 2011 (they are interested to hear about any research, experience or ideas that may be relevant to the topic)