Relationships, Responsibilities, Relevance
26th Annual Conference of the American Evaluation Association
Minneapolis, Minnesota, USA
Conference: October 24-27, 2012
Workshops: October 22, 23, 24, 28

“Evaluation takes place in complex global and local ecologies where we evaluators play important roles in building better organizations and communities and in creating opportunities for a better world. Ecology is about how systems work, engage, intersect, transform, and interrelate. Complex ecologies are comprised of relationships, responsibilities, and relevance within our study of programs, policies, projects, and other areas in which we carry out evaluations.

Relationships. Concern for relationships obliges evaluators to consider questions such as: what key interactions, variables, or stakeholders do we need to attend to (or not) in an evaluation? Evaluations do not exist in a vacuum disconnected from issues, tensions, and historic and contextualized realities, systems, and power dynamics. Evaluators who are aware of the complex ecologies in which we work attend to relationships to identify new questions and to pursue new answers. Other questions we may pursue include:

• Whose interests and what decisions and relationships are driving the evaluation context?
• How can evaluators attend to important interactions amidst competing interests and values through innovative methodologies, procedures, and processes?

Responsibilities. Attention to responsibilities requires evaluators to consider questions such as: what responsibilities, inclusive of and beyond the technical, do we evaluators have in carrying out our evaluations? Evaluators do not ignore the diversity of general and public interests and values in evaluation. Evaluations in complex ecologies make aware ethical and professional obligations and understandings between parties who seek to frame questions and insights that challenge them. Other questions we may pursue include:

• How can evaluators ensure their work is responsive, responsible, ethical, equitable, and/or transparent for stakeholders and key users of evaluations?
• In what ways might evaluation design, implementation, and utilization be responsible to issues pertinent to our general and social welfare?

Relevance. A focus on relevance leads to evaluations that consider questions such as: what relevance do our evaluations have in complex social, environmental, fiscal, institutional, and/or programmatic ecologies? Evaluators do not have the luxury of ignoring use, meaning, of sustainability; instead all evaluations require continual review of purposes, evaluands, outcomes, and other matters relevant to products, projects, programs, and policies. Other questions we may pursue include:

• How can evaluators ensure that their decisions, findings, and insights are meaningful to diverse communities, contexts, and cultures?
• What strategies exist for evaluators, especially considering our transdisciplinary backgrounds, to convey relevant evaluation processes, practices, and procedures?

Consider this an invitation to submit a proposal for Evaluation 2012 and join us in Minneapolis as we consider evaluation in complex ecologies where relationships, responsibilities, and/or relevance are key issues to address.”

From: Monday 19th March 2012 to Tuesday 20th March 2012

Venue:  Birkbeck College, Malet Street, London

Organisers: London International Development Centre (LIDC); Association of Commonwealth Universities (ACU)

Background: Higher education for international development has been, in recent years, a neglected area relative to other educational interventions. Yet higher education (HE) is necessary for the attainment of Millennium Development Goals (MDGs) and for economic development in low and middle income countries.

The programme will be posted online soon.

Who should attend:

The conference will bring together experts from three research sectors: higher education, international development and impact evaluation from academia, think tanks, government agencies and civil society organisations. PhD students are welcome if their research is relevant to the theme of the conference.

Registration is open between 2 February and 5 March 2012.
To register, please fill in and return the registration form.
Attendance is free of charge.

April 29- May 1, 2012 (Sunday-Tuesday)
Seaview Resort, Galloway, NJ, USA. (http://www.dolce-seaview-hotel.com)

Organised by the Eastern Evaluation Research Society, a Regional Affiliate of the American Evaluation Association. Flyer available here

Keynote Speaker: Jennifer Greene, University of Illinois and President of AEA Featured Speakers: Eleanor Chelimsky, U.S. Government Accountability Office and former AEA President Rodney Hopson, Dusquesne University and incoming President of AEA

Sunday Afternoon Pre-Conference Workshops and Session: Meta Analysis Ning Rui, Research for Better Schools

Focus Group Research: Planning and Implementation Michelle Revels, ICF International

Career Talk with the Experts (NEW!): An unstructured conversation about your evaluation career This session is free to participants! Sunday Evening Interactive & Networking Session: John Kelley, Villanova University Concurrent Sessions Featuring: Skill Building Sessions, Individual Presentations & Panel Sessions

A full conference program will be posted at (www.eers.org) by Mid February 2012.

The International Council on Human Rights Policy has produced two documents under the above named project(See here for details of the project):

• No Perfect Measure: Rethinking Evaluationand Assessment of Human Rights Work. Report of a Workshop, January 2012. Contents: Introduction and Context,,A Brief History,,NGO Hesitations, The Shift, Assessing the Impact of Policy Research, Impact Assessment in the context of Advocacy, Impact Assessment in the context of Capacity Building and Development, The Donor perspective, Third-Party Perspectives—Building a bridge, A note on integrating Human Rights Principles into development work, References, Selected Additional Bibliographic Resources
• Role and Relevance of Human Rights Principles in Impact Assessment: An Approach Paper. July 2011. Contents: Introduction and Context, A Brief History, NGO Hesitations, The Shift, Assessing the Impact of Policy Research, Impact Assessment in the context of Advocacy
  Impact Assessment in the context of Capacity Building and Development
  The Donor perspective, Third-Party Perspectives—Building a bridge
  A note on integrating Human Rights Principles into development work
  References, Selected Additional Bibliographic Resources

PS 14 February 2012: It appears the ICHRP website is not working at present. I have uploaded a copy of the No Perfect Measure paper here

“The Center for Evaluation (CEval) of Saarland University, Germany, is a globally active research institute for applied social science in the field of evaluation and member of the DeGEval (German Evaluation Society). On this occasion, we organize an international conference about “The Future of Evaluation in Modern Societies” on 14th and 15th June 2012 in Saarbruecken, Germany.

The objective of this event is to discuss the role of evaluation in societies comprehensively and on an international comparison for bringing different discussion strands together into a joint debate. For keynote speeches and lectures, we could already win numerous renowned scientists from the USA, Latin America, Africa and Europe.

Please find the detailed program and registration form on our homepage: http://futureofevaluation.ceval.de

You also find a review about our recent book “A Practioner Handbook on Evaluation” which will appeal to evaluation practitioners, policy-makers who conduct evaluations in their daily work, students training in applied research and organizations which are implementing projects and programs that could be the subject of an evaluation.

—————————————

Maria Albrecht,  M.A., Center for Evaluation (CEval), Saarland University, P.O. Box 15 11 50, 66041 Saarbrücken – Germany, Fon: +49 (0)681 302-3561, Fax: +49 (0)681 302-3899, www.ceval.de

16 March 2012
The Macdonald Hotel, Birmingham

[from UKES website] UKES conferences address leading issues of the day in programme and policy evaluation. The 2012 Annual Conference will address the current drive towards evaluation focused on results – frequently linked to ‘Payment by Results’ and what, in international development and elsewhere, is familiar as ‘Results-Based Management’.

Evaluators and those who commission evaluation who advocate a focus on results reflect a legitimate concern with the productivity and efficiency of programmes and the capacity of interventions to secure gains and improvements in practice and provision. They point out that programmes should be held to account to accomplish what they were designed to do and paid for, often out of public funds. A primary focus on results seeks to emphasise main effects and outcomes that have been valued and agreed. In times of austerity and unusually scarce resources, proponents of a strong focus on results argue that emphasising value for money is socially responsible.

Others argue that an over-emphasis on measuring a programme’s results neglects important questions of how results are generated in a context, whether results capture the real quality and accomplishments of a programme, and how those results may reflect the values and ambitions of all programme stakeholders. They remind us of secondary effects and ‘unintended beneficiaries’ of programmes that may not be readily captured by results. Some also raise questions about the source of criteria over what counts as a worthwhile result given that not all programme achievements can be measured, and stakeholders may differ over a programme’s objectives. 

Against this background conference participants are invited to contribute their own perspectives on the dominant issues they consider relevant to the theory and practice of evaluation in the public interest. We anticipate a lively and informative debate to stimulate professional learning and to contribute to the improvement of evaluation practice and commissioning.

Potential contributors are invited to propose discussions, seminar presentations, lectures or poster sessions which explore issues around this theme. Those issues may fall within one of the following categories – though you are invited to propose your own theme:?

• How do we define a valid ‘result’ and whose results get counted?
• How do we best measure a result – including taking account of counterfactuals?
• How do we understand where results came from, what significance they have and whether they can be replicated – i.e. what is the relation between a result and context?
• Where do benchmarks come from to measure results achievement?
• If a result is, say, a 4% improvement – how do we know whether that is a lot or a little under the circumstances?
• How do we represent the circumstances and mechanisms that give rise to a result?
• How do we account for programme accomplishments that are not represented in results?
• Is results-measurement a robust foundation for replication/extension of a programme?

A formal call for papers and proposals for sessions will be circulated shortly.  The conference will be preceded on 15 March 2012 with a choice of training workshops on specialist topics.

On 19 October 2011, Oxfam Australia hosted a ‘Big Push Forward‘ event in Melbourne with the co-conveners of this initiative – Rosalind Eyben and Irene Guijt. Sixty development practitioners, including AusAid staff and academics came together to discuss whether the concerns voiced by the Big Push Forward project are relevant in Australia.

 HOW RELEVANT ARE THE ISSUES TO AUSTRALIA?

Following an introduction from Rosalind and Irene, we  had short inputs from three speakers on how these issues resonated in our part of the world.  Dennis Altman, from the Institute of Human Security, at La Trobe University suggested that the neo-liberal language which permeates Western society has been recast in the development world. into an auditing culture, focusing on evaluation, monitoring, and counting beans.  Marc Purcell the CEO of Australia’s International NGO umbrella group ACFID noted that the commitment to international aid in Australia is extremely brittle, and that the public debate about aid in Australia has led to a deep anxiety in government about how the aid programme is being perceived. But he argued that maybe it’s no bad thing for economists to look at the work of ‘pampered NGOs’. Jess Dart, the Managing Director of consulting company Clear Horizon, felt that whilst Australian NGOs do more internal evaluation than most there was a view expressed at this year’s Australasian Evaluation Conference that ‘development is the cowboy of evaluation’.  If we can’t tell the story of what we’ve done, people will ask for results. There are lots of really good methods out there and we can use these to offer solid alternatives to tell more complex stories of transformation.

OZIFYING THE THEMES

We explored the seven themes that the Big Push Forward website is focusing upon and discussed how  to ‘ozify’ them.  Groups worked on specific clusters to generate ideas for ‘pushing forward’.  This included 5 of the original 7 clusters and one new one focused on Program Design and Evaluation. After a process of ritual dissent, facilitated by Irene, final ideas were presented.

1.       Developing different methods of reporting

• “Making sense of it all”. Look at alternative forms of aggregation and sense-making using the cluster to make a call for ideas about what’s going on and how to engage others.  This should include exploring means of more direct citizen-to-citizen dialogue

2.  Reclaiming value for money

• “ Research to Surface Valuing and Values”. Collaborate amongst agencies, to find out what has been the experience of value for money in Australia and AusAID and the piloting of new approaches.  This would allow the main approaches to be understood, and the values that underpin them to be better understood.

3.  Organisational learning and reflective practice

• “A Practitioners’ forum to build commitment to organisational learning and reflective practice” . Set up an inter-agency  forum on M&E to look at: methodologies for reflective practice and organisational learning, build capacities to ensure better reflection, documenting different models for reflective practice, discussions around conceptual model and practical implications.

4.  Communicating to the public

• “Market complexity to the marketers”. Do more internally in our organisations with those that speak to the public and help them communicate the day-to-day practice on the ground.  We could explore the use of community forums to engage the public in terms of complexity of aid.

 5.   Challenging dominant discourses

• “ Unsettling storytelling from Aboriginal Australia”. Use story-telling and narrative methods to try to unsettle some of the debates about Results Based Management in order to ask the difficult questions e.g. what does development actually mean and what does it mean to those being developed? This would highlight the difficulties and create better understanding through creating this connection for Australian people.

6.  Program design

• “ Peer and Public Review for Good Practice”. Develop a forum that creates more space for program designs that hold the qualities of community participation, co-creation, & voice in determining the results. Use a virtual peer process from different agencies and public peer review enabling commentary from others.

 WRAPPING IT UP

In our final session four people were asked to share their reflections on what they heard

Chris Nelson from AusAID and soon to be with the World Bank Justice for the Poor program suggested there were lots of ingredients but we needed to bake the cake. He also noted that there were several opportunities to engage with AusAID in the next six months in particular as there are some gaps and a need for innovation in their current results framework. He also advised of the importance of building internal networks and coalitions within institutions on this agenda.

Rosalind Eyben remarked that there seemed to be several spaces for BPF engagement: internally in our own organisations; in communities of practice between organisations; with AusAID and internationally as Australia engages more strategically within the international aid system and international policy fora. Rosalind in particular liked the idea of story-telling here in Australia in order to communicate what actually happens to people and their small organisations as the results agenda impinges upon them.

Patricia Rogers, Professor in Public Sector Evaluation at RMIT University, noted that many of the ideas suggested are based on sharing good examples of practice.  She identified some common issues that need to be addressed: Who says it’s good practice? How do we distinguish between ‘this sounds like a good idea’ to ‘this has been done and we can show it makes a difference’? How do we create space to do things differently? How might we leverage resources together and with our partners? (see for example BetterEvaluation.org)

Jo Crawford, a Research and Policy Advisor at the International Women’s Development Agency, felt it critical to consider the tension between acting now on this agenda versus taking the time to define what we need to do.  Jo made a plea for process and participation in these conversations. Otherwise values will be narrowly defined. This requires a diversity of views and voices around the table when frameworks are being established

LOTS OF ENERGY

This was a day which demonstrated a lot of energy, ideas and willingness to engage on this agenda. After the meeting one of the participants Helen Sullivan, the new Director of the Centre for Public Policy (CPP) at Melbourne University wrote the following after attending a couple of other events which touched upon the same issues.

“…there is useful work that CPP can do in helping policy makers (politicians, professionals and ‘publics’) navigate the policy world they inhabit, which is not the world as it is often constructed in text books and guides where rational, reasonable actors with the necessary time and space define, design and deliberate policy problems and solutions to a ‘good’ end, but rather comprises actors who are motivated by reason and emotion, driven by evidence and values, and where the boundary between politics and policy is often blurred.  This is not to deny the potential value of evidence-based policymaking but to assert that politics, evidence and democratic institutions co-exist and that each are informed by values and power relationships that can facilitate and limit the quality of policy and policymaking.”

Hear! Hear!

We urge all participants who were at the Melbourne meeting, and others interested in this debate in Australia, to join the Big Push Forward and share your thoughts and experiences.

Chris Roche and Susan Hornbeck

25 October 2011

Date: 27 October 2011
Venue: Bonn, Germany

Check the Impact Toolbox: http://www.ngo-ideas.net/impact_toolbox/ and
the Field Experience: http://www.ngo-ideas.net/field_experiences/

NGO-IDEAs also has an advisory role to VENRO, the German development NGO’s umbrella body. VENRO recently published a position paper “Quality before Proof” that sets “empowerment” as one of four purposes of impact observation. http://www.venro.de/evaluation.html. The symposium takes place in the wider context of this paper that has generated some discussion in Germany.

Kind regards

Bernward Causemann/NGO-IDEAs

–
Bernward Causemann, www.causemann.org

Venue: University of Copenhagen
Date: May 21-23, 2012
Invitation and Call for Papers to International Workshop
Centre for Social Science Development Research

The Evaluation Department of the Danish Ministry of Foreign Affairs and the Centre for Social Science Development Research at Institute of Food and Resource Economics at the University of Copenhagen are pleased to invite you to submit an abstract (preparatory to a full paper) to an International Workshop, which will focus on methodological and practical considerations when using case studies in evaluations of development. The workshop will be organised in collaboration with the journal Evaluation:the international journal of theory, research and practice.

A large number of development evaluations are broader, learning-oriented evaluations based on cases at country, sector or project level. A key challenge in these evaluations is how to deal with the question of external validity. Within the field of development evaluation the methodological and practical debate on how to address this issue has been relatively limited.

The Organising Committee of the International Workshop is seeking abstracts that address theoretical/methodological challenges as well as more practical experiences when using case studies in learning-oriented development evaluations. Abstracts could e.g. focus on:

• Case sampling, analytical strategies and external validity
• Experience with balancing concerns for relevance and learning with analytical depth
• Meta, synthetic and comparative studies
• Theoretical and methodological frameworks
•  Country versus donor produced case studies
•  Integrating quantitative and qualitative methods

The workshop will be focussed around international development assistance and how it might be best evaluated through case studies. Whilst the Organising Committee is interested in lessons that can be learned from other fields, priority will be given to development assistance oriented contributions that also have something innovative to say about the use of case studies
and to original methodological contributions that can be linked with development cooperation.

Different types of evaluation experience are welcome: From evaluations dealing with different projects, to multi-country thematic evaluations and meta or synthesis studies. Similarly, presentations from participants with different perspectives (evaluation officers, researchers, evaluators, evaluation users) are encouraged, so as to enhance the discussion on the strengths and weaknesses, implications and prerequisites of using case studies.

Abstracts should be 150-200 words (1 page) in length, and should contain the title, purpose of the paper/presentation, methodology, and anticipated results or conclusions. Abstracts must contain enough information for the referees to evaluate your contributions. Please make sure to provide name of author(s), affiliation, senior author’s title, e-mail address, mailing address, and phone and fax numbers. The number of participants will be limited to 60.

The following deadlines apply:

• Abstract submission (in Word or PDF format): December 15, 2011
• Notification of acceptance of Abstract: February 15, 2012
• Registration deadline: May 4, 2012

Submission: Please e-mail your abstract or paper to olewin@um.dk

Note: The workshop will not be able to cover the costs of plane tickets and accommodation for participants.

Contact Information:
Ole Winckler Andersen
E-mail: olewin@um.dk
Telephone: +45 3392 0236 (cell phone: +45 2338 6328)
Postal Address:
Evaluation Department
Ministry of Foreign Affairs
Asiatisk Plads 2
DK – 1448 Copenhagen K
Denmark

Organizing Committee

Ole Winckler Andersen, Danida’s Evaluation Department
Henrik Hansen, University of Copenhagen
Elliot Stern, Evaluation: the international journal of theory, reserch and practice

Date: 17-18 October 2011
Venue: ANU, Canberra, Australia

Concept Note, Chris Roche & Linda Kelly, 4 August 2011

The Developmental Leadership Program (DLP)[1] addresses an important gap in international thinking and policy about the critical role played by leaders, elites and coalitions in the politics of development. At the core of DLP thinking is the proposition that political processes shape developmental outcomes at all levels and in all aspects of society: at national and sub-national levels and in all sectors and issue areas.

Initial findings of the DLP research program confirm that development is a political process and that leadership and agency matter. This is of course not new, but the DLP research provides important insights into how, in particular, leadership, elites and formal and informal coalitions can play a particularly important and under-recognized role in institutional formation (or establishing the ‘rules of the game’), policy reform and development processes[2].

International aid therefore needs to engage effectively with political processes. It needs to be flexible and be able to respond when opportunities open up. It needs to avoid the danger of bolstering harmful political settlements.

Furthermore Monitoring & Evaluation (M&E) mechanisms need to be improved and made compatible with flexible programming and recognize the importance of ‘process’ as well as outcomes. Donors should invest in a range of monitoring and evaluation techniques and approaches which are more aligned with the kinds of non-linear and unpredictable processes which characterise the kinds of political processes which drive positive developmental outcomes. This is important because it can be argued that, at best, current approaches are simply not appropriate to monitor the kinds of processes DLP research indicates are important; or, at worst, they offer few incentives to international assistance agencies to support the processes that actually lead to developmental outcomes

At the same time aid donors are increasingly under pressure to adopt results-based management processes and value for money assessments. There is also growing demand for more rigorous monitoring and evaluation, and greater transparency and public accountability[3].

In the light of these issues DLP will host a seminar on the topic of ‘Monitoring and Evaluation as if Politics Matters’.

The purpose of this event is to explore the strengths and weaknesses of current approaches to M&E, and suggest what ‘intelligent mixes’ of different approaches, methods and innovations might be tested by DLP in order to better track and understand how international assistance can best support the emergence and success of developmental leaderships, elites and coalitions.

In particular the workshop will explore what effective mix of different approaches would be required to:

• Capture changes in agency, leadership, elite formation, and coalitions, i.e. the processes that generate developmental leadership
• Evaluate the developmental outcomes (positive and negative) including in particular institutional and policy reform, associated with these changes
• Assess the contribution of international agencies and aid to these processes
• Address the challenges and opportunities that emerge from the results-based, ‘value for money’, and aid-and-development effectiveness agendas which are prominent concerns in donor thinking and policies at present.

The aim of the workshop will be to produce:

• A limited mixture of ‘testable’ methods, which would be trialled by DLP and its partners over the coming period; and
• Some guidance to the international development community about the value of different evaluative approaches for different development processes in the form of a workshop report.

It is our belief that no single method will be suitable to measure the effectiveness of this work.  We will need a range and mix of approaches that draws from the best quantitative, qualitative and participatory approaches, as well as innovations being generated through social network analysis and social media.

We believe that we therefore need a mix of expertise covering a range of M&E approaches.  The workshop will therefore include experts and researchers with different approaches to Monitoring and Evaluation and political analysis, staff responsible for managing quality and M&E processes in DLP partners, including AusAID, and development practitioners working on these issues from developing countries. It would involve between 15 and 20 people.

We envisage a maximum of 4/5 speakers with an appropriate balance between the following areas.

• Theories of Change and Complexity analysis
• Gendered M&E
• Social Network Analysis and M&E of Governance questions
• Quantitative analysis, Experimental and Randomised Control Trials
• Citizen Voice and crowd-sourcing options to generate feedback
• Specialists in assessing institutional and organizational change
• Specialists with expertise in different approaches to analyzing ‘Value for Money’ questions
  • Participatory M&E and Impact Assessment

Currently the confirmed speakers are:

• Patricia Rogers, Professor in Public Sector Evaluation and leader of the research program in Evidence Based Policy and Practice in the Sustainable Health and Well-Being Research Institute at RMIT, Melbourne (www.rmit.edu.au/rd/sustainablehealthandwellbeing) and author of ‘Purposeful Program Theory’ with Sue Funnell. Patricia is also a key player in the Impact Evaluation for Development initiative. ,
• Rosalind Eyben, Fellow at the Institute of Development Studies, University of Sussex and member of the Participation, Power and Social Change team (http://www.ids.ac.uk/index.cfm?objectid=BB000CE0-5056-8171-7B0079D6144E8C25) an ex Chief Social Development Advisor at DFID. Rosalind is the instigator of the Big Push Forward initiative (http://bigpushforward.wordpress.com/),
• Irene Guijt is an international expert in the application of learning-oriented knowledge processes in international development (http://www.cognitive-edge.com/directoryrecord.php?ID=4621).  She has provided research, advisory, and training services on social and organizational learning, in particular being known for her work on innovative thinking on monitoring, evaluation and learning, most recently engaged in experimenting with ‘SenseMaker’ (http://www.globalgiving.org/stories/))
  • Shawn Powers,  Policy Manager at the Abdul Latif Jameel Poverty Action Lab at MIT a research network specializing in randomized evaluations of social programs (http://www.povertyactionlab.org/powers)
  • Dr. Maria Melody Garcia is a researcher at the German Development Institute / Deutsches Institut für Entwicklungspolitik (DIE) working on impact evaluation . She is the author of Micro-Methods in Evaluating Governance Interventions, a recent overview of  impact evaluation methodologies currently used in the field of governance. focusing specifically on experimental and quasi-experimental designs see http://www.bmz.de/en/zentrales_downloadarchiv/erfolg/BMZ_WP_Micro.pdf
  • Esme Gaussen, is an independent monitoring, evaluation and value for money specialist currently based in South Africa.  She has previously worked for the UK National Audit Office on value for money assessments of DFID-supported programmes, and developed M+E tools for the NAO’s International Technical Cooperation Programme.

The format of the workshop will involve an introduction to DLP’s research and findings by Adrian Leftwich, and a short input on the challenges raised by DLP’s partners in the first M&E workshop (by Chris Roche or Linda Kelly). We are then proposing having pairs of experts responding to these challenges suggesting how they might be addressed by DLP, from different perspectives. Speakers would be asked to prepare presnetations in advance. This would be interspersed with Questions and Answers and discussion.

This would lead on to more workshop type sessions exploring testable approaches and methods. A final session might include sharing these findings with a broader range of development practitioners and experts in Canberra.

We envisage a workshop report, that will be part of the DLP research papers series, outlining an overall approach for the M&E of ‘working politically’ as described by DLP.

Chris Roche & Linda Kelly on behalf of the Developmental Leadership Program, 31 August 2011 (they are interested to hear about any research, experience or ideas that may be relevant to the topic)