Date: 3-5 October 2012
Venue: Helsinki, Finland

EVALUATION IN THE NETWORKED SOCIETY: NEW CONCEPTS, NEW CHALLENGES, NEW SOLUTIONS

The Tenth Biennial Conference of the European Evaluation Society will be the international evaluation event of the year. It will be held in Helsinki, Finland during 3-5 October 2012 (pre-conference workshops 1- 2 October). Mark your calendars!!

Evaluators are living in times of unprecedented challenge and opportunity. The networked information environment is inducing fundamental changes in culture, politics and society. Whereas the industrial society was reliant on centralised, hierarchical, high cost information systems, the networked society is characterised by decentralised, voluntary and cheap information exchange.

The advent of social networking without borders will have fundamental implications for evaluation agendas and methods. First, it will redefine the value and legitimacy of evaluation in global social accountability networks and accelerate the internationalisation of evaluation. Second, evaluation cultures, structures and processes will have to deal  with the limitless quantity, speed and accessibility of information generated by new technologies, e.g. drawing useful meaning from huge data bases, assessing the validity of an exploding number of rating systems, league tables, etc. in ways consistent with democratic values of freedom of expression and protection of privacy.

The new information technologies offer new ways of making authority responsible and accountable as well as bringing real time citizen involvement and reliable information to bear on public policy making. What are the implications of an information economy that allows instant connectivity to thousands of program beneficiaries suddenly able to make their voices heard? Will the spread of mobile telephony to the weakest and most vulnerable members of society and the rising power of social networks act as evaluative and recuperative mechanisms or will they merely aggravate social instability? What are the risks of network capture by single or special interest groups and cooptation of evaluation?

The rise of the evaluation discipline is inextricably linked to the values central to any democratic society. How will these values be protected in a context where weak links and increasing inequalities have created new fissures in society? How will evaluation independence be protected against the pressures of vested interests intent on retaining control over the commanding heights of the society?

To help explore these and other issues relevant to the prospects of evaluation in Europe and beyond the Conference will stimulate evaluators to share ideas, insights and opinions about a wide range of topics that will throw light on the future roles of evaluation in the networked society. The Conference will help draw evaluation lessons learnt in distinct sectors and regions of the world. It will also examine the potential of alternative and mixed evaluation methods in diverse contexts and probe the challenges of assessing public interest in complex adaptive systems and networks.

To these ends the Conference will offer participants a wide choice of vehicles for the transmission of evaluation experience and knowledge: keynote speeches, paper presentations, panel debates, posters, etc.  As in past years the EES Conference will aim at a pluralistic agenda that respects the legitimacy of different standpoints, illuminates diverse perspectives and promotes principled debate. The Conference will also provide an opportunity for evaluation networks to interact and improve the coherence of their activities.

We look forward to welcoming you in Helsinki. It is one of the world leaders in modern design and it provides Europe with a a world class high tech platform. It also boasts a 450 year history and lays claim to being the warmest, friendliest, most “laid back” city of Northern Europe. Its nearby archipelago of islands offers an ideal environment for sea cruises and its neighboring old growth forests provide an idyllic setting for restful nature walks. We promise you an enjoyable as well as a professionally rewarding time!!

Ian Davies, President, European Evaluation Society
Maria Bustelo, Vice President and President Elect, European Evaluation Society

www.europeanevaluation.org 

Paper published June 2010. Produced by Innovations for Scaling Impact and Keystone Accountability. Funded by the International Development Research Center and the Packard Foundation. (Download pdf version here)

“Purpose: This paper reviews the current field of network monitoring and evaluation with the goal of identifying where progress has been made and where further work is still needed. It proposes a framework for network impacts planning, assessment, reporting and learning that can help to close some of the current gaps in network evaluation while building on the advances that have been made. This document is written for practitioners undertaking network evaluation and foundation program staff working to support networks.

Introduction and Context: Networking and networks have become increasingly and consciously utilized as organizing strategies and structures for creating social change in this world. The growing realization that no single actor, no matter how effective they are, is capable of tackling today’s social problems has spurred a flurry of international interest and investment in networks from a wide range of actors and sources. The World Bank began funding networks through its Global and Regional Partnership Program
in the mid-1990’s and is now currently supporting approximately 175 partnership programs, having spent $3.5 billion in 2006 alone.  The Climate Works Foundation has organized more than ten funders and organizations from across sectors and geographies as part of a billion dollar coordinated campaign to fight climate change. The Global AIDS Alliance has launched an
advocacy network across six African countries in a Campaign to End  Pediatric HIV/AIDS. Grant makers like the Ford Foundation, Anne E. Casey Foundation, MacArthur Foundation and many others have made the funding of networks a key part of their portfolio and grant making strategy. At the same time, the growing demand for more success from social change public and private initiatives has created an explosion of interest in and demand for increased and improved monitoring and evaluation and impact evaluation. Governments, donors and practitioners are all feeling pressure to demonstrate and report on the impact of their work. Together, these two trends have created a growing appetite for the monitoring and evaluation and impact evaluation of networks. An increasing number of methods, tools and metrics have been proposed, developed and piloted in response to this demand. While important steps have been taken, the field of network monitoring and evaluation is still, in theory and even more so in practice, in its infancy.

This paper is divided into four sections. Section one provides a literature review of the current field of network monitoring and evaluation describing various current metrics and tools for network monitoring and evaluation. The section concludes with a discussion of the key gaps in the current field. Section two examines the key characteristics of networks and the implications these have for evaluation. Section three further elaborates the specific challenges of network evaluation and desirable characteristics for a network monitoring and evaluation system. Section four introduces a framework for network monitoring and evaluation currently being developed by iScale and its partners and explains how it can be implemented.”

For questions and/or comments on this paper please contact Catrina Lucero at clucero@scalingimpact.net

by Todd Honeycutt, Issue Brief, November 2009, Number 1,  Mathematica Policy Research, Inc. 4 pages

Abstract: “Social network analysis (SNA) is a methodological approach to measuring and mapping relationships. It can be used to study whole networks, all of the ties within a defined group, or connections that individuals have in their personal communities.The resulting graph-based structures illustrate the composition and effectiveness of networks on a variety of levels. Programs that can benefit from a social network perspective are common in evaluation research. This brief introduces the concepts behind SNA and illustrates how to use this method in process and outcome evaluations.”

Final Report, October 2007, Hanne Lund Madsen, h.lund.madsen@email.dk

For The Danish Child & Youth NGO Network, The Danish NGO Education Network, Thematic Forum, Gendernet, Aidsnet

The overall aim of this study is to:

• Provide a solid basis for discussion of the future perspectives for NGO Networks including a synthesis and analysis of experiences regarding NGO networks in Denmark and aboard

• Discuss the networking concept on the basis of the study and create a framework for common positioning and decision among Danish NGO Networks regarding possible scenario for future networking cooperation.

A Reference Group appointed by the Networks involved has been engaged in the study and has provided input and reflection during the process, including participation in a mini-scenario workshop.

Some of the findings have also been shared within the larger NGO Community at a seminar on the revision of the Danida Civil Society Strategy and the observations taken into account. A debriefing has taken place with the Reference Group and the Final Report has been approved. In light of the opportunities presented in connection with the revision of the Danida Civil Society Strategy, the Reference Group is currently considering the most appropriate forum for discussion of the future architecture of the Danish Networks.

The Networks have asked the following questions that have guided the study:

Focus Question One: When and how have the networks contributed most to improving the quality of the Danish NGOs work and performance?

Focus Question Two: Which other aims and long-term visions could there be for the Danish Networks beyond that of knowledge sharing and capacity development?

Focus Question Three: Which structures and processes can be used in networks in particular structures other than the present dominant model?

Focus Question Four: How can the sustainability of the networks be ensured in the long run?

Finally, the TOR request that three different future scenarios for the Networks be identified, which can serve as a basis for reflection and clarification of positions.

The report is structured in three main parts: Chapter II provides the meta-reflections around networking with observations pertaining to the Danish experiences, Chapter III gives examples of networking structures and purposes in neighbouring countries and Chapter IV presents four scenarios for NGO relationships within networking and beyond.

by Rick Davies (April 2009).

This paper was produced for the Communication for Social Change Consortium, as a contribution to their paper for UNAIDS on reviewing approaches to monitoring and evaluation and advocating an expanded monitoring and evaluation framework for social change communication. All rights to this paper are with the Communication for Social Change Consortium (www.cfsc.org).]

Contents

1.Background..
2. What is Social Network Analysis? A brief introduction..
3. The use of SNA in the study of HIV/AIDS..
4. The use of SNA in the evaluation of HIV/AIDS interventions..
5. How could SNA be useful in the evaluation of HIV/AIDS programs?.
5.1. Within organisations: Moving from Logical to Social Frameworks.
5.2. Within organisations: Moving beyond linear models.
5.2.1 Mapping and modeling.
5.2.2 Looking inside and outside the network.
5.2.3 Matrix versus network models.
5.3. Amongst multiple organisations: Where there is no central planner.
6. The uses of theory..
7. Scalability..
8. Limitations..
9 Opportunities..
10. An Afterword..
References..

Ben Ramalingam, Enrique Mendizabal and Ed Schenkenberg van Mierop April 2008

The NFA is now undergoing a process of review. Rick Davies (manager of www.mande.co.uk) has provided ODI with a 2 page >comment< on the Background Note on the NFA

Bruce Hoppe, Ph.D. Connective Associates LLC
bruce@connectiveassociates.com
Claire Reinelt, Ph.D. Leadership Learning Community
claire@leadershiplearning.org
January 19, 2009

Abstract
Leadership development practitioners have become increasingly interested in networks as a way to strengthen relationships among leaders in fields, communities, and organizations. This paper offers a framework for conceptualizing different types of leadership networks and uses case examples to identify outcomes typically associated with each type of network. One challenge for the field of leadership development has been how to evaluate leadership networks. Social Network Analysis (SNA) is a promising evaluation approach that uses mathematics and visualization to represent the structure of relationships between people, organizations, goals, interests, and other entities within a larger system. Core social network concepts are introduced and explained to illuminate the value of SNA as an evaluation and capacity-building tool.

Full text here

Christina Prell, Klaus Hubacekb, Mark Reed, Department of Sociological Studies, University of Sheffield and  Sustainability Research Institute, School of Earth and Environment, University of Leeds, 2009

Full text here

Introduction

Many conservation initiatives fail because they pay inadequate attention to the interests and characteristics of stakeholders. (Grimble and Wellard, 1997). As a consequence, stakeholder analysis has gained increasing attention and is now integral to many participatory natural resource management initiatives (Mushove and Vogel, 2005). However, there are a number of important limitations to current methods for stakeholder analysis. For example, stakeholders are usually identified and categorized through a subjective assessment of their relative power, influence and legitimacy (Mitchell et al., 1997; Frooman, 1999). Although a wide variety of categorization schemes have emerged from the literature (such as primary and secondary (Clarkson, 1995), actors and those acted upon (Mitchell et al., 1997); strategic and moral (Goodpaster, 1991); and generic and specific (Carroll, 1989) methods have often overlooked the role communication networks can play in categorizing and understanding stakeholder relationships. Social network analysis (SNA) offers one solution to these limitations.

Environmental applications of SNA are just beginning to emerge, and so far have focused on understanding characteristics of social networks that increase the likelihood of collective action and successful natural resource management (Schneider et al., 2003; Tomkins and Adger, 2004; Newman and Dale, 2004; Bodin et al., 2006; Crona and Bodin, 2006). In this paper, we harness and expand upon this knowledge to inform stakeholder analysis for participatory natural resource management. By participatory natural resource management we mean a process that engages stakeholders on multiple levels of decision making and facilitates the formation and strengthening of relationships among stakeholders for mutual learning (Grimble and Wellard, 1997; Dougill et al., 2006; Stringer et al., 2006). To enhance stakeholder analysis, we use SNA to identify the role and influence of different stakeholders and categories of stakeholder according to their positions within the network. We do this using case study material from the Peak District National Park, UK.

Two papers by Ricardo Wilson-Grau

Evaluating the Effects of International Advocacy Networks, Ricardo Wilson-Grau and Martha Nu. “This “think piece” will first sketch the special challenges of evaluating the effects of the advocacy work of international social change networks. I will then present the approach to evaluating advocacy that I use. This essay is an adaptation of my most recent writing on the broader subject of “Complexity and International Social Change Networks,” which is a chapter in a book by the Global Partnership for the Prevention of Armed Conflict.These networks almost by definition have an advocacy component, which often is their central activity. Furthermore, my belief is that to a greater or lesser extent, the challenges and the general evaluation methodology I outline in this essay are applicable to almost all social change organisations. I leave that judgement, however, to the reader. ”

This is a paper presented at the Advocacy Impact Evaluation Workshop at the Evans School for Public Affairs, University of Washington, 4-6 December 2007, Seattle, WA, USA, sponsored by the Bill and Melinda Gates Foundation. A Spanish version “Evaluación de las redes internacionales de cambio social – Efectos y desafíos de las redes internacionales de incidencia” was published by Futuros 21: http://www.futuros21.infodetalle_articulo.asp?id_articulo=55

Evaluating International Social Change Network: A conceptual framework for a participatory approach”: Ricardo Wilson-Grau, 2007. “International networks for social change are growing in number and infuence.While they need to be able to assess the extent to which they achieve their purpose and determine ways in which to be more effective, conventional evaluation methods are not designed for such complex organisational forms, or for the diverse kinds of activity to which they are characteristically dedicated. Building on an earlier version of their paper, the authors present a set of principles and participatory approaches that are more appropriate to the task of evaluating such networks.”

Published in Development in Practice, Volume 17, Number 2, April 2007

>A report< to the Science and Human Rights Program of the American Association for the Advancement of Science Skye Bender-deMoll April 28, 2008. 47 pages

Abstract: This report investigates potential new tools and existing applications of network analysis and network mapping to assist or facilitate human rights work. It provides a very brief overview of some network concepts, quick introductions to a number of relevant fields of research, and some specific examples of how people are currently using network tools for academic and applied work. The examples serve as an overview and entry point to the research areas. As this is a developing and fragmented field, classification is difficult. However, some common points exist and a few conclusions are presented. Some of the risks and challenges of network research are discussed, along with criteria for evaluating potential future projects. Finally, several possible projects are proposed.