Next Generation Network Evaluation

Paper published June 2010. Produced by Innovations for Scaling Impact and Keystone Accountability. Funded by the International Development Research Center and the Packard Foundation. (Download pdf version here)

“Purpose: This paper reviews the current field of network monitoring and evaluation with the goal of identifying where progress has been made and where further work is still needed. It proposes a framework for network impacts planning, assessment, reporting and learning that can help to close some of the current gaps in network evaluation while building on the advances that have been made. This document is written for practitioners undertaking network evaluation and foundation program staff working to support networks.

Introduction and Context: Networking and networks have become increasingly and consciously utilized as organizing strategies and structures for creating social change in this world. The growing realization that no single actor, no matter how effective they are, is capable of tackling today’s social problems has spurred a flurry of international interest and investment in networks from a wide range of actors and sources. The World Bank began funding networks through its Global and Regional Partnership Program
in the mid-1990’s and is now currently supporting approximately 175 partnership programs, having spent $3.5 billion in 2006 alone.  The Climate Works Foundation has organized more than ten funders and organizations from across sectors and geographies as part of a billion dollar coordinated campaign to fight climate change. The Global AIDS Alliance has launched an
advocacy network across six African countries in a Campaign to End  Pediatric HIV/AIDS. Grant makers like the Ford Foundation, Anne E. Casey Foundation, MacArthur Foundation and many others have made the funding of networks a key part of their portfolio and grant making strategy. At the same time, the growing demand for more success from social change public and private initiatives has created an explosion of interest in and demand for increased and improved monitoring and evaluation and impact evaluation. Governments, donors and practitioners are all feeling pressure to demonstrate and report on the impact of their work. Together, these two trends have created a growing appetite for the monitoring and evaluation and impact evaluation of networks. An increasing number of methods, tools and metrics have been proposed, developed and piloted in response to this demand. While important steps have been taken, the field of network monitoring and evaluation is still, in theory and even more so in practice, in its infancy.

This paper is divided into four sections. Section one provides a literature review of the current field of network monitoring and evaluation describing various current metrics and tools for network monitoring and evaluation. The section concludes with a discussion of the key gaps in the current field. Section two examines the key characteristics of networks and the implications these have for evaluation. Section three further elaborates the specific challenges of network evaluation and desirable characteristics for a network monitoring and evaluation system. Section four introduces a framework for network monitoring and evaluation currently being developed by iScale and its partners and explains how it can be implemented.”

For questions and/or comments on this paper please contact Catrina Lucero at clucero@scalingimpact.net

One thought on “Next Generation Network Evaluation”

  1. In June 2009 I provided the following feedback to Keystone on a draft version of one instrument mentioned on pages 13-14 of this report, the Comparative Constituency Feedback (CCF):

    Hi Natalie

    Re the actors involved in this draft questionairre

    I went through all the questions and tried to categorise them in terms of the types of actors that were being inquired about. The results:

    1. Questions about the respondent: 7 (in Section 1)

    2. Questions about respondent’s relationships with the secretariat (or equivalent): 4 (in Section 2)

    3. Questions about respondent’s relationships with other parts of the network (other than the secretariat or equivalent): 1 (in Section 2)

    4. Questions about respondent’s relationships with the whole network: 15 (in Sections 2,3,4,5)

    The content of the questionaire seems to embody an oversimplified /outdated view of how networks are structured (i.e. the respondent, the secretariat and the whole network). In contrast to hierarchically structured organisations, (named) networks are often believed to allow their members more freedom of association with each other. Their ability to self-organise is seen as a strength, not as a risk. If so, inquiries about the nature and structure of members relationships with each other should be much more central to the design of the questionnaire. At present they are marginalised, featuring in only one question.

    The question is whether the clients of this instrument want to examine this dimension of their networks. Or are they simply secretariats and governing board members that want some crude measure of “how they are doing”?

    If they do want to pay attention to the self-organising and emergent nature of their networks, then additional (or replacement) questions will need to be asked about:(a) kinds of relationships’ respondents have with other members, (b) the kinds of other members they have relationships with. This could be done in a question with a matrix format. (BTW, it is important that the respondents should also identify which of these categories they belong to). The resulting data could be converted into network diagrams. There will be no “correct” network structure, but comparisons could be made of the structure that is found with people’s views (especially the secretariat’s) of the structure they expected to see. A generalisable measure of “good performance” here would be adequate self-knowledge of one’s own network.

    Other data on the structure of the network could collected by asking how members are connected to each other by their shared issue concerns. A menu of issues can be created and each respondent can be asked to prioritise the top 5 that are most important to them. The responses can later be analysed and displayed to show how members are connected to each other by shared concerns with one or more issues. See an example of a survey form with this function here: http://www.optimice.com.au/friends_join.php

    regards, rick davies

Comments?

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: