PAQI How-to-do-it Annex

This is a technical annex to https://www.mande.co.uk/special-issues/participatory-aggregation-of-qualitative-information-paqi/

Data processing steps

The network diagrams were produced using UCINET & NetDraw (a package). Very briefly, this involved producing the following files:

  • For relationships between sorted items:
    • Create a .txt file in a specific Dl file format, known as PARTITION, as shown in this example
      • This shows five sets of sort results, seperated by a # marker. With each set, each row shows a set of items put into one group, by the participant
    • Convert this to the first Ucinet file, using these commands: Data>Inputs text file>Input text file in DL format:
    • Aggregate the five sets of data into one items x items matrix, by using these commands: Transform>Matrix Operations>Within Dataset>Aggregations>Input dataset: [the new file you created], Sum, Break-out resultsby: rows and columns
    • Then view the saved file in Netdraw. View with link strength >1, because you want to see the connections created by multiple participants, not one.
  • For relationships between categories used
    • Take the original .txt file in PARTITION format and re-structure it as a .txt file in another Dl file format known as EDGELIST2, as shown in this example.
      • N=66 because there are 24 items and 42 categories. A1-4 are categories used by the first respondent, B1-6, by the second etc. Each row lists items put in that category
    • Convert this to the second Ucinet file, using these commands: Data>Inputs text file>Input text file in DL format:
    • This shows a categories x items matrix
    • This needs to be converted to a one mode matrix, of categories x categories. Use these commands: Data>Affiliations (2-mode to 1-mode)>Input data set:
    • Then view the saved file in Netdraw. View with link strength >1, because all categories will have at least one shared item with others.
      • PS: You can also use Netdraw to visualise the two-mode categories x items matrix (See Necheles reference below)
  • For relationships between the respondents
    • Use these commands: Tools>Similarities (e.g.correlations,)> Input Datset: name of first Ucinet file above, Measure of profile similarity: Correlation,  Compute similarities amongsts: Columns.
    • You then have a matrix of correlation values, ranging from 0 to 1. To make these easier to discriminate, when using NetDraw, it is best to multiple them by 100. Use these commands: Transform>Matrix Operations>Within Dataset>Cellwise Transformations>Multiply by constant
    • Then view the saved file in Netdraw. Focus on relationships with above average strength (because all participants will have some similarities in their classifications)

PS:  I have set up a seperate posting on the merits of different kinds of social network analysis software, including UCINET and NetDraw.

Adding qualitative “flesh” to the quantitative “bones”

The network diagrams are the structure. They are the results of all the sorting activities by all the participants. But in the process of sorting the items each participants also added qualitative information, in the form of descriptions of the categories they created. In the Indonesian example 33 category descriptions were provided by the 5 participants. This next section will describe how that qualitative information can be made accessable, as people explore the individual nodes and links in the network diagrams. This information will be in the form of node and link attributes.

With the Indonesian data  I listed the members of each grouping of items in a row, and then in an adjacent column I entered the text description of that group given by the participant. When all the groupings of one respondent were entered I started with the next respondent’s groupings on the rows below

The challenge is to then collate all text descriptions that apply to a given item and to do that for all items, in a way that is not manually time consuming. To do this I set up a list of items (in rows), and in adjacent columns I set up a logic function that in effect searched for relevant text. A copy of the Excel sheet will be attached here.

This data then needs to be  put into an attribute.txt format (example here) and then imported into Netdraw as an attribute file, when already viewing the item x  item network (File>Open>VNA text file>Attributes). Then any node can be double right clicked to view its attributes, including all the descriptions given to it by the participants (See example). Bear in mind these are descriptions of the categories it belongs to, not that specific item.

Measuring Effectiveness Conference 2009: Presentations available

The presentations from this year’s ME conference in Melbourne are available on the World Vision Australia website:

http://www.worldvision.com.au/Learn/Conferences/MeasuringEffectiveness/2009_Presentations.aspx

Twitter feeds on M&E: A List

If you know of others, please leave a comment below

  • @aeaweb/evaluation Contributors with significant focus on evaluation
  • 3ieNews 3ie seeks to improve the lives of poor people in low- and middle-income countries by providing, and summarizing, evidence of what works

Perspectives on partnership: A literature review

By Douglas Horton, Consultant, Gordon Prain, International Potato Center,
Graham Thiele, International Potato Center, November 2009. 101 pages

Hard copy available via the IPC website

Download pdf here

Abstract

This paper reports on a wide-ranging review of the literature on partnerships and other closely related forms of collaboration. It aims to contribute to knowledge of the actual and potential roles of partnership in international agricultural research for development.

The paper summarizes conclusions and insights from four distinct professional literatures: research studies; professional evaluation literature; practitioner-oriented reviews, guidelines and assessment tools; and CGIAR-related reviews, evaluations and policy documents. It identifies and analyzes key cross-cutting themes and success factors, highlights gaps in current knowledge, and identifies high-potential areas for further study. A wide range of research-based publications is reviewed, including studies in such fields as management and organizational development, public administration, economics and international development. Work in these fields covers such diverse topics as the role of inter-organizational collaboration in strategic management, public–private and cross-sector partnerships, North–South partnerships, roles of partnership in linking research with action, networking and transactions costs.

The different literatures talk little to each other and are highly  self-referential. Nevertheless, some common patterns, themes and concerns emerge related to definitions, partnership drivers and dynamics, trust and mutuality, power asymmetries and inequities, and success factors.

It is noteworthy that empirical studies of partnerships are rare, particularly in-depth case studies. Theoretical pieces seldom present empirical tests of hypotheses, and practical guidelines are seldom grounded in theory. There is a clear need for more systematic and in-depth empirical research on partnership experiences.

Although partnership is now considered an essential way of working in many fields, several authors caution that the costs of working in partnership may often exceed the benefits. Before establishing a partnership, one should identify a clear value-added proposition.

Many reports on partnership prepared for the CGIAR are available only in grey  literature, leading to difficulties in accessing them and risking a loss of knowledge. Gaps in knowledge are identified at the level of individual partnerships, the level of the organizations that participate in or manage portfolios of partnerships, and the level of research or innovation domains that are characterized by networks of partnerships.

Are Metrics Blinding Our Perception?

(from New York Times, found by Aldo Benini)

By ANAND GIRIDHARADAS

CAMBRIDGE, Massachusetts — The Trixie Telemetry company believes in hard, quantifiable truths. It believes that there is a right time and wrong time to breast-feed a baby. It believes that certain hours and rooms are better for a child’s naps than others and that data can establish this, too. It believes that parents should track how long their infants have gone without soiling a diaper and devote themselves to beating this “high score.”

To these ends, the company sells what is a coveted service in this age: a dashboard. It invites you to enter data on your baby’s life, and it produces color-coded charts, Sleep Probability Distributions, digestive analysis and such, to help parents make data-based decisions.

Don’t laugh, because Trixie Telemetry is made from the essence of our age… >read the rest of the article on the NYT website here<

Evaluation in Conflict Prone Settings: Londonderry, June 2010

Date: 7th – 11th June 2010
Venue: Londonderry, Northern Ireland
As part of its 11th International Summer School, INCORE (International Conflict Research Institute) will be running a course on Evaluation in Conflict Prone Settings.

The course covers the basics of conducting evaluations of initiatives in conflict-prone settings. The focus of the course is not limited to peacebuilding initiatives, but applies to the evaluation of the full spectrum of initiatives: development, humanitarian, private sectors, community development, and so on. Similarly, the term “conflict-prone” is broadly defined in the course. It can be applied to community work on interface areas of inner-cities, or it can be applied to more conventional, protracted, militarized conflicts. The course will be facilitated by Dr Kenneth Bush (INCORE) and Colleen Duggan (IDRC).

The 2010 International Summer School will run from 7th-11th June. The Summer School will be based on the historic Magee Campus in the city of Derry/Londonderry, on the shores of Lough Foyle in the north west of Northern Ireland, easily accessible by road, rail and air.

Previous participants of the INCORE Summer School have included USAID, European Union DG External Relations, UK Department for International Development, World Food Programme, UN Department for Peacekeeping Operations (DPKO) and others. The INCORE Summer School is recognised by UNITAR (United Nations Institute for Training and Research) Programme of Correspondence Instruction in Peacekeeping Operations, and may form part of The Certificate-of-Training In Peace Support Operations (COTIPSO) Programme.

Visit www.unitarpoci.org for further details.

FOR MORE DETAILS

Visit: http://www.incore.ulst.ac.uk/courses/ss/

CONTACT DETAILS

Email: school@incore.ulst.ac.uk

Tel: +44 (0) 28 7137 5500 Fax: +44 (0) 28 7137 5510 INCORE, University of Ulster, Aberfoyle House, Northland Road, Derry/Londonderry, Northern Ireland, BT48 7JL

ActionAid reports on “systematization”

From the ActionAid website

“Systematization is the reconstruction of and analytical reflection about an experience. Through systematization, events are interpreted in order to understand them… The systematization allows for the experience to be discussed and compared with other similar experiences, and with existing theories and, thus, contributes to an accumulation of knowledge produced from and for practice” (Systematization Permanent Workshop in AAI systematization resource pack, pg 10, 2009).

“In 2009, IASL has produced two excellent resources on systematization. The first is a resource pack, which is one of the few English language resources on this exciting methodology. The pack will inform you about the methodology, and give you a detailed orientation to how to systematize experiences. You will also find links to other systematization resources and examples, and an existing bibliography of systematization materials”

“The second resource is Advocacy for Change, a systematization of advocacy experiences related to the status of youth (in Guatemala), the right to education (in Brazil) and farming (in the United States). The systematizations allowed the actors involved to consider the evolution of the experiences and to identify lessons and insights for future interventions. The Guatemala systematization product was documented in writing and film, the US experience in writing, and the Brazil experience in film”

New INTRAC publications on M&E

Tracking Progress in Advocacy – Why and How to Monitor and Evaluate Advocacy Projects and Programmes looks at the scope of, and rational for, engaging in advocacy work as part of development interventions, then focuses on the monitoring and evaluating of these efforts – offering reasons why and when these processes should be planned and implemented, what’s involved, and who should be engaged in the process. By Janice Griffen, Dec, 2009

The Challenges of Monitoring and Evaluating Programmes offers some clarity in understanding the different uses of the term ‘programme’, and uses the different types of programme to demonstrate the issues that arise for M&E. By Janice Griffen, Dec, 2009

Pathfinder : A Practical Guide to Advocacy Evaluation

(email from Pathfinder) Hi Rick. We recently published a new resource for the advocacy evaluation field.  Our new guide, Pathfinder : A Practical Guide to Advocacy Evaluation, comes in three editions-one each for advocates, evaluators, and funders. All three editions, plus a bibliography of useful resources, are free to download and share from our website:

Pathfinder provides a “big picture” view for planning and conducting an advocacy evaluation. Drawn from Innovation Network’s research and consulting experience, the guide encourages the adoption of a “learning-focused evaluation” approach, which prioritizes using knowledge for improvement.

How wide are the ripples? The management and use of information generated from participatory processes in international non-governmental development organisations.

In early 2009 five international development NGOs participated in reflection and discussion as part of a research project called ‘How wide are the ripples?’. The aim was to explore the flow and influence of knowledge produced in grassroots participatory processes within the international NGOs which commissioned or initiated process.  This work was supported within a wider research programme called IKM Emergent, which aims to explore how knowledge is used in development, and specifically to promote the recognition and use of ‘Southern knowledge’.  In addition to working with the five participating organisations Hannah Beardon and Kate Newman (independent consultants commissioned by IKM to conduct the research) linked to a wide variety of people who reflected on the questions and shared innovative approaches, which contributed to the learning.

The research, which included a background literature review, identified the context in which INGOs are operating and some inherent obstacles to bottom-up knowledge flow in international organisations.  It also documented examples of how organisations have tried to deal with these blocks, and promote the flow of information from the grassroots to contribute to the organisational understanding of, and response to, development issues.  For example:

  • What is different or distinct about knowledge management when applied to participatory approaches?
  • Do INGOs have a special role and responsibility in bridging local knowledge and international policy regarding development?
  • What kinds of knowledge and information generated through participatory methods can be useful to others in different contexts?
  • What are the practical ways to encourage, enable and facilitate the flow of such knowledge? And
  • How can debate on these questions be encouraged and enriched in INGOs across the development sector?

The review, case studies and report are available here: http://wiki.ikmemergent.net/index.php/Workspaces:5._Participation