Training: Monitoring and Evaluation for Results

Date: May 11-15, 2009
Venue: Hotel Africa in Tunis, Tunisia

SPONSORS
World Bank Institute Evaluation Group (WBIEG) and Joint Africa Institute (JAI)

TOPICS
Introduction to Monitoring and Evaluation
Logic Models and Evaluation Questions
Indicators and Measurement
Research Designs
Data Collection
Reconstructing Baseline Data
Sampling
Data Analysis
The Practice of Impact Evaluation
Reporting Results and Utilization of Evaluations
Managing Monitoring and Evaluation Functions
Continue reading “Training: Monitoring and Evaluation for Results”

Training in Evaluation of Humanitarian Action

Date: 21st-24th June 2009
Venue: Belgium

Channel Research and the Active Learning Network for Accountability and Performance (ALNAP) are inviting participants for Training in Evaluation of Humanitarian Action, Belgium, 21st-24th June 2009 (actual training dates 22nd-24th June 2009).

This course is an introductory-to-intermediate level course and has the overall aim of assisting participants in the design of monitoring systems, and to be able to commission, manage, carry out and use small scale evaluations in humanitarian action. This 3-day training course will use the OECD-DAC evaluation criteria but also introduces new evaluation material specifically on joint evaluations and innovative learning processes as part of an evaluation process.

Continue reading “Training in Evaluation of Humanitarian Action”

Results of the CONFERENCE ON IMPACT EVALUATION HELD IN CAIRO 29 MARCH – 2 APRIL

Further contributions  to this list are welcome (linked documents, or the documents themselves). Please use the Comment facility below, or email rick at mande.co.uk

The Logic Model Guidebook: Better Strategies for Great Results

Authors Lisa Wyatt Knowlton (Ed.D.) and Cynthia C. Phillips (Ph.D.) Published October 2008. Available on Amazon.  Recommended by Jim Rugh

Excerpt: “We approach logic models as important thinking and inquiry tools, and logic modeling as a process that contributes to clarity about a sequence of interactive relationships. Logic models display relationships of many kinds: between resources and activities, activities and outcomes, outcomes and impact. This display provides an opportunity to review critically the logic of these relationships and their content. Are the underlying assumptions, choices, and relationships sensible, plausible, feasible, measurable? Logic models assist strategy and contribute to performance management through discovery of the most effective means to a specific results… The modeling process includes a cycle of display, review, analysis, critique and revision to improve the model. These actions steps, tackled with colleagues or stakeholders, can contribute significantly to more informed displays and, ultimately, more successful programs and projects.
Continue reading “The Logic Model Guidebook: Better Strategies for Great Results”

Upcoming training opportunities with Jess Dart of Clear Horizon

Date: 13-17th October 2008
Venue: Melbourne, Australia
Upcoming training opportunities with Jess Dart of Clear Horizon.

Build a People-Centred Monitoring, Evaluation, Reporting & Improvement (MERI) Framework AU$800
Two days: 13th & 14th October

Course Description: Increasingly, project teams are involved in designing monitoring, evaluation, frameworks. Having a strong program logic element, this people-centred approach uses targeted stakeholders as the organising construct to clarify project design and then develop an effective and meaningful monitoring and evaluation framework. This workshop follows a series of steps that provide a pathway for planning that takes into account clarification of program goals, articulation of how we expect these to be achieved and therefore how best to collect evidence on whether we are on track. This includes consideration of evaluation purpose, key evaluation questions, and appropriate methods for collecting information.

Most Significant Change (MSC) Technique AU$800
Two day workshop: 15th & 16th October

Course Description: MSC is a powerful tool for monitoring, evaluation and organisational learning. MSC goes beyond merely capturing and documenting participants’ stories of impact, to offering a means of engaging in effective dialogue. Each story represents the storyteller’s interpretation of impact, which is then reviewed and discussed. The process offers an opportunity for a diverse range of stakeholders to enter into a dialogue about program intention, impact and ultimately future direction. This two day workshop provides an introduction to MSC which includes designing your own MSC process.

Evaluator as Facilitator AU$500
One day workshop: 17th October

Course Description: This workshop explores the role of facilitator as evaluator when undertaking participatory evaluations. The course includes a reflection on the two roles and explores their overlap when conducting participatory evaluation processes. In particular we share our experiences and explain how to facilitate some key participatory evaluation methods such as program logic, Most Significant Change (MSC) technique and the evaluation summit. We discuss the underlying philosophies behind approaches to evaluation and facilitation and explore practical applications in a range of contexts.

Aristotle and Plato at it again? Philosophical divergence within international aid project design and monitoring & evaluation

Available from the AID-IT website

Abstract: “International aid projects are broadly concerned with fostering change. Frequently, the ‘theory of change’ within an aid project is communicated using Logical Framework Analysis, or the ‘logframe’. The logframe may be viewed from at least two philosophical perspectives-functionalist and interpretist. Functionalism is found to be useful for problem analysis and project design since it enables a deconstruction of the goal into functional components. Interpretivism is found to assist project monitoring and evaluation since it draws attention to the role of human actors within the social change process, thereby clarifying the social research plan. A bilateral aid program in the Philippines is described to illustrate the practical differences arising from the divergent philosophies.”

Editor’s comments: This paper argues for the use of more actor-oriented versions of the Logical Framework to make it easier to identify who should monitor and evaluate what in a development project. There is a substantial overlap in the arguments presented here and those I have presented in my paper on this website on the need for a Social Framework (“a Logical Framework re-designed as if people and their relationships mattered”). Continue reading “Aristotle and Plato at it again? Philosophical divergence within international aid project design and monitoring & evaluation”

Customising definitions of outputs, outcomes and impact

(from the OM email list)
I am continually challenged with the organisations with whom I work to define the terms we will use. They are by and large Northern donors and their grantees – development NGOs and social change networks. I find that the meaning of outcome and impact (and output) varies considerably and sometimes the terms are used interchangeably. Thus, one of the biggest mistakes I can make is not to define the terms right from the start.

In a recent discussion titled Outcomes vs. Impact on the American Evaluation Association listserv EVALTALK, I shared a simple instrument that I use for developing common agreement about the definitions of outputs, outcomes and impact. It can be found at:
http://www.outcomemapping.ca/resource/resource.php?id=189

Hope it is helpful to some of you planners, monitors, evaluators and those who employ them.

If you are interested in the AEA EVALTALK listserv, subscribe at www.eval.org. To use the archives, go to this web site: http://bama.ua.edu/archives/evaltalk.html. Please note that the discussion I refer to will only be archived in September.

Best wishes,
Ricardo

ricardo wilson-grau consulting
NEW: Oude Singel 184, 2312 RH Leiden, Netherlands
Rua Marechal Marques Porto 2/402, Tijuca, Rio de Janeiro, CEP 20270-260, Brasil
Tel: 55 21 2284 6889, Skype: ricardowilsongrau

Advocacy & Policy Change Composite Logic Model

Author: Coffman, Julia
Publisher: Harvard Family Research Project
Publication Date: September 2007

Abstract
Are there shared elements-goals, outcomes, indicators-across different types of advocacy work? Can we create a common vocabulary for the advocacy evaluation field?

The Composite Logic Model (“CLM”) and associated materials was developed by Julia Coffman from Harvard Family Research Project; Astrid Hendricks and Barbara Masters from The California Endowment; Jackie Williams Kaye from The Atlantic Philanthropies; and Tom Kelly from the Annie E. Casey Foundation. More than 50 funders, evaluators, and advocates also lent their expertise to refine the Model.

VisuaLyzer software: for visualising and analysing networks

There are now many different software packages available that can be used to visually represent networks, and to generate many different statistical measures of their structure. Unfortunately many of these involve a steep learning curve, and involve far more bells and whistles than I need.  VisuaLyzer is my favourite software package because it is very user friendly, and easy to use.

VisuaLyzer is produced by mdlogix, USA. You can download a trial version or buy a copy from this part of their website. For more information contact Allen Tien <allen@mdlogix.com> at mdlogix. If you do contact him, please mention you heard about Visualyser on Rick Davies’s website, MandE NEWS.

My main use of Visualyzer is to draw the organisational networks I am working with, in the course of my work as an M&E consultant on development aid programmes. These are of two types: (a) literal descriptions (maps) of the relationships as known, (b) simplified models of complex networks showing the main types of organisations and the relationships between them. Less frequently, I also import data from Excel to automatically generate network maps. This data usually comes from project documents or online surveys. I also use the combination of UCINET and Netdraw for this task.

Here is an example of a network that I drew by hand directly on screen. It represents the relationships between AMREF’s partners in the Katine project, Uganda. Click on the image to expand it a new window, then click again to get a focused image. You can represent different types of actors by varying the colour, size and shape of nodes. You can represent the different kinds of relationships between them by varying the kind of line used, its colour and thickness. If you click on a node you can enter detailed text or numerical data describing the actor’s attributes, using as many fields as needed. If you click on any link you can enter data about the attributes of that relationships. Both of these sets of data can be exported, on all actors and relationships, as an Excel file.  You can also import the same kind of data, to automatically generate a network diagram.

mdlogix describe it as “an interactive tool for entering, visualizing and analyzing network data. You can create nodes and links directly or import network data from edgelist/edgearray, Excel, or GraphML formats. Once the network is displayed, you can customize visual properties such as the colour, shape, size, and location of nodes and links to create an informative graphic representation. Images of your choice may be used to represent nodes. XY mapping of nodes as a function of node attributes is supported in layered layout. It also provides a number of analysis functions for calculating network and nodal level indices, and for finding sub-groups, partitions, communities, and roles and positions. In addition, VisuaLyzer includes powerful logic programming capabilities that allow you to investigate networks using axioms of classical set theory.”

This all sounds quite complex. But in practice it is the simplest features of Visualyzer which are the most useful. It does have a very good and easy to read Users Guide (5mb), which you may want to look at.

For more on the development of network models / descriptions and their use in monitoring and evaluation go to the Network Models section of this website.

POSTSCRIPT (1st December 2008): See also Overview of Common Social Network Analysis Software Platforms “This report was developed by the Philanthropy and Networks Exploration, a partnership between the Packard Foundation and Monitor Institute. The exploration is an inquiry into how networks can facilitate greater philanthropic effectiveness. For more information, please go to http://www.philanthropyandnetworks.org

PS2 (16th January 2009): The link to the “Overview …” doc no longer works. I have now uploaded the doc HERE, after receiving a copy via the Pakard Foundation. They also sent a link to: “Working Wikily: How networks are changing social change

The Logical Framework: A list of useful documents

Contents: 1. Explanations of the Logical Framework | 2. Wider discussions of Logic Models | 3. Critiques of the Logical Framework4. Alternative versions of the Logical Framework5. The Editor’s concerns (about uses of the Logical Framework) | 6. Online survey on views and usage of the Logical Framework

Please feel free to suggest additions or corrections to this list, by using the Comment facility at the end of this post

New section: Software

1. Explanations of the Logical Framework

2. Wider discussions of Logic Models

  • Program logic – an introduction, provided by Audience Dialogue (2007)
  • Enhancing Program Performance with Logic Models This course introduces a holistic approach to planning and evaluating education and outreach programs. Module 1 helps program practitioners use and apply logic models. Module 2 applies logic modeling to a national effort to evaluate community nutrition education. Provided by the University of Wisconsin (2007)
  • Online Logic Model training: an audiovisual presentation by Usable Knowledge, USA Twenty minutes long, with a menu that can be used to navigate to the sections of interest (2006)
  • Network Perspectives In The Evaluation Of Development Interventions: More Than A Metaphor.  Rick Davies, for the EDAIS Conference November 24-25, 2003 New Directions in Impact Assessment for Development: Methods and Practice. “In this paper I argue the case for the use of a network perspective in representing and evaluating aid interventions. How we represent the intentions of aid activities has implications for how their progress and impact can be assessed. Because our representations are by necessary selective simplifications of reality they will emphasise some aspects of change and discourage attention to others. The benchmark alternative here is by default the Logical Framework, the single most commonly used device for representing what an aid project or programme is trying to do. Five main arguments are put forward in favour of a network perspective as the better alternative, along with some examples of their use. Firstly, social network analysis is about social relationships, and that is what much of development aid is about. Not abstract and disembodied processes of change. Secondly, there is wide range of methods for measuring and visualising network structures. These provide a similarly wide range of methods of describing expected outcomes of interventions in network terms. Thirdly, there is also a wide range of theories about social and other networks. They can stimulate thinking about the likely effects of development interventions. Fourthly, network representations are very scalable, from very local developments to the very global, and they can include both formal and informal structures. They are relevant to recent developments in the delivery of development aid. Fifthly, network models of change can incorporate mutual and circular processes of influence, as well as simple linear processes of change. This enables them to represent systems of relationships exhibiting varying degrees of order, complexity and chaos. Following this argument I outline some work-in-progress, including ways in which the conference participants may themselves get involved. Finally I link this paper into its own wider web of intellectual influences and history. ” (Posted here 2003)
  • The Temporal Logic Model: A Concept Paper, by Molly den Heyer. On the IDRC website. (2002)
  • A Bibliography for Program Logic Models/Logframe AnalysisDecember 18, 2001 Compiled by: Molly den Heyer Evaluation Unit, International Development Research Centre
  • W K Kellogg Foundation Logic Model Development Guide. (2001) Using Logic Models to Bring Together Planning, Evaluation, and Action. Updated (original was published in 1998) “The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program.A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.”
  • Application of Logic Modeling Processes to Explore Theory of Change from Diverse Cultural Perspectives Ricardo Millett, Sharon Dodson, & Cynthia Phillips American Evaluation Association November 4, 2000
  • The state of the art of Logic Modelling. PowerPoint presentation by Gretchen Jordan (1999?)
  • The Logic Model for Program Planning and Evaluation, Paul F McCawley, 1997, University of Idaho Extension.

3. Critiques of the Logical Framework

  • Debunking misconceptions around the Logical Framework Approach through reviewing available literature by Munyaradzi Madziwa, August 2016
  • Critical Study Of The Logical Framework Approach In The Basque Country (2011) by ECODE, Bilbao.
  • THE USE AND ABUSE OF THE LOGICAL FRAMEWORK APPROACH A Review of International Development NGOs’ Experiences. A report for Sida. November 2005. Oliver Bakewell and Anne Garbutt, of INTRAC. “In this review, we have attempted to take stock of the current views of international development NGOs on the LFA and the ways in which they use it. We start in the next section by considering the different meanings and connotations of the term logical framework approach as it is used by different actors. In Section 3 we look at how LFAs are used by INGOs in both planning and project management. The next section reviews some of the debates and critiques around the LFA arising both from practice and the literature. In response to these challenges, different organisations have adapted the LFA and these variations on the LFA theme are outlined in Section 5. We conclude the paper by summarising the findings and reflecting on ways forward. …This review has been commissioned by Sida as part of a larger project which aims to establish new guidelines for measuring results and impact and reporting procedures for Swedish development NGOs receiving support from Sida. “

4. Alternative versions of the Logical Framework

  • Beyond Logframe: Using Systems Concepts in Evaluation, 2010, FASD (Foundation  for Advanced Studies on International Development)
  • The Social Framework, an actor-oriented adaptation of the Logical Framework, developed by Rick Davies. The sequence of rows found in a Logical Framework now represent a sequence of actors, connected to each other by their relationships, and forming a specific pathway through a wider network of actors. Narrative descriptions of expected changes, indicators of those change and means of verification are still found in the columns, but these relate to actors and their relationships. Actors can be individuals, groups, organisations or type of organisations. The assumptions column still exists, but the assumptions refers to important connections to other actors outside the specific pathway.
  • Can OM and LFA share a space? “OM (Outcome Mapping) and LFA may be useful at different levels, for diverse types of interventions or for information and in different contexts. Rather than pitting LFA and OM against each other, we need to understand what kinds of information and uses each has, as well as their advantages and disadvantages, and find ways for them to add value to each other.” See also Logical Framework Approach and Outcome Mapping: A Constructive Attempt of Synthesis. A Discussion Paper by Daniel Roduner and Walter Schläppi, AGRIDEA; Walter Egli, NADEL (ETH Zurich)
  • Logical Framework Approach – with an appreciative approach. April 2006 SIDA Civil Society Centre. “As a part of its effort to realise the intentions of Sweden’s Policy on Global Development, Sida Civil Society Center (SCSC) initiated a development project in 2005 together with PMU Interlife (the Swedish Pentecostal Mission’s development cooperation agency) and consultant Greger Hjelm of Rörelse & Utveckling. The goal was to create a working model which combines the goal hierarchy and systematics from the Logical Framework Approach (LFA)1 with the approach used in the Appreciative Inquiry tool (AI). AI is both a working method and an approach. In analysing strengths and resources, motivation and driving forces, the focus is placed on the things which are working well, and on finding positive action alternatives for resolving a situation. LFA, which is an established planning model in the field of international development, is found by many to be an overly problem-oriented model. Using this approach, one proceeds based on a situation in which something is lacking, formulates the current situation as a “problem tree”, and thus risks failing to perceive resources which are actually present, and a failure to base one’s support efforts on those resources. Working in close cooperation, we have now formulated a new working method for planning using LFA, one which is built on appreciative inquiry and an appreciative approach. The model was tested by PMU Interlife’s programme officers and their cooperating partners in Niger, Nicaragua and Tanzania during the autumn of 2005. Their experiences have been encouraging, and it is our hope that more Swedish organisations and their cooperating partners will try our model and working method.(Posted 01/07/06)
  • No more log frames!! People-Focused Program Logic Two day workshop Monday 19th and Tuesday 20th of September 2005, in Melbourne, Australia. “Purpose of the workshop: • To understand what ‘people-focused’ program logic is and how to use it • To build a people-focused program logic for their own project Who should attend? People with monitoring and evaluation interests who are working on projects with capacity building components. Course description: In this workshop, participants will build their own ‘people-focused’ logic model. To do this they will analyse the key beneficiaries of their project, build their program logic model around this analysis, and consider assumptions made in the logic. The program logic will be built around a generic theory of how capacity building works, that can be modified to include elements of advocacy and working with or through partners. Participants will also learn how this logic can be used to form the spine of their monitoring, evaluation and improvement framework. As participants will be invited to develop their own program logic model, they are encouraged to bring along others from the same project team. Examples of frameworks, and a workbook will be provided to participants” For additional information: Jo Leddy of Clear Horizon Phone: 03 9783 3662 E-mail: Jo@clearhorizon.com.au Website: www.clearhorizon.com.au See rest of the flyer for more information…(Posted 21/06/05)
  • Intertwining Participation, Rights Based Approach and Log-Frame: A way forward in Monitoring and Evaluation for Rights Based Work. Partha Hefaz Shaikh Initial Draft – Circulated for discussion. “Programme implementation through Rights Based Approach (RBA) in ActionAid Bangladesh started in 2000 and it took us quite a while to understand what it meant to implement programmes in a RBA environment. Side by side we were also grappling with issues of monitoring and evaluation of programmes implemented through a rights based approach. In order to develop a more meaningful framework that has all the elements of participation, RBA and log-frame we developed what we call “Planning and Implementation Framework Analysis (PIFA)”. ” (Posted 20/05/05)
  • A MODIFIED LOGFRAME FOR USE IN HUMANITARIAN EMERGENCIES. by Bernard Broughton (I think)
  • Family Planning Logical Framework (with two parallel processes, one feeding back into the other)
  • Build Reach into Your Logic Model. Steve Montague February 1998 “Analysts have frequently noted the importance of constructing logic models (a.k.a. logic charts, causal models, logical frameworks, and most recently performance frameworks – among other names) to explain the causal theory of a program or initiative before attempting to monitor, measure, or assess performance. …A key limitation to the logic models of the 1980s, as well as many of those in current use, has been their tendency to focus predominantly on causal chains without reference to who and where the action was taking place. “
  • Bennett’s Hierarchy (or Targetting Outcomes of Programs (TOP)). This is not a version of the LogFrame, but it is another type of logic model with multiple steps (7 levels). It has been used widely in the evaluation of agricultural extension activities in Australia. It was originally developed by Bennett in 1975.

5. The Editor’s concerns (about uses of the Logical Framework)

  1.  Long, complex, unreadable sentences, in the narrative column of the Logical Framework
    1. Often the result of compromises between many different parties who have been negotiating the contents of the Logical Framework. Net result: an unreadable document
    2. Sometimes the result of people not knowing that the whole story does not need to be told in one sentence. The row below should say what happens before (the cause) and the row above should say what happens next (the effects)
    3. Sometimes the result of people forgetting there is a column for indicators next door, where they can provide lots of interesting detail about what is expected to happen at this stage
  2. Narrative statements without people in them. E.g “Rice productivity increased”
    1. Another reason some many Logical Frameworks are so unreadable, and so boring when they are readable, is that somehow their authors have managed to leave out people. Instead we have lots of abstract and disembodied processes. And then we wonder why some people have difficulty understanding Logical Frameworks
  3. Means of Verification that refer to reports and surveys, but not who is responsible for generating and / or providing this information (and when it will be available)
    1. This problem is similar to the above, reflecting a continuing aversion to making references to real people in Logical Frameworks.
    2. One consequence is lack of clear ownership and responsibility for M&E of the changes being described at that level of the Logical Framework
  4. Insistence on there being only one Purpose level statement in a Logical Framework
    1. I have recent experience of colleagues insisting on this. For reasons I have not yet established, beyond the “it is not allowed” variety. Insisting on one Purpose and One Goal really is pushing a very linear model of reality. It does not even allow for any parallel but convergent events, such as those usually come through problem tree analyses that sometimes precede the design of a Logical framework
  5. Ambitous narrative statements coupled with modest indicators / Overly simple indicators used to describe complex developments.
    1. Such as “number of meetings held” as an indicator for the functioning of stakeholder’s advisory committee. For an alternative, see “Checklists as mini-theories of change
  6. Lists of indicators in no apparent order
    1. “A (unsorted) list is not a strategy” A sorted list can convey relative importance (most important indicator at the top), or a sequence (starting from the bottom), or multiple alternative routes to the objective in the narrative column. If there is a list, the reader should be told what sort of list it is.
  7. Broad generalisations at the Goal level
    1. Sometimes arising from confusion of a temporal hierarchy (A leads to B which leads to C which leads to D) and a nested hierarchy (A is part of B which is part of C which is part of D). The Logical Framework is supposed to be a temporal hierarchy, that tells a story. Not a pile of increasingly broad statements about the same thing
  8. Confusion over the meaning of different levels in a Logical Framework. Between Activities and Outputs, Outputs and Purpose level outcomes, and outcomes at the Purpose and Goal level.
    1. Often cause by leaving people out of the picture, as above.
    2. A workable rule of thumb, for seperating levels of the Logical Framework
      1. Activities are things that “the project” can control. The boundary of a project being defined by the reach of its contracts (with staff, consultants, suppliers and sub-contractors)
      2. Outputs are the activities of the project (if services), or their results (if goods), that people and organisations outside the project can use e.g workshops, publications, trainings, etc. Ask here: What is available to who, and in what form?
      3. Purpose level changes (outcomes), are changes in those people or organisations who have used those goods or services. Normally the project would hope to influence these (and learn about how it can have influence) but it would not be expected to control events at this level
      4. Goal level changes (outcomes), are longer term changes in those same people or organisations, or others they have subsequently interacted with.
  9. Long lists of assumptions
    1. Apparently designed to cover people’s backsides
    2. Including many events that the project should be able to influence
      1. …which therefore should be listed as one of the outputs or outcomes. I.e. brought into the central narrative of the Logical Framework
  10. Things the Logical Framework cant do very well, even in the best of hands
    1. Represent multiple parallel processes, as distinct from a single process
      1. E.g. What people are doing at multiple project locations, within a single national project
        1. Representing their interactions is even more of a challenge
    2. Represent the interactions between multiple events at the same level of a Logical framework.
      1. E.g. How different project outputs (manuals, training events, newsletters, websites, etc) feed into each other
      2. Or, how different health outcomes (at Purpose level) feed into each other, before finally contributing to Goal level changes e.g. reduced mortality
    3. Represent the interactions between multiple outputs and the many users of those outputs
      1. E.g., the range of communications products used by a range of clients of a project . Many people will use multiple products, but their usage patterns will vary. Many products will be used by multiple users, but their user groups will vary.

All these processes can be represented by network models. See the new page on developing network models of development projects. However network models are generally too complex to provide a substitute for the Logical Framework. One proposed alternative is the Social Framework, originally described here and now updated here. The Social Framework can be used to describes a pathway through a network, in a way that capable of being monitored and evaluated. Your comments are welcome.

6. Online survey into the uses of the Logical Framework

Please consider taking part in this survey. You can access the cumulative results to date at the end of the survey form. It is not long.

thanks, rick davies

%d bloggers like this: