Beyond Logframe: Using Systems Concepts in Evaluation

March 2010. Nobuko Fujita (Ed)  Foundation for Advanced Studies on International Development (FASID) Available as pdf

“Editor’s Note: The 2010 Issues and Prospects of Evaluations for International Development employs systems concepts as clues to re-assess the conventional ways of conducting evaluations and to explore how development evaluation can potentially be made more useful.

In Japan, development evaluation predominantly relies on the Logical Framework (logframe) when conducting evaluations. Evaluations based on a logframe often face difficulties. One such difficulty arises from the futile attempt to develop an evaluation framework based on a logframe, which, in many cases, was prepared as part of the early-stage planning of the project and which then does not necessarily reflect a project’s real situation at the time of evaluation. Although a logframe can be utilised initially as a tentative project plan, logframes are rarely revised even when the situation has changed. By the end of the project, the original logframe may not be an accurate embodiment of what the project is about and therefore logframes do not particularly help in terminal or ex-post evaluations.

Still, having been institutionalized by clients, logframe-based evaluations are common practice and in extreme cases, evaluators face the danger of evaluating the logframe instead of the actual project. Although widely used for its simplicity, logframes can end up becoming a cumbersome tool, or even a hindrance to evaluation.

Various attempts have been made to overcome the limitations of the logframe and some aid organizations such as USAID, UNDP, CIDA and the World Bank have shifted from the logframe to Results-Based Management (RBM). Now GTZ  is in the process of shifting to a new project management approach designed on RBM and systems ideas.

In the first article, “Beyond logframe: Critique, Variations and Alternatives,” Richard Hummelbrunner, an evaluator/consultant from Austria, sums up the critique of logframe and the Logical Framework Approach (LFA), and explores some variations employed to overcome specific shortcomings of LFA. He then outlines a systemic alternative to logframe  and introduces the new GTZ management model for sustainable development called “Capacity WORKS.” Richard has dealt with LFA and possible alternatives to LFA at various points along his career, and he is currently involved in GTZ’s rollout of Capacity WORKS as it becomes the standard management model for all BMZ 5 projects and programmes.

What does he mean by “systemic alternative”? In the second article, “Systems Thinking and Capacity Development in the International Arena,” Bob Williams, a consultant and an expert in systems concepts, explains what “thinking systemically” is about and how it might help evaluation. He boils down systems ideas into three core concepts (inter-relationships, perspectives, and boundaries), and relates these concepts to various systems methods.

In December 2009, FASID offered a training course and a seminar on this topic in Tokyo. Through the exchange of numerous e-mails with the instructors prior to the seminar, it occurred to me that the concepts might be more easily understood presented as a conversation. That is what we tried to do in the third article, “Using Systems Concepts in Evaluation – A Dialogue with Patricia Rogers and Bob Williams –.” These two instructors of the FASID training course and workshop explain in simple conversational style where and how we can start applying systems concepts in development evaluation.

This issue also carries a report of two collaborative evaluations of Japanese Official Development Assistance (ODA) projects. The first case presents an innovative joint evaluation conducted collaboratively with Vietnamese stakeholders. The evaluation took place in 2009 – 2010 as the last year of a three-year evaluation capacity development project coordinated by the Japan International Cooperation Agency. The second case covers a joint evaluation study of another Japanese ODA project in Lao PDR with a local Lao administration for which neither logframe nor OECD DAC five criteria was used. Instead, an evaluation framework was developed from scratch, based entirely on the beneficiaries’ interests and perspectives. In both cases, a partner country’s participation in the evaluation necessitated considerable changes in perspectives of evaluation practice. I hope they provide examples of how boundaries and perspectives, as discussed theoretically in the first three articles, relate to development evaluation in practice.”

5 thoughts on “Beyond Logframe: Using Systems Concepts in Evaluation”

  1. Dear All,

    I would be interested in learning from others on how they are responding to a growing need to show value for money? This is something that doesn’t stand out in log-frames, but yet we know this is a target and rather important part of an evaluation. Maybe it fits in efficiency, but DFID for example is asking for more in relation to VFM.

    Thanks.
    Renee

  2. Hi all, it is always good to re-think of other methods to manage and evaluate projects

  3. On the “Attribution Gap” between outcomes and impact as shown in Fig. 7: The GTZ results model, on page 26

    This is a useless and misleading simplification. There are attribution gaps everywhere, along the whole supply chain connecting project managers to project workers and on the intended beneficiaries.

    In each interaction between people involved in aid interventions there are other influences effecting what the participants do next. Many of these influences are mediated via the participants’ relationships with others in sorrounding networks of actors, both present and past.

    This is one reason, amongst others, why a more network (versus stage) view of development interventions is needed

    http://mande.co.uk/2008/uncategorized/the-social-framework-as-an-alternative-to-the-logical-framework/

  4. I don’t think the comment in the article’s introduction that “organizations such as USAID, UNDP, CIDA and the World Bank have shifted from the logframe to Results-Based Management (RBM)” accurately reflects the rest of the ideas in the publication, or how these organizations would necessarily characterize their approaches. As the publication itself points out later in the conclusion the “LFA… is a component of results based management” and many organizations have used it, while still putting it in the context of results-based management.

    And, as Oliver Bakewell and Anne Garbutt pointed out in their paper on “The Use and Abuse of the Logical Framework Approach” in 2005, the logical framework approach is much more than the logframe, which is just “the matrix which summarises the main elements of the programme of work and connects them to each other.”

    So even where the matrix may not be appropriate, there still may be room for a participatory approach to identifying problems, and trying to work towards activities that will address them, an idea that underlies the original conception of the Logical Framework Approach – the LFA.

Comments?

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: