Everyone seems to be agreed on the importance of lesson learning. But how can we tell when lessons have been learned ? This is not an academic question if NGOs are going to devote more staff time and money in order to improve their lesson learning capacity, and if they are going to seek donor support to do so.

The focus of this brief paper is on description: How will we recognise something when we bump into it ? The reason for this focus is that the subject of organisational learning, perhaps more that other areas of development practice, seems to be swamped with prescriptive statements. The same is also the case with books on organisational learning aimed at managers of private sector organisations. Normally we need to be able to describe a process reasonably well before we can go about making prescriptions about how to improve that process. We also need a capacity to describe events if we then want to go out and test if those prescriptions worked ! Good descriptions can have practical value. Defined in narrower terms, we need to find indicators of when learning is taking place, and of different aspects of that process.

(The ideas that have been developed here are strongly influenced by the work of Gregory Bateson (1979), an anthropologist)

Examples

Although not yet included in this document, in the Group Session some illustrative examples will be drawn from a recent review of ActionAid's research into the participatory identification of indicators, and the evaluation strategies of another ActionAid project in Vietnam. Some use has also been made of the literature on the subject of organisational learning that has been produced by UK NGOs in the last few years. A brief list of some of the papers available is given at the end of this paper, along with details of the reviews of the ActionAid initiatives.

Definitions

An example of a prescriptive definitions of organisational learning, meant to be applied to NGOs, can be found in Britton's paper "The Learning NGO" written for a Learning NGO Workshop in 1995.

"...a learning organisation is one which actively incorporates the experience and knowledge of it's members and partners through the development of practices, policies , procedures and systems in ways which continuously improve it's ability to set and achieve goals, satisfy stakeholders, develop it's practice, value and develop its people and achieve its mission." (Britton, 1995)

This definition implies that learning organisations can be distinguished from non-learning organisations. Although it is a comprehensive definition I am not sure how easy it would be to apply, to differentiate one such organisation from the other.

An alternative definition of organisational learning can be suggested, one that is descriptive

"Learning involves the selective retention of information. Information is a difference that makes a difference - either in understanding and/or behaviour. From the huge volume of their day to day experiences organisations selectively retain some information and not others. Evidence of this process of selective retention can be seen in the form of changes to their understanding and/or behaviour."

This definition suggests that all organisations learn. But it implies that organisations could be differentiated in terms of the content of what they learn, how they are in-formed.

There are two sides to this view of learning. On the one hand there is the active process of learning taking place at the moment: the selective attention given to particular experiences and the immediate impact these selected experiences have on an organisation. On the other hand there is a more passive process, the retention of information accumulated in the past. This is not an academic distinction - aspects of both can be identified in practice (see below).

Lessons

What is a lesson ? A lesson can be seen as a "if-then" type of statement. When we did x then y happened.. If people say they have identified a lesson learned from an experience then we should be able to find this sort of structure in their statements or reports.

Lessons in this form can range from the very general to the highly specific. For example, from "user fees are bad" to "if user fees are introduced in x, y, and z circumstances then more than 90% of poor households are likely to..." I think that the more specific expressions of lessons are preferable, but there may be merits to more general statements. What do group session participants think ?

Active learning involves the identification of lessons that are news. They involve new information (differences that make a difference). Old lessons will be embedded in the way the organisation works: in organisational structures, systems, roles and procedures. These are evidence that the organisation has become in-formed by its past experiences.

Identifying new lessons (active learning)

There are four questions which can be used to identify new lessons that people within organisations may have learned. They are in the form of a scale, providing increasing certainty that a lesson has been learned (and it's details).

1. What changes have taken place in...(in the area of practice which is of concern)...? If no changes can be identified within a period of concern then it is unlikely that the person interviewed has learned anything. (But not impossible).
2. Which of these is most important ? If they cannot differentiate different events it is unlikely that they would have had much impact (i.e made a difference). The exception might be two very important events with similar consequences.
3. Why is it important ? What difference has it made ? Expected rather than actual consequences are not lessons learned, but more like hypotheses, still to be tested by future observation. If consequences cannot be identified then it is unlikely, but not impossible, that any difference has been made.
4. What difference has it made to their behaviour or that of others ? What people are doing as distinct from thinking or feeling. Impact on actual behaviour could be seen as more important. People may have thoughts or feelings but not act upon them, or have their actions influenced by them.

This approach to identifying "lessons learned" is different from simply asking people what lessons they (or their organisation) has learned in the last x months. This tends to produce generalisations, which then have to be tied to the ground by asking for examples.

Generalisation

Generalisation is part of the process of learning. It is about the application of lessons from one specific situation to a wider range of situations. There two different interpretations of this process. One is when the person who has learned the lesson argues that it has wider applicability - S/he is generalising. This is the most common use of the word. The other is when a number of other people hear about the lesson learned (either in it's details or as a generality) and then decide it is important and should be taken note of. It is this second form of generalisation which is likely to make the wider difference, and which we should therefore try to identify.

How do we tell when lessons are being generalised in this way, from one setting to many others ? When interviewing people to find out what they have learned from others we should ask similar questions to the above ("Identifying lessons learned"). The only difference will be that the source of news (changes) is another person/organisation. The focus is on what the respondent has heard from or about this other person/organisation's experiences. Much of what has been heard will not be remembered, and of that which has remembered only some will have made any difference to the respondents understanding or behaviour.

The answers to these questions provide us with largely qualitative information about what has been learned. The distribution of these answers amongst the people we might interview provides a more quantitative description. They tell us about the breadth of learning.

Lateral learning, beyond the organisation.

Learning from others can be more formalised. An increasingly common practice in business firms is bench marking. This involves identifying another firm with conspicuously good practice in a certain area and then seeking to learn from that firm and reach that same, or better level of practice. It is possible that this process takes place less formally as well.

My impression is that this rarely happens in the NGO sector, if at all, on the formal level. If so, why is this the case, when it does happen in the private sector ? Or if it does, what is the best example that can be identified ? Comments from Group Session participants please ?

When trying to identify the existence of lateral learning between NGOs we can ask people in one NGO which other NGO has the best practice in x area ? And how is it better than what this NGO is doing ? If these differences cannot be identified then it is unlikely that lateral learning even of the more informal kind is taking place.

Depth of learning.

Learning can take place at two levels. Firstly, how to implement a specific project activity in the best way, and secondly, what the NGO should be aiming to achieve with such an activity, or others in its place. Changes in the latter are what has been called second-order learning. It is possible to document this type of learning.

Ranking exercises can be used to identify NGO staff members' views of the relative success of different project activities. Asking respondents to compare pairs of ranked items, and identify the ways in which one is more successful than another will provide the criteria used to judge success/failure. It is then possible to treat these criteria as items which themselves can then be ranked by the same respondent, in terms of their relative importance. Changes in the ranking of these criteria, over a period of time, is evidence of second order learning.

Memory

It was mentioned above that old lessons learned will be embedded in the way an organisation works: in organisational structures, systems, roles, and procedures. They will have been in- formed by past experiences. The difficulty with understanding this aspect of learning in organisations is that because these features have often become "part of the woodwork" their meaning (differences they make) are not always easy to identify, though they may all be felt to be important and necessary. Things that don't change very much tend to become part of peoples' tacit knowledge.

One way of exploring what is being memorised by the structures and procedures of an organisation is to ask

"What difference would it make if...(insert a change)...?

• .. this procedure was discontinued ?
• .. the staff of this (x) section of the organisation were no longer in a separate unit but included with those in y ?
• ...the work of this (x) meeting was absorbed into that of y ?

Answers to these questions emphasise that while the lessons of the past are built into structure and process of organisations they are being re-membered, re-constructed, re-interpreted every time someone thinks about them. Retaining the lessons of the past is not so static a process after all. It may be the original purpose of a system has been forgotten but the current interpretations are still seen as sensible by the organisation.

Attention

Organisations selectively attend to their daily experience, and from within what is attended some things are learned and amongst these some are remembered longer than others. It is possible to identify where attention is being directed, without going into a great deal of detail about what is actually being learned.

Important events are often attended to frequently. Some of the most frequently monitored information in savings and credit projects is loan repayment rates. These are focused on because they are felt to have important consequences. Asking how often a person or organisation collects x information, or what the most frequently collected information is, may indicate where attention is normally focused.

Events/objects which are important will generally be differentiated in greater detail than those which are not. This capacity can be tapped by simply asking "What kinds of x are there ?", where x is the area of knowledge the interviewer is concerned with. Amongst each type (or category) identified, the question can be repeated "What type of x (type 1 2,...) are there ?", etc. The areas where a person is most able to continue to differentiate types, rather than say "They are all the same" is likely to be an area where they have a special concern, or at least substantial past experience.

If there is felt to be a risk that such knowledge is just a reflection of past knowledge rather than the current focus of attention. it is possible to then ask what is the latest news, for each of the categories, and to see where the most current news is concentrated.. This is not foolproof.

These aspects of attention can also be seen in documents and reports as well as identified through interviews. Some areas will be discussed in depth, others will not. Some information will be very up to date (or in time series) and others will not.

Specialisation

This concentration of attention into certain areas is a form of specialisation. Specialisation also occurs in the tasks carried out by an individual, and by sections of an organisation. Specialisation is a form of accumulated knowledge. This should be visible in organagrams and job descriptions, if they are still in use. Within organisations there are some areas which have developed specialised knowledge and others which have not. The extent of specialisation in any area may change over time. These variations and trends can be identified crudely by looking at the allocation of staff, and the tasks undertaken by staff.

Capacity to learn

Asking staff to describe variations in the way in which a particular project activity is carried out from location to location, such as loan approvals, can be indicative of an organisations capacity to learn. No awareness of differences, because they can't be seen or are not supposed to be there, means little opportunity to learn which are better than others, and thus improve performance in that area. Procedures are often described as uniform but in practice are not so, either because there are local needs that have to be taken account of, or just because of the limitations of human skills, and error. While variations in some areas may be strictly policed e.g recording of expenditures of money, and understandably so, invariant routines across all areas of organisational activity would be very unusual. Staff will vary in their ability to identify variations in practice and variations may be more evident in some areas than others.

Systems

Logical frameworks

These are normally seen to be all about planning, and predictability, not about the new and unexpected. They are not normally seen as conducive to learning new lessons. How would we recognise a Logical Framework that was oriented to lesson learning ?

Two suggestions:

1. The outputs row (first column) would include "lessons learned" by project staff as one of the outputs. The second column would require indicators of this outcome. This would be qualitative (and possibly quantitative ) information on the differences made by the lessons learned. The third column would detail how these are identified and documented.
2. Goal or purpose statements would involve beneficiaries views of value and impact. In most project locations peoples views are likely to change over time, including their standards of value. This will present project managers with a moving target and if so will require repeated adaptation by NGOs working with them.

Comments from Group Session participants please.

Management Information Systems

How would we recognise a MIS that was conducive to learning ? Some speculations:

1. There would data available on the use of the MIS, frequently updated.
2. The amount of use of the system be increasing over time. This would suggest it's contents are making a difference to the lives of the people using the system. This would especially be the case if there is evidence of increasing repeat use, not just more people using the system.
3. The system structure would show evidence of change in response to user's response to its content (or non-use). There would be evidence of it being in-formed by experience.
4. Structures of categories used in the MIS would include opportunities for open-ended responses and "any other" type categories in order to catch unexpected information, variations from what was expected. These are new opportunities for learning.
5. Statistical information collected by the system would be accompanied or linked to interpretations of the same. The differences that are made by differences would be clear.
6. Specialisation would be evident, in the more detailed differentiation of content in some areas compared to others, and in the relative frequency with which different information is collected.
7. Over time some areas of information collection would be abandoned. There would be a forgetting of low priority experiences.

Comments from Group Session participants please.

How would we judge when a whole organisation and a whole NGO sector is learning ?

Some final contentious suggestions:

1. In the private sector many organisations either cease to exist or are merged with others. This implies that some organisations simply cannot adapt sufficiently to meet current conditions. The incidence of mortality and merger in many NGO sectors seems to be low. Why is this and what does it mean ? Does it mean that the performance of 99% of NGOs is felt to be satisfactory. Would it be better if there was a higher failure rate (reflected in closures or mergers) ? Comments from Group Session participants please.
2. It could be argued that if an organisation is continuously learning then more specialisation of roles and structures will appear over time. This is a contentious suggestion because in some country NGO sectors at least there does not seem to be a great deal of evidence of specialisation amongst NGOs, as whole organisations. Why is this and what does it mean ? Is this a toleration of amateurism ? Or is a an adaptive response to some features of the environment in which NGOs work ? Comments from Group Session participants please.

References

ActionAid (1997) The 1997 Wealth Ranking Exercises. RDA 1: Mai Son District, Son La Province. Edited by Rick Davies.

ActionAid (1997) The Summary Report. Methods and Indicators for Measuring the Impact of Poverty Reduction. An ODA Funded ActionAid Reserach Project.

ActionAid (1997) Guidance Notes on Increasing the Participation of the Poor in the Assessment of the Impact of Development Interventions. Methods and Indicators for Measuring the Impact of Poverty Reduction. An ODA Funded ActionAid Reserach Project.

Bateson, G. (1979) Mind and Nature: A Necessary Unity.. Dutton. New York

Britton, B (1995) The Learning NGO. Paper prepared for the Learning NGO Workshop.

Edwards, M (1996) Becoming a learning Organisation, or the Search for the Holy Grail ? Paper prepared for the Aga Khan Foundation Canada Round Table Systematic Learning: Promoting Pubic Support for Canadian Development Cooperation.

Howes, M., Roche, C. (1996) How NGOs Learn: The Case of Oxfam UK and Ireland. Paper presented at the NGO study group of the Development Studies Association, 12 July 1996.

Roche, C. (1995) Institutional Learning in Oxfam: Some Thoughts. Oxford. Oxfam.

Mande NEWS: Lesson Learning moved to this new address on January 19th 1998. Since then there have been visits to this page. See MandE Statistics for more details of the usage of MandE NEWS.