An evolutionary approach to facilitating organisational learning:

An experiment by the Christian Commission for Development in Bangladesh.

Rick Davies, Centre for Development Studies, Swansea SA2 8PP, Wales, UK. Fax/phone: 44 223 841367. E-mail: rick@shimbir.demon.co.uk. URL: http://www.swan.ac.uk/cds/rd1.htm

This paper has been published, with some variations, in Mosse, D.. Farrington, J., and Rew, A. (1998) Development as Process: Concepts and Methods for Working with Complexity. London. Routledge/ODI, pages 68-83; and in Impact Assessment and Project Appraisal, Vol. 16. No. 3, September 1998, pages 243-250.

INTRODUCTION

The Christian Commission for Development in Bangladesh (CCDB) is a medium sized Bangladeshi non-government organisation (NGO) with almost 550 staff. It's annual budget of approximately US$4 million is funded by a consortium of Protestant donor agencies, in addition to its own internally generated income. Its main programme is the Peoples Participatory Development Programme (PPRDP), which involves more than 46,000 people in 785 villages in 10 districts. Approximately 80% of the direct beneficiaries are women. Development assistance is made available to participants in three forms: group based savings and credit facilities used to meet the needs of individual households, grant assistance given to the same groups on a pro-rata basis and intended for community level developments, and skills training, mainly for livelihood purposes. The large scale and open ended nature of these activities poses a major problem for the design of any system intended to monitor process and outcome.

In 1994 an experiment in participatory monitoring was conducted with CCDB's PPRDP programme that involved the deliberate abandonment of the use of "indicators", a central concept in orthodox approaches to monitoring. The key features of the new approach to project monitoring have been outlined in very brief form below. Implementation took place in four PPRDP project areas in Rajshahi zone of western Bangladesh, where 140 CCDB staff are working with approximately 16,500 people grouped into 503 shomiti (associations).

The experiment was the outcome of a voluntary collaboration between the author as an independent researcher and CCDB. CCDB adopted the approach because, according to the Director, it appeared to fit their needs. The experiment was not funded by or encouraged by CCDB's donors, nor was the author in any way answerable to those donors. The author worked with a male staff member from the Training Unit, located in the Dhaka office of CCDB, and saw himself as answerable to the CCDB Director. This immediate allegiance was moderated by a longer term need to generate information of value to a PhD thesis. .

The first section of this paper outlines the methodology by detailing the steps involved in it's operation. This is followed by a summary of the state of the monitoring system as of March 1995, a year after the first steps were taken to establish it. A series of contrasts are then made between this participatory monitoring system (PMS) and what are described as orthodox approaches to project monitoring. Questions are then raised about the evaluation of monitoring systems. Finally, an interim conclusion about the value of the experiment is stated, and two issues for further exploration are identified.

AN OUTLINE OF THE PROCESS, AS IMPLEMENTED

Nine steps were involved. These are outlined in detail below.

Step 1 The selection of domains of change to be monitored.

Through the Director of CCDB the author facilitated a brief process whereby Dhaka based senior staff identified three broad areas or types of changes they thought CCDB needed to monitor at the project level. The number was limited to three in order to keep procedures simple at the earliest stages of what was initially an experiment.

The three types of change selected were phrased as follows:

"Changes in people's lives"

"Changes in people's participation"

"Changes in the sustainability of people's institutions and their activities."

None of these types of change were precisely defined. Their boundaries were deliberately left "fuzzy". Initially it would be up to the field level staff to interpret what they felt was a change belonging to any one of these categories. One additional type of change was included. This was "any other type of change" as judged important by the project level staff. The intention was to leave one completely open window through which field level staff could define what was important and report accordingly. In the case of the first three domains of change it was the Dhaka head office staff who had proposed the boundaries or window within which events would be reported.

Step 2 The reporting period

Since the first trial of the method in March 1994 changes have been reported for each of the months of April onwards in each of the four Rajshahi project areas. An experiment was made with fortnightly reporting in April but this was found to be too demanding of staff time at both the project office and head office level.

Step 3 The participants

There were four groups of participants in the monitoring system: (a) the shomiti members in the Rajshahi area, (b) the project staff in the Rajshahi area, (c) the senior staff in the head office of Dhaka, and (d) CCDB's donors, particularly those participating in the annual Round Table Meeting (RTM). The structure of their participation determined how the information from the monitoring system was analysed. This is discussed in detail under Step 5 below.

Step 4 Phrasing the question

The basis to the monitoring system was a simple question in the following form:

"During the last month, in your opinion, what do you think was the most significant change that took place in the lives of people participating in the PPRDP project ?

The respondent was then asked to give an answer (written in Bangla) in two parts. The first part was descriptive: what happened, who was involved, where did it happen, when did it hap pen. The intention was that there should be enough information written down so that an inde pendent person could visit the area, find the people involved and verify that the event took place as described.

The second part of the answer was expected to be explanatory. The respondent must explain why they thought the change was the most significant out of all the changes that took place in that month. In particular, what difference did it make already, or will it make in the future ?

Significance was not expected in any absolute sense, but rather in a relative sense, evident when the various changes that were observed to have taken place in the same reporting period were compared to each other.

It was not expected that the explanation of significance will be objective. On the contrary, it would be a subjective expression of the respondents values and concerns. The purpose of the explanation was to help bring these values into the public realm where they could be examined, compared and selected.

The process of sampling changes that was involved was purposive rather than random. The monitoring system was not reporting the average state of the PPRDP, but rather what was tak ing place on the outer perimeter of the programme's experience, the most significant events. If the reported change was a negative one, then it was a type of change the PPRDP would want to move away from, to avoid in the future. If it was a positive one, then it was a type of change that the PPRDP would want to see become more central to their programme, more typical of their activities as a whole, in the future.

Step 5 The structure of participation

In March 1994 a workshop was held with the senior staff of the four Rajshahi project offices to plan the implementation of the monitoring system. Each Project Office (PO) was told that at the end of each month thereafter they would be expected to report to the head office one sig nificant change in each of the four domains of change. Each Project Office was then asked to draw up a plan for how their field staff would, each month, identify a range of potentially im portant changes and how these would then be analysed in order to identify the most important. This change would then be sent by the Project Office to the head office in Dhaka. For research reasons no requirements or constraints were imposed on who could or could not be involved in the identification of significant changes within each of the Project Offices. However, it would not have undermined the basic methodology if the head office had imposed a specified com mon process. For example, all Project Offices must involve shomiti representatives in this pro cess. In this experiment Project Offices were not told that they had to include beneficiaries, or that they could not include beneficiaries in this process. They were also told they were free to copy from each others plans if they wished.

Some options concerning methods of selecting from an array of significant changes were out lined, specifically the possibility of using hierarchy (immediate bosses) or teams (of peers) to make the selection of the most significant change out of all they had identified. There was no requirement that the same approach be used in each project area. Nor was there any require ment that the plan individual Project Offices made would have to be rigidly adhered to thereaf ter. However, it was insisted that if the plan was changed then the new plan should be made clear to the head office. The central requirement was that however the changes were identified and then selected to be sent to Dhaka, it should be transparent and accountable to those reading the selected accounts. In practice, an average of 15 changes were documented by the field staff at each Project Office, each month, out of which four were then selected by more senior staff in the same Project Office as the most significant and then sent on to Dhaka.

The same process was repeated at the Dhaka head office. The four sets of four changes (one set of 4 from each Project Office) were brought to the head office each month. The task of the head office staff was to select the 4 changes from the 16 which they thought were the most significant of all. In other words, the single most significant change in peoples lives, in peoples participation, in sustainability, and change of any other type. The choice of participants was left to the Director. In practice between four and eight senior staff attended each of the monthly meetings. The process whereby the choice was made by the Dhaka participants was left up to that group. In practice they decided that each participant would rate each story out of 10, and the ratings would then be aggregated to produce the group response. The rating process was preceded by an active group discussion of each account of change. The single requirement was that they must document and explain their choice, including who was involved in that process, and communicate it back to the staff in the four Project Offices. In practice, the Dhaka office meeting and discussion took about 3 hours of staff time per month.

The third level in this process of analysis involved the donors who attended the 1994 Round Table Meeting (RTM) in Dhaka in November 1994. By the end of September the CCDB head office had selected 24 accounts of significant changes (4 domains x 6 months). Those changes were collated in the form of four chapters in a report. The introduction outlined the methodology behind their collection (as here), and each chapter thereafter focused on one do main of change (with accounts of change ordered chronologically within the chapters). The appendices detailed an analysis of shomiti and staff participation in the monitoring system. It was proposed that donors should read each chapter and select the one change in each chapter which they think was the most significant according to their own values and concerns. As with other participants, they should document the reasons for their choices. In practice, the presence of wide range of people at the RTM enabled the 6 months' changes in the first domain, that of the lives of the people, to be analysed in this way by 5 sub-groups (2 donors, 1 senior staff, 1 junior staff and 1 shomiti representatives group).

The structure of participation described above meant that a very wide range of people's life experiences at the shomiti level were subject to an iterated process of analysis (choice- explanation-choice...) that eventually selected a small number of stories of high value. At each level of CCDB a range of stories of change were available and subject to a range of interpreta tions. From amongst these some were selected, retained and forwarded on to the next level in the organisational hierarchy. This process was re-iterated from the level of field workers, senior Project Office staff and senior Dhaka office staff. The process of iterated variation - selection - retention, taking place here within the environment of a single organisation, is the essence of what has been described by Campbell (1969) and others as the evolutionary algorithm.

Step 6 Feedback

After each month's changes were evaluated by CCDB head office staff their judgement of the most significant changes, and the reasons behind those judgements, have been fed back to the Project Offices concerned. Similarly, the results of the sub-group discussions at the RTM were also fed back via a plenary session.

The purpose of regular feedback was so that those identifying the changes in the first instance can take into account the views of CCDB senior staff when in the process of evaluating changes. They could either passively adapt their search for significant change according to the perceived concerns of the head office, or more actively seek better examples and provide better explanations for the significance of the types of changes that they think are most significant. It was intended that if feedback was provided as planned the monitoring system should take the form of a slow but extensive dialogue up and down the CCDB hierarchy each month. In more evolutionary terms it can be seen as a process of co-evolution of interpretative frameworks within an organisational ecology.

Step 7 Verification

Those changes that were identified as the most significant of all were precisely those stories where the most effort needed to be invested in verifying the factual details of the event. Verification visits to the sites of the described events can perform a policing function, ensuring that field staff are kept honest in their report writing. They also provide an opportunity to gather more detailed information about the event which was seen as specially significant, and if some time after the event, a chance to see what has happened since the event was first documented (another aspect of impact). Initially follow up visits were made by the author, with his Training Unit counterpart. Later in his absence CCDB sent head office staff from the CCDB Information Unit.

The next two steps are optional extras, not ones central to the process.

Step 8 Quantification

This can take place at two stages. Firstly when an account of change was being described it was quite possible to included quantitative information as well as qualitative information. Secondly, it was possible to quantify the extent to which changes identified as the most significant in one location or zone have taken place in other locations, within a specific period. In the case of one significant change identified in March 1994 (concerning a women shomiti member's purchase of land in her own name) a letter was sent by the Programme Coordinator to all 10 PPRDP Project Offices seeking information on the numbers of identical incidents that they were aware of having taken place in their project area in the past year. However, no need was seen to repeat this particular question every month thereafter, as in traditional monitoring systems.

Step 9Monitoring the monitoring system

This step was not essential, but is described here nevertheless. Using records generated by the above process it was possible to monitor changes over time in the proportion of shomities and households that the monitoring system is effectively sampling. An analysis in November 1994 showed that after 6 months of operation accounts of change had been written concerning 43% of the shomities. The proportion has continued to grow since then. Similarly it is possible to monitor the degree to which different types of staff (gender, age, position, education) are ac tively involved in the process of monitoring change, and of those actively involved, with what degree of success. Success in this case being defined as having an account of change being se lected as most significant at the Project Office and Dhaka level. CCDB has yet to do this type of analysis formally, but informally there is awareness of differences between staff in their par ticipation and success. Finally, with an existing data base detailing features of the shomities that exist it is possible for specialist staff, such as the CCDB Research Unit, to identify the correla tion between changes reported as taking place and objective features of the shomities, such as gender, size, and savings levels.

OUTCOMES SO FAR

CCDB has experimented on two previous occasions with monitoring methods. One system was developed in 1989 by an external consultant and was wholly quantitative in its focus, poten tially generating large volumes of tabulated data. It was never implemented. The second was jointly developed in 1993 by a new expatriate staff member working with the staff of the CCDB Information Unit. It focused on the use of rating scales by field staff which were intended to identify the variations in five predetermined types of participation taking place in the peoples organisations. It was not implemented.

Although initially planned to operate for the six months until the RTM meeting the monitoring system described above was continued on afterwards, on the instructions of the Director. De lays in the reporting of changes from the project level have not increased, but Dhaka staff were one month behind in their analysis of changes, at the time of the last visit to CCDB in March 1995. In January 1995 CCDB decided on its own initiative to extend the system to include 3 more of the 10 PPRDP areas. A training workshop was organised by the writers CCDB Train ing Unit counterpart in mid-January, making use of Rajshahi zone staff who have experience with the method. Mention has been made of extending it to other CCDB projects and to spe cialist support units with the CCDB Dhaka office.

Rather than the contents becoming increasingly identical over time, and the system reaching a form of steady state, new changes have continued to be reported. The most notable of which concerned shomiti involvement in a intra-family conflict over contraception use, reported in December 1995. Rather than the Project Office with the most sycophantic Project Officer be ing the most successful (as defined above), the reverse has been the case, success seems to be more correlated with independence of opinion. While the Director had previously identified one Project Office as the most successful, on the basis of its good credit repayment record, the same Project Office was the least successful in terms of it's ability to generate, through the PMS, a large number of highly rated accounts of significant change.

In contrast to the very limited use made of the output of the CCDB Research Unit over the same period, staff in the Information Unit and the Materials Development Units of the Dhaka office have made extensive use of stories of change in CCDB publications, videos and educa tional materials. In addition, project office staff took visiting donor representatives to shomities featuring the reported significant changes immediately prior to the RTM in November.

The specific settings of the parameters of the monitoring system have not remained static. In March 1995 CCDB staff were in the process of considering a focus on "changes in the project management". In response to demand from the field and head office the Dhaka staff who give the highest or lowest ratings for a change which in aggregate has been selected as most signifi cant had been asked to give their explanations. As a result of an informal participatory evalua tion of the system carried out by the author in February 1995 CCDB staff have become aware of the fact that collectively they see a much wider range of objectives for the monitoring system than were initially conceived by the author and the Director (who wanted evidence of impact). In summary, the monitoring system has survived and is itself undergoing evolution, both in the specifics of its procedures and in its perceived purpose.

The fate of the system is by no means secure however. Differences of opinion (some explicit) exist within senior CCDB staff, as to the value of the process. In the absence of clear signs of external demand for the information that it is producing, for example, from CCDB's donors, it's future is uncertain.

The survival, use and extension of the system exceeded the researchers initial expectations. Two of the researchers own expectations were not met. In each account of change that went up the organizational hierarchy there were associated arguments for that choice, added when ever that change was selected. It was expected that middle and especially senior staff would focus on the associated arguments and through their own choices, selectively reinforce specific types of arguments. Decision premises would thus be subject to evolution over time. In practice it seemed that all levels of staff focused primarily on the descriptions, the empirical events, and gave only passing attention to the criteria of choice used by those below them. Two explana tions are possible. One is of "inappropriate" management style, that middle and senior staff were guilty of micro-management. The other is that the behaviour reflects the staff members understanding of how the system was supposed to work, that they were in fact required to fo cus on the events themselves first of all. Explanation of the option of focusing on the criteria of choice used by more junior staff may have been needed.

The other expectation was that because of the diversity of possible significant changes each set of staff faced they would be required to be critical. They would have to make choices and this would involve comparisons, and thus judgements of relative merit. Participation in meetings at both Project office and Dhaka Office levels showed that staff could become very animated and assertive in their opinions of relative merit. However, documentation of the choices made, es pecially the reasons given, only rarely reflected any sense of doubt or criticism, or made any explicit comparison with other changes that had been examined.

A diversity of perceptions offers an organisation choice of direction. One aspects of that diver sity is the extent to which the changes that are reported are explicitly negative or positive. Per haps 90%-95% of the changes that were reported were seen by participants as being about pos itive changes. During the informal evaluation of the PMS it was clear that staff at all levels were quite aware of this aspect of its functioning. Field level staff explained the risks to their job security involved in more critical reporting of events. Senior staff, less explicitly, indicated concern about donor and governing board responses to negative changes. For CCDB staff the balance of reporting as it existed was functional. Coming from an academic perspective it was somewhat disappointing for the researcher, who initially saw the presence of critical content as a potential indicator of the systems success. Two means of managing balance in reporting were potentially available. One was where negative changes were reported, senior staff could select them and quite consciously provide feedback which lauded the act of reporting as a reason for its selection, along with the specific contents of the report. One problem with this approach, already encountered, is that subtlety of communication style (intended to lower risk) will mean that what is intended as a report of a negative change is in fact not recognised as such by the senior staff. The other means was for CCDB staff to establish a fourth separate and specific domain where "negative changes" had to be reported. Here the researchers interests as a re searcher meant that this option was not spelled out as clearly, or encouraged, as much as it might have been.

ORTHODOX VERSUS EVOLUTIONARY APPROACHES TO MONITORING

Current approaches to monitoring found in the larger NGOs, as well as bilateral and multilat eral aid organisations are heavily influenced by a planning ethos that places substantial empha sis on rationality, prediction and control (Davies, J,. 1994). The approach documented in this paper is in many respects the opposite. Seven differences are outlined below.

1. Perhaps the central feature of planning base methods of monitoring is the use of "indicators", and the need for a common understanding about them within the organisation if the monitoring system is to work as intended. That understanding ideally includes: (a) the meaning of the "indicator", what it is meant to represent, (b) the worth of monitoring the event represented by the indicator rather than any other possibilities, and (c) the existence of the event, did it take place ?. Confusion, especially over their existence or meaning is seen as a threat to the systems functioning. Within the conventional approach it is believed that differences in the subjective perspectives of events, and the underlying value concerns of different observers need to be controlled or ignored.

Under the evolutionary approach agreement on meaning of events is an outcome at the end of a process (a months cycle or more), never final in its form, and subject to revision in the light of new experience. The identification of differences in interpretation is a central to the whole process, they are to be brought to the surface and explored, not to be ruled out. In some respects, specially the unfinished and tentative nature of knowledge, the epistemology of the evolutionary approach is a more post modern outlook, whereas the conventional approach is closer to positivism.

The PMS acknowledges the fact that different sets of values and interpretative frameworks co- existence at different levels within an NGO, and between the NGO and its donors. The NGO field level staff in effect are creating menus of possible views of the world within broad categories defined by their bosses (Step 1). Their immediate superiors choose from this menu of views and in the process create a smaller menu based on their view of the world. This menu in turn is chosen from by those above them, according to their views. Although clearly located in a hierarchical structure of power it gives a significant power to those at the base, more so than under conventional systems. It also still enables the donors to address their concerns without counter-productively riding on the backs of the NGO. The senior staff of CCDB did not address gender issues in their original specification of the three domains of change. Nevertheless, a large proportion of the accounts of change that were processed up the hierarchy had very clear issues of gender equity embedded in them. In the Round Table Meeting between CCDB and its donors the most important differences between them in their interpretation and valuation of the significant changes focused on the gender issues within the descriptions of those changes. These were far more subtle and complex than an indicator ap proach to gender issues would have revealed.

2. Planning based systems are almost universally very quantitative in their content. Quantitative analysis is based on the ability to enumerate things or events. Enumeration requires a basic as sumption about the identify of events being enumerated i.e each of six apples is in fact an ap ple. At the very basic level of counting quantification is also about the homogenisation of expe rience. Differences between apples are extraneous and irrelevant. One apple and one orange can only be added by regarding them both simply as pieces of fruit. Within the daily experience of organisation those events which are countable are summarised by a process of inclusion. Broad swathes of experience are summarised by totals and averages.

The need to summarise is understandable given that most formal organisations have a hierar chical shape. The daily experience of clients and staff at an organisation's base has to be con verted into a form manageable by the very small number of staff at the apex. In CCDB the Dhaka office staff had to make sense of 16,500 person years of beneficiary experience of CCDB each year, in the Rajshahi area alone. The question is whether there are not better means of doing this than quantification via indicators.

Within the evolutionary approach experience is summarised by selection rather than by inclu sion, it focuses on the exceptional rather than the commonality, and it seeks to differentiate rather than homogenise. It is about defining the meaningful edges of experience rather than identifying a central tendency. Within this perspective an observer of a basket of five or six dif ferent types of fruit might selectively summarise that experience by reporting that the orange is best, because it is the sweetest. Other observers might selectively summarise the same experi ence from a different perspective, reporting that the apple should be removed because it has begun to rot and will spoil the other fruit if left there. Philosophising together they might agree that the rot is the more important feature because unattended to it will preclude a range of later experiences, including those of sweetness.

3. Under the planning based approach to monitoring events of concern are identified before their occurrence, rather than afterwards. In conventional systems "indicators" are established at the beginning of a project, and data in the form of statistics is gathered repeatedly throughout the life of the project, concerning the frequency of those events. In more recent revisions of this approach, described as "process" approaches, by the Overseas Development Administra tion (ODA) and others, the identification of indicators may be delayed until the project is estab lished, and may in the course of the life of the project, be redefined more than once. The pro cess is strongly deductive in orientation: start with a conception of the desired state and work down from there what might be the empirical indicators of its occurrence.

The opposite is an inductive approach, where indicative events are abstracted out of recent ex perience, and this process is renewed with each new reporting period of the monitoring system. Instead of being predictable it is open ended.

4. In most monitoring systems events of concern are defined by people distant from those events which are to be monitored. Typically the identification of indicators is carried out by senior staff in organisations, either in specialist research units or senior executive staff. In some cases it is made by executives in head offices in other countries, such as was the case with Action Aid's attempt to identify a common set of indicators for its global programme in the early 1990's. Distance can also be in the form of differences in tribe, caste, class, language and education as well as in geography, between those identifying the indicators and those whose lives they are expected to relate to. Reformist approaches have consisted of taking the indicator identification process down the hierarchy, in some cases, to the beneficiaries themselves whose views are sought, through the use of PRA methods. Action Aid, ACCORD and other northern NGOs have taken this route in more recent times. The problem with such an approach is the difficulty the NGO then finds, while operating within the conventional framework, in summa rising the information produced by a diversity of locally identified indicators.

The alternative approach is to give those closest to the experience being monitored (e.g the field staff) the right to pose to those above them a range of competing interpretations of those events. The role of those in power over them then becomes to respond, on a selective rather than inclusive basis, to the menu of options provided to them anew each month. Diversity be comes a source of opportunity rather than a conundrum.

5. Normally the analysis of events documented by an organisations monitoring system is carried out on a centralised basis, at senior levels of the organisation. Typically, field level workers do not analyses the data they collect, rather they simply forward information up their hierarchies for others to analyse. In the language of computer design this is a serial and central processor based approach. From a Marxist point of view it might be seen as a form of alienation.

The alternative, which can be described as parallel and distributed processing, is seen both in recent approaches to computer design, and theorising about information processing in biologi cal systems, and markets. Information is not stored or processed on a centralised basis, but is distributed throughout the organisation, and processed locally. Staff not only collect informa tion about events but they make their own evaluation of that information, according to their own local perspective.

6. Normally when conventional monitoring data is analysed it is in a form and location that strips it of context. Centrally located analysts of tables of statistics sent from field offices are usually well removed from the site of field experience and have a very limited and static predesigned context in which to interpret the meaning of the events that are summarised. Typically few text comments accompany statistics sent up from field workers.

The alternative makes use of what Geertz (1973) has called "thick, description", closely textured accounts of events, placed in their local context, and where the role of the observer and their subjectivity, is visible. In the world of ordinary people these often take the form of stories or anecdotes. Within the evolutionary approach to monitoring outlined here these "stories" are accompanied by their readers interpretations.

7. Most monitoring systems are largely static structures. Indicators remain essentially the same each reporting period, and the same questions are asked again and again. The focus remains the same. Those involved in the monitoring system are also seen as unchanging, they simply do their duty. The possibility of independent (constructive or subversive) staff adaptations of and to the monitoring system is denied.

With the evolutionary approach the contents of the monitoring system are potentially far more dynamic and adaptive (The reality of practice will of course vary from organisation to organisation). Events reported reflect both a changing world and a changing sets of perceptions within the organisation about what is important within their world. Where quantitative data is sought on the incidence of an event found to be significant by a number of levels of the organi sation it can be done so on a once off basis, there is no intrinsic need to repeat the same inquiry each reporting period thereafter. Though there are relatively static domains of change being monitored the fuzzy nature of their definition, and the process of definition by current best ex ample, means the overall focus of the monitoring can move over time. Finally, the monitoring system is simply another arena of organisational life where staff are expected to adapt, but one where their adaptations should be more visible, and therefore open to more deliberate and se lective reinforcement.

EVALUATING MONITORING SYSTEMS

The evaluation of a monitoring system is as problematic as the task of evaluating the impact of CCDB's activities on the lives of it's beneficiaries. The month to month functioning of the monitoring system is an event subject to observation and interpretation just as much as that of the women shomiti member who purchased land in her own name. There are multiple observers with varying criteria of concern, some within CCDB and some outside. All are making judgements based influenced by their current context and past history. What weight should each of those judgements be given and how can they be aggregated or summated ?

An approach based on an evolutionary epistemology would start with the fact that the system has survived and treat that as evidence of its value in an aggregate sense. It has some degree of fitness with it's environment. But this is a minimalist judgement. In the words of Belew (1991:9), "the dumbest smart thing you can do is stay alive". Associated with this complex recurrent event called a monitoring system are the many interpretations of its meaning, including values and preferences which sustain it, some of which are more prevalent than others, and whose prevalence changes over time. Success, i.e some form aggregate value, might also be judged, though with much more difficulty, by identifying changes in prevalence of these sustaining values. It could be argued that sustainability on this level is more important than the mere survival of a set of organisational routines known as a "participatory monitoring system". But on the other hand it could be argued that those values need some form of substantiation to be fully meaningful and thereby to survive as a set.

While the CCDB monitoring system shows that the structured and public exercise of choice is a means of surfacing and managing some values within an organisation, it remains to be established how such a process should be structured when there is a need to evaluate an intra- organisational event in a wider social and organisational context. Donors, government, consultants, and academics all have may have an interest in the CCDB experiment, yet they are much more independent from each other than staff within a single hierarchically structured organisation. One possibility, yet to be explored, is the use of a market as a means of summarising judgements. As with the use of hierarchy, its value could be improved by making information about events taking place in that market as transparent and public as possible.

CONCLUSIONS AND ISSUES REMAINING

The monitoring system that has been developed not only addressed the needs of CCDB as an organisation to monitor the impact of its activities. The concepts underlying its functioning are also relevant to the wider issue of how to best analyse qualitative (as distinct from quantitative) data, and how to do this on a participatory rather than solitary basis. The approach proposed suggest that a key issue is inter-subjectivity, the extent to which different observers of events or phenomenon agree and disagree with each other. In this paper the focus has been on the per ception of change and of differences between those perceptions. The associated practical issues are how to process such information and at the same time how to represent it.. It is suggested that iterated processes of variation - selection - retention, i.e the use of an evolutionary algo rithm, are of value in dealing with both tasks.

If this approach to organisational learning is to be developed two areas need more in-depth ex ploration. One is the concept of public and private domains of knowledge in organisation. Studies of simulated multi-agent systems have suggested that the capacity for learning involves a balance between order and disorder (Kauffmann, 1994). The public domain of knowledge in an organisation could be seen as highly ordered, one containing "that which everyone knows that everyone knows" and one where areas of agreement on meaning provide a further level of order. Many more private domains also exist in organisations, and in their plurality and differ ence can be seen to embody more organisational disorder. Many of these private domains are likely to remain so, because of the contradictions in interests which could not co-exist in public. What is the role of monitoring systems in relationship to these public and private domains ? Widened agreement on an issue, and awareness of that agreement, will almost by definition assist successful joint action by people holding those views. Increased awareness of differences is more ambivalent. It may enable innovations in organisations' objectives and practice or it may threaten the very cohesion, and survival, of the organisation. The question is then "How can these risks be managed creatively ?"

The second issue is that of internal and external demand for information. The CCDB experi ment took place because of internal perceptions of need, but that awareness included knowl edge of donors views and concerns. The impact of external demands for information is medi ated not only by internal processes of interpretation within NGOs, but also processes of inter pretation within their donor organisations. Variations clearly exist in the extent to which donors see their demands as legitimate or questionable (e.g ODA versus Christian Aid), and burdens to be minimised or lightened versus requests which are potentially educative and enabling. Em phasis also varies in the extent to which they see themselves as seeking confirmation of pre-es tablished expectations versus identification and understanding of news about innovation. The fate of developments such as those within CCDB will undoubtedly be influenced by how these tensions are resolved. In order to understand and possibly influence this process attention will also have to be directed by researchers and consultants to the nature of information demands within donor organisations, to the influences on these demands, and how they can be managed creatively.

--o0o--

Subsequent evolution of the CCDB system

Since 1994 a number of people and organisations have experimented with variations of the method implemented by CCDB. Their names and contact addresses are given below, for those who might want to contact them.

• Leslie Dove is helping the Women's Health and Safe Motherhood Project implement the system in the Philippines. Five domains of change are being used: (a) Changes in women's health practices, (b) Changes in women's decision making, (c) Changes in women's participation and involvement in the community, (d) Any other changes, (d) Lessons learned.
• Jessica Dart, a Ph.D student at the University of Melbourne. Jessica has been adapting the method to Target 10, a dairy extension project in Victoria, Australia. See "Target 10 Evaluation Stories May 1998 - May 1999" for detailed examples of the products so far.
• Irene Guijt, of IIED, has encouraged it's use by NGOs in Brazil.
• Terry Bergdall, consultant based in Zambia. Applied a variation of the method during the design stage of a rural development project in Ethiopia.
• Helen Wedgwood, and others at ITDG, applied the method as a way of integrating information coming in to the head office in the UK, from the country and regional offices.
• Michael Schroll, of IIRR in Kenya, has used the method in M&E training workshops with NGOs.
• And CCDB continues to use the method, though the original system has been expanded and altered more than once. For information contact Kalipada Sarker, who initially helped set up the system, and who has been managing it since then

Mailing List

Please Note: There is now a "MostSignificantChanges" egroup dedicated to information sharing and discussion about the use of the method described in this paper.

REFERENCES

Belew, R.K. 1991. Artificial Life: A Constructive Lower Bound for Artificial Intelligence. IEEE Expert, 6.1:8-15.

Campbell, D.T. 1969. Variation, Selection and Retention in Socio-cultural Evolution. General Systems, 16:69-85.

Davies, J. 1994. Information, Knowledge and Power. IDS Bulletin. 25.2:1-12.

Geertz, C. 1973. The Interpretation of Cultures. Basic Books, USA.

Kauffman, S.A. 1993. The Origins of Order: Self-organization and Selection in Evolution. Oxford University Press. Oxford.

=================================

Revised June 25, 1996