## A digression on complexity and networks…

….a side argument from the Rick on the Road post: Cynefin Framework versus Stacey Matrix versus network perspectives

In that post I said

PS1:Michael Quinn Patton’s book on Developmental Evaluation has a whole chapter on “Distinguishing Simple, Complicated, and Complex”. However, I was surprised to find that despite the book’s focus on complexity, there was not a single reference in the Index to “networks”. There was one example of a network model (Exhibit 5.3) , contrasted with a Linear Program Logic Model…” (Exhibit 5.2), in the chapter on Systems Thinking and Complexity Concepts. [I will elaborate further]

One interpretation: Complexity arises through the interaction of many agents having some degree of autonomy. With no autonomy there is simple order (complete predictability), with complete autonomy there is chaos (no predictablity). How do we define autonomy? One view: Autonomy = The number of possible relationships an actor can have with others. When realised, this can be measured in terms of  network density (a Social Network Analysis (SNA) measure). Two cariacature examples of the extremes: 1. An army, with a hierarchical chain of command,  is highly ordered. Here the network structure is  sparse (i.e.  a tree structure) and low in density. 2. “Economic man” , who is free to interact with anyone, in order to maximise his/her utility. Here all possible relationships can be realised, as everyone interacts with everyone. Complexity is the territory in between where actors have some degree of choice of who they interact with. And where there is some degree of predictability. When realised, those choices can also be described in terms of different kinds of network structures. So if we want to explore complex systems we need to look at the structure of networks of actors, both as “initial conditions” affecting what happens next and as “final states”, reflecting what has happened over a given period of time. I.e. an empirical approach, not mysticism :-)

PS: The concept of autonomy could probably be further differentiated, in terms of relationship choices, as follows : (a) the range of relationships available to an actor, already discussed above (b) the freedom to choose amongst those that are available, (c) the range of behaviors available within a given relationship. But how do you measure freedom (b) ? One measure might be the degree to which any choices made are uncorrellated with other events. The diversity of choices made could also be important. Diversity suggests freedom from constraint (more on this theme here).

## Results Based Management Explained (by the ADB)

Results Based Management (RBM) can mean different things to different people. A simple explanation is that RBM is the way an organization is motivated and applies processes and resources to achieve targeted results.

Results refer to outcomes that convey benefits to the community (e.g. Education for All (EFA), targets set in both Mongolia and Cambodia). Results also encompass the service outputs that make those outcomes possible (such as trained students and trained teachers). The term ‘results’ can also refer to internal outputs such as services provided by one part of the organization for use by another. The key issue is that results differ from ‘activities’ or ‘functions’. Many people when asked what they produce (services) describe what they do (activities).

RBM encompasses four dimensions, namely:

• specified results that are measurable, monitorable and relevant
• resources that are adequate for achieving the targeted results
• organizational arrangements that ensure authority and responsibilities are aligned with results and resources
• processes for planning, monitoring, communicating and resource release that enable the organization to convert resources into the desired results.

RBM may use some new words or apply specific meanings to some words in general usage. Check introduction to RBM presentation[PDF | 56 pages].

RBM references that provide more background

• A diagram showing relationship between goals and outcomes
• United Nations Development Program RBM overview
• Canadian International Development Agency RBM overview
• RBM diagnostic tool for Cambodia and Mongolia

## Making government budgets more accessible and equitable

(from ID21)

Involvement in the budget process in poor countries has traditionally been limited to a select group of political actors. But this has changed over the last decade with legislators, civil society groups and the media playing a more active role. What impact is broader engagement having?

Research from the Institute of Development Studies, UK, examines the substance and impact of applied budget work undertaken by civil society groups. The research draws on six case studies of independent budget work in Brazil, Croatia, India, Mexico, South Africa and Uganda. One focus of the research is how civil society budget work influences government budget priorities and spending in a way that benefits poor and socially excluded groups.

Budget work is carried out by various types of organisations including non-government organisations (NGOs), networks and social movements, and research organisations. All the groups examined in the case studies share a commitment to increasing the influence of poor and marginalised groups in the budget process and ensuring that budget priorities reflect the needs of these groups.

The six organisations all engage in certain core activities centred on data analysis and dissemination, advocacy and capacity building. Most work on national and state-level budgets, though several groups also work at the local government level.

The research shows that independent budget work has the potential to deepen democracy by strengthening accountability, fostering transparency and encouraging participation. It can also increase financial allocations in areas that contribute to social justice and equity outcomes and ensure that public money is efficiently spent.

The research also reveals the limits to budget work. Any increases in financial allocations secured as a result of advocacy initiatives are likely to represent a small share of overall government spending. Also, the scope of budget work to influence financial allocations depends on the openness and flexibility of the budget process (spending priorities may not be open to change).

The impacts of budget work identified by the research include:

• improving the transparency of budget decisions and budget processes and increasing the accountability of state actors
• increasing awareness and understanding of budget issues
• improving budget allocations in a way that benefits poor and socially excluded groups
• ensuring better use of spending, for example in areas such as health and education, and reducing corruption (by tracking expenditures)
• diversifying the range of actors engaged in budget processes (for example, legislators, civil society groups and the media)
• strengthening democracy and deepening participation.

The research concludes that:

• Budget work has been successful in a range of areas, including improving equity and social justice outcomes.
• The technical nature of the budget process limits the scope for broadening citizen participation.
• The challenge for budget groups is how to scale-up and replicate the successful impacts achieved to date.
• Influencing budget policies requires a combination of sound technical knowledge, effective communications and strategic alliances.
• Promoting the voice of poor and socially excluded groups is an important indirect effect of budget work.

Source(s):
‘Budget Analysis and Policy Advocacy: The Role of Non-governmental Public Action’, IDS Working Paper 279, IDS: Brighton, by Mark Robinson, 2006 Full document.

Funded by: UK Economic and Social Research Council

id21 Research Highlight: 16 August 2007

Further Information:
Mark Robinson
Policy and Research Division
UK Department for International Development (DFID)
1 Palace Street
London SW1E 5HE
UK

Tel: +44 (0)20 70230000
Fax: +44 (0)20 70230636
Contact the contributor: mark-robinson@dfid.gov.uk

## Results Based Management (RBM): A list of resources

### CIDA website: Results-based Management

Results-based Management (RBM) is a comprehensive, life-cycle approach to management that integrates business strategy, people, processes, and measurements to improve decision-making and to drive change.

The approach focuses on getting the right design early in a process, implementing performance measurement, learning and changing, and reporting on performance.

• RBM Guides
• RBM Reports
• Related Performance Sites

• ### ADB website: Results Based Management Explained

Results Based Management (RBM) can mean different things to different people. A simple explanation is that RBM is the way an organization is motivated and applies processes and resources to achieve targeted results.

Results refer to outcomes that convey benefits to the community (e.g. Education for All (EFA), targets set in both Mongolia and Cambodia). Results also encompass the service outputs that make those outcomes possible (such as trained students and trained teachers). The term ‘results’ can also refer to internal outputs such as services provided by one part of the organization for use by another. The key issue is that results differ from ‘activities’ or ‘functions’. Many people when asked what they produce (services) describe what they do (activities).

RBM encompasses four dimensions, namely:

• specified results that are measurable, monitorable and relevant
• resources that are adequate for achieving the targeted results
• organizational arrangements that ensure authority and responsibilities are aligned with results and resources
• processes for planning, monitoring, communicating and resource release that enable the organization to convert resources into the desired results.

RBM may use some new words or apply specific meanings to some words in general usage. Check introduction to RBM presentation[PDF | 56 pages].

RBM references that provide more background

### UNFPA website: Results-Based Management at UNFPA

There is a broad trend among public sector institutions towards Results-Based Management–RBM. Development agencies, bilateral such as Canada, the Netherlands, UK, and the US as well as multilateral such as UNDP, UNICEF and the World Bank, are adopting RBM with the aim to improve programme and management effectiveness and accountability and achieve results.

RBM is fundamental to the Fund’s approach and practice in fulfilling its mandate and effectively providing assistance to developing countries. At UNFPA, RBM means:

• Establishing clear organizational vision, mission and priorities, which are translated into a four-year framework of goals, outputs, indicators, strategies and resources (MYFF);
• Encouraging an organizational and management culture that promotes innovation, learning, accountability, and transparency;
• Delegating authority and empowering managers and holding them accountable for results;
• Focusing on achieving results, through strategic planning, regular monitoring of progress, evaluation of performance, and reporting on performance;
• Creating supportive mechanisms, policies and procedures, building and improving on what is in place, including the operationalization of the logframe;
• Sharing information and knowledge, learning lessons, and feeding these back into improving decision-making and performance;
• Optimizing human resources and building capacity among UNFPA staff and national partners to manage for results;
• Making the best use of scarce financial resources in an efficient manner to achieve results;
• Strengthening and diversifying partnerships at all levels towards achieving results;
• Responding to the realities of country situations and needs, within the organizational mandate.

#### OECD report: RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A REVIEW OF EXPERIENCE BACKGROUND REPORT

In order to respond to the need for an overview of the rapid evolution of RBM, the DAC Working Party on Aid Evaluation initiated a study of performance management systems. The ensuing draft report was
presented to the February 2000 meeting of the WP-EV and the document was subsequently revised.
It was written by Ms. Annette Binnendijk, consultant to the DAC WP-EV.

This review constitutes the first phase of the project; a second phase involving key informant interviews in a number of agencies is due for completion by November 2001.

158 pages, 12 page conclusion

this list has a long way to go….!

## The road to nowhere? Results based management in international cooperation

Results-based management has become a fact of life for development agencies. They might hope to learn from the experience of the US Agency for International Development (USAID) which has already gone down this road. It is indeed instructive that USAID has come back up the road again saying ‘there’s nothing down there’. But development agencies have been rightly criticised in the past for paying too little attention to the final impact of their activities, so we would like to support a results-based approach. But we should not do so blindly when it suffers from the severe limitations outlined below. Serious attempts to link agency performance to developmental outcomes must rely upon the log-frame. The log-frame is not a universal panacea but, used properly, can force agencies into a critical examination of the nature of their programmes and projects, and the results they achieve.

This posting is available in full at the Euforic website

## M&E blogs: A List

• EvalThoughts, by Amy Germuth, Durham, NC, United States, President of EvalWorks, LLC a woman-owned small evaluation and survey research consulting business in Durham, NC.
• Evaluation and Benchmarking. “This weblog is an on-line workspace for the whole of Victorian government Benchmarking Community of Practice.”
• M&E Blog, by…?
• Aid on the Edge of Chaos, by Ben Ramalingam
• Design, Monitoring and Evaluation, by LARRY DERSHEM – Tbilisi, Georgia
• Managing for Impact: About “Strengthening Management for Impact” for MFIs
• Genuine Evaluation: “Patricia J Rogers and E Jane Davidson blog about real, genuine, authentic, practical evaluation”
• Practical Evaluation, by Samuel Norgah
• AID/IT M&E Blog: “…is written by Paul Crawford, and is part of a wider AID/IT website”
• Blog: Evaluateca: Spanish language evaluation blog maintained by Rafael Monterde Diaz. Information, news, views and critical comments on Evaluation
• Empowerment Evaluation Blog “This is a place for exchanges and discussions about empowerment evaluation practice, theory, and current debates in the literature” Run by  Dr. David Fetterman”
• E-valuation: “constructing a good life through the exploration of value and valuing” by Sandra Mathison,Professor, Faculty of Education, University of British Columbia
• Intelligent Measurement. This blog is created by Richard Gaunt in London and Glenn O’Neil in Geneva and focuses on evaluation and measurement in communications, training, management and other fields.
• Managing for Impact: Let’s talk about MandE! “Welcome to the dedicated SMIP ERIL blog on M&E for managing for impact!An IFAD funded Regional Programme, SMIP (Strengthening Management for Impact) is working with pro-poor initiatives in eastern & southern Africa to build capacities to better manage towards impact. It does so through training courses for individuals, technical support to projects & programmes, generating knowledge, providing opportunities for on-the-job-training, and policy dialogue.”
• MCA Monitor Blog “…is a part of CGD’s MCA Monitor Initiative, which tracks the effectiveness of the US Millennium Challenge Account. Sheila Herrling, Steve Radelet and Amy Crone, key members of CGD’s MCA Monitor team, contribute regularly to the blog. We encourage you to join the discussion by commenting on any post”
• OutcomesBlog.Org “Dr Paul Duignan on real world strategy, outcomes, evaluation & monitoring Dr Paul Duignan is a specialist in outcomes, performance management, strategic decision making, evaluation and assessing research and evidence as the basis for decision making. He has developed the area of outcomes theory and its application in Systematic Outcomes Analysis, the outcomes software DoView and the simplified approach to his work Easy Outcomes. He works at an individual, organizational and societal level to develop ways of identifying and measuring outcomes which facilitate effective action. For a bio see here.
• Rick on the Road: “Reflections on the monitoring and evaluation of development aid projects, programmes and policies, and development of organisation’s capacity to do the same. This blog also functions as the Editorial section of the MandE NEWS website
• The Usable Blog “A blog on “Thoughts, ideas and resources for non-profit organizations and funders about the independent sector in general and program evaluation in particular” By Eric Graig “
• The MSC Translations blog is maintained by Rick Davies, and is part of the MandE NEWS website. The purpose of this blog is:1. To make available translations of the MSC Guide in languages other than English. 2. To solicit and share comments on the quality of these translations, so they can be improved.The original English version can be found here The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use
• Zen and the art of monitoring & evaluation “This blog is some of the rambling thoughts of Paul Crawford, a monitoring & evaluation (M&E) consultant for international aid organisations” Paul is based in Australia.

And other lists of M&E blogs

## Improving health services through community score cards. A case study from Andhra Pradesh, India

Case study 1, Andhra Pradesh, India : improving health services through community score cards
MISRA, Vivek et al , August 2007

This eight page note summarises the findings, processes, concerns, and lessons learned from a project in Andhra Pradesh – one of six pilot projects aimed at the application of specific social accountability tools in different contexts of service delivery

## SYSTEMS CONCEPTS IN EVALUATION: AN EXPERT ANTHOLOGY

Bob Williams and Iraj Imam (eds.)
EdgePress/American Evaluation Association (2007)

Systems Concepts in Evaluation: An Expert Anthology brings
together a wide range of systems concepts, methodologies and
methods and applies them to evaluation settings. This book

• What is a systems approach?
• What makes it different from other approaches?
• Why is it relevant to evaluation?

The 14 chapters cover a wide range of systems concepts and methods. Most chapters are case study
based and describe the use of systems concepts in real life evaluations. The approaches and methods
covered include:

• System Dynamics (both quantitative and qualitative)
• Cybernetics and the Viable System Model
• Soft Systems Methodology
• Critical Systems Thinking

There are also overview chapters that explore the history and diversity of systems approaches and their
potential within the evaluation field. There is a substantial introduction by Gerald Midgley to the key
developments in systems concepts and methods over the past 50 years, and this explores the
implications for evaluation of each of those developments.

Although focused on evaluation, the book is a valuable source for anyone interested in systems concepts,
action research and reflective inquiry. It is useful for both teaching and practice.

Chapters :
Introduction, Iraj Imam, Amy LaGoy, Bob Williams and authors
Systems Thinking for Evaluation, Gerald Midgley
A Systemic Evaluation of an Agricultural Development: A Focus on the Worldview Challenge,
Richard Bawden
System Dynamics-based Computer Simulations and Evaluation, Daniel D Burke
A Cybernetic Evaluation of Organizational Information Systems, Dale Fitch, Ph.D.
Soft Systems in a Hardening World: Evaluating Urban Regeneration, Kate Attenborough
Using Dialectic Soft Systems Methodology as an Ongoing Self-evaluation Process for a
Singapore Railway Service Provider, Dr Boon Hou Tay & Mr Bobby, Kee Pong Lim
Evaluation Based on Critical Systems Heuristics, Martin Reynolds
Human Systems Dynamics: Complexity-based Approach to a Complex Evaluation, Glenda H
Eoyang, Ph.D.
Evaluating Farm and Food Systems in the US, Kenneth A Meter
Systemic Evaluation in the Field of Regional Development, Richard Hummelbrunner
Evaluation in Complex Governance Arenas: the Potential of Large System Action Research,
Danny Burns
Evolutionary and Behavioral Characteristics of Systems, Jay Forrest
Concluding Comments, Iraj Imam, Amy LaGoy, Bob Williams and authors

PUBLICATION AND PURCHASE DETAILS

NAME : Systems Concepts in Evaluation : An Expert Reader
EDITORS : Bob Williams and Iraj Imam
PAGES : 222pp

ISBN 978-0-918528-22-3 paperback
ISBN 978-0-918528-21-6 hardbound

PUBLISHER :

EdgePress/American Evaluation Association (2007)

PURCHASE

Available via Amazon : Hardback only. \$US36 plus postage