Image map created by yEd

In January 2017, I was recruited to manage the Household Economic Strengthening component of a USAID program in Nigeria. Its objective was that all orphans and vulnerable children would be able to access “comprehensive and coordinated services and are able to realize their full rights”

I had to manage state “household economic strengthening” coordinators across 5 Northern states and 2 Southern states, who in turn were coordinating 50 officers attached to the civil society organizations across the local governments where the project was implemented. Their efforts were targeted towards 675,000 children and 175,000 caregivers

My immediate concern was to check the level of knowledge and implementation of the different economic strengthening interventions at the state levels. So, I set up a scaled strength and weakness checklist of the different type of viable economic strengthening interventions for the coordinators. I wanted everyone to realise “one-size does not fit all” even if a group of people is all categorized as vulnerable. There are types and levels and as such should be provided support accordingly.

We came up with a scoring system for the categories of household, taking into account context difference. We then monitored the types of services provided to each. Built into this framework was an expectation that as households progressed, they would access varied packages of services best suited to their needs.

In some states I was able to meet the state coordinators and local officers during their state monthly and quarterly review meetings, to see what was happening. Elsewhere, I relied mainly on program reports and visits. Overall most of my contributions were made from a “helicopter view”.

I thought that MSC could help us gather evidence and ideas for potential follow-up projects, for program development and to learn about how program participants felt. My intention was to help the coordinators develop a simple template for the collection of stories at the state level, which would be shared with the local government officers to demonstrate tangible achievements at the household level and to be reported quarterly.

Based on the economic strengthening team's decision to first focus on a range of activities that were important for achieving outcomes that fit rightly with the project objectives especially as this needed to be in alignment with what is deemed necessary by beneficiaries irrespective of their category of vulnerability;coming up with the number of persons and households to be provided cash transfers as the initially appropriate service proved challenging.Thus,to avoid a situation of having insufficient program resources that matches the targeted population based on the prevalent level of vulnerabilty,the difficult decision of narrowing the vulnerabilty criteria appeared to be the best option.Even though it did not go down well with some programme and community stakeholders,the importance of empowering the most vulnerable and an emphasis on needs based programming brought everyone to some level of agreement."

Most program staff have worked on other USAID projects and generally understand and support M&E. I shared my idea with other leads and management at our staff meeting, for everyone’s input and buy-in, since we might make changes based on what we learn through using MSC. The Chief of Party and Deputy Chief of Party immediately raised questions of cost and resources, since we already had a robust M&E system in place, approved by USAID, with many indicators and an expensive web/phone reporting application. The M&E Director said the program’s complexity means that qualitative stories will not be representative data.

As you know, there’s been a lot of backlash from communities and local partners when we send foreigners. Plus, we wind up spending more money on the translation than anything else! So I tried to hire a local evaluator in order to conduct the MSC. That’s USAID policy now, right? Self reliance. Well, of course I couldn’t find anybody trained in it. Then I had to try and find a trainer. They couldn’t come out until Q3, at the earliest. Rainy season. That wasn’t going to work. I started thinking, am I going to have to do this myself?

‘The coordinators organized and facilitated a learning sharing event and shared a clear objective of a simple template with the different categories households to collect change stories. Different categories twenty households representatives were conceptualized clearly the area of storyline about the economic changes and the services received to change it. Last six months timeframe were agreed by all to include the changes and how they did the changes in everybody’s storyline. Some of the households representative were wrote their induvial story in a small card and some of them were told change story to a facilitator for keeping the records’.

I wanted to know the level of awareness of the caregivers regarding the various services provided to them. I also wanted to find out which of the services they had availed and their feedback regarding its usefulness to them. I wanted to probe about the services that they had discontinued and reasons for doing so. To find out if they had become advocates of the service, I built in questions to find out if the caregivers had talked to others about the services – to whom; how many; and what they had told them to persuade or dissuade them.

When I discussed with the coordinators about collecting the stories, I found certain resistance. There were several reasons cited. It would be time consuming, how could communities be involved in the process and government officials give importance to quantitative data. Then I shared experiences of MSC in explaining how change comes about and in which context and thus, can contribute to an effective monitoring and evaluation. I organised a short workshop for them and gave them a taste of MSC in relation to the work they were doing. Thus, through a lot of effort I was able to get buy in from most of them.

I capacitated five state coordinators about the process of implementing MSC. These coordinators trained 50 local officers about the process of collecting the MSC stories using the template during the quarterly review meeting with local government officials. Each local officers facilitated local government officials to identify the stories demonstrating tangible achievements at the household level. From the list of selected stories, they discussed, why each story is significant and selected one MSC story per quarter. Each local officers sent one story per quarter to the state coordinators who in turn selected one MSC and sent me.

"I immediately perform the following tasks quickly:
1. Write some sample story eliciting questions to ask participants to tell their stories, such as ""What was your best and worst experience using the services provided?""
2. Write some questions about their stories, such as ""Who are the participants and what did they do with the service provided?""
3. Write some questions about participants, such as ""What is your age and role in the story?""
4. Sketch the story form using the above questions.

This sample story form is uploaded to a shared drive and emails sent to all state coordinators so that they and their local officers can customise for their local context. Once everyone involved has done their part, I will consolidate all inputs into a common form that can be used for stories collection across all 7 states."

I contacted all of the coordinators with my idea for a story template. In the districts where I was able to attend monthly meetings, the coordinators gave me some time to discuss the story template. People contributed a number of ideas for it. They asked "Who is going to collect the stories?" "How much work will it be?" "How many stories do we need?" "What will we use them for?" I was able to answer these questions by asking one of them to share a story, playing the role of a mother in a household. I 'collected' the story from the mother, stopping periodically to point out how the dialogue influences the kind of story that is shared. This enabled the coordinators in different locations to decide on the domains that were important to collect, and how to construct the template. In the locations where I wasn't able to attend monthly meetings, I had to rely on visits. There were far less people at these visits, and often I was meeting with people in sequence rather than having everyone together in a room. I tried to present the same information about the story template, but this proved difficult and time consuming. Some people felt that I had pre-empted what they needed to cover in the visit; and people had diverse views about what should go into the template. I was left wondering how I would be able to manage to get a consensus on the template when there was such a wide range of variation in the types of contacts I had across the 7 states

I realized that my “helicopter view” from program documents was missing the detail I needed to understand the real circumstances the families were enduring, so I decided to ask the coordinator from one of the Southern states to choose some of the most experienced officers to visit families. I thought we could come together after the visits and begin to design the template. Jaiyesmi visited one of her families that had taken in two sisters –orphans- children of her late neighbor. The mother said “I can’t afford to feed two more people. We were already only eating potatoes, but what should I do, leave the children in the street?” Jaiyesmi wrote that the family told her they had asked for more help from the aid workers and were turned away because the children were from the wrong tribe. Now the family is telling tales in the village about the program.

I am working with Women’s CSO, Youth Networks and the Ministry of Gender to implement a capacity building package for girls in communities practicing female genital mutilations (FGM).
As we experienced issues with the indicators developed for the measurement of social norm change, I suggested the collection of stories on change within communities.

I discussed the dilemma with colleagues. Those most excited about learning opportunities agreed to collaborate on a small MSC “proof of concept” to convince everyone (program, USAID, local leaders) to support a larger effort. I could quickly train a few motivated people instead of having to cover all seven states.

Breaking resistance of the coordinators and creating positive environment was a big achievement. I prepared a report of the workshop and shared with other coordinators who had not attended the workshop to make them appreciate the need of qualitative data to add value to the quantitative data that they were providing.

I completed a sample story that explained some of the findings from our existing M&E data. This convinced them that explanatory value was just as important as representativeness

In thinking about a uniform template for locations,I experienced a lightbulb moment when over discussions at lunch,a colleague asked? "What are your top three questions?".So I mailed all the coordinators who similarly responded with a focus on "what people did with services provided","changes experienced in quality of service accessed" and "services considered as the best".

"per reviewing all individual history, it is reflected thst the project intervention contributed to increase monthly households income at significant level in targeted communities. The findings were validated with the community households and determined the key outcome changes to share with the project higher management for their further analysis".

As soon as the stories started coming in, we wasted no time in trying to look for patterns in the stories people told and in their answers to questions. And then build on these patterns by adding observations, interpretations and ideas that will catalyze thought and discussion during the next sensemaking phase of the project.

To foster interest in and commitment to participating I requested for a meeting of both program and M&E staff. Through examples of MSC, I explained it’s focus on learning and deliberation amongst staff as well as engagement with communities. Though some felt it would be additional work, most agreed that it would add depth in understanding our own interventions.

As a next step, I hold another workshop to explore and develop consensus on how they visualize the most tangible changes brought by the project at the household level. Based on their ideas, we developed a template that can be used to collect MSC stories from the household."

I sent my draft template to the seven state coordinators and asked them to discuss it with their 50 coordinators, at the next regular meeting. I wanted to know what the coordinators would think of the guide that seemed too much like a questionnaire and whether it would be better to go back to the MSC principle of using an open ended approach to let the caregivers describe what the most important result/change was. Then we could figure out how to sort the stories by the different types of households and interventions.

I also explained that in complex setting, quantitative monitoring approaches did not provide the global either unexpected changes induced by the intervention.
The MSC technic will enable programme managers to learn about the change including unexpected outcomes from the perspectives of beneficiaries and other stakeholders.

It took some time to train local people, because we had no resources and they can't take time away from work. But the learning was incredibly useful and the MSC approach had more credibility when local evaluators presented what they did to the other states.

Discussion in the workshop/meeting revealed that the significant changes happed successfully due to project's income generating intervention and communities active participation to do the activities. Facilitator further asked to all storyteller about the specific outcome changes on 'increasing households income' and wanted to know how many of them have the similar results changes and then asked to look at the community level to estimate how many percentage of households have the similar changes due to involving in project activities. How much they are happy about the changes and how they will improve more households income in future. The summary of findings shared with the senior management to review, validate and comparing with other storyline to select the most significant change one

Now that I knew we could get some stories of change and impact, the next big challenges were (1) helping the project and USAID understand and use the information, and (2) supporting the coordinators in their use of the information to increase understanding among local and state government officers about complexity of household vulnerabilities in each context, and ways to improve program implementation across the complicated landscape. These are related challenges because everyone is involved in assessing the value of the interventions and deciding how the program should evolve to make the most difference under these circumstances: people engaged in the program (participants, beneficiaries); coordinators managing activities and information at field level; local and state officials working to reduce poverty; program staff implementing activities; and USAID allocating funding and other resources across a range of needs. But the coordinators are really the heart of the transformation into using MSC data in fruitful ways. Plus, they are working most directly with local and state officials, who ultimately need to take ownership of diverse needs across their citizens. But over the next six months I found that many of the coordinators did not stay in their positions. Turnover in coordinators meant that most new staff did not know how to implement or use MSC to make sure the important stories would emerge. So the qualitative data was only coming from some states, and these states were not necessarily the ones that had the most urgent or severe needs. I felt we were not capturing the information the program (and local officials and donors) really needed to know.

Based on the feedback of the workshop report, I prepared a Guide for potential users. A video documentation and Frequently Asked Questions (FAQs) was also shared. To get better perspective of the caregivers, I examined the reports submitted by the coordinators who were asked to prepare their Plans of Action for taking the process forward.

After sharing an update during the program management meeting,a recurring concern was,"How do we integrate collection of stories with the existing method of data collection and monitoring for easy and regular analysis?" The project director then proceded to set up a task team which included focal persons from each of the various thematic program areas chaired by the M&E Advisor to propose how best to proceed within two weeks.An additional task was for the consideration of time,staff and cost implications as well as how best and when to inform senior management and the donor in the event of alterations to the already approved plans and budget.

The MSC itself, through the collection of stories, increased the involvement of the stakeholders and the ownership of the programme.
When we started analysing the stories at all level of the programme and shared some results with the programme management, there was some excitement about by how different beneficiaries value the interventions."

The Deputy Chief of Party was still unimpressed, but the Chief of Party and the M&E Director agreed to a small pilot of MSC in one or two locations. Their conditions were set thusly: information received could be used for success stories and PR purposes, and the MSC would be procured through an earmarked grant for higher education partnerships.

The higher management of the program carefully reviewed, analyzed and validated the changes at further downstream and community level that project contributed to improve the household's income stream and other wellbeing in the targeted communities. Both quantitative data and qualitative messages clearly defined the progress of the projects and program and compared with the desired progress were sat for the time period. The further information and validation used for modifying the real significant changes at community and project areas and decided to share the contribution with wider audiences through writing and publishing a significant story and presenting through different print and social media as evidence of changes and learning due to the project's income provision activities.

One common observation is that unpredictable situations are where you need trust. We interpret this to mean that one trust others more in unpredictable situations. Alternatively, it could mean that the greater the level of trust, the less unpredictable the situation is.

The programme managements are highly excited to looking summary progress of the project interventions and made a plan to improve the gaps identified during MSC process. They have included in a plan for adaptation of MSC to other intervention areas and widely dissemination and publishing plan of MSC findings through annual report, blogs, website, newsletter, print media, donors and other agencies for fund generation to scaling up the project.

We co-produced a list of what people valued from the interventions. Then we discussed whether we were currently measuring what people valued. Decisions were taken to revise the monitoring and evaluation to assess people-valued outcomes.

After doing situation analysis and consultation with key stakeholders, organized a meeting of the M&E advisors of various thematic areas to arrive at a consensus for taking the process forward. Had to do advocacy as some of them were not fully convinced of the usefulness of MSC. However, by the end of the meeting, they had actively participated in preparing a draft Plan of Action to share

1.9.1.1 Following the presentation of the task team that existing report structures be maintained, the support of the thematic program officer or coordinator with highest number of related stories submitted per time was further recommended, to assist with sorting .It was then advised that the state teams be saddled with the second stage of reviewing and filtering stories based on the specific themes related to the questions earlier agreed upon on a select day during the report submission period and attached as appendices to the standard M&E report. The selected stories will then undergo further review before being documented by the advocacy and business development unit at the country project level .Though the committee suggested that it was important to first document lessons at the project level rather than immediately inform the donor and senior management since the process is to pilot innovation,one question everyone kept asking was "How will the process improve programming at all levels?"."Making it compulsory for every state team and program unit to incorporate or tweak and justify at least one activity that is reflective of the MSC findings into work and activity plans during quarterly reviews, for starters", echoed the Project Director.

(Viewpoint: 1 year later, program participant in area with high coordinator turnover, not represented in MSC data collection) We were hearing the stories of some other areas and understanding how they were experiencing the activities, but we did not have the same economic opportunities or partners in our region, and our culture is very different. We could see the discouragement of our leaders, both community leaders and local government officials, and we were all frustrated by some of the program changes that did not seem right for our people. Some households that were already better off were able to benefit from the program but most of us facing the most severe challenges had no help.

The process itself proved valuable. Conducting interviews with beneficiaries to determine how change was unfolding on the ground gave us a much better sense of the scope, scale, and public opinion of the project from location to location. It was quite different, more so than we realized. We discovered a much wider discrepancy among implementation than we would have detected in indicator measurement alone. The enumerator training proved to be an asset, too.