Narrative Assessment

Foreword

Narrative Assessment is a new way to monitor, evaluate, and communicate about advocacy. It has come about in view of a need that we have seen among civil society organisations doing advocacy work, as well as their donors. With Narrative Assessment, we want to help organizations to bring out what advocates really do, the challenges they face and address, and the meaning of their achievements. This can be helpful to bring teams and networks together and develop more shared understanding. It can help in communication between organizations and donors. In addition, it can help organizations to communicate with their audiences and support bases.

This manual is an introduction and practical guideline for program or project managers, advocates, M&E professionals and Communications staff. We think it is also relevant for donors seeking ways to monitor, evaluate and communicate about the advocacy programs they support.

In this manual, we tell what Narrative Assessment is and what it can be used for. It also charts the different steps involved with Narrative Assessment, and offers practical advice on how to carry out a Narrative Assessment.

For those with a further interest, we also offer an in-depth discussion of how Narrative Assessment relates to other methods.

Narrative Assessment is a new idea and we hope it will grow and develop from the experiences and lessons learnt in practice. If you are interested in trying out Narrative Assessment or learning more, do not hesitate to contact us at margit.vanwessel@wur.nl and wenny.ho@hivos.org.

Margit van Wessel and Wenny Ho

Asking questions was a good process to help reflect. The conversations were really useful, they gave you the time to reflect which is usually not possible.

1. Introduction

The unseen nature of advocacy work

Many organizations use advocacy to work towards a better society. Monitoring and evaluation aim to create knowledge about the advocacy and its effectiveness. This is necessary to be able to plan, learn and adjust course. In many cases, it is also necessary to communicate about advocacy and be accountable to, for example, institutions and citizens supporting an organization’s advocacy.

However, monitoring, evaluating and learning are notoriously difficult when it comes to advocacy. Causal relations between actions and results are hard to establish. Influencing often takes place behind closed doors and it can be risky to talk openly about it. Those targeted by advocacy often may not be ready or available to discuss being influenced or not by specific actors, actions or events. Intervention effects can be expected to be one among numerous other causal strands, making it hard to determine or claim their contribution. In addition, the targets of advocacy —policy makers, publics, private sector actors— are moving targets, continually subject to various influences. Moreover, as change processes often play out simultaneously at different levels, and involve multiple actors, actions and events, it may be difficult to identify evidence and interpret the contribution of outcomes to desired changes.

In addition, to advocate effectively, advocates need to navigate unpredictable and unknowable dynamics. These dynamics may provide opportunities for advocates or throw up sudden barriers that limit opportunities to attain desired changes, notwithstanding best efforts and capabilities. It also means that advocates often have to adapt to changes in the contexts where they work.

In practice and research, much of the difficulty of assessing advocacy results from the complexity of the change processes in which advocacy is involved .

In line with this, there is no generally agreed view on ‘what works when’ in advocacy. There are some shared understandings on how to act effectively. For example, there is the notion of working based on an analysis of political context, of working within windows of opportunities; of engaging with policymakers’ needs and understandings rather than just having confrontational approaches; of providing credible evidence and usable ideas; and of forming coalitions. However, none of these are sufficient for success.

Using the narrative assessment stories in the reflection workshop created a feeling of sharedness, that you are not alone.

Consequently, strategizing by advocates often is not based on sure-fire knowledge of cause and effect, but on practice-based and contextualised judgment as to what possibilities for change exist and what strategy could effectively influence certain targets in a certain context, at a certain moment.

Therefore, much knowledge about the unfolding of advocacy work in relation to a change process is tacit in nature: it exists and develops in advocates’ minds and in interactions between advocates and with other stakeholders.

Narrative Assessment

Narrative Assessment is not a new concept. The term ‘Narrative Assessment’ is also used in other domains, to refer to varied forms of qualitative assessment. For example, in special education, educators make ‘narrative assessments’ of students’ progress that cannot be measured conventionally, through standardized testing. While our context is very different, the rationale for our form of Narrative Assessment is related: current monitoring and evaluation (M&E) methods primarily focus on results, but do not address the often-complex relations between results and efforts.

Narrative Assessment proposes to strengthen existing M&E methods by a systematic approach to making sense of the realities underlying results. It focuses on unpacking the dynamics and contribution of advocacy work, to inspire learning and to support program adaptation and communication.

Narrative Assessment overcomes two problems of current monitoring and evaluation of advocacy: a focus on results that in themselves provide little insight about their meaning or the quality of efforts, and unrealistic assumptions about availability and usefulness of objective evidence. Narrative Assessment helps advocates, their organizations and donors to bring to light tacit knowledge and experience for sharing and wider learning and practice. The following chapters will discuss the nature of Narrative Assessment, what it can offer, and how it can be used in practice. An appendix contains further discussion, relating Narrative Assessment to wider theory and practice in advocacy evaluation.

2. Starting points of Narrative Assessment

Narrative Assessment in brief

Narrative Assessment centers on the stories that advocates themselves have to tell about their work, while building their validity by critically co-creating and examining them. Narrative Assessment stories can help advocates and others involved to understand what happened, and why, and come together to see new directions for the future. Narrative Assessment also makes it possible to communicate to partners, donors and other publics in a convincing and meaningful way. It has the following building blocks: 

• Narrative Assessment illuminates the dynamics of advocacy for a program, showing how over a certain time period, advocates make sense, decide, strategize, use their knowledge and skills, act and learn.
• Narrative Assessment makes visible the dynamics and challenges that make achievements and disappointments understandable by jointly creating stories that explain what happened, how the advocate dealt with events, and how that links to advocacy results.
• By co-constructing advocates’ stories in the light of a program’s Theory of Change, Narrative Assessment assesses whether and how outcomes are relevant steps in a pathway of change, and how challenges are to be understood. It thereby helps to revise a program’s Theory of Change and adjust future action. Theory of Change has been widely hailed as an approach to planning, monitoring and evaluation in the field of international development by which more justice can be done to complexity. Moving away from constricting and imposed log frames, Theory of Change has come to be seen as an approach by which funders, practitioners and evaluators can do better justice to complex change processes that development interventions seeks to contribute to.  Narrative Assessment can contribute to this. Advocacy results are often steps in the direction of impact, rather than results that have a direct impact on constituencies. The nature and significance of an advocacy outcome can therefore often only be interpreted in light of the hypothesized pathways of change and desired outcomes that are described in a Theory of Change.
• Narrative Assessment involves examining advocates’ stories, drawing, among others, on Narrative Inquiry theory.  This makes Narrative Assessment a scientifically robust method that can bring out strengths, limits and gaps in analyses behind strategy and capacities to carry out programs.

Narrative Assessment builds on existing M&E methods

Current M&E methods such as Most Significant Change, Outcome Mapping and Outcome Harvesting produce findings on results. Narrative Assessment can significantly strengthen evaluations by building on their findings. It is most suitable for zooming in on selected aspects or developments in an advocacy program (a set of interrelated advocacy activities over a period, carried out by an organization or collective of organizations) that merit further study and communication. In this way, Narrative Assessment complements and deepens rather than substitutes other program monitoring and evaluation methods in use. It can also be carried out independently from existing methods. The appendix provides further reading on Narrative Assessment and existing M&E methods.

Premises

Narrative assessment is distinct from other M&E approaches. Three premises make Narrative Assessment specifically suitable for advocacy.

First, in a challenging and changing environment, advocates’ efforts are often countered or aided by forces much more influential than their actions. Small advocacy efforts may bring big wins, whereas huge efforts may be required to merely halt or delay a negative development. In advocacy work, these often invisible dynamics are the key to establishing causal relations between advocacy efforts and outcomes with some degree of plausibility. However, they are generally black-boxed in current M&E approaches. This blocks understanding how results were achieved, leading to unsupported assumptions of direct linkages between advocacy and its results. Narrative Assessment makes the otherwise largely invisible dynamics of the advocate’s environment an explicit focus of attention. Second, stories enable connecting and interpreting the complex and contradictory signals of a messy world.  Narrative Assessment stories bring out dilemmas and challenges faced by advocates and the way they have acted upon them using their experiences, skills and knowledge. In this way, Narrative Assessment stories uniquely illuminate the dynamics of advocacy and the real contribution of efforts. This in turn, allows for insightful descriptions of advocacy work in relation to advocacy results. Third, Narrative Assessment is not just about storytelling. It uses scientific criteria of narrative inquiry to ask hard questions, tease out the relevance of failures and achievements, critically examine claims, and test and validate stories, thereby strengthening their plausibility and thereby, credibility.

Stories and their assessment

Stories

A story is an account of events over time, brought together into a coherent whole, conveying certain meanings from the standpoint of the narrator. Stories are interpretations of human action in practical, concrete situations. They simultaneously function as explanation, justification and instruction. A plausible story creates order and sense in a shifting, unpredictable and incoherent environment. By making past actions understandable in this way, stories create the conditions for further action.
What qualifies as a story? Contrary to other story-based evaluation methods, Narrative Assessment specifically employs concepts and theory from narrative analysis and narrative inquiry, by which the full communicative potential and qualities of stories are put to use.

A Narrative Assessment story is built up from the following elements:

• It contains an element of transformation;
• It presents this transformation as a movement over time;
• It contains actions by which this transformation happens, carried out by characters;
• These actions take place in a specific setting

These four elements are brought together in a plot (possibly involving crises and turning points). This plot has a point: a key message to take away from the story. 

These are human interest stories, like ‘brilliant failures’, there are more credible and real.

Narrative Assessment methodology revolves around co-creation of stories by advocates and Narrative Assessment facilitators, making sure all these elements are considered in stories that advocates tell about the past unfolding of programs. The facilitator encourages the advocate to tell their story, with the message they think is important to share. The facilitator also makes sure the advocate tells how context mattered in what happened; how change happened; what actions came in; and by what actors. By asking critical questions, Narrative Assessment facilitators also help advocates to reflect on what happened and what their role has been. They also help to bring out the observations and analysis of what happened that can make the story believable for others. These stories, co-constructed and critically assessed, bring out the nature and meaning of processes and outcomes.

Stories are supremely useful for reflecting on and assessing advocacy. By building stories, advocates can take along others on their journey, as key characters in a story that unfolds over time. Stories can cover all situations that advocates face, make sense of and navigate. Stories allow them to share their knowledge of the situation, of other story characters, the context involved, and of the transformation of which they were part and in which they had a role to play. These stories can vary greatly. They can be about relations built over time, and how they made a difference. They may be about just one window of opportunity and how it was swiftly acted upon with great results. They can tell about the ways constricted political space for action hampered a program unexpectedly, or about the ways powerful adversaries acted against a program’s objectives, and how this was responded to. Importantly, in Narrative Assessment stories, disappointments and failures are not negatives one would rather hide; they form meaningful parts of the stories.

Furthermore, stories are useful for assessing advocacy because advocacy often seeks to advance change in situations that, from the program’s perspective, are an affront to moral principles of some sort (e.g. the right to clean water, to land, justice, basic human rights, or protection from violence). In addition, advocates appeal to a shared moral standpoint when they call out for support, or make propositions with regard to the worthiness of their efforts or the significance of their success. In such situations of assumed or observed moral affront that demands action, stories can convey the meaning of the work being done. 

In addition, stories make available the lived experiences of advocacy staff, providing vibrant, multifaceted, and situational rendering of meaning.  By relating a story to the program’s Theory of Change, its meaning can be constituted in terms of what is relevant for wider audiences (colleagues, partners, communities and donors).

Assessment of stories

A story is an account of what happened, idealizing and cleaning up, attributing causation, highlighting and lowlighting. A story imposes meaning and coherence upon a disorderly and ambiguous reality. Stories thus create order. They provide meaning and direction, also countering the threat of becoming overwhelmed by disorder and indeterminacy.  However, this entails certain risks pertaining to credibility. While stories impose meaning and coherence on messy situations, they may do so in ways that glorify the program, advocate or achievements without much concern for building plausibility. Narrative Assessment includes a critical examination of that meaning to maximally develop the plausibility of stories. Plausibility is to be determined by the ‘verisimilitude’ of stories – their quality of being believable. Narrative Assessment stories therefore need to be co-constructed between the advocate as lead author and the Narrative Assessment facilitator in the role of critical friend. The facilitators are to seek detail, assessing consistency and plausibility of statements, embedding in context, clarification, and where possible, signs of evidence. In this way, Narrative Assessment facilitators test the believability of stories against alternative interpretations, undermining gaps, and apparently empty claims. In this way, Narrative Assessment offers a new direction for conceptualizing rigor, drawing on the narrative inquiry research tradition.

3. What does Narrative Assessment offer? To whom?

Bringing out the reality of advocacy

Narrative Assessment uses stories to clarify how advocacy works and how it relates to advocacy outcomes. The stories unveil what happened, how and why. They tell, for example, which decisions were made and how, and which strategies were followed, even while consequences were not clear, adversaries made their own strategic moves, and contexts kept changing.

The heart of Narrative Assessment is thus formed by the stories of what advocates have to tell about their work. Stories revolve around their experiences, knowledge and skills that go into making sense, deciding and acting on opportunities, challenges and dilemmas. Through the co-construction of stories by advocates and a Narrative Assessment facilitator, the causal links between advocacy and outcomes are explored and substantiated to develop plausible accounts of contributions to change. This way of working does justice to the dynamics of advocacy within specific contexts and tries to assess, and bring out, the plausibility of claims made in, for example, Outcome Harvesting.

However, Narrative Assessment does not have to focus on, or start with, outcome findings. Stories can also be built around other dynamics. For example, stories can tell how a program developed. They can tell about the challenges a program faced with changes in a political context, how these were responded to; or describe how hard lessons were learnt.

Advocacy knowledge and skills are deeply rooted in advocates’ experience and knowledge how to maneuver in complex and dynamic contexts and in their engagement with actors and political arenas. They are gained and shared through learning in action;  by experience, facing failures, and learning from doing so; through in-service training and real-time hands- on coaching. Current M&E approaches do not acknowledge or facilitate the identification and sharing of this knowledge.

Stories generated through Narrative Assessment, putting at the center advocates’ work and the judgements on which it is based, bring out this otherwise overlooked, tacit knowledge of advocates about the unfolding advocacy dynamics and how to navigate developments. Furthermore, in order to understand the nature and meaning of advocacy processes and results one should have a clear understanding of their specific contexts. The stories provide this, being always embedded in a concrete setting, and thus also doing justice to the crucial role of contexts in advocacy initiatives and outcomes.

Reflection and learning within programs

Narrative Assessment stories support advocates to reflect on and address questions of strategy, gaps in strategic thinking, and capacities of those involved. For example, the stories make explicit advocates’ strategic considerations and the assumptions underlying these. This enables the revisiting of starting points and claims, for example, on effectiveness of strategies, underlying analyses, or capacities available to the team. A subsequent collective inquiry into and sense-making of these considerations, assumptions, starting points and claims, and how they relate to advocacy results, puts them to the test. This drives learning and reflection. Narrative assessment thus links advocacy results with advocates’ frustrations, joys, dilemmas, disappointments and challenges, seeking to explain the connections and attach meaning to them.

Different factors and developments may contribute to successes or to disappointing results. Identification of relevant organizational factors can contribute to strengthening of capacities; identification of relevant external conditions can contribute to strengthening or reconsidering strategies or allies as well as to enhanced understanding and resolving in the face of disappointing progress. In the end, Narrative Assessment stories of how strategic maneuvering relate to advocacy results to build knowledge on the effectiveness of advocacy strategies, and help further the knowledge and evidence base for developing more robust theories of change.

Engaging stakeholders

Through Narrative Assessment stories, stakeholders can see the meaning of advocacy results, and can recognize otherwise ignored differences among them in interpretation and understanding of the advocacy process. In this way, Narrative Assessment enables sharing, opening up dialogue, building of mutual understanding and aligning interventions around effective advocacy strategies.

Mobilization of support

Reporting advocacy results can easily render advocacy meaningless for publics beyond a very small set of insiders. Lack of impact on constituencies can be mistaken for lack of significance if this significance is not articulated. An adjustment to a policy document may be a result of great, long-term effort by advocates. However, making clear of what that change actually may mean to constituencies or society requires special attention. This is not just because policy processes are technical and may need explaining. Advocacy achievements are often small, intermediate steps that have real meaning only in the light of a larger future outcome. Advocacy often require further policy influencing in order to attain clear change for the ultimate beneficiaries such as increased access to clean water, or a living wage. Narrative Assessment can contribute to making internal as well as external communication more realistic and appreciative by bringing out the meaning of these intermediate advocacy results.

Drawing on the communicative potential of stories, Narrative Assessment stories contribute to realistic, yet appealing communication about advocacy towards donors, supporters, constituents and wider publics. This can help serve accountability needs and generate support for future action.

4. Narrative assessment step by step

This chapter describes three steps taken in every Narrative Assessment: (1) preparation; (2) narrative interviews; (3) distilling stories from them.

Step 1. Preparation

Advocates, M&E specialists or others conducting the assessment, and possibly other stakeholders (organizational staff, allies and partners) define the parameters of the assessment.

1. Defining the purposes of the assessment. For example, a Narrative Assessment that is to help in revisiting a Theory of Change would differ from one that is to clarify accountability needs in relations with a donor.
2. Delimiting the program or the part of it that is to be monitored or evaluated – establishing the unit of analysis. Different possible purposes may determine the selection criteria:
   1. The advocacy trajectory that has taken place belongs to the heart of a program, for example, in terms of centrality to objectives, exemplary nature of the work done, or challenges faced;
   2. Because of what happened, the organization expects to unearth an interesting story that it wishes to share internally or communicate externally;
   3. Because of what happened (e.g. unexpected disappointments or successes) an organization finds there are important lessons to learn.
3. Defining what is to be incorporated into the assessment: which actors, processes, actions, within which time period and which setting.
4. Identifying the advocates who have carried responsibility for programs and whose inside information or role make their participation key to the assessment.
5. Identifying the Theory of Change underlying the (part of) the program that is to be assessed: objectives; pathways of change; actors; intervention strategies; contexts, assumptions.
6. Selecting the Narrative Assessment facilitators, and organizing the Narrative Assessment team. Narrative Assessment facilitators can be internal M&E or program management staff, or external persons, e.g. consultants.
7. If needed, training the facilitators in holding a Narrative interview, and creating and using the Narrative Assessment stories.
8. Building basic mutual understandings on the assessment between advocates and Narrative Assessment facilitators involved, so that a foundation of trust can be created.

You can really explain what you are doing, and put a human face to it.

Step 2. Narrative interviews

1. Identifying interviewees, depending on the purpose of the Narrative Assessment. Advocates who have been in the middle of the action carrying out or managing an advocacy program are suitable candidates.
2. Introducing Narrative Assessment to the interviewees. Many interviewees will be used to conventional reporting methods. Narrative Assessment seeks to draw out the generally implicit knowledge and experience of advocates, and seeks to give ample room to the challenges they face. Telling stories from that perspective often requires a shift in sharing for the interviewees. A preparatory conversation and supportive interviewing can help bring about and sustain this shift.
3. Within the chosen purpose of the Narrative Assessment, defining the topic of the story to be told in consultation with the interviewees. This can be done from different starting points, depending on:
   1. Intended use (see further below);
   2. Possibilities of building from findings produced through evaluation methods in use, like Outcome Harvesting or Most Significant Change (see the appendix: Further Reading);
   3. Analysis of the program and its context. If a program has faced challenges, or has seen important successes, it may be helpful to collect stories on the way challenges unfolded and were responded to, or how strategic maneuvering by advocates contributed to successes, for example.
4. Conducting and recording or noting down the narrative interviews. Interviewees are invited to tell the story behind a success, challenge or other type of development that appears relevant for the program. A clearly defined topic is to be decided upon. In addition, to make sure that stories can result from these interviews, a temporal perspective is crucial. Interviewees are therefore to be explicitly invited to tell how something relevant that happened evolved over time. Stories can, for example, be about:
   1. The way an outcome or interrelated set of outcomes came about;
   2. How key challenges came about and were addressed;
   3. How a program developed over time in light of advancing insight into its political context.

As the conducting of the narrative interviews is crucial for the quality of Narrative Assessment, details are provided on two key responsibilities of the facilitator, namely (1) building the plausibility of narrative assessment stories and (2) interviewing.

Plausibility

To build the plausibility of stories, Narrative Assessment facilitators encourage the interviewees to substantiate their stories with examples, observations and analytical reflections that support their claims about the unfolding of programs within a context and process. We call this ‘narrative substantiation’. Using Narrative Inquiry principles, there are three ways to build plausibility:

1. Detail: stories become more plausible by providing detail to what happened, in particular when it comes to claims regarding outcomes, or the way a program contributed to outcome, and how challenges were engaged with;
2. Context: stories become more plausible when they are rooted in the context in which developments unfolded, and stories align with what we know about these contexts and other developments taking place in that context. This requires the facilitator to explore rival explanations of what is being told;
3. Consistency: stories become more plausible when they are internally consistent, and are also consistent with what we know about external developments. In this, consistency is closely linked with context. 

These three ways of building plausibility through detail, context, and consistency are integrated in the assessment part of Narrative Assessment through the facilitator’s role in co- construction of stories. As the advocate being interviewed brings in the content for the story, the Narrative Assessment facilitator contributes to the plausibility through critical inquiry. This assessment role of the facilitator in the Narrative Assessment methodology should cover, among other things, the strategic intent of the advocate, his or her analysis of options given the context, use of capacities and knowledge by the advocate in line with these, and of inside information of what happened.

The Interview - Interactions and challenges

Obtaining good stories requires that the facilitator gives space to the interviewee. Before the interview starts, the facilitator and the interviewee agree on what the story is to be about by exploring and establishing what interesting story or stories there are to tell. It is for the interviewee to decide what can be a story to tell. Even with advocacy being a continuing change process, to make sure a temporal account is developed, the interviewee also needs to establish points in time when a story may best begin and end. Opening questions to facilitate this exploration are, for example:

• What was the most important development in your program over the past year?
• What is the most interesting story that could be told about your work since this program started? What is the most important thing you learnt about the program’s Theory of Change?
• When do you think this story should start? How did this begin?

While Narrative Assessment facilitators are there to support the development of the story, critical questions can be asked to explore consistency gaps, questionable silences or claims and so tease out details to strengthen plausibility and the degree of verisimilitude of the stories. The facilitator in this way works to ensure that context, dynamics, challenges and key actors are in fact brought into a story so it reflects the complex reality. Examples of questions are:

• What made this work there, at that moment?
• Were there challenges? How did you learn about them? How did you deal with them?
• Who were important players in this? What did they do that mattered?
• How was the political reality different from what you expected when you started?
• What makes you think that what you and your team did made a difference? What makes you say that?

The Narrative Assessment facilitator may face various challenges, especially as the critical questions aim to surface that what generally may be ignored or seen as too detailed or unimportant. An interviewee may be unable or unwilling to tell a story that fulfils the requirement of plausibility, by not sharing the needed context or detail. At times, one may suspect that an interviewee seeks to advance an agenda through the telling of a story. An interviewee may not wish to speak out for strategic reasons or because of lack of trust. This all may limit possibilities to build an insightful and plausible story through Narrative Assessment. A possible measure to counter these problems can be the creation of a Narrative Assessment facilitation team of relative outsiders and insiders. In any case, the sense of security that may be needed for an open interview, depends also on the organization of the interviewee, and on how open the organization is to learning and sharing. Rather than pushing an interviewee to open up, the facilitator needs to stress that the story is to be his or hers.

Transcription of the narrative interviews, in full if possible or in a more limited sense. Full transcripts help ensure that a more complete, detailed and thereby more plausible and insightful story emerges from the Narrative Assessment interview. Taking notes and a recording for back-up and seeking out important details is an adequate alternative.

Step 3. Distilling stories from the narrative interviews

Since the narrative interviews are to focus on the story of a meaningful development, stories will evolve around a plot concerning this development, and will offer a message with regard to reasons, lessons or meanings regarding what happened. Since the interviews are action- oriented, asking ‘what happened’, actors, actions, context and transformation will also emerge during the interview. Even so, those writing up a story will need to transform the interview transcripts or notes into a story that is concise enough and meaningful to busy staff and other audiences to read. While the stories need to be engaging, they need to be also gauging what happened in a complex reality. This requires condensation of the story, while retaining the different elements from the perspective of the interviewee, and putting him or her at the heart of the story as protagonist whose knowledge and experience is presented through the story, while also bringing out critical detail and consistency.

Stories can develop from a single interview, to tell the story of an advocate. It is also possible to develop stories about programs, incorporating and connecting stories told by different advocates.

Further assessment of the plausibility of stories

As discussed in step 2, during the interview, the facilitator collects information for building a story as plausible as possible. After the interview, the facilitator assesses the quality and completeness of the information given. Depending on this assessment, the facilitator decides if a plausible story can be written. To strengthen plausibility further, the assessment can include the identification of available evidence to further corroborate claims on outcomes, contributions, and challenges to programs. This is done while accepting that often the evidence will be limited, that views on what constitutes evidence will depend on the context, and that capacity for finding and tracing evidence will be limited. Also, ambitions to integrate evidence may differ per use: stories developed for learning and reflection may require less corroboration of claims than stories developed for accountability purposes. When Narrative Assessment builds on Outcome Harvesting, evidence is already provided through the supporting sources that are required and through the substantiation process of outcome statements.

In short, the assessment of stories involves the following steps, depending on need:

1. Assessment of the overall story of advocacy processes (including strategy, embedding in context, challenges and opportunities met), and the advocacy program’s contribution in relation to outcomes. This assessment explores, identifies and tests narrative substantiation for claims made in stories, building the validity of these stories as discussed above.
2. Depending on the level of evidence-building deemed necessary: identification and assessment of available objective evidence supporting or challenging stories, including on the program’s contribution to the outcomes.
3. Depending on the level of evidence-building deemed necessary: identification of gaps in substantiation.
4. Adjustment of stories based on these steps. Narrative substantiation and evidence can, for example, be integrated into stories. Stories may be adjusted based on the assessment of claims on processes, advocates’ actions, and contribution to results,or, where necessary, statements of uncertainty may be incorporated into stories.

BOX 1 IN BRIEF

In brief, the role of the Narrative Assessment facilitator is to:

1. Support the storytelling advocate to tell his/her story - asking for clarifications, examples, interpretation and meaning of, for example, developments;
2. Increase reflexivity of the story, that is, - through constructive yet critical questions - the facilitator helps the interviewee to move beyond ‘and then this happened, and then that happened’;
3. Tease out the tacit knowledge elements, for example, of the context, of strategies used - help the storyteller to emerge these; increase plausibility by teasing out details and contextualizing
4. Keep an eye on the story building elements:
   1. element of transformation
   2. movement over time
   3. actions that contribute to the transformation, carried out by characters (actors)
   4. taking place in a specific setting, context
   5. the actions, transformation and actors woven together in a plot - with a take away message
5. Identify gaps and weaknesses of the story. Strengthen evidence where possible.
6. Write up the story

5. Uses of Narrative assessment stories

Narrative assessment has three main uses:

• Enriching periodic reflection, learning and planning;
• Strengthening accountability and dissemination (external communication) including reporting;
• Mid-term and final evaluation.

This section describes these uses and the steps taken in narrative assessment each use.

Enriching periodic reflection, learning and planning

Learning and reflection through the co-construction and assessment of stories generally take place at an individual level. Narrative Assessment stories can be used to bring individual learning to a program level by incorporating activities designed for collective reflection, sense-making and learning. Through these activities Narrative Assessment also supports decision making in multi-stakeholder settings.

Various workshop designs can enhance a common understanding of the program objectives, effectiveness of strategies, assumptions and priorities. Workshop objectives may be:

• Sharing of preliminary reflections regarding interventions, the handling of challenges, and acting on windows of opportunity;
• Reflection on evidence of successes or failures and their implications;
• Dialogue on the way forward, e.g. by refining the Theory of Change based on the assessment of effectiveness of strategies.

BOX 2 EXAMPLE

The example of a Green and Inclusive Energy Reflection and Planning meeting

Introduction

The Green and Inclusive Energy program is part of a wider initiative implemented by a consortium of Hivos, IIED and Article 19. The program advocates a transformation towards inclusive decentralized renewable energy systems. It works at three different levels: country-level (Indonesia, Zimbabwe, Malawi, Kenya, Tanzania and Nicaragua - now replaced by Guatemala), regional level (Central America), and global level. For its program management the program uses Outcome Harvesting and Theory of Change thinking.  Small teams including an advocacy officer work with partners in the countries, guided by a joint Theory of Change. The reflection and planning meeting followed a first cycle of Outcome Harvesting (elaboration of outcome statements, substantiation and analysis).

Step 1. Narrative interviews

To prepare for the annual reflection and planning meeting of the Green and Inclusive Energy program, narrative interviews with all advocacy officers were held on the outcomes to which they contributed. Each advocacy officer selected one or more outcome statements for the interview. An interview guide was elaborated to provide some focus for the narrative assessment facilitators.

There were two major interview topics: advocacy dynamics to unpack the relations between the outcome achieved and the influencing factors; and exploring what went into the actions in terms of, among others, decision-making and underlying assumptions. Where appropriate, attention was directed to partnership and collaboration issues.

Step 2. Narrative assessment stories

From the verified transcripts, emerging common themes were identified by the Narrative Assessment facilitators and agreed with the program manager, who in this case is the ‘owner’ of the annual reflection and planning meeting, and with the meeting committee. Around each theme, stories were then carved out from the transcripts, with some interviews contributing to several themes. This resulted in the editing of 11 stories of 1/2 to 1.5 page organized in four theme clusters: working with government, inclusiveness in advocacy, working with partners, and innovative advocacy approaches. Each story contained the six story elements mentioned earlier: transformation, a movement over time, actions by which this transformation happens, in a specific setting, a plot (often involving crises and turning points) and a take away message. A theme cluster often covered stories from more than one country or level.

Step 3. The reflection workshop design

With the stories as a basis, a workshop was designed together with the meeting committee. The workshop participants consisted of staff from partners and the program with different functions, and included those with whom a narrative interview was held. It was decided to organize sharing, reflection and sense- making in four groups around the theme clusters. Each theme cluster was moderated by a facilitator, among whom were the Narrative Assessment facilitators, to focus on:

• Connecting the stories and bring out commonalities;
• Sense-making of what emerges.

Before the start of the workshop, participants received all outcome statements and all 11 stories, so they were aware of the achievements of the program and had an equal information base.

Step 4 The reflection workshop

Each group worked with their cluster stories around reflection questions such as:

• What resonates with your own work and thinking?
• What do these stories tell us about our strategies?

The facilitator helped the group to reach deeper levels of reflection by linking, comparing or contrasting issues emerging from stories, and by highlighting strands or threads. In this way, the group could jointly make sense of what was revealed and capture the emerging insights.

There were two rounds of group work so each participant could discuss two theme clusters. The insights from the group work was then shared in plenary for collective reflection on what convergence and agreements emerged across the stories and themes. The workshop participants then looked at the Theory of Change to locate the collective agreements on aspects such as interventions, challenges, assumptions and outcome areas.

Evaluation of the use of Narrative Assessment stories in the reflection workshop

For a reflection meeting, the NA stories proved to be of added value as advocates felt that the stories helped to create the right perspective: they could understand not the end result alone, but also the context around it, and appreciate the challenges confronted. They felt that the unpacking of advocacy dynamics enabled them to finally explain what they are doing, and show a human face to their work. They stated that these NA stories ‘are our own stories, not stories that we are used to telling donors’. The stories made their work more understandable to each other. Showing how advocates dealt with failure and challenges, jumped on opportunities, strategized and went through lows and highs did not only increase real and realistic learning, but also mutual empathy and the insight in their own work as part of a collective endeavor.

Those who were involved in an interview felt that this was really useful, as it forced them to take the time to reflect, which otherwise a priori they would have considered impossible. The critical questions asked helped to generate a good process of reflection, forced them to think beyond the bare facts in a story.

And while one story already allowed to draw many conclusions, bringing together many stories from across the program was reported to make the reflection even more interesting and useful for learning - including from what does not work - and stimulating new ideas. Common themes emerged from the many stories, for example about leadership, and about being flexible and adaptive as an advocate. Having the stories from across the program also created a feeling of partnership; that one is not alone pushing the cart.

The reflection meeting brought many insights with implications for the program’s Theory of Change in the countries. The next step for the advocates and their teams will be to adjust their strategies and Theory of Change based on the insights.

Strengthening accountability and dissemination

The stories developed and assessed through Narrative Assessment can form a firm and appealing basis for communication to diverse internal and external audiences. By communication, an organization accounts for past actions, while also appealing to support further actions. Story-based communication is highly suitable for communicating to relate to the future as much as to the past. Moving over time, a story is a proposition of the meaning of past actions. It can also be a proposition to come along on a next stage of a journey. To offer such a proposition, stories need to present a view of a plausibly successful way forward.

By revealing the advocacy dynamics and relating them to outcomes achieved, Narrative Assessment stories can offer credible interpretation of how a change is a step in the right direction, and how that may help to set the stage for further desired change. That credibility and the plausibility of the stories and derived public documents are robust as the advocates’ knowledge, ability and insights are embedded in stories co-created through critical inquiry and rigorous analysis.

Where Narrative Assessment stories do not report actual or expected future success, they can show how ineffectiveness or failure came about and could be overcome. The stories can make understandable how an unwelcome and unexpected turn of events, struggles among the advocates themselves to get a hold on complex issues, or other factors have influenced the advocacy work and results. The insights from such stories help advocates to explain their case internally.

The journey that organizations and programs are undertaking with their advocacy is shared with partners, communities, donors and the wider public using Narrative Assessment stories. These stories can also offer legitimation and justification for continued support and further investment. This is important in view of the public and political debate regarding the effectiveness of development interventions. The Narrative assessment stories not only satisfy the need of media and supporters for narratives and case studies to give context to the numbers and statistics. Crucially, communication based on Narrative Assessment stories can strengthen arguments for continued investment while at the same time contributing to a realistic understanding and acceptance of advocacy outcomes for what they are: steps on an often long and winding journey.

Different modes of presentation may be used: blogs, websites or reports, for example. Advocates can also present their stories in short videos.

Stories should be adapted to audiences and objectives. External communication with donors and advocacy experts will differ from communication with the general public, for example in its use of evidence, technical aspects or contextual information. Because of sensitivity and confidentiality requirements, stories for internal use will differ from stories for the wider public.

The stories must only be used with a clear awareness of what they are: highly personal accounts that should only be shared with wider audiences after obtaining permission from the interviewee, possibly after adjustment for sensitive matters. Narrative Assessment can best be considered an internal exercise first and foremost, using the information for external audiences subsequently and only in so far as advocates agree, in a form and with content agreed with them. In this way, trust and confidentiality can be established as required for achieving the openness needed for a successful Narrative Assessment.

BOX 3 EXAMPLE

The example of Green and Inclusive Energy reporting

In their program documents, organizations often make claims about what they are about (‘innovative advocacy’) or what they will contribute to (‘inclusive energy systems’). Often these claims that may not elaborated or substantiated. For Hivos’ Green and Inclusive Energy program, we made claims explicit and selected the most crucial ones. From the stories for internal use created from the narrative interviews, a few were selected that corresponded to a claim and then reworked for external communication purposes. This reworking entailed adjusting the colloquial tone of conversation that had been maintained in the Narrative assessment stories for internal use, to make the story suitable for a general public.

Before the stories were shared externally, the story-owner reviewed them and details deemed too sensitive were removed. Especially, those details that could bring along risks or hamper further progress of advocacy work, required attention.

The stories that were generated in this way show how the claim made was taken on in a real life case. In this case, the story was also related to evidence of results achieved in an outcome statement, coming with its own evidence base.

The below example is an outcome statement and Narrative Assessment story on this outcome. These were shared with the donor in the annual report of 2017.

Green and Inclusive Energy – what do we want to achieve

More than a billion people live without access to reliable, affordable and clean energy. Through a strategic partnership with the Dutch government, Hivos is focusing on lobby and advocacy that will influence the public and political debate on energy, with the ultimate aim of transitioning toward greener and more inclusive energy systems.

What have we achieved? One example of Zimbabwe from 2016.

In July-November 2016, the Ministry of Energy in Zimbabwe held consultative meetings on the renewable energy policy across all 10 provinces to engage a wider audience. The meetings contributed importantly to the renewable energy policy compared to previous consultations held by the government by bringing in people’s concerns.

Why is this achievement important?

The Sustainable Energy for All (SE4ALL) initiative emphasizes the contribution of all stakeholders in the energy sector, including the traditional and non-traditional energy organizations. As such, it was imperative to ensure there was a wider consultation of these groups in the development of the national SE4ALL Action Agenda and the Investment Prospectus (A document that identifies and develops a set of implementable programs and projects, including their investment requirements that can be presented to potential private and public investors).

It is also important to ensure that all policy development processes are well informed by those who are affected, in this case those who lack access to energy. It is from this perspective that the gender and women’s groups as well as other productive groups which are directly affected by the availability/non- availability of energy, for example agriculture, were consulted during the provincial meetings and were represented during the national SE4ALL platforms (for example the SE4ALL Validation workshop).

How did we achieve this? The story of Reginald, our regional advocacy officer

In the past, policy development in the energy sector in Zimbabwe only involved important stakeholders at high level. I remember that I wrote an email to the director of the department working on renewable energy, arguing that the grassroots had to be involved as well. He said this was a good idea, but he said we had to meet. We, as Hivos, then got together with our partners and together developed this idea for the government to have consultative meetings on renewable energy policy with all provinces. In this way, the policy would be informed by grassroots voices. So, this is what we proposed when we met with the Ministry.

If we want to adopt renewable energy at a very fast pace, and we only have about 15 years to reach 100% adoption of renewable energy, then we need to include the people on the ground! These people will have to make this move towards renewable energy, so they must know and understand reasons why. This is what we argued in a petition to the ministry. Also, we argued that energy is a developmental issue, not just a matter of energy. Therefore, you need to relate energy to different development domains like agriculture and gender as well.

High-level government officials were quite receptive to these ideas that we proposed and this helped us a lot. We pursued with our arguments and giving examples. In this way, we managed to convince other key people along the way. As a consequence, we indeed organized a series of meetings in the provinces in partnership with the ministry and other CSOs. We as Hivos had a leading role in organizing these interactions, but other important stakeholders, like UNDP and our civil society partners, were also involved. We invited people from various departments, local administrators and people from different sectors, while making sure that people from local communities and people who had carried out small and medium-size renewable energy projects were also present. Also, we would bring in people who have benefited from renewable energy projects.

In this way, we had real life testimonies on how renewable energy worked positively in their constituencies, sharing important knowledge on some key issues including on what could be improved, and what they would want to see in a new governmental renewable energy policy. For example, because of these meetings, local project people could share difficulties that they faced regarding the costs involved, for example, with environmental impact assessments required by the Zimbabwean authority overseeing water bodies, and water charges to drive turbines for hydropower units. These costs have been prohibitive for some RE projects. It was also helpful to see rural community members, who were proud of their projects, convince other communities about the value of such projects. Peer education and peer reviews really give good results.

The meetings made a lot of difference. If you compare Zimbabwe’s energy policy of 2012 and the local renewable energy policy of 2016, there are a lot of differences. You find inclusion of community views, and the understanding that energy is a developmental issue. We also found that since all this happened, now we are always included in policy processes around renewable energy. We see our role in the future of this policy also: to stay involved for the implementation to ensure that weaknesses are addressed; to take the policy back to the people also, so they remain involved to help speed up with the adoption of renewable energy across the country.

Use of Narrative Assessment stories and insights in mid-term and final Evaluation

Authors and readers of reports about programs often have related complaints. Reporting can be a time-consuming chore. Reading and processing reports can be unrewarding as the commonly found tables and narrations are hard to make sense of, conveying the meaning of what has been done, achieved or not achieved. Narrative Assessment offers a way to make reporting and evaluation broader, more meaningful. Stories and insights from across stories can be integrated into reports to convey the nature of the work being done, as well as the nature and significance of successes and challenges.

Mid-term reports can also include a section exploring ways forward for staff, program partners and donors, who would read it with the insight in the capacities, challenges and opportunities of the program gained from the earlier report sections.

For end-of-project reporting this applies as well. In addition, the stories and insights can offer a sense of advocacy as a (challenging) journey. Advocacy achievements are often small intermediate steps that get their real meaning only in the light of a larger future picture that can be sketched through the stories, providing a sense of the significance of future support - depending on the plausibility of stories and the insights drawn from them.

The creation process of Narrative Assessment stories for use in mid- or end-term evaluation is similar to that of stories for reflection, learning and planning. A difference is that the entry point for the narrative assessment interviews may be either clusters of outcomes or reflection on a certain period of time, for example on how a project started and evolved.

To meet requirements of rigor, evidence-building (discussed in the appendix) may have a more prominent role in reporting than in some of the other uses.

You can also learn from what doesn’t work.

During an interview for a program report, the Narrative Assessment facilitator uses the same inquiry principles as for other uses, including asking critical questions and exploring verisimilitude and plausibility of the unfolding story. The interview can be more taxing for the facilitators as the interviewees may have forgotten details or wish to remember only certain developments or actions. It should be emphasized that the interviews are not about who is at fault, but to gain understanding of the unfolding advocacy process in interaction with the context. The Narrative Assessment stories thus generated cover less detail than those that focusing on a single change, e.g. one outcome statement in Outcome Harvesting. Instead, they provide insights into a longer phase of the overall journey, or a larger change built from other outcomes in case outcome clusters were the interview topic.

BOX 4 EXAMPLE

Example of the mid-term review 2018 of Sustainable Diets for All (SD4ALL)

Introduction

The Sustainable Diets for All program is part of a wider initiative implemented by a consortium of Hivos, IIED and Article 19. The program advocates a transformation towards sustainable food systems. It works in four countries distributed over four regions (Indonesia, Zambia, Uganda and Bolivia) and at the international level. Similar to the Green and Inclusive Energy program, it uses Outcome Harvesting and Theory of Change thinking.  A small team including an advocacy officer works with partners in each country, guided by a joint Theory of Change. The mid-term review meeting followed a second cycle of Outcome Harvesting.

Step 1

In preparation of the mid-term review, the SD4All program manager suggested holding Narrative Assessment interviews with all five advocacy officers. After a first interview, in conversation with the program manager, the Narrative Assessment facilitators decided to focus the interviews not on particular outcomes, but rather on how to develop an advocacy program. Six interviews were held via meeting software.

Step 2

From the verified transcripts, six stories were created around emerging themes identified by the Narrative Assessment facilitators. All themes related to the evolving program. In this case, the stories were between 1.5 and five pages. Again, each story contained the story elements mentioned earlier: transformation, a movement over time, actions by which this transformation happens, in a specific setting, a plot (often involving crises and turning points) and one or more take-away messages.

Step 3

As the meeting was part of a mid-term review, some instruments from Theory U  were used that can support deeper reflection, such as levels of listening and dialogue walks. The team worked in two groups on the following questions:

• What do the stories tell about your own journey and the journey of the SD4All program?
• What do the stories tell you about how important challenges and opportunities for SD4all have been taken on?
• What do the stories tell you about best possible ways forward for SD4all
  • at the level of your country or region?
  • at international level?
• With the stories in mind, think with your heart and mind about what SD4all should try to achieve in the coming 2.5 years.
• Based on the stories and the discussion so far, can you come up with one to three concrete ideas for SD4all to take up, that can be made operational this week?

In plenary, the team then discussed the following:

• What opportunities do you see emerging and where?
• Where and on what should SD4All therefore put more weight?
• How can we translate our views so far into concrete actions: during this week, and in the coming two years?

This session then led into one on the consequences for the SD4All overall, country theories of change, and priorities for the year and the remaining project period.

Evaluation of the Narrative Assessment process and experience

As the program works with consortium partners, these required more information on the context of the new approach. SD4All is a new program on an ambitious topic. The Narrative Assessment stories helped to unearth insights into the struggle to start such a program, to show the strategies and reasoning used, and recognize the complexities of starting a new program with a heritage from the past. Interviewees remarked how they changed from thinking that the interview was a test, to coming into a reflective and learning mood through the facilitated interview. They said that the interviews helped them to remember what was already forgotten in the rush of the day. However, not all interviewees found it easy to transmit the peculiarities of their context despite their understanding its importance. As this was a first pilot it took quite some time, whereas a regular use of the approach could make it more agile. For all participants the process itself was as important as what it produced. It was recommended to extend this approach to the partner organizations as well, and to organize a training of narrative assessment facilitators.

6. Competencies needed for Narrative Assessment

Narrative Assessment, while building on something as subjective as stories, in no way means abandonment of rigor. Building on theoretical foundations of narrative inquiry and other qualitative research approaches, Narrative Assessment offers a new direction for conceptualizing rigor. Plausibility is to be determined by the lifelikeness of stories, their ‘verisimilitude’ – its quality of being believable. To ensure this, Narrative Assessment requires some key competencies. 

Narrative Assessment facilitators

Narrative Assessment facilitators can be M&E specialists, or other persons trained in qualitative evaluation or research approaches. They should have the capacity to help advocates build their stories and to put those down in a form that is appealing and informative to audiences.

To help build stories and assess their plausibility, Narrative Assessment requires co- construction by advocates and Narrative Assessment facilitators. While advocates are in the lead in the story-telling, facilitators are critical examiners teasing out detail, interpretation, and where possible, signs of evidence. This entails demands on facilitators. Their professional role is grounded in their capacity to simultaneously build and examine stories, while balancing the need for trusting, open relations with advocates with the need for critical distance. The readiness and capability of the Narrative Assessment facilitator to probe and examine is crucial to make sure that Narrative Assessment stories surpass commonly told success stories that simply shine a favorable light on programs.

Preferably, facilitators are also familiar with advocacy work. They also need to be informed about the program involved and its Theory of Change, if available. Finally, stories will have to incorporate sufficient contextual information in order to be comprehensible and convey the meaning intended. Advocates may be well able to share their knowledge to provide this context. However, this contextual knowledge will need to be critically assessed in terms of the plausibility of the analysis and the possibility of alternative viewpoints that may raise questions or run counter to a story. To perform this task of critical assessment facilitators need a minimum degree of contextual knowledge, or draw on external resources to gain it. In cases where this is not possible, the plausibility of the story may still rely on story quality, including e.g. detail and contextual information as provided by the advocate. However, weak contextual embedding of the story makes it more vulnerable to challenge.

The second key competency for Narrative Assessment facilitators is the skill to translate an interview transcript or notes into a story that contains the six story elements and fulfils the criteria of verisimilitude. This requires the ability to capture verbal information in a written storyline, ordering and polishing information, enough to produce an engaging story, but not too much so the story will still reflect the personal account of the storyteller. Listening abilities have to go hand in hand with the skills to probe and to write coherently, succinctly and engagingly.

Advocates

Advocates need to have the motivation and will to share the stories, give an account of their work in two senses: to tell, and to be answerable. They must be able to recall what happened, from record or otherwise, and to substantiate their claims. This demands not perfect memory or perfect knowledge. It does require the ability and openness to bring out what they do know and why they think things have happened in a certain way. It also requires admitting uncertainty, exploring different understandings, and accepting that there will be moments they will be challenged. The advocate and the Narrative Assessment facilitator need to build and maintain a constructive and mutually supportive working relationship, while also offering the space to question, as demanded by the quest for high plausibility of stories. Finally, it is not easy for a fish to understand that it is swimming in water; the interview requires from the advocate a certain capacity to reflect and comment on one’s own context. The Narrative Assessment facilitator needs to be able to tease out this reflexivity. In some cases, advocates may find that Narrative Assessment requires some adjustment as they may find that Narrative Assessment differs much from commonly used forms of reporting.

Communication staff

Because of the wide-ranging usability of stories for different internal and external purposes, depending on the stage of the assessment, or its purpose, it may be helpful to involve staff members who have different forms of involvement and responsibilities, e.g. project, M&E and communication staff. While M&E and project staff can be directly involved in the co-construction of stories and their assessment, communications staff can be involved at different stages. One option is engagement at the start of the assessment, in selection of program parts for Narrative Assessment. Communication staff can be involved to discuss the purposes of the assessment and the ways in which assessment of different parts of a program could contribute to communication with different types of audiences. Communications staff can also be engaged when stories are completed and their usefulness for external communications can become clearer. Communication staff can also be involved as members of the Narrative Assessment team throughout the process, as they are often trained to recognize a good story and have the skills to judge whether a story ‘sticks’ – they may be of help establishing which advocates may have what stories to tell that may best support certain goals. Competency requirements for communications staff include the capacity to assess the way different stories can contribute to communication with different audiences, and to adapt those stories to the purpose and audience.

Programme or project managers

Program managers are crucial for creating space for reflection, through the Narrative Assessment interviews and the reflection meetings. They need to be resilient and flexible in the face of uncertainty, as the interviews may unearth unexpected issues. Finally, they need a capacity for adaptive management so the insights and proposals from the Narrative Assessment stories and reflection meetings can be used to their full potential.

With one story, you can already reach a lot of conclusions. Using a number of stories together makes it even more interesting.

Appendix: Further reading

Narrative Assessment is a new M&E method. Below, we discuss why we think a new method was necessary. We also discuss how we think Narrative Assessment differs from other current methods. However, Narrative Assessment also connects meaningfully with existing approaches and methods that are commonly used. In the final section, we show how.

Narrative Assessment and existing M&E methods

Existing methods and their limits

It is increasingly recognized that conventional M&E methods are not very suited for dealing with complex interventions such as advocacy. For example, experts recognize that causal pathways often cannot be fully known prior to the intervention; that what needs to be measured may not be known prior to an intervention, and that processes shaping a program and its results are not stable. 

Practitioners, scholars and consultants have increasingly started to take complexity as a starting point for understanding development work itself, as well as its monitoring and evaluation. A number of recent publications stress the complexity-related challenges involved with advocacy and its monitoring and evaluation.  Recently, M&E methods have been developed that seek to engage with complexity, including Complexity-responsive Evaluation,  Complexity-Aware Monitoring,  Developmental Evaluation,  Contribution Analysis,  Outcome Mapping  and Outcome Harvesting.  While those M&E approaches have much to offer, they are not specifically geared to advocacy. They have certain limitations when it comes to monitoring and evaluation of advocacy. Current M&E approaches do not address four key issues, discussed below:

• Measurement and evidence;
• The meaning of achievements and failures;
• Learning;
• Communication.

Measurement and evidence

Current M&E methods focus on measurement of achievements, often using indicators to classify these. Outcome indicators used for advocacy are often quantitative in nature, such as ‘number of elected officials who publicly support the campaign’. Also qualitative indicators are employed, including e.g. strengthened organizational capacity, or uptake of certain arguments in political debate. To a degree, such indicators can be used to ‘measure’ achievements of advocacy programs. However, rather intangible outcomes such as increased credibility of the organization in the eyes of policymakers, or increased support for a certain policy position among policymakers are hard to measure. At the same time, these are crucial achievements.

In addition, these methods do not critically reflect on the issues of evidence. They easily gloss over the fact that when it comes to advocacy, evidence of outcomes is often not available or accessible (intangible, invisible or politically sensitive). Some methods that focus on evidence-gathering, (contribution analysis and process tracing being the most prominent ones), are appealing in their sensitivity to complex contexts and processes. At the same time, implementation of such methods, especially when it comes to complex programs involving many actors (as often found in advocacy for development) is highly resource-intensive, while not being able to resolve key mentioned challenges of evidence availability and accessibility that would often be confronted.

The meaning of achievements and failures

While advocacy achievements may be measured in some cases or to some degree, they need to be interpreted to make sense. For example, a number of elected officials publicly supporting a campaign does not tell us whether the campaign has actually changed their viewpoints. Qualitative tools measuring change do exist, including e.g. Most Significant Change, Outcome Harvesting and Outcome Mapping. However, while such approaches can bring out results in qualitative terms, they still pay little attention to the question what an outcome means in light of the Theory of Change of a program. As mentioned earlier: advocacy results are often steps in the direction of impact, rather than results that have a direct impact on constituencies. The nature and significance of an advocacy outcome can therefore often only be interpreted in light of the hypothesized pathways of change and desired outcomes that are described in a Theory of Change. Secondly, current tools are highly intervention-centered, paying little attention to the context in which a particular intervention is to make a difference and how contextual factors influences and interacts with intervention activities. In advocacy, an organization’s intervention will be but one factor in a complex process involving many actors and factors. Current tools thereby tend to provide a rather decontextualized presentation of a program and its results when it comes to advocacy. What a program’s achievements mean in light of the broader context in which it operates remains hard to say.

Learning

Focusing on outcomes first and foremost, existing methods provide little insight into the actual work of advocates. The judgments and analyses behind strategies contributing to achievements and failures remain black-boxed. As a consequence, they cannot support decision-making and strategizing about next advocacy steps: what went into the mix of efforts to achieve outcomes, what dilemmas and challenges came up, and how these were confronted and eventually solved (or not). Existing methods therefore offer little opportunity for identification of lessons to strengthen future advocacy theories, approaches and strategies.

Communication

Existing M&E methods for advocacy are mostly focused on producing reports for internal and external accountability purposes. There are two basic limitations with this, constricting the usefulness of such reports for communication.

Internal communication in organizations and networks
Existing methods do little for internal communication about advocacy work within organizations and networks. Reports that focus on what comes out of the ‘black box’ of advocacy do not convey the nature and significance of advocacy work and the knowledge and skills involved. In addition, current M&E methods ignore the fact that advocacy is not done by one organization on its own, but engages multiple actors. They do little to bring together stakeholders (one’s organization, allies, and partners), to interpret the significance of the advocacy work and its link with its achievements. They do not address questions of interpretation of achievements and failures and challenges are not collectively addressed. Collectively obtained experiences and lessons are therefore not used for planning next actions. Unpacking the inside of the ‘black box’ will help organizations to understand and learn about advocacy work. Explaining advocacy and the relation with its results to the internal audiences, and externally to supporting donors and the public at large will build legitimacy and organizational credibility. Collective interpretation and sense-making of advocacy work in relation to results has the potential to strengthen coalitions and networks and the impact of their collaboration.

External communication

Current M&E methods do little for external communication. They tend to lead to reports that are technical in nature and therefore inaccessible for audiences beyond a small circle of experts. These reports are not well-suited to convey advocacy achievements, which often are steps towards impact rather than forms of impact. Advocacy results easily remain out-of- view or meaningless for supporters and wider publics. In addition, lack of impact on constituencies can be mistaken for a lack of significance if the significance of the results is not articulated. For example, achieving an adjustment to a policy document may result from great collective efforts made over many years. However, it should be clarified how that change is or may become helpful to constituencies or society at large. This is not simply because policy processes are technical. Again, the outcomes of advocacy for development are often interim in nature, banking on future policy processes to attain the ultimate legitimacy of positive impacts such as increased access to clean water, food security or a living wage. In existing M&E methods, this is not taken into account.

How Narrative Assessment builds on existing approaches and methods

While Narrative Assessment offers an alternative approach, it also builds on existing approaches and methods that are currently in use for planning, monitoring and evaluation: narrative methods, collective sense-making approaches, and Theory of Change. This is outlined below.

Narrative methods

Several existing methods for monitoring and/or evaluation include stories, such as Performance Story Reporting;  Most Significant Change;  Collaborative Outcomes Reporting,  and Narratives of Change.  The potential of stories to learn about change in complex development contexts is widely seen. However, stories have been part of evaluation methodology to only a limited degree, mostly putting the capturing of program impact in participants’ or beneficiaries’ stories at its core. 

So far, ideas on the importance of stories and development of methods have not led to a story-centered approach specifically adapted to advocacy monitoring and evaluation. Importantly, current methods do not engage with the fact that advocacy achievements mostly do not lead to direct impact, but to steps in a longer process of change. These steps can best be understood from a Theory-of-Change perspective, best accessed by engaging the advocates themselves whose strategic maneuvering we would need to understand.

Existing methods also do not help making advocacy understandable or realistically justify investments. They not incorporate inquiry into practical judgement and advocacy processes. In complex situations with many unknowns, and a key dependence on the nature and quality of advocates’ practical judgments, understanding advocacy and its results would demand attention to just that. The nature and quality of that judgment, incorporating analyses of strategic options and considerations, chances at success, while balancing opportunity, cost and risk can only be conveyed through stories. 

Finally, important here is that existing story-based approaches in evaluation do not make optimal value of the communicative potential of stories. Nor do they adequately address the risks that need to be acknowledged and dealt with when it comes to stories, in particular their tendency to idealize, highlight and lowlight certain aspects, acts or events, rather than simply providing a factual account of ‘what happened’. 

In short: while stories help to provide a view of the reality of advocacy work and achievements the way no existing method can, Narrative Assessment stories build in a robustness through the critical inquiry techniques used by the Narrative Assessment facilitator.

An example illustrating the approach is Outcome Harvesting. Narrative Assessment’s way or working can build on outcomes created through Outcome Harvesting. Outcome Harvesting is an evaluation method that similarly collects achievements with an open-ended approach, includes incorporation of evidence of what has been achieved, and works backward to determine whether and how the project or intervention contributes to the change.  Outcome Harvesting is flexible, accommodating the complexity of change and adaptation in interventions. However, Outcome Harvesting does not center on interpretation, the narrative construction of the ways in which outcomes and the path towards them are to be understood and described from the Theory of Change (whether formally named like that or not) that advocates work with. It constructs the connections between outcomes only to a limited degree, and importantly, does not focus on making comprehensible how they could come about. Outcome Harvesting also does not involve reflection on challenges confronted during programs - why outcomes did not come about, what dynamics and conditions may have contributed to that, and how such issues were engaged with. Finally, Outcome Harvesting does not address challenges involved with gathering evidence for advocacy evaluation. In a way, Narrative Assessment seeks to form a response to Outcome Harvesting to address these issues, innovating upon it.

The process of developing the stories, the questions forced us to think, they are not for donors which is what we are used to tell.

Collective sense-making methods

A number of methods involve collective sense-making through interpreting outcomes (e.g. Most Significant Change; Collaborative Outcomes Reporting). Narrative Assessment can build on these, from the starting point that bringing together different viewpoints and interpretations helps to build more robust knowledge and evidence regarding the effectiveness of advocacy programs. An innovative dimension in Narrative Assessment is the role given to critical inquiry of sense-making drawing on Narrative Inquiry, accepting that robustness will be aided by an optimal substantiation of claims provided in stories (taking the starting point that, of necessity, advocates’ practical judgment must make an important contribution to that substantiation).

Theory of Change

Narrative Assessment is closely linked to Theory of Change in two ways, Firstly, the construction of meaning involved in the co-construction of stories (what happened, why, how relevant are our achievements, how can disappointments and challenges be understood) takes place in light of Theory of Change. Secondly, reflection on the strengths, limits and gaps in analyses and capacities to achieve change is to feed into reflection on the ways past action holds up against Theory of Change, and to inform future action. Theory of Change allows focus on longer-term impacts rather than short-term results. As Theories of Change make assumptions and pathways of change explicit, a Theory-of-Change process facilitates reflection, interaction and adjustment when it comes to assumptions and understandings of how change happens. This supports critical reflection on how an intervention may actually contribute to desired change. It can take into account that achievements and change caused by other influences create new conditions for programs. Programs can be expected to improve their effectiveness with the capacity to act on these conditions and adjust in such situations, rather than by holding on to planned actions.  It also facilitates interaction and adjustment across different actors in a program. Theory of Change in an interactive process of deliberation and critical reflection may help to confront and engage with differences.  Theory of Change is hereby hailed as potentially helpful in articulating assumptions and pathways of change, and dialogue and adjustment over time, as programs unfold, as well as evaluation.  Many reports and practical guidelines engage with Theory of Change and its potential merits in terms of its focus on assumptions and pathways of change. This is considered fundamental to devise, understand and adapt interventions.

The stories show learning points, also about challenges. You can appreciate that more.

Theory of Change can be a fundamental starting point for advocacy monitoring and evaluation because of the complexity of the processes of change that advocacy programs seek to contribute to, and the complicated nature of interventions. Relating to a program’s Theory of Change, advocates can articulate how outcomes fit specific pathways of change, allowing for flexibility in accounting for the way programs unfold. The interpretation of outcomes by relating to Theory of Change also makes it possible to specify and clarify Theory of Change as directly relevant for the outcomes at hand. It also facilitates establishing where program theory had not worked out as foreseen, and how this can be explained.

In advocacy monitoring and evaluation, it is commonly accepted that outcomes will often only be steps in the direction of desired change. Programs are often oriented towards normative and legal frameworks, often at multiple levels, concerning, for example, human rights, biodiversity, gender and reproductive health and rights, or climate change mitigation. Time frames in program theory are typically many years longer than the duration of programs.

With advocacy outcomes largely remaining at a remove from impact, relevance for constituencies can and should be theorized, banking on future changes that in the end are to contribute to impact. Outcomes can be accepted as relevant when they can be constructed, plausibly, as steps in the direction of desired change; a step in a longer journey that can be assessed in the context of many other steps, including future ones.

By bringing Narrative Assessment together with Theory of Change thinking, both the theorizing (especially hypothesizing future pathways) is strengthened, as well as the review of assumptions about strategies and pathways built into the Theory of Change.

About the authors

Margit van Wessel works as assistant professor at the Chair Group Strategic Communication of Wageningen University & Research. She is interested in civil society advocacy and its evaluation, and initiated the development of Narrative Assessment drawing on her experiences as an advocacy researcher and evaluator, and her communication science background. She holds a PhD in Social Science and an MSc in Human Geography from the University of Amsterdam. Margit can be contacted at margit.vanwessel@wur.nl.

Currently, Wenny Ho is a senior advisor in Design, Monitoring & Evaluation and Learning with Hivos, a Dutch INGO. Previous positions were, among others, with the Dutch Ministry of Foreign Affairs (Research Department) and ActionAid International (International Advisor Accountability). She holds a PhD in International Relations and Change Management from the University of Amsterdam, and a MSc in Forestry of the University of Wageningen. Wenny specializes in action research and multi-stakeholder processes. She can be contacted at wenny.ho@hivos.org.

The stories help to create the right perspective. You really can understand not only the end result, but the whole story.

Glossary

HIVOS

Humanistisch Instituut voor Ontwikkelingssamenwerking. Hivos as a humanist institution believes that human life in its many forms is valuable, and that people are filled with potential.

OUTCOME HARVESTING

Outcome Harvesting collects (‘harvests’) evidence of what has changed (‘outcomes’) and, then, working backwards, determines whether and how an intervention has contributed to these changes.

MOST SIGNIFICANT CHANGE

The Most Significant Change technique is a qualitative and participatory form of M&E. It is based on the collection, systematic selection and analyses of stories of significant changes attributed to an intervention.

THEORY OF CHANGE

Theories of change are the ideas and hypotheses (‘theories’) people and organizations have about how change happens. These theories can be conscious or unconscious and are based on personal beliefs, assumptions and a necessarily limited, personal perception of reality.

ADVOCACY

Any action that speaks in favor of, recommends, argues for a cause, supports, defends, or pleads on behalf of others. Advocacy can include many activities that a person or organization undertakes including e.g. social media campaigns, public speaking, publishing research and lobbying.

OUTCOME MAPPING

Outcome Mapping is a set of tools used for planning, monitoring and evaluating interventions aimed at bringing about social, economic or technological change. The idea is that to succeed, an intervention needs to involve multiple stakeholders. OM connects ‘outputs’ to ‘outcomes’ by focusing on the patterns of action and interaction among stakeholders.

NARRATIVE ASSESSMENT

Narrative Assessment is a systematic monitoring and evaluation approach to making sense of the realities underlying advocacy results. It starts from the stories of advocates themselves and focuses on unpacking the dynamics and contribution of advocacy work, to inspire learning and to support program adaptation and communication.

NARRATIVE INQUIRY

Narrative inquiry is a form of research that centers on the stories research participants tell about their experiences. It is based on the premise that, as human beings, we come to understand and give meaning to our experiences through story.

CO-CREATION

Co-creation is way of working, or form of strategy, that brings different parties or actors together to jointly work on a mutually valued outcome or proposal. Co-creation brings a blend of perspectives and ideas from the different actors to generate proposals and solutions that supposedly are richer.

SENSE-MAKING

Sense-making is the process by which people give meaning to their collective experiences. There are various tools to support collective sense-making.

PLAUSIBILITY

The degree to which key stakeholders believe the proposed logic of the story is correct or at least, plausible.

VERISIMILITUDE

Quality (of stories) of being believable.

NARRATIVE ASSESSMENT FACILITATOR

Person who conducts the Narrative Assessment interview and possibly also creates the story from that. This persons is versed in posing critical questions in a constructive manner to help the advocate tell his story in a credible and plausible way.

IIED

International Institute for Environment and Development is a policy and action research organization, based in the UK.

ARTICLE 19

Article 19 is a British human rights organization founded in 1987 with a specific mandate and focus on the defense and promotion of freedom of expression and information worldwide. The organization takes its name from Article 19 of the Universal Declaration of Human Rights.

The stories help to create the right perspective. You really can understand not only the end result, but the whole story.