Evaluation Through Writing and Images

The Limpopo National Park (LNP) project evaluation was unusual in that it was both written and filmed. AFD was counting on the fact that the film images, interviews and animated maps would give a less austere rendering of the goals of the funded development, the physical environment and wildlife, the villages’ economic development and the concrete results achieved.

As part of a learning objective, this approach aimed to turn the evaluation into a fresh opportunity for dialogue among the project stakeholders and, further afield, to reach development professionals, especially those interested in biodiversity-related issues.
 

Methodological difficulties

Several difficulties arose in this “mixed” evaluation. The first involved ensuring a methodological coherence between the two evaluations (written and filmed) as it needed to lead to the same evaluative conclusion. The solution devised was to co-develop the evaluation methodology and systematically ensure an exchange of viewpoints at every stage of this method.

It was decided at the outset that the written and filmed evaluations would take place simultaneously and follow exactly the same construction and validation steps. As a result, just one purchase order was issued to recruit an integratedteam.

Further reading: Rehabilitating Limpopo National Park

One difficulty encountered during data collection: the challenge of fully coordinating the two evaluations carried out by an integrated team that had distinct specialties (written and filmed), in the same field and with the same actors. It was decided that joint interviews would be held, especially during framework interviews, to gather general information on the issues under discussion and the interviewee’s role in the project.

However, the written evaluation interviews needed to go into greater detail than was relevant for the filmed evaluation. Thus, to limit the number of interviews and boost efficiency, the teams divided up the interviews depending on the role of the project stakeholder interviewed.

A complementary view of the projects results

The LNP filmed evaluation confirms that this filmed approach shows the practical aspects, the dynamics and context of the project but is ill-suited to analysing organisational, financial and methodological processes. It gives value to stakeholders’ (beneficiaries or not) contextualised accounts and actual words. In short, the informative value of images is more powerful but less precise than the written word.

The approach based on a concomitant written and filmed evaluation of the Limpopo project reinforced the evaluative analysis thanks to the exchange of the team’s different viewpoints. The filmed interviews were used to validate the secondary data collected and thus refine the analyses of the stakeholders’ interactions as well as the challenges and policies related to the project. The film more easily revealed the various or even opposing perspectives of some stakeholders.

These divergences were often less obvious in the written report, mainly as this calls on the intermediation of an author. The film also influences the narrative structure as it gives more detail on the history of the project and its challenges, whereas the written evaluation presents an analysis based on pre-defined evaluative criteria. Although the film does not replace the detailed sections on the institutional, budgetary, political and technical aspects of the written evaluation, it completes and strengthens the messages, notably for the project stakeholders.

An effective tool for awareness-raising and dialogue

At the feedback sessions on the conclusions of the LNP evaluation, the film – which explores the highly complex topic of national park protection together with its multiple stakeholders – made it easier to share the results and encourage debate, especially among the communities and staff in charge of protecting LNP. Misunderstandings were resolved and the dialogue on the LNP’s future development has continued. Finally, the filmed evaluation also helped to draw the public’s attention to its messages – a public that extends beyond the project stakeholders to include, for example, students or citizens interested in development. In short, a broader and more varied audience.

To continue the filmed evaluations

What are the prospects for filmed evaluations? The complementarity of written and audiovisual formats could be further developed in order to use the specific qualities of both words and images in the evaluative approach. Using the medium (written or filmed) that is best suited to understanding and evaluating each project component would provide a fairer and more relevant perception.

This approach could be formulated as a specific methodological reference that would help to frame the contexts in which a filmed evaluation would be much more relevant and clarify the methodology combining both formats. The AFD teams will test this in 2019 with new projects under development.

This article was originally published in the AFD 2017/2018 Evaluations Report