DRC: In Kinshasa, how an evaluation is put together

The implementation of a project to evaluate AFD’s actions mobilizes extensive resources. An example with a scientific evaluation of the health and socioeconomic impacts of drinking water networks in Kinshasa, the sprawling and resilient capital of a fragile State.

Kinshasa, one of the three largest cities in Africa. Access to drinking water in the capital of the Democratic Republic of Congo is a challenge for residents in the outlying neighborhoods of this urban monster with an unclear number of residents. Kinshasians in areas where there has been little urbanization, or none yet, often have no water distribution network. They get their water supply from springs or streams which flow in the depressions of a changing and unstructured landscape.

AFD has been supporting water supply in these new neighborhoods not served by the water authority since 2006. A decentralized management system has been set up by the local NGO ADIR. It is based on community participation structured around users’ associations, ASUREPs. A second implementation phase started in 2016 for four years, called PILAEP 2. The main objective is to connect 400,000 residents in 26 neighborhoods to water networks and thereby meet an essential need: high-quality and accessible drinking water.

The scientific evaluation research project conducted by AFD aims to scientifically evaluate this water infrastructure, its health impacts (diarrheal diseases, stress) and socioeconomic impacts (water chore for women, schooling for children, local governance, etc.). In the context of a conflict-affected State and failed administration, where very few studies have been conducted on this scale, its management requires a strong commitment from all partners.

As development partners, it is always necessary to measure the impact of what we do. These impacts can be positive, which we hope, or negative.

Didier Grébert, AFD Director in Kinshasa

Previous Pause Next

1 of 3

Evaluation, a time machine

“I once waited a month to receive a reply by post to an e-mail entitled ‘urgent’. The letter was covered in official stamps”, remembers with a smile Claire Zanuso, an economist, responsible for impact evaluations at AFD, who is on her way to Kinshasa. Managing a scientific impact evaluation in a politically unstable context like in Kinshasa requires patience and return trips to the field to speed up the implementation. “This process”, she continues, “is especially important in the context of a fragile State such as DRC where the ownership and sustainability of operations are particularly important issues.” She is leaving to attend training for interviewers and the launch of the baseline survey for this evaluation.

“We build a time machine to identify the impacts which can strictly be attributed to the operation supported by AFD in an area where other development actors are present”, she explains. “It involves setting up a rigorous protocol which conditions the validity of the exercise. It aims to compare life with and without the PILAEP project, in other words, before the construction of the water networks, then two years after the commissioning. We compare the impacts of the project with those who have benefited from it and a control group, which has not benefited from it. This amounts to building a counterfactual, in scientific jargon, which we hope will be as rigorous as possible.”

As partners, AFD has chosen the Congolese National Institute of Statistics (INS) and the DIAL Mixed Research Unit, which is attached to the Research Institute for Development (IRD) and Paris-Dauphine University. While this adventure is conditioned by a requirement for accountability vis-à-vis taxpayers and the French State, it is also a learning process through the “by-products” it creates. For example, it helps the teams gain a better understanding of the project context through capitalization work on the previous phases. Often little is known about the outlying neighborhoods. The data about them are sparse and rapidly become obsolete, as they are subject to the challenge of the predominant informality and rapid growth. The residents of these neighborhoods are under-represented in national surveys.

This process also contributes to improving the management through detailed knowledge of the living conditions of populations in the outlying neighborhoods, of the everyday life of the project and its sticking points, in particular thanks to statistics, qualitative surveys and the production of mapping tools. “It involves”, concludes Claire Zanuso, “contributing to strengthening the statistics of our partners, the evaluability of projects, and disseminating a culture of evaluation with our teams and all the project stakeholders.” Co-construction over the long term. The time for science.

The search for quality data available in real time

The DIAL researchers worked for several months on defining an evaluation protocol adapted to this versatile context, identify the survey areas in the field and jointly build with INS the collection tools, the questionnaires and the application to collect data on tablets. One of the issues for the scope of the survey: the collection of quality data. “DIAL is a research laboratory which has a long tradition of conducting surveys in Africa and all over the world”, says Camille Saint-Macary, a researcher at DIAL.

It first and foremost involved training interviewers, paying special attention to the diversity of teams. To launch the survey, two 10-day training sessions were organized in Kinshasa to familiarize the INS interviewers with the questionnaire. The objective: inform them about what is being sought and teach them how to ask the right questions, minimizing biases. They must also be able to understand the situations they will come across and know how to respond to them. “This is difficult yet key work”, explains Camille Saint-Macary, “They are the ones who are going to collect stories in the field, then transpose them into data, which will subsequently be used as material for the analysis. It is for this reason that we pay attention to ensuring they work in good conditions, with time.”

To facilitate the remote management of the survey, the questionnaires have been transposed to an application which can be accessed from a tablet. The answers are entered via this application. “DIAL has been working with Congo’s INS for a long time. We have access to quite effective tools to transfer field data to an INS server connected to DIAL’s server in Paris. We will be able to ensure that the survey runs smoothly virtually in real time and remotely, check that we don’t have any major inconsistencies and put things right if we see that the teams are experiencing difficulties.”

Before PILAEP 1, life without drinking water

Albert Djodjo is Chairman of the Board of Directors of an ASUREP. He has been living in the Mikonga neighborhood for 22 years. “Before PILAEP 1 started, we used to rely on water from the river. Every morning, right at 6 o’clock, mothers and children used to go down to the river to fetch water and bring it home”, says Albert Djodjo, “The water wasn’t fit for consumption. When it rained, we could spend the whole night outside collecting rainwater.”

As the neighborhood is located in valleys, it was unsafe water drained by the rain which was drawn by its residents. “There were only two pipes connected to wells available for 28,000 residents”, he points out. Tensions, negotiations, fights, cases of rape on the paths leading to the river or at the collection points, this was all part of the daily lot of the water chore.

“In terms of health”, Albert continues, “there were many cases of water-borne diseases. There were cases of diarrhea on each plot.” This situation was largely due to water pollution, the lack of sanitation and hygiene practices. In addition, there was typhoid fever, malaria and cholera epidemics.

In this neighborhood, there were only 6 or 7 schools to send children to. There were only boys playing in the schoolyards because, adds Albert Djodjo, “Our girls first had to go and fetch water. It took two to three hours for them to come back. If they made the effort to go to school, they were likely to be punished for being late.”

The survey area, an art in itself

The objective of this two-month survey, from the end of March to May 2018, is to provide an inventory prior to the commissioning of the water network in the PILAEP 2 neighborhoods. A team of 60 Congolese interviewers is questioning 18,000 residents in 3,000 households and 10 survey areas. Recruiting interviewers, counting the households, interviewing and collecting data are among the stages covered by INS.

The teams are ultimately seeking to understand if the fact of reducing the distance to the water point – and therefore the time devoted to the water chore – increases, for example, the school enrolment rate for girls, income-generating activities and empowerment for women, and reduces the violent acts committed against them and the stress they experience on a daily basis. Finally, the improvement in the health of residents, especially children, is also evaluated. Another area involves questioning the quality of the governance established via users’ associations (ASUREPs).

The survey work is an art in itself. It involves being accepted, gaining the trust of people so that they answer questions and that these answers provide material for analysis. “When you introduce yourself for the first time”, says Jérôme Anyasi, an interviewer, “people don’t know why you’ve come to their homes. If you don’t manage to clearly introduce the purpose of the study, they can refuse. Once they’ve accepted, if you talk for a bit too long, people might get fed up and decide to do something else.”

Some subjects are also sensitive, adds the interviewer: “People don’t necessarily feel like giving certain information, like the value of their property”. The same goes for information about governance, corruption or violence in the neighborhood. In the current political context, which is deteriorating, the people interviewed may also be scared of getting arrested. A 3-day pilot test allowed the wording of the questions to be examined in order to make adjustments to the questionnaire. After making final adjustments to the collection tools and a launch meeting with the teams, it was possible to launch the survey.

Advocacy for sustainable and high-quality investments

RDC, Kinshasa, enfants, Kabamba — *© Ines Kabamba / AFD*

Through this evaluation, AFD is seeking to bridge a knowledge gap concerning the effectiveness of sustainable investments in drinking water networks on public health, economic development and social ties. In terms of cholera, the international community generally focuses on short-term approaches via minor operations (distribution of chlorine tablets, filter kits) and emergency measures (immunization campaigns).

This impact evaluation, combined with another evaluation conducted in Uvira in South Kivu (DRC), aims to contribute to feeding into the international debate on the distribution between short and long-term investments and the importance that needs to be given to the quality of water and sanitation services in the context of the Sustainable Development Goals. As pointed out by Martin Leménager, head of the PILAEP project team, “It’s great to have a water network in your neighborhood, but it’s not enough. There mustn’t be unexpected cut-offs in the service and the water mustn’t contain bacteria!”.

DRC: in Kinshasa, how an evaluation is put together

partager