Key concepts and definitions

EVALUATION

Key concepts and definitions

I - What is meant by the impact of an intervention?

There are two commonly accepted definitions of what is the impact of a development intervention. The first is used by the Development Assistance Committee of the OECD (DAC) : “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”. The second is used more particularly by economists and can be summarised as: “All the effects that a development intervention has on beneficiaries that can strictly be attributed to this action”.

The first definition focuses on the temporal dimension by placing the impact at the end of a results chain. The second definition is based on the allocation criterion by isolating among the effects those which have been caused by the intervention in question. The second definition is used in the framework of impact evaluations, but the use of both definitions can sometimes lead to ambiguities.

The impact of a development action can be studied at the level of its final beneficiaries in which case it concerns changes in their well-being. It can also be studied at the level of a community, institution, region, etc.

II - How to measure impacts?

The impact of a development intervention cannot be measured by simply observing the well-being of a population over time. Indeed, other events may have contributed to changes observed in the well-being indicator, which means it is difficult to distinguish the share than can only be attributed to the intervention. For this reason, impact measurement is based on a reconstruction of what would have happened without the intervention or the “counterfactual situation”. It is the comparison of two levels of well-being “with” and “without” the intervention which gives the measurement of its impact.

The counterfactual situation is unobservable since the population affected by the intervention cannot be observed at the same time as if there had been no intervention. Different methods make it possible to get round this difficulty by providing an estimation of the counterfactual situation: this involves finding a population that is as similar as possible to the one concerned by the intervention (or “treatment group”) and that evolves without such an intervention. This population is called the comparison group or control group.

The statistical construction of a comparison group may be conducted after the development action has begun. The most common methods include: matching beneficiaries to non-beneficiary individuals that share similar socio-economic characteristics; separating individuals between treatment groups and control groups by so-called natural events that are not related to the project itself or to the characteristics of the beneficiaries, thus making the two populations comparable; purging the initial differences between beneficiaries and non-beneficiaries by methods comparing the differences in evolution (rather than in the final level) in the well-being indicator (double difference).

A control group may also be built prior to the selection of project beneficiaries. In this case, for the requirements of the impact evaluation, the selection is generally made in a random manner (Randomised Control Trial). This method is similar to the blind tests commonly used in medical spheres and can provide unambiguous differences in well-being between the treatment group and the control group.

When the question asked for the impact evaluation does not require an accurate statistical measurement, the construction of counterfactual theories that take the evolution of the project environment into account can sometimes be insufficient.

III - What is an impact evaluation?

An impact evaluation is an evaluation of a project, programme or policy based in particular on the measurement of its impact by using a counterfactual. However, an impact evaluation is generally not restricted to simply measuring the impact and breaking it down into different categories of beneficiaries and different types of action.

A quality impact evaluation must also provide answers to questions bearing upon the target of the intervention – which populations were concerned-, its mechanisms – how did the impact develop- and its determinants- how can the presence or absence of impact be explained according to the contexts and categories of beneficiaries. The impact evaluation therefore combines the rigorous measurement of the effects of the intervention on well-being with an understanding of the behavioural and/or environmental mechanisms that link the intervention to the changes in the well-being of beneficiaries.