Debates

EVALUATION

Debates

I - What are the recent changes to impact evaluations?

In 1997, the Mexican government launched a new type of social transfer program called Progresa in which transfers were conditioned upon sending children to school and going to health centres for consultations (Conditional Cash Transfers, CCT). A rigorous impact evaluation of the programme was conducted and was based on a partly random selection of municipalities benefitting from the programme as early as its first expansion phase and others that only benefitted during subsequent expansion phases. Thanks to evidence of Progresa’s positive results provided by the impact evaluation, this programme was maintained by successive governments and was even extended to urban areas that had previously been excluded. The results of the evaluation were also used to justify the implementation of similar programmes in 27 other countries.

The Progresa impact evaluation also had a strong influence on the use of such evaluation methods in development spheres. These approaches were inspired by medical approaches and were, for example, used in the 1960s to assess social policy transfers in the United States, yet they had until then been used little in poor countries. Following the Progresa impact evaluation, donors such as the World Bank, encouraged by research institutes such as the Center for Global Development or the Abdul Latif Jameel Poverty Action Lab, financed a large number of evaluations based on these so-called robust statistical methods. More recently, the International Initiative for Impact Evaluation (3ie) was launched by foundations, bilateral donors and governments from the South in order to finance over a dozen rigorous evaluations each year. These evaluations will cover interventions in a wide-range of sectors including health, education and agriculture.

This new generation of impact evaluations will provide an unambiguous evaluation of project impacts because they are based on transparent and rigorous statistical methods. In addition, a real knowledge accumulation process is expected - a recognised weak point in classic evaluations – that would be associated with transparency, opposability and debates among peers which are characteristics of the scientific approach adopted. Finally, in addition to these methods, the associated experimental approach applied to development actions is something new – to test a policy, or several variations to a policy, on population groups and rigorously measure its effects before considering changing scale. Esther Duflo speaks about creative experimentation in this respect as both a new way of evaluating and a new way of designing development policies.

II - Why is the impact evaluation subject to debate?

The development of impact evaluations has caused debate among the development community. This debate stems from the ambition of this new tool: to produce hard evidence of what works by using an experimental measurement of the impacts of the targeted actions in order to improve future policies. Every one of these words has provoked debate among specialists:

Is the evidence produced more conclusive – harder – by using this method rather than another ?
Does the aim of grasping what works by simply measuring effects not stem from an excessive faith in quantitative methods ?
Does experimentation in social policies not raise ethical issues ?
Does the sum of knowledge on the impacts of targeted actions provide information on the impacts of complex interventions ?
Can the knowledge acquired be generalised to other contexts ?
Will impact evaluations be used more by decision-makers than other types of evaluation ?

All these questions are legitimate and sometimes provoke lively – and often fruitful – debate. Challenges to this method generally relate less to the impact evaluation as a new knowledge tool for development than to the sometimes excessive ambition that some attribute to it.