Impact evaluation: Wikis

Advertisements
  

Note: Many of our articles have direct quotes from sources you can cite, within the Wikipedia article! This article doesn't yet, but we're working on it! See more info or our list of citable articles.

Encyclopedia

From Wikipedia, the free encyclopedia

Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones.[1] In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would participants’ well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, “a comparison between what actually happened and what would have happened in the absence of the intervention.”[2]

Impact Evaluation helps us to answer key questions for evidence-based policy making: what works, what doesn’t, where, why and for how much? It has received increasing attention in policy making in recent years in both Western and developing country contexts.[3] It is an important component of the armory of evaluation tools and approaches and integral to global efforts to improve the effectiveness of aid delivery and public spending more generally in achieving outcomes. Originally more oriented towards evaluation of social sector programs in developing countries, notably conditional cash transfers, impact evaluation is now being increasingly applied in other areas such as the agriculture, energy and transport.

Contents

Counterfactual Evaluation designs

Counterfactual analysis enables evaluators to attribute cause and effect between interventions and outcomes. The ‘counterfactual’ measures would have happened to beneficiaries in the absence of the intervention, and impact is estimated by comparing counterfactual outcomes to those observed under the intervention. The key challenge in Impact Evaluation is that the counterfactual cannot be directly observed, but must be approximated with reference to a comparison group. There are a range of accepted approaches to determining an appropriate comparison group for counterfactual analysis, using either prospective (ex ante) or retrospective (ex post) evaluation design. Prospective evaluations begin during the design phase of the intervention, involving collection of baseline and end-line data from intervention beneficiaries (the ‘treatment group’) and non-beneficiaries (the ‘comparison group’), and may also involve selection of individuals or communities into treatment and comparison groups. Retrospective evaluations are usually conducted after the implementation phase, and may exploit existing survey data, although the best evaluations will collect data as close to baseline as possible, to ensure comparability of intervention and comparison groups.

There are five key principles relating to internal validity (study design) and external validity (generalizability) which rigorous Impact Evaluations should address: confounding factors, selection bias, spillover effects, contamination, and impact heterogeneity.[4]

Confounding occurs where certain factors, typically relating to socio-economic status, are correlated with both exposure to the intervention and, independent of exposure, are causally related to the outcome of interest. Confounding factors are therefore alternate explanations for an observed (possibly spurious) relationship between intervention and outcome.

Selection bias occurs where intervention participants are non-randomly drawn from the beneficiary population, and the criteria determining selection are correlated with outcomes. Unobserved factors, which are associated with access to or participation in the intervention, and are causally related to the outcome of interest, may lead to a spurious relationship between intervention and outcome if unaccounted for. Self-selection occurs where, for example, more able or organized individuals or communities, who are more likely to have better outcomes of interest, are also more likely to participate in the intervention. Endogenous program selection occurs where individuals or communities are chosen to participate because they are seen to be more likely to benefit from the intervention. Ignoring confounding factors can lead to a problem of omitted variable bias. In the special case of selection bias, the endogeneity of the selection variables can cause simultaneity bias.

Spillover (referred to as contagion in the case of experimental evaluations) occurs when members of the comparison (control) group are affected by the intervention. Contamination occurs when the comparison group has access to another intervention which also affects the outcome of interest.

Impact heterogeneity refers to differences in impact due by beneficiary type and context. External validity is crucial for any lessons learning from Impact Evaluation, and rigorous Impact Evaluations will assess both the extent to which different groups (e.g. the disadvantaged) benefit from an intervention as well as the potential effect of context on impact.

Impact evaluation designs are identified by the type of methods used to generate the counterfactual and can be broadly classified into three categories – experimental, quasi-experimental and non-experimental designs – that vary in feasibility, cost, involvement during design or after implementation phase of the intervention, and degree of selection bias. White (2006)[5] and Ravallion (2008) [6] discusses alternate Impact Evaluation approaches.

Under experimental evaluations the treatment and comparison groups are selected randomly and isolated both from the intervention, as well as any interventions which may affect the outcome of interest. These evaluation designs are referred to as randomized control trials (RCTs). In experimental evaluations the comparison group is called a control group. When randomization is implemented over a sufficiently large sample with no contagion by the intervention, the only difference between treatment and control groups on average is that the latter does not receive the intervention. Random sample surveys, in which the sample for the evaluation is chosen on a random basis, should not be confused with experimental evaluation designs, which require the random assignment of the treatment.

The experimental approach is often held up as the ‘gold standard’ of evaluation, and it is the only evaluation design which can conclusively account for selection bias in demonstrating a causal relationship between intervention and outcomes. Randomization and isolation from interventions are seldom practicable in the realm of social policy, and may also be ethically difficult to defend,[7] although there may be opportunities to utilize natural experiments. Bamberger and White (2007)[8] highlight some of the limitations to applying RCTs to development interventions. Methodological critiques have been made by Scriven (2008)[9] on account of the biases introduced since social interventions cannot be triple blinded, and Deaton (2009)[10] has pointed out that in practice analysis RCTs falls back on the regression-based approaches they seek to avoid, and so are subject to the same potential biases. Other problems include the often heterogeneous and changing contexts of interventions, logistical and practical challenges, difficulties with monitoring service delivery, access to the intervention by the comparison group and changes in selection criteria and/or intervention over time. Thus, it is estimated that RCTs are only applicable to 5 per cent of development finance.[11]

  • Quasi-experimental design

Quasi-experimental approaches can remove bias arising from selection on observables and, where panel data are available, time invariant unobservables. Quasi-experimental methods include matching, differencing, instrumental variables and the pipeline approach, and are usually carried out by multivariate regression analysis.

If selection characteristics are known and observed then they can be controlled for to remove the bias. Matching involves comparing program participants with non-participants based on observed selection characteristics. Propensity score matching (PSM) uses a statistical model to calculate the probability of participating on the basis of a set of observable characteristics, and matches participants and non-participants with similar probability scores. Regression discontinuity design exploits a decision rule as to who does and does not get the intervention to compare outcomes for those just either side of this cut-off.

Difference-in-differences or double differences, which use data collected at baseline and end-line for intervention and comparison groups, can be used to account for selection bias with under the assumption that unobservable factors determining selection are fixed over time (time invariant).

Instrumental variables estimation accounts for selection bias by modelling participation using factors (‘instruments’) that are correlated with selection but not the outcome, thus isolating the aspects of program participation which can be treated as exogenous.

The pipeline approach (stepped-wedge design) uses beneficiaries already chosen to participate in a project at a later stage as the comparison group. The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest. However, in practice, it cannot be guaranteed that treatment and comparison groups are comparable and some method of matching will need to be applied to verify comparability.

  • Non-experimental design

Non-experimental Impact Evaluations are so-called because they do not involve a comparison group which does not have access to the intervention. The method used in non-experimental evaluation is interrupted time-series, which compares intervention groups before and after implementation (pre-test post-test) or simply post-test analysis of the intervention group. This is the weakest evaluation design, because in order to show a causal relationship between intervention and outcomes convincingly, the evaluation must demonstrate that any likely alternate explanations for the outcomes are irrelevant. However, there remain applications to which this design is relevant, for example in calculating time-savings from an intervention which improves access to amenities.

Estimation methods

Estimation methods broadly follow evaluation designs. Different designs require different estimation methods to measure changes in well-being from the counterfactual. In experimental and quasi-experimental evaluation, the estimated impact of the intervention is calculated as the difference in mean outcomes between the treatment group (those receiving the intervention) and the control or comparison group (those who don’t). The single difference estimator compares mean outcomes at end-line and is valid where treatment and control groups have the same outcome values at baseline. The difference-in-difference (or double difference) estimator calculates the difference in the change in the outcome over time for treatment and comparison groups, thus utilizing data collected at baseline for both groups and a second round of data collected at end-line, after implementation of the intervention, which may be years later.

Impact Evaluations which compare average outcomes in the treatment group, irrespective of beneficiary participation (also referred to as ‘compliance’ or ‘adherence’), to outcomes in the comparison group are referred to as intention-to-treat (ITT) analyses. Impact Evaluations which compare outcomes among beneficiaries who comply or adhere to the intervention in the treatment group to outcomes in the control group are referred to as treatment-on-the-treated (TOT) analyses. ITT therefore provides a lower-bound estimate of impact, but is arguably of greater policy relevance than TOT in the analysis of voluntary programs.[12]

Debates in Impact Evaluation

While there is agreement on the importance of Impact Evaluation, and a consensus is emerging around the use of counterfactual evaluation methods, there has also been widespread debate in recent years on both the definition of Impact Evaluation and the use of appropriate methods (see White 2009[13] for an overview).

Advertisements

Definitions of Impact Evaluation

The International Initiative for Impact Evaluation (3ie) defines rigorous Impact Evaluations as: ”analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context”.[14]

According to the World Bank’s DIME Initiative, “Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program. Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental designs”.[15]

Similarly, according to the US Environmental Protection Agency impact evaluation is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program.[16]

According to the World Bank's Independent Evaluation Group (IEG), impact evaluation is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project.[17]

Impact Evaluation has been defined differently over the past few decades.[18] Other interpretations of Impact Evaluation include:

  • An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, or a process evaluation which focuses on implementation;
  • An evaluation carried out some time (five to ten years) after the intervention has been completed so as to allow time for impact to appear; and
  • An evaluation considering all interventions within a given sector or geographical area.

Common definitions of ‘impact’ used in evaluation generally refer to the totality of longer-term consequences associated with an intervention on quality-of-life outcomes. For example, the Organization for Economic Cooperation and Development’s Development Assistance Committee (OECD-DAC) defines impact as the “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.[19] A number of international agencies have also adopted this definition of impact. For example, UNICEF defines impact as “The longer term results of a program – technical, economic, socio-cultural, institutional, environmental or other – whether intended or unintended. The intended impact should correspond to the program goal.”[20] Similarly, Evaluationwiki.org defines impact evaluation as an evaluation that looks beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.[21]

Technically, an evaluation could be conducted to assess ‘impact’ as defined here without reference to a counterfactual. However, this would be more appropriately referred to as outcome monitoring. The NONIE Guidelines on Impact Evaluation[22] adopt the OECD-DAC definition of impact while referring to the techniques used to attribute impact to an intervention as necessarily based on counterfactual analysis.

Methodological debates

There is intensive debate in academic circles around the appropriate methodologies for Impact Evaluation, between proponents of experimental methods on the one hand and proponents of more general methodologies on the other. William Easterly has referred to this as ‘The Civil War in Development economics’. Proponents of experimental designs, sometimes referred to as ‘randomistas’,[23] argue randomization is the only means to ensure unobservable selection bias is accounted for, and that building up the flimsy experimental evidence base should be developed as a matter of priority.[24] In contrast, others argue that randomized assignment is seldom appropriate to development interventions and even when it is, experiments provide us with information on the results of a specific intervention applied to a specific context, and little of external relevance.[25] There has been criticism from evaluation bodies and others that some donors and academics over-emphasize favoured methods for Impact Evaluation[26], and that this may in fact hinder learning and accountability.[27]

Theory-Based Impact Evaluation

While knowledge of effectiveness is vital, it is also important to understand the reasons for effectiveness and the circumstances under which results are likely to be replicated. In contrast with ‘black box’ Impact Evaluation approaches, which only report mean differences in outcomes between treatment and comparison groups, Theory-Based Impact Evaluation involves mapping out the causal chain from inputs to outcomes and impact and testing the underlying assumptions.[28] [29] Most interventions within the realm of public policy are of a voluntary, rather than coercive (legally required) nature. In addition, interventions are often active rather than passive, requiring a greater rather than lesser degree of participation among beneficiaries and therefore behavior change as a pre-requisite for effectiveness. Public policy will therefore be successful to the extent that people are incentivized to change their behaviour favourably. A Theory-Based approach enables policy-makers to understand the reasons for differing levels of program participation (referred to as ‘compliance’ or ‘adherence’) and the processes determining behavior change. Theory-Based approaches use both quantitative and qualitative data collection, and the latter can be particularly useful in understanding the reasons for compliance and therefore whether and how the intervention may be replicated in other settings. Methods of qualitative data collection include focus groups, in-depth interviews, participatory rural appraisal (PRA) and field visits, as well as reading of anthropological and political literature.

White (2009b)[30] advocates more widespread application of a theory-based approach to impact evaluation as a means to improve policy relevance of Impact Evaluations, outlining six key principles of the theory-based approach: 1. Map out the causal chain (program theory) which explains how the intervention is expected to lead to the intended outcomes, and collect data to test the underlying assumptions of the causal links. 2. Understand context, including the social, political and economic setting of the intervention. 3. Anticipate heterogeneity to help in identifying sub-groups and adjusting the sample size to account for the levels of disaggregation to be used in the analysis. 4. Rigorous evaluation of impact using a credible counterfactual (as discussed above). 5. Rigorous factual analysis of links in the causal chain. 6. Use mixed methods (a combination of quantitative and qualitative methods).

Examples of impact evaluations

While experimental Impact Evaluation methodologies have been used to assess nutrition and water and sanitation interventions in developing countries since the 1980s, the first, and best known, application of experimental methods to a large-scale development program is the evaluation of the Conditional Cash Transfer (CCT) program Progresa (now called Oportunidades) in Mexico, which examined a range of development outcomes, including schooling, immunization rates and child work.[31] [32] CCT programs have since been implemented by a number of governments in Latin America and elsewhere, and a report released by the World Bank in February 2009 examines the impact of CCTs across twenty countries.[33]

More recently, Impact Evaluation has been applied to a range of interventions across social and productive sectors. 3ie has launched an online database of impact evaluations covering studies conducted in low- and middle income countries. Other organisations publishing Impact Evaluations include Innovations for Poverty Action, the World Bank's DIME Initiative and NONIE. The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years.[34]

Organizations promoting Impact Evaluation of Development Interventions

In 2006, the Evaluation Gap Working Group[35] argued for a major gap in the evidence on development interventions, and in particular for an independent body to be set up to plug the gap by funding and advocating for rigorous Impact Evaluation in low- and middle-income countries. The International Initiative for Impact Evaluation (3ie) was set up in response to this report. 3ie seeks to improve the lives of poor people in low- and middle-income countries by providing, and summarizing, evidence of what works, when, why and for how much. 3ie operates a grant program, financing impact studies in low- and middle-income countries and synthetic reviews of existing evidence updated as new evidence appears, and supports quality impact evaluation through its quality assurance services.

A number of additional organizations have been established to promote impact evaluation globally, including Innovations for Poverty Action, the World Bank’s Development Impact Evaluation (DIME) Initiative and the Network of Networks on Impact Evaluation (NONIE).

Systematic Reviews of Impact evidence

A range of organizations are working to coordinate the production of Systematic Reviews. Systematic reviews aim to bridge the research-policy divide by assessing the range of existing evidence on a particular topic, and presenting the information in an accessible format. Like rigorous Impact Evaluations, they are developed from a study Protocol which sets out a priori the criteria for study inclusion, search and methods of synthesis. Systematic reviews involve five key steps: determination of interventions, populations, outcomes and study designs to be included; searches to identify published and unpublished literature, and application of study inclusion criteria (relating to interventions, populations, outcomes and study design), as set out in study Protocol; coding of information from studies; presentation of quantitative estimates on intervention effectiveness using forest plots and, where interventions are determined as appropriately homogeneous, calculation of a pooled summary estimate using meta-analysis; finally, systematic reviews should be updated periodically as new evidence emerges. Systematic reviews may also involve the synthesis of qualitative information, for example relating to the barriers to, or facilitators of, intervention effectiveness.

Organizations supporting the production of systematic reviews include the Cochrane Collaboration, which has been coordinating Systematic Reviews in the medical and public health fields since 1993, and publishes the Cochrane Handbook which is definitive Systematic Review methodology guide. In addition, the Campbell Collaboration has coordinated the production of Systematic Reviews of social interventions since 2000, and the International Initiative for Impact Evaluation (in partnership with the Campbell Collaboration) is funding Systematic Reviews of social programs in developing countries. Other organizations supporting Systematic Reviews include the Institute of Education’s EPPI-Centre and the University of York’s Centre for Reviews and Dissemination.

The body of evidence from Systematic Reviews is large and available through various online portals including the Cochrane library, the Campbell library, and the Centre for Reviews and Dissemination. The available evidence from Reviews of development interventions in low- and middle-income countries is being built up by organisations such as the International Initiative for Impact Evaluation's synthetic reviews programme.

Sources and external links

See also

References

  1. ^ World Bank Poverty Group on Impact Evaluation, accessed on January 6, 2008
  2. ^ White, H. (2006) Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank, World Bank, Washington, D.C., p. 3
  3. ^ Briceno, B. and Gaarder, M. (2009) Institutionalizing Evaluation: A review of international experience, DFID/3ie, New Delhi
  4. ^ International Initiative for Impact Evaluation (3ie), Principles for Impact Evaluation
  5. ^ White, H. (2006) Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank, World Bank, Washington, D.C.
  6. ^ Ravallion, M. (2008) Evaluating Anti-Poverty Programs
  7. ^ Ravallion, M. (2009) Should the Randomistas Rule? The Economists' Voice, Volume 6, Number 2
  8. ^ Bamberger, M. and White, H. (2007) Using Strong Evaluation Designs in Developing Countries: Experience and Challenges, Journal of MultiDisciplinary Evaluation, Volume 4, Number 8, 58-73
  9. ^ Scriven (2008) A Summative Evaluation of RCT Methodology: & An Alternative Approach to Causal Research, Journal of MultiDisciplinary Evaluation, Volume 5, Number 9, 11-24
  10. ^ Deaton (2009) Instruments of Development: Randomization in the Tropics, and the Search for the Elusive Keys to Economic Development, NBER Working Paper No. w14690
  11. ^ Bamberger, M. and White, H. (2007) Using Strong Evaluation Designs in Developing Countries: Experience and Challenges, Journal of MultiDisciplinary Evaluation, Volume 4, Number 8, 58-73
  12. ^ Bloom, H. (2006) The core analytics of randomized experiments for social research. MDRC Working Papers on Research Methodology. MDRC, New York
  13. ^ White, H. (2009) Some reflections on current debates in impact evaluation, Working paper 1, International Initiative for Impact Evaluation, New Delhi
  14. ^ International Initiative for Impact Evaluation (3ie) (2008) Principles for Impact Evaluation. 3ie, New Delhi
  15. ^ World Bank (n.d.) The Development IMpact Evaluation (DIME) Initiative, Project Document, World Bank, Washington, D.C.
  16. ^ US Environmental Protection Agency Program Evaluation Glossary, accessed on January 6, 2008
  17. ^ World Bank Independent Evaluation Group, accessed on January 6, 2008
  18. ^ White, H. (2006) Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank, World Bank, Washington, D.C.
  19. ^ OECD-DAC (2002) Glossary of Key Terms in Evaluation and Results-Based Management Proposed Harmonized Terminology, OECD, Paris
  20. ^ UNICEF (2004) UNICEF Evaluation Report Standards, Evaluation Office, UNICEF NYHQ, New York
  21. ^ Evaluation Definition: What is Evaluation? - EvaluationWiki
  22. ^ Leeuw, F. and Vaessen, J. (2009) Impact Evaluations and Development: NONIE Guidance on Impact Evaluation, World Bank, Washington, D.C.
  23. ^ Ravallion, M. (2009) Should the Randomistas Rule? The Economists' Voice, Volume 6, Number 2
  24. ^ Banerjee, A. V. (2007) ‘Making Aid Work’ Cambridge, Boston Review Book, MIT Press, MA
  25. ^ Bamberger, M. and White, H. (2007) Using Strong Evaluation Designs in Developing Countries: Experience and Challenges, Journal of MultiDisciplinary Evaluation, Volume 4, Number 8, 58-73
  26. ^ http://www.europeanevaluation.org/download/?noGzip=1&id=1969403 EES Statement on the importance of a methodologically diverse approach to impact evaluation
  27. ^ http://www.odi.org.uk/resources/odi-publications/opinions/127-impact-evaluation.pdf The 'gold standard' is not a silver bullet for evaluation
  28. ^ White, H. (2009b) Theory-based impact evaluation: Principles and practice, Working Paper 3, International Initiative for Impact Evaluation, New Delhi
  29. ^ Leeuw, F. and Vaessen, J. (2009) Impact Evaluations and Development: NONIE Guidance on Impact Evaluation, World Bank, Washington, D.C.
  30. ^ White, H. (2009b) Theory-based impact evaluation: Principles and practice, Working Paper 3, International Initiative for Impact Evaluation, New Delhi
  31. ^ Gertler, P. (2000) Final Report: The Impact of PROGRESA on Health. International Food Policy Research Institute, Washington, D.C.
  32. ^ Behrman, J., Sengupta, P. and Todd, P. (2002) Progressing Through Progresa: An Impact assessment of a school subsidy experiment in Mexico.
  33. ^ Fiszbein, A. and Schady, N. (2009) Conditional Cash Transfers: Reducing present and future poverty: A World Bank Policy Research Report, World Bank, Washington, D.C.
  34. ^ Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank, 2006
  35. ^ Savedoff, W., Levine, R. and Birdsall, N. (2006), ‘When will we ever learn: Improving lives through impact evaluation’, Report of the Evaluation Gap Working Group, May, Center for Global Development, Washington, D.C.

Advertisements






Got something to say? Make a comment.
Your name
Your email address
Message