2014年-世界发展银行全球_Evaluation_of_Development_Programs___Randomized_Controlled_Trials_or_Regressions__14页_146kb

Summary of "Evaluation of Development Programs: Randomized Controlled Trials or Regressions?"

Core Content

This paper by Chris Elbers and Jan Willem Gunning discusses the evaluation of complex development programs that involve multiple interventions, arguing that traditional methods such as Randomized Controlled Trials (RCTs) may not be suitable for assessing the total program effect (TPE) due to the correlation between program assignment and individual treatment effects, as well as the difficulty of controlling for interactions and spillovers.

Main Points

• Program Evaluation Complexity: Programs often involve multiple interventions, making it difficult to evaluate them using separate component evaluations. The TPE is proposed as an extension of the Average Treatment Effect on the Treated (ATET), which accounts for treatment heterogeneity and the correlation between program participation and individual treatment effects.

• RCT Limitations: RCTs are less suitable for program evaluation in real-world settings where program assignment is not centrally controlled. They are also less effective when there are spillover effects or when the treatment is not uniformly applied.

• Regression Techniques: Regression methods can be used to estimate the TPE using observational data from a representative sample. This involves modeling the relationship between outcome variables and the interventions, as well as other control variables, and accounting for treatment heterogeneity through interactions and adjustments in the model.

• Modeling the TPE: The TPE is defined as the expected value of the product of individual treatment effects and the program intervention. The paper presents a regression model that allows for the estimation of the TPE by incorporating interactions between treatment variables and control variables.

• Spillover Effects: Spillover effects, such as those from neighboring villages or regions, can affect the outcomes of the program. The paper suggests that including proxies for these spillovers in the regression model can help identify the full program impact.

• RCT vs Regression: The paper argues that while RCTs can estimate the TPE, they are less powerful and may not capture the actual program assignment process. Additionally, RCTs may fail to account for the correlation between treatment variables and control variables when the program is implemented at a lower administrative level.

Key Information

• TPE Formula:
  $$
  \mathrm{TPE} = E \beta_i (P_{i1} - P_{i0})
  $$
  This formula accounts for the average treatment effect on the treated, considering individual heterogeneity.

• Regression Model:
  $$
  \Delta y_i = \alpha \Delta X_i + \beta_i \Delta P_i + \gamma + \Delta \varepsilon_i
  $$
  The model is extended to include interactions and spillovers, as shown in:
  $$
  \Delta y_i = \gamma + \theta_1 \Delta X_i + \theta_2 \Delta P_i + \theta_3 \Delta X_i \otimes \Delta P_i + \theta_4 \Delta P_i \otimes \Delta P_i + \omega_i
  $$

• Empirical Example: The paper uses data from a health insurance program in Vietnam to illustrate the TPE estimation. The data includes changes in health and economic outcomes across households over two survey periods (1992-3 and 1997-8).

• Results: A naive difference-in-differences approach yields smaller and less significant effects compared to the TPE estimation, which accounts for heterogeneity and selection on the gain.

Structure of the Paper

1. Introduction of TPE: The paper introduces the concept of the total program effect (TPE) as an extension of the average treatment effect on the treated (ATET), suitable for complex interventions.
2. Correlation Between P and X: It discusses how correlations between program variables and control variables can affect the estimation of the TPE and how regression techniques can be adapted to account for this.
3. Spillover Effects: The paper addresses spillover effects, which are common in real-world program evaluations, and suggests how to incorporate them into the regression model.
4. RCT vs Regression: A comparison between RCTs and regression methods is made, highlighting the limitations of RCTs in capturing the full program impact.
5. Empirical Example: The TPE is estimated using the Vietnam health insurance program data, showing the advantages of the regression approach over naive methods.
6. Conclusion: The paper concludes that regression methods are more appropriate for evaluating complex development programs, especially when there are interactions, spillovers, or non-random assignment of interventions.

Key Takeaways

• TPE is more comprehensive: It accounts for treatment heterogeneity and the correlation between treatment and control variables.
• RCTs have limitations: They are less effective when the program assignment is not random and when there are spillover effects.
• Regression methods are preferred: They allow for a more accurate estimation of the total program effect by considering the real-world context of program implementation and individual differences in treatment response.