兰德-AEP-Data-Note-Technical-Appendix_10页_228kb

AEP Data Note Technical Appendix Summary

Core Content

This document provides a technical appendix for the American Educational Panels (AEP) Data Notes published by the RAND Corporation in 2018 and 2019. It outlines the methodology used to collect and analyze data from the American Teacher Panel (ATP) and American School Leader Panel (ASLP), which are nationally representative samples of K-12 educators.

The AEP was established in 2014 and expanded significantly during the 2016-2017 school year. The May 2017 Measurement Learning and Improvement (MLI) Survey was administered to the full ATP and ASLP samples, generating 13,839 complete responses from teachers and 4,515 from school leaders.

Main Weighting and Replication

• Weighting Variables: A total of 81 weighting variables are used to ensure that the sample reflects the national population of teachers and school leaders.
• Main Weight: Calculated by modeling response probabilities across various educator and school characteristics, then calibrating the sample to match known national demographics.
• School Characteristics: Include school level, size, urbanicity, socioeconomic status, and student demographics.
• Educator Characteristics: Include gender, race/ethnicity, and years of experience.
• Replicate Weights: 80 replicate weights are used to calculate jackknife standard errors for margin of error estimates.

Survey Data and Response Rates

• ATP Response Rate: 66% (13,839 complete responses out of 20,986 invitations).
• ASLP Response Rate: 43% (4,515 complete responses out of 10,585 invitations).
• Response Rate Definitions: A teacher response was considered complete if they answered at least 50% of the core survey, while a principal response required at least 33% completion.

Data Analysis Strategy

• Primary Estimation: Simple weighted averages are used to calculate national-level estimates and subgroup comparisons.
• Regression Model: For subgroup comparisons, a regression model is used where the survey measure is regressed on subgroup indicators:
  $$
  Y_{is} = \beta_0 + \beta_1 X_{is} + \varepsilon_{is}
  $$
  Here, $Y_{is}$ is the survey response, $X_{is}$ is a subgroup indicator, $\beta_0$ is the reference group mean, and $\beta_1$ is the differential response for the subgroup.
• Supplemental Model: A more comprehensive model is used that includes school and individual characteristics, as well as state fixed effects:
  $$
  Y_{is} = \gamma_0 + \gamma_1 X_{is} + \boldsymbol{L}{is} \gamma_2 + \boldsymbol{W}{is} \gamma_3 + \alpha_s + \omega_{is}
  $$
  This helps to control for potential confounding variables and assess the robustness of findings.

Handling of Multiple Comparisons

• Type I Error Risk: The document acknowledges the risk of false discovery when conducting multiple comparisons within similar domains.
• Benjamini-Hochberg Correction: This correction is applied to account for multiple hypothesis testing and ensure statistical significance remains robust.

Limitations

• Self-Reported Data: All findings are based on self-reported data from teachers and school leaders, which may be influenced by social desirability bias.
• Bias Correlation: Biases could be correlated with specific educator characteristics, such as the urbanicity of the school.
• Representative Nature: While the findings reflect educators' perceptions, they may not fully capture the underlying causes of observed patterns.

Key Information

• The AEP includes over 20,000 teachers and 10,000 school leaders.
• The MLI Survey covers topics such as educator preparation, working conditions, curriculum use, and student interventions.
• The survey was developed in collaboration with the Bill & Melinda Gates Foundation and includes questions from the 5Essentials Survey.
• Data are used by RAND, BMGF, and state education agencies for comparative analysis.
• The document emphasizes the use of survey weights and replicate weights to ensure accurate estimation and error calculation.
• Some states were oversampled, and for others, standard errors were aggregated by census region to maintain robustness.

Conclusion

The technical appendix provides a detailed explanation of the data collection, weighting, and analysis strategies used in the AEP Data Notes. It highlights the importance of using calibrated weights and replicate weights for accurate estimation and the application of multiple hypothesis testing corrections to mitigate the risk of false discoveries. The document also acknowledges the limitations of self-reported data and the potential for social desirability bias, while emphasizing the value of the findings in understanding educators' perspectives and working conditions.