2013年-世界发展银行全球_Pilot_Implementation_of_Statistical_Models_for_Estimation_of_the_Value-Added_of_Bulgarian_Schools_Using_National_Student_Assessment_Data_27页_1mb

Summary of the Pilot Implementation of Statistical Models for Estimating the Value-Added of Bulgarian Schools

Core Content

This report outlines the pilot implementation of statistical models to estimate the value-added of Bulgarian schools using national student assessment data. The initiative was undertaken by the World Bank in collaboration with the Bulgarian Ministry of Education, Youth and Science (MEYS) and the Center for Control and Assessment of the Quality of School Education (CKOKUO). The goal is to provide a technical analysis of the value-added approach, its potential, and its limitations within the Bulgarian education system.

The value-added measure (VAM) is defined as a statistical indicator that isolates the contribution of schools to student achievement from factors outside their control, such as socio-economic background and family characteristics. It is based on the idea that by comparing students' test scores at two different points in time, the impact of the school can be estimated after accounting for prior performance and other contextual variables.

Main Points

• Objective: To explore the feasibility of using value-added analysis to assess school performance in Bulgaria, based on national student assessment data.
• Data Sources: The pilot used data from national assessments in Grade 4 (2009) and Grade 7 (2012), covering 1918 schools and approximately 48,529 students.
• Subjects Analyzed: Bulgarian language and literature, and Mathematics, due to their consistent use across education stages.
• Student Background Variables: Limited to age, gender, language spoken at home, and school type. A questionnaire was designed to collect more detailed data but faced low response rates.
• Models Tested: Classical linear regression models and multilevel models. The multilevel model was found to be less reliable due to sensitivity to starting values and data inconsistencies.
• Key Findings:
  • Test scores from Grade 4 and 7 have a strong positive correlation.
  • Girls outperform boys in assessments.
  • Students who speak Bulgarian at home tend to perform better in both subjects.
  • Roma-speaking students have lower performance in both subjects.
  • Turkish-speaking students have no impact on math scores but a slight negative impact on Bulgarian language scores.
  • Private schools have a positive impact on student performance, likely due to higher social status.
  • Schools of type 3 and 4 (professional, sport, and arts schools) have the worst effect on performance.
• Testing Instrument Issues: The current assessment tools do not produce normally distributed results and have limited variance, which affects the accuracy of value-added estimates.
• Recommendations:
  • Conduct further analyses to address data limitations and inconsistencies.
  • Consider re-running models with higher response rate questionnaire data for a subset of schools.
  • Expand the range of student-level contextual data through surveys or administrative systems.
  • Improve the reliability and validity of the testing instruments.
  • Repeat the value-added analysis with the next student cohort to assess model stability over time.
  • Institutionalize the use of assessment data for policy and accountability purposes.

Key Information

• Scope: The pilot aimed to estimate school value-added using longitudinal data from two assessments.
• Methodology: Linear regression models were used, with the inclusion of school and student characteristics.
• Data Limitations: The dataset suffered from missing data and inconsistencies, which affected the reliability of the value-added estimates.
• Administrative Data: The Ministry provided data in three files, structured by student and school IDs, with test scores and background variables.
• Unique Identifier: The EGN (national ID number) was used to link student data across assessments.
• Policy Context: The new draft law on Preschool and School Education aims to introduce performance-based financing and accountability measures, making value-added analysis a relevant tool for policy development.

Conclusion

The pilot implementation of value-added models in Bulgaria has provided initial insights into how schools contribute to student achievement. While the models are technically feasible, the limited data and inconsistencies have constrained their accuracy. The findings suggest that value-added analysis could be a valuable tool for improving school accountability and performance, but it requires more reliable data and a robust assessment framework. The report encourages further research and institutional support to enhance the use of assessment data for educational improvement and policy decisions.