兰德-Validation-Study-of-the-TNTP-Core-Teaching-Rubric_45页_601kb

报告摘要

Summary of the Validation Study of the TNTP Core Teaching Rubric

Core Content

This report presents the findings of a multiyear validation study of the TNTP Core Teaching Rubric (TNTP Core), an observation-based measure of teachers' instruction aligned with the Common Core State Standards (CCSS). The study was commissioned by TNTP and conducted by the RAND Corporation, focusing on two key aspects: rater content expertise and predictive validity of TNTP Core scores in relation to student achievement.

The TNTP Core rubric evaluates four domains of instructional practice: Culture of Learning, Essential Content, Academic Ownership, and Demonstration of Learning, each scored on a five-point scale. Unlike traditional rubrics that focus on teacher behaviors, TNTP Core emphasizes student behaviors as the basis for scoring.

Main Findings

Correlation with Student Achievement: The relationship between TNTP Core scores and student achievement gains is modest, with correlations ranging from 0 to 0.12 for math and 0.03 to 0.21 for ELA. A correlation of 0.2 indicates that TNTP Core scores explain only 4% of the variance in value-added scores.
Scoring Consistency and Bias: Observers often disagree on ratings, and this disagreement may be influenced by their content expertise. Raters with more specialized knowledge in a subject (e.g., math specialists) tend to give more severe ratings than generalist raters.
Reliability and Uncertainty: There is significant uncertainty in TNTP Core scores due to rater variability and lesson selection bias. Even with four observations per year and two raters per lesson, the scores may not accurately represent a teacher's overall instructional practices.
Implications for Use: The study highlights the need for multiple raters and multiple observations to reduce uncertainty and improve the validity of TNTP Core scores. It also suggests that higher rater certification standards and ongoing calibration and validation processes are necessary for accurate and consistent scoring.

Key Recommendations

Raise Rater Certification Standards: The current 75/50 threshold for rater certification may be too low. The report suggests considering 90% agreement-within-one and 70% exact agreement with a master rater to improve consistency and reduce bias.
Implement Monitoring Systems: Schools and districts should build systems to monitor score quality under operational conditions. This includes post-certification calibration and validation exercises, such as scoring the same lesson video and comparing with a master rater.
Use Diverse Rater Backgrounds: To ensure a comprehensive understanding of instructional practices, teachers should be observed by both content experts and generalists. This diversity helps to reduce the impact of individual rater biases.
Collect Additional Validity Evidence: Validity and reliability are dynamic processes, not static properties. Schools should consider collecting additional evidence such as content knowledge assessments, outcome-based data, and response process data to support the use of TNTP Core in various contexts.

Study Design Overview

The study involved volunteer teachers from over 20 districts and charter school networks in eight states.
Teachers were observed three times per year, with video recordings of their instruction.
Two samples were used: the TNTP sample (volunteer teachers) and the MET sample (teachers from the Measures of Effective Teaching study).
Value-added scores were used as an alternative measure of instructional quality, capturing teachers' contributions to student performance in math and ELA.

Limitations

The study is based on a small sample of volunteer teachers, which may limit the generalizability of the findings.
The research design focuses on three aspects of validity: scoring consistency, generalization of scores, and predictive validity.
It does not cover all aspects of validity important for the implementation of TNTP Core, such as longitudinal validity or impact on teacher development.

Conclusion

The study underscores the modest predictive validity of TNTP Core scores and the challenges in achieving reliable and consistent ratings. It calls for improved rater training and certification, ongoing monitoring of score quality, and the use of multiple raters and observations to enhance the utility of TNTP Core in evaluating and improving teacher instruction.

展开完整摘要