The Mathematical Quality of Instruction (MQI) in Kindergarten: An Evaluation of the Stability of the MQI Using Generalizability Theory

Research Findings: We evaluated the score stability of the Mathematical Quality of Instruction (MQI), an observational measure of mathematics instruction. Three raters each scored, independently, 100 video-recorded lessons taught by 20 kindergarten teachers in the spring. Using generalizability theory analyses, we decomposed the MQI’s score stability into potential sources of variation (teachers, lessons, raters, and their interactions). The 13-item (3-domain) Ambitious Mathematics Instruction scale and the Whole Lesson scale each explained about one third of the variance attributed to differences in the main construct of interest (teachers’ instructional strategies). The MQI’s Errors and Imprecision scale was not relevant at the kindergarten level; there were virtually no errors and/or ambiguities observed across the 100 mathematics lessons. In a series of decision studies, we examined improvements in reliability with combinations of up to 6 raters and 8 lessons. Only the Richness of Mathematics domain scores and the Whole Lesson scores achieved acceptable reliabilities. Practice or Policy: The findings have important implications for the use of observation measures to document teachers’ mathematics practices in the early years of school.