An intelligent tutoring system-style assessment software that diagnoses the underlying causes of students’ mathematical mistakes
Ivy Liang, John Leddo
Traditional large-scale and high-stakes assessments have focused largely on whether test takers give the correct or incorrect answers to questions. Early instructional software followed this paradigm. The introduction of intelligent tutoring systems (ITSs) led to an emphasis on discovering where students were making mistakes and explaining the mistakes through matching them to pre-defined error catalogs. The deficiencies with this approach were an emphasis on identifying only procedural mistakes and not validating whether the matched errors were, in fact, the true causes of mistakes. The present paper describes an ITS-style assessment software that diagnoses causes of errors by assessing underlying and pre-requisite concepts a student needs to solve a problem. The assessments focus on a variety of knowledge types: abstract concepts, procedures, and ability to apply concepts to problems. The software even assesses whether a mistake was caused by carelessness or mistyping information from the problem. The software was evaluated by comparing its agreement in diagnosing causes of students’ mistakes with that of experienced teachers. Results showed that the software’s agreement percentage was in the 90s and statistically equal to that of experienced teachers’ inter-rater agreement.