APPLICATION OF THE IRT AND TRT MODELS TO A READING COMPREHENSION TEST

Abstract

The purpose of the present study is to apply the item response theory (IRT) and testlet response theory (TRT) models to a reading comprehension test. This study applied the TRT models and the traditional IRT model to a seventh-grade reading comprehension test (n = 8,815) with eight testlets. These three models were compared to determine the best model for a testlet-based reading comprehension assessment. The goodness-of-fit indices such as -2 log likelihood, Akaike information criterion, and Bayesian information criterion were utilized as model comparison indices. The standardized local dependence X2 statistic was computed for a comparison of local dependence among the three different models. Scatter plots were obtained to evaluate parameter-estimation consistency among models. Correlations and mean differences between the estimated parameters were also examined to detect and quantify the magnitude of inaccuracy due to the use of a worse-fitting model. Finally, items were evaluated based on the item parameters from the TRT models and compared to the results from the Coh Metrix.

According to the three goodness-of-fit indices, the bi-factor model was the best-fitting model and the testlet-effects model was the second best-fitting model among the three models, showing a superiority of the TRT models for the testlet-based assessments compared to the more traditional IRT model. The standardized local dependence X2 statistic revealed significant local dependencies when the traditional IRT model was applied to the data, while the application of the TRT models resolved the problematic local dependencies. Items requiring the comprehension ability of context-related vocabulary showed high local dependencies when the 3-plm was applied. Comparisons of the item and person parameters and their standard error estimates showed a slight underestimation and less stable standard errors of the item discrimination parameters under the locally dependent condition. The results of the item parameters from the TRT models supported the advantages of the TRT models in selecting good items for accurate assessments. This study provided replicable evidence that the TRT models are crucial for testlet-based reading comprehension assessments. For future study, various comprehension tests containing more questions for each testlet as well as smaller sample sizes might be useful.