Examining Construct Invariance of the Cross Cultural Assessment Data Using Generalizability Theory

When provided with the reliability or generalizability information for assessment results, the results are assumed to be equally reliable for all subgroups of the population. However, this assumption may not hold when... [ view full abstract ]

When provided with the reliability or generalizability information for assessment results, the results are assumed to be equally reliable for all subgroups of the population. However, this assumption may not hold when systematic differences exist among groups of examinees. The purpose of this study is to examine group differences in the reliability of international assessment results using generalizability analyses. Specifically, the central research question guided this study was whether we can safely assume that the test results are reliable equally across diverse student groups?  

The data were responses from a total of 1,626 students of four countries on TIMSS 2011 8th grade mathematics achievement and attitudes scales. The sample consists of Korea (N = 371) and United States (N = 656) and the data were the sample of students who took the first booklet of the test. This study considered four separate generalizability analyses: (1) a univariate p × i analysis that ignored both the content and cognitive domain facets, (2) a multivariate analysis that took cognitive domains into consideration as a fixed facet but ignored the content facet, (3) a univariate p × (i:h) analysis that took into account the content facet but ignored the cognitive facet, and (4) a multivariate  analysis that took into account both content and cognitive domain facets.

Results for all designs consistently suggest a decease of about .05 in both generalizability and dependability indices for the United States group. Thus, our study conveys the important practical message that the assessment results are not equally reliable for all subpopulations. The results from this study will help test administrators and practitioners appropriately evaluate international students’ assessment results with respect to fairness and validity.