For well over 100 years, factor analysis has played a prominent role in the development of intelligence tests and intelligence theory. As noted by Keith, Caemmerer, and Reynolds (2016), “Given this link, evidence that factor... [ view full abstract ]
For well over 100 years, factor analysis has played a prominent role in the development of intelligence tests and intelligence theory. As noted by Keith, Caemmerer, and Reynolds (2016), “Given this link, evidence that factor analyses conducted with intelligence measures are incorrect or misguided would have important implications for intelligence theory and the development and use of intelligence tests” (p. 38).
In a seminal critique, Frazier and Youngstrom (FY; 2007) questioned whether test developers and researchers were identifying more factors in factor analysis than are adequately specified and concluded that many contemporary tests may be over-factored. Keith, Caemmerer, & Reynolds (2016) challenged the results and conclusions of the FY study and argued that conventional MAP and PA procedures should not be regarded as a “gold standard” for determining the number of factors to retain in EFA and that the FY study should not be used as evidence that intelligence tests are over-factored. To date, these issues remain unresolved.
Since the publication of the FY study over a decade ago, virtually every major intelligence battery has been revised and several new tests have been developed. Accordingly, the purpose of the current study is to provide an update and extension to FY in order to evaluate the number of factors measured by current and previous editions of cognitive ability tests using a series of EFA/CFA decision rules. It is believed that the present results will be instructive for comparing the number of factors suggested for retention by empirical decision rules (old and new) to the number of factors proposed by test authors. Additionally, as this is the first study to systematically apply PA-PAF to actual cognitive data, it is believed that the results will also be useful for determining whether that indicator is more accurate at recovering viable factors in intelligence test data.
The present study evaluated covariance matrices obtained for 27 intelligence test batteries using 11 different EFA/CFA factor retention criteria (e.g., MAP, PA-PCA, PA-PAF, CFI, TLI, RMSEA). First, the original data from the FY study were obtained and evaluated to help calibrate the present findings. Once the FY results/procedures were replicated, these analyses were applied to the 12 tests that have been published since 2007. Additionally, descriptive information for each test battery was obtained to evaluate historical trends in test size, number of factors measured, and subtests/subscales to factor ratios.
Results continue to support a historical increase in the size and number of factors measured by intelligence tests. However, a leveling effect was observed in tests published more recently suggesting that this trend may have reached an apex. For all tests, agreement among the various criteria were low (ICC = .43) complicating the identification of a “gold standard” method. Consistent with the simulation results furnished by Keith et al. (2016), more conservative criteria such as MAP and PA-PCA appeared to under-factor when compared to other criteria. However, the alignment between PA-PAF and CFA results was inconsistent with publisher theory. Given the lack of consistency between methods, over-reliance on empirical extraction criteria may be problematic. As a consequence, researchers are encouraged to carefully evaluate the alignment between local fit and theory when explicating structural solutions for cognitive test data.
