Tilburg University
Psychometric problems with the method of correlated vectors applied to item scores (including some nonsensical results)
Wicherts, J.M. Published in: Intelligence DOI: 10.1016/j.intell.2016.11.002 Publication date: 2017 Document Version
Peer reviewed version
Link to publication in Tilburg University Research Portal
Citation for published version (APA):
Wicherts, J. M. (2017). Psychometric problems with the method of correlated vectors applied to item scores (including some nonsensical results). Intelligence, 60, 26–38. https://doi.org/10.1016/j.intell.2016.11.002
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
during the testing program (under a 20-minute time limit). Appendix B reports sensitivity analyses for both the STAI/STAS and 5PFT samples that addressed whether results were sensitive when using another way of dichotomizing the Likert scales. Because the items in the SPM are ordered in five sets of a dozen items that are each ordered in difficulty, I ordered the STAI/STAS and 5PFT items accordingly on the basis of independent samples. To get those independent samples, I used the total samples (N = 528 for the STAI/STAS and N = 6776 for the 5PFT), ordered cases on the basis of a random number between 0 and 1, and selected the first half of both the 5PFT sample and the STAI/STAS sample. These independent samples were composed of 276 and 3346 respondents for the STAI/STAS and 5PFT, respectively. I used these independent samples to order the items in both new scales in terms of difficulty. Specifically, I first determined the dichotomized item’s p-values for both scales in these independent samples and subsequently rank-ordered 12 items in terms of p-values in each of five sets of 12 items. I then rank-ordered these five sets according to the overall sum score, with easiest items in the first set of 12 items and the most difficult items in the last set of 12 items. In this way, the rank-order of the 60 items in the newly formed scales mimicked the rank-order of the items in terms of difficulty in the SPM. Subsequently, I used the remaining respondents to conduct the MCV analyses. The final STAI/STAS sample included 252 respondents and the final 5PFT sample included 3330 respondents. Because the item orderings were based on separate (independent) samples, they can be considered a design feature of the two new 60-item scales measuring anxiety and anger (STAI/STAS) and personality (5PFT). The p-values and the item total correlations from the analysis samples are given in Tables A1 and A2 in Appendix A. All the raw data is also uploaded to the OSF page accompanying the
Table 1
Guttman scale item parameters and classical test theory item statistics for five items in three groups as depicted in Figure 1 showing the sensitivity of CTT statistics to latent ability levels.
Low Middle High phi coefficients
DIFF p itc p itc p itc L-H M-H L-M
Table 2.
Correlations between item-total correlations across six samples that took the SPM and the mean sum score on the SPM in each sample
Roma Libyan 1 Libyan2 African Indian White M
Table 3.
Origins, references, and descriptive statistics for samples used to test Jensen Effects with MCV
Sample Origin N M rel. age
Table 4.
MCV correlations for the 28 group comparisons, including those in the Dutch samples based on the STAI/STAS or the 5PFT instead of the SPM.
Roma STAI Libyan 2nd Libyan st. African 5PFT Indian White
Table 5.
MCV correlations for the 28 group comparisons, including those in the Dutch samples based on the STAI/STAS or the 5PFT instead of the SPM in which the MCV correlations were based on loadings from Lynn et al. (2004) or the 5PFT sample.
Roma STAI Libyan 2nd Libyan st. African 5PFT Indian White
Appendix A Additional Tables
Table A1
Item statistics of the STAI and STAS (N=252).
Item p ITC Item p ITC Item p ITC
stai26 0.92 0.41 stai18 0.55 0.42 stas8 0.59 0.23
stai35 0.82 0.48 stai17 0.54 0.56 stai38 0.48 0.54
stai27 0.78 0.60 stai14 0.46 0.49 stai40 0.54 0.49
stai29 0.68 0.51 stai13 0.36 0.56 stas1 0.33 0.35
stai36 0.68 0.65 stai8 0.84 0.49 stas2 0.35 0.35
stai32 0.64 0.50 stai10 0.77 0.60 stas5 0.27 0.43
stai34 0.69 0.63 stai11 0.72 0.57 stas6 0.25 0.33
stai31 0.66 0.58 stai5 0.72 0.56 stas3 0.24 0.40
stai30 0.63 0.58 stai1 0.71 0.57 stas10 0.81 0.23
stai33 0.62 0.59 stai2 0.54 0.54 stas9 0.66 0.34
stai28 0.48 0.59 stai4 0.65 0.48 stas17 0.37 0.46
stai25 0.37 0.51 stai3 0.57 0.51 stas20 0.17 0.38
stai19 0.82 0.56 stai12 0.37 0.56 stas13 0.13 0.37
stai16 0.72 0.60 stai7 0.38 0.57 stas15 0.13 0.35
stai20 0.73 0.65 stai6 0.29 0.45 stas14 0.16 0.36
stai23 0.76 0.57 stai9 0.19 0.44 stas11 0.09 0.36
stai15 0.75 0.58 stai37 0.81 0.36 stas12 0.08 0.35
stai21 0.71 0.59 stas4 0.71 0.31 stas18 0.07 0.24
stai22 0.66 0.56 stai39 0.75 0.37 stas16 0.06 0.28
stai24 0.66 0.43 stas7 0.67 0.22 stas19 0.06 0.26
Note: ITC: item-total correlation.
Table A2
Item statistics of the 5PFT (N=3300) and the item labels from the original scale.
Item p ITC Item p ITC Item p ITC
Table A3.
Results of Table 4 in which undefined item-total correlations (because of p=1 or p=0) were imputed with the value 0.
Roma STAI Libyan 2nd Libyan st. African 5PFT Indian White
Table B1
Sensitivity analysis showing MCV correlations for the 28 group comparisons, including those in the Dutch samples based on the STAI/STAS or the 5PFT instead of the SPM.
Roma 5PFT Libyan 2nd Libyan st. African STAI Indian White
Figure 4. Maximum values of Phi coefficients, unstandardized group difference in p, and the item-total correlation as a function of the p-value of the items. 0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" 0" 0.1" 0.2" 0.3" 0.4" 0.5" 0.6" 0.7" 0.8" 0.9" 1" ma xi mu m&va lu e&o f&p hi ,&d iff eren ce&i n&p &a nd &p oi nt 5b is eri al &co rrel a8o n& p5value&of&item&
