University of Groningen
Practical Significance of Item Response Theory Model Misfit Crisan, Daniela
DOI:
10.33612/diss.128084616
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date:
2020
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Crisan, D. (2020). Practical Significance of Item Response Theory Model Misfit: Much Ado About Nothing?.
University of Groningen. https://doi.org/10.33612/diss.128084616
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.
Download date: 25-06-2021
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 1PDF page: 1PDF page: 1PDF page: 1
Practical Significance of Item Response Theory Model Misfit
Much Ado About Nothing?
Daniela Crișan
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 2PDF page: 2PDF page: 2PDF page: 2
© 2020 Practical significance of item response theory model misfit: Much ado about nothing?
Daniela R. Crișan, University of Groningen
ISBN: 978-94-034-2743-0 (print version) ISBN: 978-94-034-2744-7 (electronic version)
Cover design: Ipskamp Printing Printed by: Ipskamp Printing
All rights reserved. No parts of this publication may be reproduced or transmitted in any form by any means, without permission from the author.
Practical Significance of Item Response Theory Model Misfit
Much Ado About Nothing? PhD thesis
to obtain the degree of PhD at the University of Groningen
on the authority of the
Rector Magnificus Prof. C. Wijmenga and in accordance with
the decision by the College of Deans. This thesis will be defended in public on
Thursday 2 July 2020 at 16.15 hours
by
Daniela-Ramona Crișan
born on 12 February 1990 in Oradea, Romania
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 3PDF page: 3PDF page: 3PDF page: 3
© 2020 Practical significance of item response theory model misfit: Much ado about nothing?
Daniela R. Crișan, University of Groningen
ISBN: 978-94-034-2743-0 (print version) ISBN: 978-94-034-2744-7 (electronic version)
Cover design: Ipskamp Printing Printed by: Ipskamp Printing
All rights reserved. No parts of this publication may be reproduced or transmitted in any form by any means, without permission from the author.
Practical Significance of Item Response Theory Model Misfit
Much Ado About Nothing?
PhD thesis
to obtain the degree of PhD at the University of Groningen
on the authority of the
Rector Magnificus Prof. C. Wijmenga and in accordance with
the decision by the College of Deans.
This thesis will be defended in public on Thursday 2 July 2020 at 16.15 hours
by
Daniela-Ramona Crișan
born on 12 February 1990 in Oradea, Romania
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 4PDF page: 4PDF page: 4PDF page: 4
Supervisor Prof. R. R. Meijer
Co-supervisors Dr. J. N. Tendeiro Dr. D. van Ravenzwaaij
Assessment committee Prof. P. de Jonge
Prof. L. A. van der Ark Prof. R. Watson
Table of contents
Chapter 1. Introduction ... 9
1.1. Context ... 11
1.2. Topic of the thesis ... 12
1.3. Outline of the thesis ... 13
Chapter 2. Investigating the practical consequences of model misfit in unidimensional IRT models ... 15
2.1. Introduction ... 17
2.2. Methods ... 21
2.2.1. Independent variables ... 21
2.2.2. Dependent variables ... 22
2.2.3. Model-fit items ... 23
2.2.4. Model-misfit items ... 24
2.2.5. Model-fit checks ... 24
2.2.6. Design and implementation ... 25
2.3. Results ... 26
2.3.1. Model-fit checks ... 26
2.3.2. Effect of misfit on model parameters ... 27
2.3.3. Effect of misfit on the rank ordering of persons ... 30
2.3.4. Effect of misfit on criterion-related validity estimates ... 33
2.4. Discussion ... 35
2.4.1. Practical implications ... 36
2.4.2. Limitations and future research... 37
Chapter 3. Practical consequences of model misfit when using rating scales to assess the severity of attention problems in children ... 39
3.1. Introduction ... 41
3.1.1. Using sum scores to assess AP severity ... 42
3.1.2. IRT as a psychometric tool for assessing AP ... 43
3.1.3. Present study ... 44
3.2. Methods ... 45
3.2.1. Sample ... 45
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 5PDF page: 5PDF page: 5PDF page: 5
Supervisor Prof. R. R. Meijer
Co-supervisors Dr. J. N. Tendeiro Dr. D. van Ravenzwaaij
Assessment committee Prof. P. de Jonge
Prof. L. A. van der Ark Prof. R. Watson
Table of contents
Chapter 1. Introduction ... 9
1.1. Context ... 11
1.2. Topic of the thesis ... 12
1.3. Outline of the thesis ... 13
Chapter 2. Investigating the practical consequences of model misfit in unidimensional IRT models ... 15
2.1. Introduction ... 17
2.2. Methods ... 21
2.2.1. Independent variables ... 21
2.2.2. Dependent variables ... 22
2.2.3. Model-fit items ... 23
2.2.4. Model-misfit items ... 24
2.2.5. Model-fit checks ... 24
2.2.6. Design and implementation ... 25
2.3. Results ... 26
2.3.1. Model-fit checks ... 26
2.3.2. Effect of misfit on model parameters ... 27
2.3.3. Effect of misfit on the rank ordering of persons ... 30
2.3.4. Effect of misfit on criterion-related validity estimates ... 33
2.4. Discussion ... 35
2.4.1. Practical implications ... 36
2.4.2. Limitations and future research... 37
Chapter 3. Practical consequences of model misfit when using rating scales to assess the severity of attention problems in children ... 39
3.1. Introduction ... 41
3.1.1. Using sum scores to assess AP severity ... 42
3.1.2. IRT as a psychometric tool for assessing AP ... 43
3.1.3. Present study ... 44
3.2. Methods ... 45
3.2.1. Sample ... 45
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 6PDF page: 6PDF page: 6PDF page: 6
3.2.2. Measures – CBCL/6-18 Attention Problems Scale ... 45
3.2.3. Measures - Outcomes ... 46
3.2.4. Outline of the analyses ... 46
3.3. Results ... 47
3.3.1. Sample descriptives ... 47
3.3.2. Model violations and psychometric evidence against interpreting sum scores as unidimensional indicators of AP severity ... 48
3.3.3. Practical consequences of ignoring model violations on the predictive accuracy of long-term outcomes... 56
3.4. Discussion ... 59
3.5. Appendix ... 64
Chapter 4. On the practical consequences of misfit in Mokken scaling ... 67
4.1. Introduction... 69
4.1.1. Mokken Scale Analysis ... 70
4.1.2. How is Mokken Scale Analysis used in practice? ... 72
4.1.3. Practical significance ... 73
4.2. Methods ... 73
4.2.1. Independent variables ... 73
4.2.2. Design ... 74
4.2.3. Data generation ... 74
4.2.4. Dependent variables ... 76
4.2.5. Implementation ... 77
4.3. Results ... 78
4.3.1. Scale reliability and rank ordering ... 78
4.3.2. Person classification ... 80
4.3.3. Bias in criterion-related validity estimates... 82
4.4. Discussion... 85
4.4.1. Take-home message... 86
4.4.2. Limitations and future research ... 87
4.5. Appendix ... 88
Chapter 5. The Crit value as an effect size measure for violations of model assumptions in Mokken Scale Analysis for binary data ... 91
5.1. Introduction... 93
5.1.1. The monotonicity assumption in MSA ... 94
5.1.2. The invariant item ordering assumption in MSA ... 96
5.1.3. Aim of the study ... 98
5.2. Methods ... 98
5.2.1. Model-fit data generation ... 99
5.2.2. Model-misfit data generation ... 100
5.2.3. Independent variables and outcome variables ... 102
5.2.4. Implementation ... 104
5.3. Results ... 104
5.3.1. Crit for violations of monotonicity ... 104
5.3.2. Crit for violations of invariant item ordering ... 108
5.4. Discussion ... 111
5.4.1. Take-home message ... 112
5.5. Appendix ... 114
Chapter 6. Discussion ... 123
6.1. Introduction ... 125
6.2. Summary of the main findings ... 127
6.2.1. Person rank ordering ... 127
6.2.2. Person selection and classification ... 127
6.2.3. Predictive validity ... 128
6.2.4. Scale score reliability ... 128
6.2.5. The Crit index as an effect size measure for IRT model misfit ... 129
6.3. Theoretical considerations ... 129
6.4. Practical considerations ... 131
6.5. Limitations and future research ... 132
6.6. Final remarks ... 133
References ... 135
Samenvatting (in Dutch) ... 155
Acknowledgements ... 161
About the Author ... 163
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 7PDF page: 7PDF page: 7PDF page: 7
3.2.2. Measures – CBCL/6-18 Attention Problems Scale ... 45
3.2.3. Measures - Outcomes ... 46
3.2.4. Outline of the analyses ... 46
3.3. Results ... 47
3.3.1. Sample descriptives ... 47
3.3.2. Model violations and psychometric evidence against interpreting sum scores as unidimensional indicators of AP severity ... 48
3.3.3. Practical consequences of ignoring model violations on the predictive accuracy of long-term outcomes... 56
3.4. Discussion ... 59
3.5. Appendix ... 64
Chapter 4. On the practical consequences of misfit in Mokken scaling ... 67
4.1. Introduction... 69
4.1.1. Mokken Scale Analysis ... 70
4.1.2. How is Mokken Scale Analysis used in practice? ... 72
4.1.3. Practical significance ... 73
4.2. Methods ... 73
4.2.1. Independent variables ... 73
4.2.2. Design ... 74
4.2.3. Data generation ... 74
4.2.4. Dependent variables ... 76
4.2.5. Implementation ... 77
4.3. Results ... 78
4.3.1. Scale reliability and rank ordering ... 78
4.3.2. Person classification ... 80
4.3.3. Bias in criterion-related validity estimates... 82
4.4. Discussion... 85
4.4.1. Take-home message... 86
4.4.2. Limitations and future research ... 87
4.5. Appendix ... 88
Chapter 5. The Crit value as an effect size measure for violations of model assumptions in Mokken Scale Analysis for binary data ... 91
5.1. Introduction... 93
5.1.1. The monotonicity assumption in MSA ... 94
5.1.2. The invariant item ordering assumption in MSA ... 96
5.1.3. Aim of the study ... 98
5.2. Methods ... 98
5.2.1. Model-fit data generation ... 99
5.2.2. Model-misfit data generation ... 100
5.2.3. Independent variables and outcome variables ... 102
5.2.4. Implementation ... 104
5.3. Results ... 104
5.3.1. Crit for violations of monotonicity ... 104
5.3.2. Crit for violations of invariant item ordering ... 108
5.4. Discussion ... 111
5.4.1. Take-home message ... 112
5.5. Appendix ... 114
Chapter 6. Discussion ... 123
6.1. Introduction ... 125
6.2. Summary of the main findings ... 127
6.2.1. Person rank ordering ... 127
6.2.2. Person selection and classification ... 127
6.2.3. Predictive validity ... 128
6.2.4. Scale score reliability ... 128
6.2.5. The Crit index as an effect size measure for IRT model misfit ... 129
6.3. Theoretical considerations ... 129
6.4. Practical considerations ... 131
6.5. Limitations and future research ... 132
6.6. Final remarks ... 133
References ... 135
Samenvatting (in Dutch) ... 155
Acknowledgements ... 161
About the Author ... 163
544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan 544201-L-bw-Crisan Processed on: 9-6-2020 Processed on: 9-6-2020 Processed on: 9-6-2020
Processed on: 9-6-2020 PDF page: 8PDF page: 8PDF page: 8PDF page: 8
515082-L-os-lameris 515082-L-os-lameris 515082-L-os-lameris
515082-L-os-lameris Processed on: 3-11-2017Processed on: 3-11-2017Processed on: 3-11-2017Processed on: 3-11-2017
