Document Type

Dissertation

Degree

Doctor of Philosophy

Major

Psychology, Industrial and Organizational

Date of Defense

4-15-2020

Graduate Advisor

Stephanie Merritt, PhD

Co-Advisor

Garett Foster, PhD

Committee

John Meriac, PhD

Cody Ding, PhD

Abstract

Research and development of modern psychometric methods such as item response theory have drastically changed the way we understand and carry out the measurement of psychological constructs. Despite this, there has been relatively little adoption by psychological researchers to incorporate these methods into their research. While multiple explanations are surely valid, one oft stated reason is the large sample size requirements of these methods. The sample size requirements of item response theory are needed so that effective estimation of item parameters can be carried out. In an attempt to make these modern measurement methods more accessible and feasible to psychological researchers, this study investigated the extent to which subject matter experts and trained novices could effectively rate the location parameter of items to use as starting parameters in the item parameter estimation process. Rather than starting with random values, as is the default approach, starting with more accurate item locations was hypothesized to result in just as accurate item parameters that do not require typical sample sizes for these models. A pseudo-simulation process was carried out to estimate parameter recovery at various sample sizes when using SME and trainee ratings of item locations as starting parameters. Results suggest that while SMEs and trainees were not able to perfectly align item location parameters with statistical estimations, person estimates derived when using these as starting parameters yielded quite similar results to the parameters from the default MML procedure. Similar results were uncovered across sample sizes. Additionally, as sample size decreased from 500 to 200, recovery results became less stable indicating that even with SME and trainee estimates of item location used as starting parameters, sample size issues still remained when estimating item parameters.

Share

COinS