Evaluating and extending statistical methods for estimating the construct-level predictive validity of selection tests

Background:
In this thesis the problem of range restriction was addressed using the United Kingdom Clinical Aptitude Test (UKCAT) and Professional and Linguistic Assessments Board (PLAB) test in the selection of undergraduate medical school entrants and International Medical Graduates (IMGs) in the UK as motivating examples. Methods for correcting for bias in the estimate of predictive validity due to range restriction (particularly Multiple Imputation (MI) and Full Information Maximum Likelihood (FIML)) were evaluated for the predictive validity, single hurdle concurrent and multiple hurdle validity designs under varying degrees of strictness in selection. For MI, the impact of the composition of the imputation model was also investigated.

Methods:
The performance of MI and FIML was tested through Monte Carlo simulations and validated using PLAB data.

Results:
Generally, MI and FIML were found to be equivalent in performance and superior to other methods of correcting for range restriction bias for selection ratios of ≤ 20% only in instances where data were multivariate normal. The inclusion of highly predictive variables in the imputation model increased the precision of MI.

Conclusion:
MI and FIML are viable alternatives for tackling bias in the estimate of predictive validity for direct range restricted data that satisfies the assumption of multivariate normality. Caution should be taken to avoid their application in instances where the assumption of multivariate normality is violated.