Guidelines for Bootstrapping Validity Coefficients in ATCS Selection Research

Civil Aerospace Medical Institute2000PDFTech ReportRussell, C. JDean, MBroach, D.United States. Department of Transportation. Federal Aviation Administration. Office of Aviation. Civil Aerospace Medical InstituteUniversity of OklahomaUnited States. Department of Transportation. Federal Aviation Administration. Office of Aviation. Office of Aerospace MedicineUnited States. Department of Transportation. Federal Aviation Administration. Office of Aviation. Civil Aerospace Medical InstituteUnited StatesAir traffic controllersAptitude testsCoefficientsConfidence intervalsGuidelinesLiterature reviewsMethodologyRepresentative samples (Statistics)Selection and appointmentStatistical samplingTrainingValidityBootstrappingAir Traffic Control Specialist SelectionDOT/FAA/AM-00/15AM-97-B-HRR-509Final ReportThis technical report 1) reviews the literature on bootstrapping estimation procedures and potential applications to the selection of air traffic control specialists (ATCSs), 2) describes an empirical demonstration of procedures for estimating the sample size required to demonstrate criterion-related validity in ATCS selection, and 3) provides summary guidelines and recommendations for estimating sample size requirements in ATCS selection test validation using bootstropping procedures under conditions of direct and indirect range restriction. Bootstrapping estimates the sampling distribution of a statistic by iteratively resampling cases from a set of observed data. Confidence intervals are constructed for the statistic, providing an empirical basis for inferential statements about the likley magnitude of the statistic. Correlations between scores on the written ATCS aptitude test battery and subsequent performance in initial qualification training for a large sample of 10,869 controllers hired between 1986 and 1992 were bootstrapped in an empirical demonstration of the methodology. Finally, a three-step sequence of procedures is described for use in future bootstrap estimates of confidence intervals. Recommendations for sample size requirements in future ATC criterion validity studies include: 1.Results suggest samples of at least N = 175 to ensure the 90% confidence interval for rxy does not contain 0. Assumptions of bivariate normality in traditional parametric estimation procedures are not justified in the current data. Note that this observation may result in confidence intervals that are wider or narrower for any given sample size than intervals obtained from traditional parametric estimation. 2.Corrections for direct range restriction did not substantively influence whether the bootstrapped 90% confidence interval contained 0. Future applications should assess whether this holds true. 3.Given the apparent absence of bivariate normality in the current data, similar bootstrapping procedures should be used to assess whether the 90% confidence intervals for p - po and RY.X1X2 - RY.X1 contain 0. Overall, the results suggest that bootstrapping of validity coefficients in controller selection research may be technically feasible. However, legal considerations may limit practical use of the methodology until accepted professional guidelines, standards, and principles are revised to accommodate innovative methodologies.120