Development of a situational judgment test to supplement current US air force measures of officership.

Aptitude requirements for US Air Force officer commissioning include completion of a college degree and minimum scores on the Air Force Officer Qualifying Test (AFOQT) Verbal and Quantitative composites. Although the AFOQT has demonstrated predictive validity for officer training, the Air Force has striven to improve predictive validity and diversity. To this end, a Situational judgment Test (SJT) was added to the AFOQT in 2015. SJT development was consistent with recommendations to broaden the competencies assessed by the AFOQT with the goal of providing incremental validity, while reducing adverse impact for historically underrepresented groups. To ensure content validity and realism, SJT development was based on competencies identified in a large-scale analysis of officership and input from junior officers in scenario and response generation and scoring. Psychometric evaluations have affirmed its potential benefits for inclusion on the AFOQT. An initial study showed the SJT to be perceived as highly face valid regardless of whether it was presented as a paper-and-pencil test (with narrative or scripted scenarios) or in a video-based format. Preliminary studies demonstrated criterion-related validity within small USAF samples, and a larger Army cadet sample. Additionally, operational administration of the SJT since 2015 has demonstrated its potential for improving diversity (i.e., reduced adverse impact relative to the AFOQT Verbal and Quantitative composites). Predictive validation studies with larger Air Force officer accession samples are ongoing to assess the incremental validity of the SJT beyond current AFOQT composites for predicting important outcomes across accession sources.

Development of a serious gaming approach for cyber aptitude assessment.

Numerous traditional assessments have been developed to determine suitability of US military recruits for cyber careers. Cyber career field managers expressed a concern there may be well-qualified candidates that lack cyber knowledge, and therefore are not identified with knowledge-based tests. Technological advances such as serious gaming may provide opportunities to assess constructs traditional methods do not effectively measure. The purpose of this effort was to identify potential gains in validity that could be achieved beyond traditional methods through the use of serious games for several cyber jobs (both for enlisted and officer positions). Throughout this phase of research, an extensive literature review of military and civilian assessments targeted cyber occupations. Then, military subject matter experts in these career fields provided input and guidance (e.g., focus on aptitudes and traits as knowledge and skill are rapidly outdated). A gap analysis between all measures of such constructs identified a short list of candidates for measurement in a serious game. A survey of 800 airmen in the 1N4X1A, 3D1X2 and 17DEX/SX career fields was conducted; 290 respondents identified six constructs to be the focus for serious game assessment. The game was developed, and constructs validated on a sample chosen to model Air Force enlisted recruits. Additional psychometric data from enlistees and cyber trainees will be gathered once COVID-19 restrictions are lifted.

Using natural language processing to increase prediction and reduce subgroup differences in personnel selection decisions.

The purpose of this research is to demonstrate how using natural language processing (NLP) on narrative application data can improve prediction and reduce racial subgroup differences in scores used for selection decisions compared to mental ability test scores and numeric application data. We posit there is uncaptured and job-related constructs that can be gleaned from applicant text data using NLP. We test our hypotheses in an operational context across four samples (total N = 1,828) to predict selection into Officer Training School in the U.S. Air Force. Boards of three senior officers make selection decisions using a highly structured rating process based on mental ability tests, numeric application information (e.g., number of past jobs, college grades), and narrative application information (e.g., past job duties, achievements, interests, statements of objectives). Results showed that NLP scores of the narrative application generally (a) predict Board scores when combined with test scores and numeric application information at a level of correlation equivalent to the correlation between human raters (.60), (b) add incremental prediction of Board scores beyond mental ability tests and numeric application information, and (c) reduce subgroup differences between racial minorities and nonracial minorities in Board scores compared to mental ability tests and numeric application information. Moreover, NLP scores predict (a) job (training) performance, (b) job (training) performance beyond mental ability tests and numeric application information, and (c) even job (training) performance beyond Board scores. Scoring of narrative application data using NLP shows promise in addressing the validity-adverse impact dilemma in selection. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Correction for range restriction: Lessons from 20 research scenarios.

Data are often available only for recruits, a range-restricted sample. This creates the potential for mistaken inferences and poor decisions. This is because inferences and decisions are about the population, not the sample. Despite these problems, researchers must try to determine statistical values as if the sample was not range-restricted. Although range restriction correction methods have been available for over a century, often they are not applied or are applied incorrectly. Technical psychometric discussions of range restriction have not improved researcher practice. As an alternative, realistic scenarios are presented to illustrate and explain the consequences of (1) failing to correct correlations, (2) using the wrong correction formula, (3) correcting when information about previous selection variables is unavailable, (4) using an inappropriate unrestricted sample, (5) incorrectly computing the confidence interval for corrected correlations, and (6) interpretation of results. Although there are situations under which correction has little effect, correction still provides better estimates of relations among variables. It also improves theoretical understanding and interpretation of real-world results.

Tailored Adaptive Personality Assessment System (TAPAS) as an indicator for counterproductive work behavior: Comparing validity in applicant, honest, and directed faking conditions.

The predictive validity of the Tailored Adaptive Personality Assessment System (TAPAS), the U.S. Army's first computer-adaptive personality test incorporating multidimensional pairwise preference items, has been demonstrated for training performance in both the Army and Air Force. While the unique TAPAS format has been described as more resistant to applicant faking than traditional self-report personality measures, evidence regarding the magnitude of applicant score distortion on TAPAS, and how such distortion (if present) may affect reliability and validity, has been limited. To address this gap, the present study compared operational TAPAS scores of Air Force enlisted recruits (administered pre-accession to applicants) to their post-accession retest scores under honest and directed faking ("fake good") conditions (based on re-administration of TAPAS during Basic Military Training). Data are presented on the relationship of applicant pre-accession scores to their retest scores under honest conditions (a form of test-retest reliability) and the magnitude of mean score differences in applicant, honest, and directed faking conditions is documented. Further, the validity of the TAPAS as an indicator for counterproductive work behaviors (CWB) was evaluated. Results indicate that TAPAS scores are relatively stable over time and the TAPAS methodology appears to reduce score distortion. In addition, the results suggest that the validities of the TAPAS scores as CWB correlates are comparable across honest and directed faking testing conditions and generally in line with those found for traditional Likert-type self-report Big Five measures.

Compilation of Cognitive and Personality Norms for Military Aviators.

INTRODUCTION: The assessment of individuals on abilities or other characteristics is based on comparison to a representative sample. General population norms provide an appropriate reference group when the distribution of scores in the sample can be expected to be similar to those for the general population (e.g., comparing high school students at a particular school to national high school norms on a college entrance test). Specialized norms are needed, however, when subsets of the population differ from the population at large. Military pilot trainees represent a special population; they are highly screened on cognitive ability and other characteristics thought to be related to job performance. Other characteristics (e.g., personality) are thought to be "self-selected," resulting in distinctive profiles. Normative tables were developed for U.S. Air Force pilot trainees for two widely used tests, the Multidimensional Aptitude Battery-II (MAB-II) and NEO Personality Inventory-Revised (NEO PI-R). METHODS: The MAB-II and NEO PI-R were administered to large samples of USAF cadets, ROTC students, and officers selected for pilot training. RESULTS: The mean MAB-II full-scale IQ was about 1.5 SD above the adult population norm and was much less variable, supporting the need for specialized norms. Tables showing the percentile equivalents are provided for use by clinicians. DISCUSSION: Use of these tables, in addition to, or in lieu of, commercially published norms, will prove helpful when clinical psychologists perform assessments on pilots; in particular when evaluating them for return-to-duty status following a disqualifying condition that may have affected cognitive functioning or emotional stability. Carretta TR, King RE, Ree MJ, Teachout MS, Barto E. Compilation of cognitive and personality norms for military aviators. Aerosp Med Hum Perform. 2016; 87(9):764-771.

Personnel Selection Influences on Remotely Piloted Aircraft Human-System Integration.

INTRODUCTION: Human-system integration (HSI) is a complex process used to design and develop systems that integrate human capabilities and limitations in an effective and affordable manner. Effective HSI incorporates several domains, including manpower, personnel and training, human factors, environment, safety, occupational health, habitability, survivability, logistics, intelligence, mobility, and command and control. To achieve effective HSI, the relationships among these domains must be considered. Although this integrated approach is well documented, there are many instances where it is not followed. Human factors engineers typically focus on system design with little attention to the skills, abilities, and other characteristics needed by human operators. When problems with fielded systems occur, additional training of personnel is developed and conducted. Personnel selection is seldom considered during the HSI process. Complex systems such as aviation require careful selection of the individuals who will interact with the system. Personnel selection is a two-stage process involving select-in and select-out procedures. Select-in procedures determine which candidates have the aptitude to profit from training and represent the best investment. Select-out procedures focus on medical qualification and determine who should not enter training for medical reasons. The current paper discusses the role of personnel selection in the HSI process in the context of remotely piloted aircraft systems.

Predictive validity of pilot selection instruments for remotely piloted aircraft training outcome.

INTRODUCTION: Demand for remotely-piloted aircraft (RPA) support has increased dramatically over the last decade. Initial efforts to meet the demand focused on cross-training experienced manned aircraft pilots and funneling recent Specialized Undergraduate Pilot Training (SUPT) graduates to RPA pilot training. This approach reduced the number of personnel available for manned airframes and is no longer sustainable. In 2009, the USAF established an RPA career field and the Undergraduate RPA Training (URT) course to train officers with no prior flying experience to be RPA pilots. URT selection methods are very similar to those for SUPT. Some important factors for URT applicants are medical flight screening and aptitude tests [Air Force Officer Qualifying Test (AFOQT) and Pilot Candidate Selection Method (PCSM)]. The current study examined the predictive validity of the AFOQT pilot and PCSM composites for URT completion. METHOD: Subjects were 139 URT students with AFOQT and PCSM scores. The training criterion was URT pass/fail and the pass rate was 74.8%. RESULTS: Both the AFOQT pilot (r = 0.378) and PCSM (r = 0.480) composites demonstrated good predictive validity. DISCUSSION: No minimum qualifying PCSM score exists for URT. Had a minimum PCSM score of 25 been used, the pass rate would have been 80.2%; 12 more eliminees would have been screened out compared with the current AFOQT pilot minimum qualifying score of 25. Although current selection methods are effective, based on results of several RPA job/task analyses, the Air Force is examining the utility of other measures to supplement current methods.

Improved military air traffic controller selection methods as measured by subsequent training performance.

INTRODUCTION: Over the past decade, the U.S. military has conducted several studies to evaluate determinants of enlisted air traffic controller (ATC) performance. Research has focused on validation of the Armed Services Vocational Aptitude Battery (ASVAB) and has shown it to be a good predictor of training performance. Despite this, enlisted ATC training and post-training attrition is higher than desirable, prompting interest in alternate selection methods to augment current procedures. The current study examined the utility of the FAA Air Traffic Selection and Training (AT-SAT) battery for incrementing the predictiveness of the ASVAB versus several enlisted ATC training criteria. METHOD: Subjects were 448 USAF enlisted ATC students who were administered the ASVAB and FAA AT-SAT subtests and subsequently graduated or were eliminated from apprentice-level training. Training criteria were a dichotomous graduation/elimination training score, average ATC fundamentals course score, and FAA certified tower operator test score. RESULTS: Results confirmed the predictive validity of the ASVAB and showed that one of the AT-SAT subtests resembling a low-fidelity ATC work sample significantly improved prediction of training performance beyond the ASVAB alone. DISCUSSION: Results suggested training attrition could be reduced by raising the current ASVAB minimum qualifying score. However, this approach may make it difficult to identify sufficient numbers of trainees and lead to adverse impact. Although the AT-SAT ATC work sample subtest showed incremental validity to the ASVAB, its length (95 min) may be problematic in operational testing. Recommendations are made for additional studies to address issues affecting operational implementation.