Exploring the Utility and Fairness of a Multi-institutional Situational Judgment Test.

OBJECTIVE: To pilot a situational judgment test (SJT) developed to assess empathy, integrity, and teamwork and explore differences in performance between first-year (P1) and fourth-year (P4) student pharmacists based on gender, race or ethnicity, and geographical region. METHODS: An SJT was developed to assess empathy, integrity, and teamwork, using best practices. The SJT was piloted tested with P1 and P4 student pharmacists at 5 institutions. Scores were adjusted for elevation and scatter to minimize the subgroup differences in response styles. RESULTS: The unadjusted mean SJT scores for P1 and P4 student pharmacists were 79.7 % and 80.7 %, respectively. The adjusted scores for P1 and P4 student pharmacists were 86.3 % and 87.2 %, respectively. Small differences were observed across institutions which can likely explained by multiple factors. The mean scores were higher among students who were older, female, identified as White, and spoke English as their first language. The subgroup differences persisted even after adjusting scores for elevation and scatter. CONCLUSION: Performance on an SJT designed to assess empathy, integrity, and teamwork was similar to others in the literature; however, differences in performance were observed across subgroups. Additional research is needed to determine whether these differences reflect test bias or other possible factors.

The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests.

BACKGROUND: Situational Judgment Tests (SJTs) are commonly used in medical school admissions. However, it has been consistently found that native speakers tend to score higher on SJTs than non-native speakers, which can be particularly problematic in the admission context due to the potential risk of limited fairness. Besides type of SJT, awareness of time limit may play a role in subgroup differences in the context of cognitive load theory. This study examined the influence of SJT type and awareness of time limit against the background of language proficiency in a quasi high-stakes setting. METHODS: Participants (N = 875), applicants and students in healthcare-related study programs, completed an online study that involved two SJTs: one with a text-based stimulus and response format (HAM-SJT) and another with a video-animated stimulus and media-supported response format (Social Shapes Test, SST). They were randomly assigned to a test condition in which they were either informed about a time limit or not. In a multilevel model analysis, we examined the main effects and interactions of the predictors (test type, language proficiency and awareness of time limit) on test performance (overall, response percentage). RESULTS: There were significant main effects on overall test performance for language proficiency in favor of native speakers and for awareness of time limit in favor of being aware of the time limit. Furthermore, an interaction between language proficiency and test type was found, indicating that subgroup differences are smaller for the animated SJT than for the text-based SJT. No interaction effects on overall test performance were found that included awareness of time limit. CONCLUSION: A SJT with video-animated stimuli and a media-supported response format can reduce subgroup differences in overall test performance between native and non-native speakers in a quasi high-stakes setting. Awareness of time limit is equally important for high and low performance, regardless of language proficiency or test type.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions.

OBJECTIVE: The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness. METHODS: An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics. RESULTS: Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics. CONCLUSION: An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Broad vs. narrow traits: a scoping review of measuring personality traits in teacher selection using the situational judgment test.

Situational Judgment Tests (SJTs) have gained popularity and are commonly used as a measurement technique in a variety of professions, particularly those that include hiring, promoting, and professional development. In various educational sectors around the world, SJTs are being utilized as a measure by which to choose individuals who possess the requisite non-academic attributes for the profession. The objective of this review is to identify and analyze the traits that are measured in teacher selection using SJTs, in terms of both broad and narrow traits. This review uses a scoping review approach comprising five stages which are: identifying the research question, identifying relevant studies, selecting the studies, charting the data and reporting the results. Seven empirical research studies on teacher selection using SJTs were identified in which broad and narrow traits are used differently in selection. In the studies, a broad trait-conscientiousness-and seven narrow traits-organization and planning; empathy and communication; adaptability and resilience; mindset; emotional regulation; professional ethics; and enthusiasm and motivation-were recognized. Analysis revealed, in line with other prior studies, that the traits discovered tended to be used as the foundation for teacher selection criteria. The use of broad and narrow traits as the basis for teacher selection criteria has implications for the selection of the "best" teacher candidates because both broad and narrow traits generally do not accurately measure the precise characteristics needed. Future studies should focus on measuring more precise characteristics without overlap between the targeted characteristics, in light of the conclusions from this review.

Development of cross-service training and job performance criterion measures for the U.S. Military.

As part of an effort to provide standardized criterion measures across the Armed Services, the current research effort developed a set of service-wide criterion measures for first-term enlisted personnel using a recently developed model of cross-service job performance. Two concurrent work streams developed criterion measures which provide complete construct coverage of the cross-service performance model. Using existing service-specific instruments, methods for development of the Cross-Service Situational judgment Test (CSSJT), the Cross-Service Job Performance Ratings Scales (CSPRS), and two cross-service self-report measures (end of training and in unit surveys) are described and discussed.

Examining Validity for the Pharmacy Affective Domain Situational Judgment Test.

Objective. With the integration of the affective domain in the pharmacy accreditation standards, it is important for programs to have methods for formatively assessing student competency in these areas. The objective of this study was to examine the validity evidence for a recently developed situational judgment test to assess the affective domain contained in the Accreditation Council for Pharmacy Education (ACPE) Standards 2016.Methods. After pilot testing items in multiple pharmacy schools, a revised 15-item situational judgment test instrument was developed measuring the affective domain as it relates to the pharmacy profession. The resulting Pharmacy Affective Domain Situational Judgment Test (PAD-S) was administered to students at three institutions and internal structure validity evidence was examined using item descriptives, Cronbach alpha, and results from a Rasch measurement model.Results. There were 559 valid responses included in the study. The items were of varying difficulty, with Rasch results confirming the hierarchy of items and suggesting that items were contributing to the measure. The Cronbach alpha was 0.70, suggesting acceptable reliability. However, the reliability items from the Rasch model were lower (0.65 and 0.66), indicating that the ranking of ability was limited, which may be due to fewer items of high difficulty.Conclusion. Overall, the PAD-S performed well as a measure of the affective domain. The PAD-S may be a useful formative instrument as part of a comprehensive assessment plan and may be less resource intensive than other measures.

The Relationship Between a Multiple Mini-interview and Situational Judgment Test for Admissions.

Objective. To explore the relationship between a multiple mini-interview (MMI) and situational judgment test (SJT) designed to evaluate nonacademic constructs.Methods. A 30-question ranked-item SJT was developed to test three constructs also measured by MMIs during a pharmacy school's admissions process. First-year pharmacy students were invited to complete the SJT in fall 2020. One hundred four students took the SJT (82.5% response rate), with 97 (77% of possible participants) having MMI scores from the admissions process. Descriptive statistics and other statistical analyses were used to explore the psychometric properties of the SJT and its relationship to MMI scores.Results. Seventy-four percent of students identified as female (n=72), and 11.3% identified with an underrepresented racial identity (n=11). The average age, in mean (SD), was 21.8 (2.1) years. Students' mean (SD) scores were 85.5 (3.1) (out of 100 points) on the SJT and 6.1 (1.0) (out of 10 points) on the MMI. Principal components analysis indicated that the SJT lacked construct validity and internal reliability. However, reliability of the entire SJT instrument provided support for using the total SJT score for analysis (α=.63). Correlations between total SJT and MMI scores were weak (r p<0.29).Conclusion. Results of this study suggest that an SJT may not be a good replacement for the MMI to measure distinct constructs during the admissions process. However, the SJT may provide useful supplemental information during admissions or as part of formative feedback once students are enrolled in a program.

Evaluating the Whole Applicant: Use of Situational Judgment Testing and Personality Testing to Address Disparities in Resident Selection.

PURPOSE OF REVIEW: Urology program directors are faced with increasing numbers of applications annually, making holistic review of each candidate progressively more difficult. Efforts to streamline evaluation using traditional cognitive metrics have fallen short as these do not predict overall resident performance. Situational judgment tests (SJTs) and personality assessment tools (PATs) have been used in business and industry for decades to evaluate candidates and measure non-cognitive attributes that better predict subsequent performance. The purpose of this review is to describe what these assessments are and the current literature on the use of these metrics in medical education. RECENT FINDINGS: SJTs relative to PATs have more original research. Data suggests that SJTs decrease bias, increase diversity, and may be predictive of performance in residency. PATs are also emerging with data to support use with ability to assess fit to program and certain traits identified more consistently among high-performing residents and correlation to performance on ACGME milestones. PATs may be more coachable than SJTs. SJTs and PATs are emerging as techniques to supplement the current resident application review process. Early evidence supports their use in undergraduate medical education as does some early preliminary results in graduate medical education.

Identifying Potential Attrition during the Residency Applicant Screening Process Using a Situational Judgment Test.

OBJECTIVE: As USMLE Step 1 moves to pass/fail, residency programs are seeking alternate interview selection processes. Attrition in general surgery is reported as high as 26%. Thus, it is important to ensure that programs are selecting and matching applicants with shared values. Situational judgment tests (SJTs) measure educational and cultural values by posing ambiguous situations and individuals rate the effectiveness of possible reactions. SJTs have previously been shown to identify job applicants with shared values while promoting diversity. Scoring categories are high, moderate, or low values congruence. We sought to explore predictive validity of the SJT relative to program attrition. DESIGN: Residents who matched into our program between 2018 and 2021 completed the SJT. We tracked attrition. SETTING: UT Health San Antonio, Texas PARTICIPANTS: Fifty-six categorical general surgery residents RESULTS: Per SJT ratings, the numbers of residents who had high, moderate, and low values congruence were 27, 16, and 13, respectively. Attrition numbers for residents who scored high and moderate congruence were similar, indicating that these ratings were indistinguishable. As such, we combined those 2 categories to create a 2 × 2 matrix and used signal detection theory as a framework for analysis. Overall attrition was 16.1% (9/56). Of the 43 residents who scored high or moderate congruence, 90.7% remained in the program. There was a 9.3% chance of attrition for these residents. Of the 13 residents who scored low congruence, 38.5% attrited. While scoring as low congruence on the SJT does not definitively indicate attrition, it does indicate that attrition is 4.14 times more likely for these residents (chi-square, p = 0.0121). CONCLUSIONS: One of the most important aspects of residency applicant selection and interviewing is mitigating risk by identifying applicants who carry a high risk of attrition. The SJT significantly identifies at-risk applicants.

The role of the emotive, moral, and cognitive components for the prediction of medical students' empathic behavior in an Objective Structured Clinical Examination (OSCE).

OBJECTIVES: Investigate whether medical students' emotive abilities, attitudes, and cognitive empathic professional abilities predict empathic behavior in an Objective Structured Clinical Examination (OSCE). METHODS: Linear and multiple regressions were used to test concurrent validity between Interpersonal Reactivity Index (IRI), Jefferson Scale of Physician Empathy (JSPE-S), Situational Judgement Test (SJT-expert-based score (SJT-ES), SJT-theory-based score (SJT-TS)) and empathic behavior in an OSCE measured by Berlin Global Rating (BGR) and Verona Coding Definitions for Emotion Sequences (VR-CoDES). RESULTS: Highest amounts of explained variance of empathic behavior measured by VR-CoDES were found for the SJT-ES (R2 = 0.125) and SJT-TS (R2 = 0.131). JSPE-S (R2 = 0.11) and SJT-ES (R2 = 0.10) explained the highest amount of variance in empathic behavior as measured by BGR. Stepwise multiple regression improved the model for BGR by including SJT-ES and JSPE-S, explaining 16.2% of variance. CONCLUSIONS: The instrument measuring the emotive component (IRI) did not significantly predict empathic behavior, whereas instruments measuring moral (JSPE-S) and cognitive components (SJT) significantly predicted empathic behavior. However, the explained variance was small. PRACTICE IMPLICATIONS: The instrument measuring the emotive component (IRI) did not significantly predict empathic behavior, whereas instruments measuring moral (JSPE-S) and cognitive components (SJT) significantly predicted empathic behavior. However, the explained variance was small. In a longitudinal assessment program, triangulation of different instruments assessing empathy offers a rich perspective of learner's empathic abilities. Empathy training should include the acquisition of knowledge, attitudes, and behavior to support learner's empathic behaviors.

Interested in situational judgment tests? Preparing pharmacy educators for potential challenges.

PURPOSE: There is a growing interest in situational judgment tests (SJTs) for assessments and the authors offer their insights to help others develop and utilize these instruments while navigating common challenges. DESCRIPTION: This manuscript provides important considerations to those interested in SJTs using the "What? So What? Now What?" reflection framework. The authors offer their experience developing, implementing, and evaluating SJTs designed to measure empathy, adaptability, and integrity. ANALYSIS/INTERPRETATION: Effective SJT design can be organized around five key steps: attribute definition, item design, key development and scoring, test administration, and data analysis and interpretation. There are general recommendations to optimize SJT design to enhance reliability and validity of the results; however, there are often other challenges that have not been previously described. CONCLUSIONS: SJTs offer an exciting tool for measuring social and behavioral attributes; however, their infancy in health professions education warrants caution. Like any assessment, there are several challenges to consider when designing, implementing, and evaluating SJTs in pharmacy education. IMPLICATIONS: Individuals interested in SJTs are encouraged to ensure the construct measured is well-defined, there are clear and supported decisions around item and key design, and the appropriate psychometric analyses are conducted.

Assessing distinguishable social skills in medical admission: does construct-driven development solve validity issues of situational judgment tests?

BACKGROUND: Social skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJTs) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs. METHOD: The final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior. RESULTS: The proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures. CONCLUSIONS: Results suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended.

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues.

BACKGROUND: Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile. METHODS: Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508). RESULTS: The SJT demonstrated good internal consistency (Cronbach's alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity. CONCLUSIONS: This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery.

OBJECTIVE: The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences. DESIGN: PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded. RESULTS: Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation. CONCLUSIONS: As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

Development and Implementation of Situational Judgment Tests as an Evaluation Method for Training Oncology Physicians: Application in the KOKON-KTO Training.

Situational judgment tests (SJTs) are often used in aptitude testing and present practice-specific challenges. Their implementation into online training programs provides the opportunity to assess learning progress and improve training quality. In this study, text-based SJTs for oncology physicians were developed, validated, and implemented into the KOKON-KTO training which uses a blended learning training format to teach oncology physicians how to consult cancer patients on complementary and integrative medicine (CIM). The SJT was implemented to measure the e-learning results. In the development and validation phase, a total of 15 SJTs (each SJT including 1 best choice answer based on training content and 4 distractors; 9 SJTs for oncologists and 6 SJTs for oncology gynecologists only) were developed by an interprofessional team (n=5) using real-case vignettes and applying an in-depth review process. Best answers were validated by experts (oncologists and oncology gynecologists) with experience in advising cancer patients on CIM. In the implementation and evaluation phase, SJTs were answered by KOKON-KTO training participants (n=19) pre- and post e-learning. Results were analyzed using descriptive measurements, item difficulties, and Cohen's d for effect size pre- and post-training. The experts (n=12, 49.8% gynecologists) agreed with best choice answers (69.4% for oncology gynecology; 81.5% for oncology) in 12 out of 15 SJTs. Comparing pre- and post-training scores, KOKON-KTO training participants were able to improve knowledge substantially (effect sizes for oncologists d=1.7; oncology gynecologists d= .71). Future studies need to increase the number of experts and SJTs in order to apply further psychometric measurements. As part of the KOKON-KTO study, this project is registered as DRKS00012704 on the "German Clinical Trials Register" (Date of registration: 28.08.2017).

A Situational Judgment Test to Assess Students' Achievement of Accreditation Council for Pharmacy Education Standards 3 and 4.

Objective. For pharmacy students to successfully meet competencies related to Accreditation Council for Pharmacy Education Standards 3 and 4, it is essential for pharmacy programs to assess student progression in the affective domain. The purpose of this study was to develop and assess the validity of a situational judgment test (SJT) for evaluating student mastery of Standards 3 and 4.Methods. A multi-institutional faculty team developed an 18-item SJT that consisted of scenarios asking the respondent to rank the effectiveness of four response options mapped to Standards 3 and 4. The research team systematically reviewed the literature, created items, and deliberated until consensus was achieved. Subject matter experts (SMEs) reviewed and provided feedback on the instrument. Students from two institutions were recruited to participate in cognitive interviews about the finalized instrument. Cognitive interview data were analyzed to identify themes.Results. After edits were made to the instrument based on SME feedback, students (n=18) in the cognitive interviews identified item length as a concern and commented on item/response clarity and comprehensiveness. Data from the cognitive interviews were used to modify the SJT to reduce the length and clarify items. The result was two shorter versions of the instrument, both with similar mapping to all elements in Standards 3 and 4.Conclusion. Early steps in validating the SJT suggested that the instrument may be a promising tool to assess student progression in the affective domain. The SJT instrument is intended to provide evidence of student pharmacist development that occurs in the didactic, experiential, and co-curricular portions of pharmacy education. The instrument can serve as one part of a comprehensive assessment plan.

Use of Situational Judgment Tests to Teach Empathy, Assertiveness, Communication, and Ethics.

Objective. The objectives of this study were to develop situational judgment test (SJT) scenarios for use in teaching empathy, assertiveness, and interprofessional communication in second-year pharmacy (P2) students and ethics in third-year pharmacy (P3) students, determine whether the SJTs developed were effective at measuring students' communication skills and ethical judgment, compare the performance of individual third-year students to the performance of teams of third-year students on the ethics SJT, and evaluate student feedback about SJTs as a teaching tool.Methods. Pharmacy faculty developed five SJT scenarios related to communications and five SJT scenarios related to ethics and piloted the scenarios with P2 and P3 students, respectively. Second-year students completed SJTs individually, while P3 students completed SJTs individually and in teams. Scenarios and responses were discussed with faculty after completion of the SJTs, and students completed a questionnaire to provide feedback on the SJTs.Results. The communications SJT was completed by 59 P2 students with a mean score of 67.5%. The ethics SJT was completed by 57 P3 students with a mean score of 80.1%. The ethics SJT was also completed by 10 teams of P3 students resulting in a mean score of 93.2%. Students indicated the SJT content was realistic and the tests provided the opportunity to reflect on how to approach challenging situations.Conclusion. Situational judgment tests were useful for teaching empathy, assertiveness, interprofessional communication, and ethics in pharmacy students. Future research should focus on predictive validity of SJTs for these content areas.

Improving the effectiveness of embedded behavioral health personnel through situational judgment training.

Embedding behavioral health (EBH) personnel into operational units has emerged as a major trend within the US military. These positions require skillsets in addition to those needed in a clinic setting. Little or no empirically based training has yet been developed to ensure preparedness to serve in EBH roles. Based on the demonstrated effectiveness of situational judgment tests (SJT) in training for other domains, we developed and evaluated 60 SJT scenarios for use in the US Air Force's formal EBH training course. SJTs were based on real-world experiences of seasoned EBH professionals, and were used for structured, scenario-based discussions during training as well as a 30-item pre- and posttest to assess learning. Use during training was associated with significantly improved scores in six EBH competencies (ethical decision-making, leadership consultation, balancing professional relationships, unit engagement, triage to correct services, and teaming). In a follow-up survey, students reported scenario-based discussions were very helpful, and reported improved abilities and confidence. This study demonstrates the potential utility of the SJT methodology to prepare mental health professionals to serve in EBH positions. Further development and possible applications are discussed, such as expansion to other training venues, job candidate screening, and assessment for readiness for EBH practice.

Does the ability to identify the construct on an empathy situational judgment test relate to performance? Exploring a new concept in assessment.

INTRODUCTION: The ability to identify criteria (ATIC) refers to an examinee's capacity to distinguish the construct being evaluated. Previous research indicates ATIC can be predictive of performance on some assessments. This exploratory study investigated the relationship between a participant's ability to identify criteria and their performance on an empathy situational judgment test (SJT), an assessment format used to measure social and behavioral attributes. METHODS: A 12-item empathy SJT was completed by 15 students and 15 pharmacists. During a cognitive interview, participants were asked what they believed each exam question measured. Responses were coded to determine whether participants stated "empathy" (indication of ATIC). The point-biserial correlation coefficient was calculated to explore the relationship of ATIC (correctly or not correctly identifying the item measured empathy) and performance on the SJT (total score). RESULTS: Participants identified empathy 33.3% of the time, and it was the construct most often identified. Pharmacists (27.5%) identified empathy less often than students (39.2%). When empathy was identified as the construct, it was most often reported for items in a non-healthcare setting (56.3%) rather than a healthcare setting (43.7%) and for questions targeting affective empathy (71.3%) rather than cognitive empathy (28.7%). There were no statistically significant relationships with correctly identifying the construct and performance on individual items and the overall test. CONCLUSIONS: There is inconclusive evidence that ATIC relates to performance on an empathy SJT. Additional research is needed to evaluate the role of ATIC and assessment performance to corroborate study results.

Nominal Factor Analysis of Situational Judgment Tests: Evaluation of Latent Dimensionality and Factorial Invariance.

Situational judgment tests have gained popularity in educational and psychological measurement and are widely used in personnel assessment. A situational judgment item presents a hypothetical scenario and a list of actions, and the individuals are asked to select their most likely action for that scenario. Because actions have no explicit order, the item generates nominal responses consisting of the actions selected by the individuals. This article shows how to factor-analyze the nominal responses originated from such a test, including the estimation of the number of latent factors and a factor invariance analysis in a multiple group design. The method consists of applying the MNCM, a multidimensional extension of the nominal categories model by Bock. The article includes the results of two studies: (1) a simulation study about Type-I error rate, statistical power, and recovery of the parameters in a multigroup factorial invariance design and (2) a real data example using responses to a situational judgment test measuring gender stereotypes to illustrate the approach. Results suggest the use of the Akaike information criterion, Bayesian information criterion, and corrected Bayesian information criterion indices to guide the selection of the number of factors with nominal responses. All the analyses are conducted using the computer program Mplus. The code is included as Supplemental Material (available online) for the readers so that they can adapt it to their own purposes.