Conducting an objective structured clinical examination under COVID-restricted conditions.

BACKGROUND: The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. The purpose of this study was to gather evidence to support the reliability and validity of the modified NAC Examination. METHODS: The modified NAC Examination was delivered to 2,433 candidates in 2020 and 2021. Cronbach's alpha, decision consistency, and accuracy values were calculated. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs. female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs. non-CSA), postgraduate training (PGT) (no PGT vs. PGT), and language of examination (English vs. French). Criterion relationships were summarized using correlations within and between the NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. RESULTS: Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were statistically significant. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). CONCLUSIONS: The NAC Examination yields reliable total scores and pass/fail decisions. Expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that required modifications due to COVID.

Cheating in OSCEs: The Impact of Simulated Security Breaches on OSCE Performance.

Construct: Valid score interpretation is important for constructs in performance assessments such as objective structured clinical examinations (OSCEs). An OSCE is a type of performance assessment in which a series of standardized patients interact with the student or candidate who is scored by either the standardized patient or a physician examiner. BACKGROUND: In high-stakes examinations, test security is an important issue. Students accessing unauthorized test materials can create an unfair advantage and lead to examination scores that do not reflect students' true ability level. The purpose of this study was to assess the impact of various simulated security breaches on OSCE scores. APPROACH: Seventy-six 3rd-year medical students participated in an 8-station OSCE and were randomized to either a control group or to 1 of 2 experimental conditions simulating test security breaches: station topic (i.e., providing a list of station topics prior to the examination) or egregious security breach (i.e., providing detailed content information prior to the examination). Overall total scores were compared for the 3 groups using both a one-way between-subjects analysis of variance and a repeated measure analysis of variance to compare the checklist, rating scales, and oral question subscores across the three conditions. RESULTS: Overall total scores were highest for the egregious security breach condition (81.8%), followed by the station topic condition (73.6%), and they were lowest for the control group (67.4%). This trend was also found with checklist subscores only (79.1%, 64.9%, and 60.3%, respectively for the security breach, station topic, and control conditions). Rating scale subscores were higher for both the station topic and egregious security breach conditions compared to the control group (82.6%, 83.1%, and 77.6%, respectively). Oral question subscores were significantly higher for the egregious security breach condition (88.8%) followed by the station topic condition (64.3%), and they were the lowest for the control group (48.6%). CONCLUSIONS: This simulation of different OSCE security breaches demonstrated that student performance is greatly advantaged by having prior access to test materials. This has important implications for medical educators as they develop policies and procedures regarding the safeguarding and reuse of test content.

Weighting checklist items and station components on a large-scale OSCE: is it worth the effort?

BACKGROUND: Past research suggests that the use of externally-applied scoring weights may not appreciably impact measurement qualities such as reliability or validity. Nonetheless, some credentialing boards and academic institutions apply differential scoring weights based on expert opinion about the relative importance of individual items or test components of Observed Structured Clinical Examinations (OSCEs). AIMS: To investigate the impact of simplified scoring models that make little to no use of differential weighting on the reliability of scores and decisions on a high stakes OSCE required for medical licensure in Canada. METHOD: We applied four different weighting models of various complexities to data from three administrations of the OSCE. We compared score reliability, pass/fail rates, correlations between the scores and classification decision accuracy and consistency across the models and administrations. RESULTS: Less complex weighting models yielded similar reliability and pass rates as the more complex weighting model. Minimal changes in candidates' pass/fail status were observed and there were strong and statistically significant correlations between the scores for all scoring models and administrations. Classification decision accuracy and consistency were very high and similar across the four scoring models. CONCLUSIONS: Adopting a simplified weighting scheme for this OSCE did not diminish its measurement qualities. Instead of developing complex weighting schemes, experts' time and effort could be better spent on other critical test development and assembly tasks with little to no compromise in the quality of scores and decisions on this high-stakes OSCE.

Dried blood spots via remote testing as a possible future application in the doping control process.

The use of DBS as biological matrix in combination with a novel technical remote testing app specifically developed for the doping control process shows that testing is becoming easier and variable through DBS. Supplemented by the remote testing solution, the system could significantly increase in the unpredictability of test dates and sample quantity.

Bone resorption is induced on the second day of bed rest: results of a controlled crossover trial.

The aim of the study was to analyze the kinetics of short-term changes in bone turnover. We studied in a randomized crossover design the effects of 6 days of bed rest on eight healthy male subjects (mean body wt: 70.1 +/- 5.7 kg; mean age: 25.5 +/- 2.9 yr). The metabolic ward period was divided into three parts: 4 ambulatory days, 6 days of either bed rest or non-bed rest periods, and 1 recovery day. The diet was identical in both bed rest and non-bed rest phases. Continuous urine collection started on the first day in the metabolic ward to analyze excretion of bone resorption markers, namely C-telopeptide (CTX) and N-telopeptide (NTX), creatinine, urea, and 3-methylhistidine. On the second ambulatory day and on the fifth day of bed rest or during the non-bed rest phase, blood was drawn to analyze bone formation markers and amino acid concentrations. Urinary calcium excretion was increased as early as the first day of bed rest (P < 0.01). CTX and NTX excretion stayed unchanged during the first 24 h of bed rest compared with the non-bed rest period. However, already on the second day, both resorption markers had increased significantly. NTX excretion increased by 28.7 +/- 14.0% (P < 0.01), whereas CTX excretion rose by 17.8 +/- 8.3% (P < 0.001). Creatinine, urea, and 3-methylhistidine excretion did not change. We conclude that 24 h of bed rest are sufficient to induce a significant rise in osteoclast activity in healthy subjects.