Criteria to evaluate graduate nurse proficiencies in obtaining a health history and perform physical assessment in simulation-based education: A narrative review.

BACKGROUND: Simulation is a technique being used increasingly in healthcare education which offers opportunities to evaluate nursing proficiencies. The use of valid and reliable instruments is recognised as the foundation for a robust assessment, however competency-based health assessment courses for graduate nurses can consequently become reductionist in measuring proficiencies. OBJECTIVE: The specific review question was: In simulation-based education, what are the criteria that evaluate graduate nursing student's competence in obtaining a health history and performance of patient assessment? METHODS: Eleven studies were included in the review. Papers were critically appraised with The Joanna Briggs Institute quasi-experimental studies checklist. Bloom's taxonomy was used to structure this narrative review. RESULTS: Seven papers evaluated cognition through questionnaires and two papers used a Likert-scale to determine self-perceived knowledge. Six papers evaluated psychomotor skills with a behavioural checklist. Diversity of application was factored into the studies when testing affective skills. Three papers used Likert-scales to evaluate preparedness, six papers used Likert-scales to evaluate self-confidence and one used a Likert-scale to evaluate autonomy. Three papers used a checklist to evaluate professionalism. Four papers used faculty member/ standardised patient feedback. CONCLUSION: Reductionist evaluation instruments create a barrier when evaluating competency. The limited validity and reliability of assessment instruments in simulation, as well as the lack of standardisation of affective skills assessment, presents a challenge in simulation research. Affective skills encompass attitudes, behaviours and communication abilities, which pose a significant challenge for standardised assessments due to their subjective nature. This review of the simulation literature highlights a lack of robustness in the evaluation of the affective domain. This paper proposes that simulation assessment instruments should include the standardisation of affective domain proficiencies such as: adaptation to patients' cognitive function, ability to interpret and synthesise relevant information, ability to demonstrate clinical judgement, readiness to act, recognition of professional limitations and faculty/standardised-simulated patient feedback. The incorporation of the affective domain in standardised assessment instruments is important to ensure comprehensive assessment of simulation particularly in the development of health history and physical assessment proficiencies. Attention to all of the domains in Blooms taxonomy during simulation assessment has the potential to better prepare professionals for the patient care setting.

The feasibility and reliability of frailty assessment tools applicable in acute in-hospital trauma patients: A systematic review.

BACKGROUND: Assessing frailty in patients with an acute trauma can be challenging. To provide trustworthy results, tools should be feasible and reliable. This systematic review evaluated existing evidence on the feasibility and reliability of frailty assessment tools applied in acute in-hospital trauma patients. METHODS: A systematic search was conducted in relevant databases until February 2020. Studies evaluating the feasibility and/or reliability of a multidimensional frailty assessment tool used to identify frail trauma patients were identified. The feasibility and reliability results and the risk of bias of included studies were assessed. This study was conducted and reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and registered in Prospective Register of Systematic Reviews (ID: CRD42020175003). RESULTS: Nineteen studies evaluating 12 frailty assessment tools were included. The risk of bias of the included studies was fair to good. The most frequently evaluated tool was the Clinical Frailty Scale (CFS) (n = 5). All studies evaluated feasibility in terms of the percentage of patients for whom frailty could be assessed; feasibility was high (median, 97%; range, 49-100%). Other feasibility aspects, including time needed for completion, tool availability and costs, availability of instructions, and necessity of training for users, were hardly reported. Reliability was only assessed in three studies, all evaluating the CFS. The interrater reliability varied between 42% and >90% agreement, with a Krippendorff α of 0.27 to 0.41. CONCLUSION: Feasibility of most instruments was generally high. Other aspects were hardly reported. Reliability was only evaluated for the CFS with results varying from poor to good. The reliability of frailty assessment tools for acute trauma patients needs further critical evaluation to conclude whether assessment leads to trustworthy results that are useful in clinical practice. LEVEL OF EVIDENCE: Systematic review, Level II.

Performance assessment using sensor technology.

Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills.

Influence of Standardized Procedures on the Reliability of Hip Clinical Assessment.

OBJECTIVE: This study evaluated a standardized and personalized approach to verify the effects of conditions on intrarater and interrater reliability, standard error of measurement, and minimal detectable difference for provocative tests and range-of-motion (ROM) tests used in hip pain assessment: flexion-adduction-internal rotation (FADIR), flexion-abduction-external rotation-extension (FABER), and hip internal rotation with 90° of hip flexion (hip IR). METHODS: Nineteen participants (mean [± SD] age = 24 ± 2 years; 10 women and 9 men) without lower limb or back pain were recruited. Three raters evaluated each participant during 2 testing sessions, 1 day apart. Raters performed the 3 tests in 4 conditions: classic (C), controlled pressure duration (CPD), subject-specific position (SSP), and mixed (M = CPD + SSP). RESULTS: For intrarater reliability, the CPD condition showed the highest intraclass correlation coefficients (ICCs; mean and 95% confidence interval [CI]) for hip IRROM (0.83; 95% CI, 0.53-0.94) and FADIRROM (0.75; 95% CI, 0.60-0.89). The SSP condition showed the highest ICCs for FABERheight (0.71; 95% CI, 0.42-0.87) and FABERROM (0.62; 95% CI, 0.27-0.83). Concerning interrater reliability, the classic condition presented the highest ICCs for FABER variables (height: 0.54; 95% CI, 0.28-0.76; ROM: 0.58; 95% CI, 0.32-0.79) and hip IR ROM (0.72; 95% CI, 0.51-0.87). The CPD condition showed the highest ICC for FADIRROM (0.57; 95% CI, 0.32-0.78). CONCLUSION: In the conditions of this study, CPD showed the highest ICCs for hip IRROM and FADIRROM, and SSP showed the highest ICCs for FABERheight and FABERROM.

Interrater agreement and reliability of clinical tests for assessment of patients with shoulder pain in primary care.

Background: There is limited information about the agreement and reliability of clinical shoulder tests. Objectives: To assess the interrater agreement and reliability of clinical shoulder tests in patients with shoulder pain treated in primary care. Methods: Patients with a primary report of shoulder pain underwent a set of 21 clinical shoulder tests twice on the same day, by pairs of independent physical therapists. The outcome parameters were observed and specific interrater agreement for positive and negative scores, and interrater reliability (Cohen's kappa (κ)). Positive and negative interrater agreement values of ≥0.75 were regarded as sufficient for clinical use. For Cohen's κ, the following classification was used: <0.20 poor, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 good, 0.81-1.00 very good reliability. Participating clinics were randomized in two groups; with or without a brief practical session on how to conduct the tests. Results: A total of 113 patients were assessed in 12 physical therapy practices by 36 physical therapists. Positive and negative interrater agreement values were both sufficient for 1 test (the Full Can Test), neither sufficient for 5 tests, and only sufficient for either positive or negative agreement for 15 tests. Interrater reliability was fair for 11 tests, moderate for 9 tests, and good for 1 test (the Full Can Test). An additional brief practical session did not result in better agreement or reliability. Conclusion: Clinicians should be aware that interrater agreement and reliability for most shoulder tests is questionable and their value in clinical practice limited.

Clinimetric properties of a translated and culturally adapted Norwegian version of the Patient and Observer Scar Assessment Scale for use in clinical practice and research.

PURPOSE: To translate and culturally adapt the Patient and Observer Scar Assessment Scale, POSAS, to Norwegian and explore its test-retest, intra- and inter-tester reliability. METHODS: POSAS was translated into Norwegian following international guidelines in collaboration with an international translation bureau. Twenty-six adults and 24 children were recruited from a burns outpatient clinic. Three observer-categories: doctor, nurse and physiotherapist, assessed the patients' scars and scored the Observer scale for estimating inter-tester reliability. Photos of the scars were taken and used to score the Observer scale a second time for examining intra-tester reliability. The patients or parents/next of kin rated their scar on the Patient scale at the clinic and after two days at home for examining test-retest reliability. Intraclass correlation (ICC) and Kappa were used for statistical analysis. RESULTS: A Norwegian version of POSAS (POSAS-NV) was developed. Inter-tester ICC of the Observer parameters varied between 0.203 and 0.728, and for the total sum score, ICC=0.528 (0.280-0.708). Intra-tester ICC of the Observer scale ranged between 0.575 and 0.858. The Patient scale demonstrated high test-retest reliability. CONCLUSIONS: Intra-tester reliability of the Observer scale and test-retest reliability of the Patient scale of POSAS-NV were found satisfactory, but not inter-tester reliability of the Observer scale.

Validity of Postural Sway Assessment on the Biodex BioSway™ Compared With the NeuroCom Smart Equitest.

CONTEXT: Current tools for sideline assessment of balance following a concussion may not be sufficiently sensitive to identify impairments, which may place athletes at risk for future injury. Quantitative field-expedient balance assessments are becoming increasingly accessible in sports medicine and may improve sensitivity to enable clinicians to more readily detect these subtle deficits. OBJECTIVE: To determine the validity of the postural sway assessment on the Biodex BioSway™ compared with the gold standard NeuroCom Smart Equitest System. DESIGN: Cross-sectional cohort study. SETTING: Clinical research laboratory. PARTICIPANTS: Forty-nine healthy adults (29 females: 24.34 [2.45] y, height 163.65 [7.57] cm, mass 63.64 [7.94] kg; 20 males: 26.00 [3.70] y, height 180.11 [7.16] cm, mass 82.97 [12.78] kg). INTERVENTION(S): The participants completed the modified clinical test of sensory interaction in balance on the Biodex BioSway™ with 2 additional conditions (head shake and firm surface; head shake and foam surface) and the Sensory Organization Test and Head Shake Sensory Organization Test on the NeuroCom Smart Equitest. MAIN OUTCOME MEASURES: Interclass correlation coefficient and Bland-Altman limits of agreement for Sway Index, equilibrium ratio, and area of 95% confidence ellipse. RESULTS: Fair-good reliability (interclass correlation coefficient = .48-.65) was demonstrated for the stance conditions with eyes open on a firm surface. The Head Shake Sensory Interaction and Balance Test condition on a firm surface resulted in fair reliability (interclass correlation coefficient = .50-.59). The authors observed large ranges for limits of agreement across outcome measures, indicating that the systems should not be used interchangeably. CONCLUSIONS: The authors observed fair reliability between BioSway™ and NeuroCom, with better agreement between systems with the assessment of postural sway on firm/static surfaces. However, the agreement of these systems may improve by incorporating methods that mitigate the floor effect in an athletic population (eg, including a head shake condition). BioSway™ may provide a surrogate field-expedient measurement tool.

Clinical Assessment and Thickness Changes of the Oblique and Multifidus Muscles Using a Novel Screening Tool and Exercise Program: A Randomized Controlled Trial.

CONTEXT: Training and assessment of the abdominal and trunk muscles are widely used in the clinical setting. However, it is unknown what types of exercises are most effective in activation of both the global and local stabilizers in these regions. OBJECTIVE: The purpose of this study was to establish the reliability of a novel clinical screening tool (sling screen) to assess the muscles of the abdomen and trunk. The second aim was to use the clinical screening tool and musculoskeletal ultrasound to compare the effects of a rotary-based exercise program that targets both the global and local muscles to the effects of a traditional exercise program on the activation of the abdominal and trunk muscles. DESIGN: Double-blind, randomized controlled trial. SETTING: Sports medicine facility. PARTICIPANTS AND INTERVENTIONS: Thirty-one healthy participants were randomly allocated to receive a single-session rotary-based or traditional "core" exercise program. MAIN OUTCOME MEASURES: The participants were assessed at the baseline and immediately postintervention. The primary outcome measures were muscle thickness examined by musculoskeletal ultrasound and clinical examination of muscle activation using a screening tool. The data were collected by blind assessors. Reliability and validity of a clinical screening tool (sling screen) were also assessed. RESULTS: The analysis of the covariance tests showed a significant increase in oblique thickness for the rotary exercise group. All participants displayed a significant increase in multifidus thickness. The Wilcoxon signed-rank tests revealed a significant increase in clinical assessment scores in the rotary exercise group but not the traditional exercise group. Reliability of the sling screen ranged from moderate to good. CONCLUSION: This clinical trial provides evidence that a rotary-based exercise program may be more effective in producing increases in oblique muscle thickness than traditional "core" exercises in young, healthy adults. The sling screen tool was able to identify these muscle thickness changes. Future studies should investigate how these results correlate to injury risk, other populations, and also how to implement the sling screen into clinical practice.

Quality of clinical assessment and child mortality: a three-country cross-sectional study.

This analysis describes specific gaps in the quality of health care in Central Africa and assesses the association between quality of clinical care and mortality at age 2-59 months. Regionally representative facility and household surveys for the Democratic Republic of the Congo, Cameroon and Central African Republic were collected between 2012 and 2016. These data are novel in linking facilities with households in their catchment area. Compliance with diagnostic and danger sign protocols during sick-child visits was observed by trained assessors. We computed facility- and district-level compliance indicators for patients aged 2-59 months and used multivariate multi-level logistic regression models to estimate the association between clinical assessment quality and mortality at age 2-59 months in the catchment areas of the observed facilities. A total of 13 618 live births were analysed and 1818 sick-child visits were directly observed and used to rate 643 facilities. Eight percent of observed visits complied with 80% of basic diagnostic protocols, and 13% of visits fully adhered to select general danger sign protocols. A 10% greater compliance with diagnostic protocols was associated with a 14.1% (adjusted odds ratio (aOR) 95% CI: 0.025-0.244) reduction in the odds of mortality at age 2-59 months; a 10% greater compliance with select general danger sign protocols was associated with a 15.3% (aOR 95% CI: 0.058-0.237) reduction in the same odds. The results of this article suggest that compliance with recommended clinical protocols remains poor in many settings and improvements in mortality at age 2-59 months could be possible if compliance were improved.

Use of a virtual patient simulation program to enhance the physical assessment and medical history taking skills of doctor of pharmacy students.

INTRODUCTION: To evaluate if utilizing a virtual simulated patient program in a required pharmacy course improves the health history taking and physical assessment skills of first-year professional pharmacy students. METHODS: During a 14-week course students registered for a web-based virtual patient simulation program. Students were assigned a virtual patient case to complete during weeks one and 13 of the course. Scores from the pre- and post-interventions were compared focusing on the students' ability to perform a physical exam, take a medical history, provide patient education, and display empathy. Students were also asked to self-assess their skills after having been exposed to the virtual simulation program throughout the semester. RESULTS: The total number of paired observations used in the final analysis of the pre- and post-virtual patient test case was 171 students. The overall student performance index, which includes the scoring of subjective/objective data and patient education/empathy, improved from 52% at week one to 78.42% at week 14. The individual scoring of subjective/objective data and patient education/empathy significantly improved from week one to week 13. Approximately 76.54% of the class strongly agreed or agreed that the virtual simulation program made them feel more confident in their ability to obtain a medical history from patients. CONCLUSIONS: Virtual patient simulation programs are effective teaching methods for students to develop and improve upon their medical history taking and physical assessment skills.

Improving the assessment of communication competencies in a national licensing OSCE: lessons learned from an experts' symposium.

BACKGROUND: As the communication competencies of physicians are crucial for providing optimal patient care, their assessment in the context of the high-stakes Objective Structured Clinical Examination (OSCE) is of paramount importance. Despite abundant literature on the topic, evidence-based recommendations for the assessment of communication competencies in high stakes OSCEs are scarce. As part of a national project to improve communication-competencies assessments in the Swiss licensing exam, we held a symposium with national and international experts to derive corresponding guidelines. METHODS: Experts were invited on account of their recognized expertise either in teaching or assessing communication competencies, or in conducting national high-stakes OSCEs. They were asked to propose concrete solutions related to four potential areas for improvement: the station design, the rating tool, the raters' training, and the role of standardized patients. Data gene.rated in the symposium was available for analysis and consisted of video recordings of plenary sessions, of the written summaries of group work, and the cards with participants' personal take-home messages. Data were analyzed using a thematic analysis approach. RESULTS: Nine major suggestions for improving communication-competencies assessments emerged from the analysis and were classified into four categories, namely, the roles of the OSCE scenarios, rating tool, raters' training, and simulated patients. CONCLUSION: In the absence of established evidence-based guidelines, an experts' symposium facilitated the identification of nine practical suggestions for improving the assessment of communication competencies in the context of high-stakes OSCEs. Further research is needed to test effectiveness of the suggestions and how they contribute to improvements in the quality of high-stakes communication-competencies assessment.

Interobserver Agreement in the Assessment of Clinical Findings in Children with Headaches.

OBJECTIVE: To determine the interobserver agreement of history and physical examination findings in children undergoing evaluation in the emergency department (ED) for headaches. STUDY DESIGN: We conducted a prospective, cross-sectional study of children aged 2-17 years evaluated at 3 tertiary-care pediatric EDs for non-traumatic headaches. Two clinicians independently completed a standardized assessment of each child and documented the presence or absence of history and physical examination variables. Unweighted κ statistics were determined for 68 history and 24 physical examination variables. RESULTS: We analyzed 191 paired observations; median age was 12 years, with 19 (9.9%) children younger than 7 years. Interrater reliability was at least moderate (κ ≥ 0.41) for 41 (60.3%) patient history variables. Eleven (61.1%) of 18 physical examination variables for which κ statistics could be calculated had a κ that was at least moderate. CONCLUSIONS: A substantial number of history and physical examination findings demonstrated at least moderate κ statistic values when assessed in children with headaches in the ED. These variables may be generalizable across different types of clinicians for evaluation of children with headaches. If also found to predict the presence or absence of emergent intracranial abnormalities, the more reliable clinical findings may be helpful in the development of clinical prediction rules or risk stratification models that could be used across settings for children with headaches.

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies.

BACKGROUND: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. METHOD: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. RESULTS: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. CONCLUSIONS: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements.

Burns and Scalds Assessment Template: standardising clinical assessment of childhood burns in the emergency department.

OBJECTIVES: The Burns and Scalds Assessment Template (BaSAT) is an evidence-based proforma coproduced by researchers and ED staff with the aim of (1) standardising the assessment of children attending ED with a burn, (2) improving documentation and (3) screening for child maltreatment. This study aimed to test whether the BaSAT improved documentation of clinical, contributory and causal factors of children's burns. METHODS: A retrospective before-and-after study compared the extent to which information was recorded for 37 data fields after the BaSAT was introduced in one paediatric ED. Pre-BaSAT, a convenience sample of 50 patient records of children who had a burn was obtained from the hospital electronic database of 2007. The post-BaSAT sample included 50 randomly selected case notes from 2016/2017 that were part of another research project. Fisher's exact test and Mann-Whitney U tests were conducted to test for statistical significance. RESULTS: Pre-BaSAT, documentation of key data fields was poor. Post-BaSAT, this varied less between patients, and median completeness significantly (p<0.001) increased from 44% (IQR 4%-94%) to 96% (IQR 94%-100%). Information on 'screening for maltreatment, referrals to social care and outcome' was poorly recorded pre-BaSAT (median of 4% completed fields) and showed the greatest overall improvement (to 95%, p<0.001). Documentation of domestic violence at home and child's ethnicity improved significantly (p<0.001) post-BaSAT; however, these were still not recorded in 36% and 56% of cases, respectively. CONCLUSION: Introduction of the BaSAT significantly improved and standardised the key clinical data routinely recorded for children attending ED with a burn.

Value of clinical examination in the assessment of penetrating neck injuries: a retrospective study of diagnostic accuracy test.

BACKGROUND: There are many high-volume trauma centers in limited resource environments where a thorough clinical examination of patients may contribute to a more economical, accurate, and widely applicable method of determining the proper management of patients with penetrating neck injuries. The purpose of this study was to validate thorough physical examination as a reliable diagnostic tool in these patients. METHODS: We performed an observational retrospective study of a diagnostic accuracy test where we compared clinical findings (symptoms and soft signs on admission of the patient) with the definitive findings according to the gold standard test for each particular situation (selective studies, clinical observation and surgical exploration). The study was conducted at Hospital Occidente Kennedy (HOK) between August 2009 and June 2010. RESULTS: The sample consisted of the clinical records of 207 (n = 207) patients who went to the emergency room for penetrating neck wounds at Hospital Occidente Kennedy (HOK). Of the total sample, 36.2% (n = 75) of patients were considered "asymptomatic" as they didn't present with any soft signs of injury. Vascular soft signs were present in 57% (n = 118) of the patients, soft signs of the airway and the upper gastrointestinal tract were present in 15.9% (n = 33) and 21.3% (n = 44) of the patients respectively. The sensitivity and negative predictive value (NPV) of any soft sign to determine injuries which require surgical repair was 97.4% [CI] [86.5-99.5%] and 98.7% [CI] [92.8-99.8%] respectively, with a range of confidence [CI] of 95%. CONCLUSIONS: Our study's main findings suggest that patients with neck injuries and no vascular, airway, or gastrointestinal soft sign can be safely managed with a conservative approach. It is important to emphasize the value of the clinical examination since there are many contexts in the modern world where a considerable amount of the population is afflicted by neck trauma and treated under conditions where technological resources are limited.

Validity of the Sock Test for Sitting Balance: A Functional Sitting Balance Assessment.

There is a relationship between sitting balance and self-care functioning, but no occupation-based sitting balance assessment. The objective of this study was to determine whether the Sock Test for Sitting Balance (STSB) is a valid balance assessment for use by therapists in acute care. Results of the STSB were compared with those of the Adapted Functional Reach (AFR), the Functional Independence Measure (FIM™) chair transfer, and the Kansas University Sitting Balance Scale (KUSBS), in both hospitalized and healthy participants. There is a significant relationship between STSB results and AFR results for all participants (rs = -.382, p = .012). Relationships between STSB results and score on the FIM™ Chair Transfer subscale (p = .001), and score on the KUSBS (p = .003) are significant. The STSB can differentiate between a hospitalized population and healthy population (p = .017). Criterion and known-groups validity are demonstrated, but should be confirmed with further testing.

How Consistent Is Competent? Examining Variance in Psychomotor Skills Assessment.

PURPOSE: Direct assessment of trainee performance across time is a core tenet of competency-based medical education. Unlike variability of psychomotor skills across levels of expertise, performance variability exhibited by a particular trainee across time remains unexplored. The goal of this study was to document the consistency of individual surgeons' technical skill performance. METHOD: A secondary analysis of assessment data (collected in 2010-2012, originally published in 2015) generated by a prospective cohort of participants at Montreal Children's Hospital with differing levels of expertise was conducted in 2017. Trained raters scored blinded recordings of a myringotomy and tube insertion performed 4 times by junior and senior residents and attending surgeons over a 6-month period using a previously reported assessment tool. Descriptive exploratory analyses and univariate comparison of standard deviations (SDs) were conducted to document variability within individuals across time and across training levels. RESULTS: Thirty-six assessments from 9 participants were analyzed. The SD of scores for junior residents was highly variable (5.8 out of a scale of 30 compared with 1.8 for both senior residents and attendings [F(2,19) = 5.68, P < 0.05]). For a given individual, the range of scores was twice as large for junior residents than for senior residents and attendings. CONCLUSIONS: Surgical residents may display highly variable performances across time, and individual variability appears to decrease with increasing expertise. Operative skill variability could be underrepresented in direct observation assessment; emphasis on an adequate amount of repetitive evaluations for junior residents may be needed to support judgments of competence or entrustment.