Bronchoscopy Teaching Without a Gold Standard: Attending Pulmonologists' Assessment of Learners, Supervisory Styles, and Variation in Practice.

BACKGROUND: Despite the growing role of simulation in procedural teaching, bronchoscopy training largely is experiential and occurs during patient care. The Accreditation Council for Graduate Medical Education sets a target of 100 bronchoscopies to be performed during pulmonary fellowship. Attending physicians must balance fellow autonomy with patient safety during these clinical teaching experiences. Few data on best practices for bronchoscopy teaching exist, and a better understanding of how bronchoscopy currently is supervised could allow for improvement in bronchoscopy teaching. RESEARCH QUESTION: How do attending bronchoscopists supervise bronchoscopy, and in particular, how do attendings balance fellow autonomy with patient safety? STUDY DESIGN AND METHODS: This was a focused ethnography conducted at a single center using audio recording of dialog between attendings and fellows during bronchoscopies, supplemented by observation of nonverbal teaching. Interviews with attending bronchoscopists and limited interviews of fellows also were recorded. Interviews were transcribed verbatim before analysis. We used constant comparative analysis to analyze data and qualitative research software to support data organization and thematic analysis. Education researchers from outside of pulmonary critical care joined the team to minimize bias. RESULTS: We observed seven attending bronchoscopists supervising eight bronchoscopies. We noted distinct teaching behaviors, classified into themes, which then were grouped into four supervisory styles of modelling, coaching, scaffolding, and fading. Observation and interviews illuminated that assessing fellow skill was one tool used to choose a style, and attendings moved between styles. Attendings accepted some, but not all, variation in both performing and supervising bronchoscopy. INTERPRETATION: Attending pulmonologists used a range of teaching microskills as they moved between different supervisory styles and selectively accepted variation in practice. These distinct approaches may create well-rounded bronchoscopists by the end of fellowship training and should be studied further.

Using the 4Ms framework to teach geriatric competencies in a community clinical experience.

BACKGROUND: As the population of older adults in the US steadily increases and becomes more diverse, there is an urgent need to integrate geriatric competencies into baccalaureate nursing education. PURPOSE: To integrate the Institute for Healthcare Improvement 4 Ms Framework into an existing baccalaureate nursing community clinical experience to build geriatric and interprofessional competencies and promote positive health outcomes. METHODS: As part of the Geriatric Workforce Enhancement Program, 15 students worked with bilingual social workers and community health workers in an affordable housing urban highrise, assessed building residents and implemented personalized plans of care using the 4Ms framework (what matters to the individual, medications, mentation, and mobility). RESULTS: Students demonstrated competence conducting cognition and depression screening, medication review, and functional and fall risk assessments. Student self-rated achievement of learning objectives ranged from 4.3 to 4.8 (1-5 scale). A retrospective pretest-posttest survey suggested learning about the importance of interprofessional teamwork, and integration of person-centered values when providing care to older adults in the community. Students reflected on barriers to health for older adults in low socioeconomic states and the importance of improving care across the continuum. CONCLUSION: The 4Ms framework provided a valuable construct to guide the community experience and teach geriatric evidence-based practice to nursing students.

Impact of an electronic-based assessment on student pharmacist performance in a required therapeutics course.

INTRODUCTION: The use of technology in the classroom has continued to grow, and with the advancement of classroom management systems and online exam software, there are opportunities to administer exams electronically. This study assessed the impact of electronic-based assessments on examination scores in a required therapeutics course. METHODS: This was a retrospective, single-centered, observational study including second professional year pharmacy students enrolled in a required, one semester therapeutics course. Four assessments were administered each semester. Lecture content and exam format, a mixture of multiple-choice questions and free response written cases, did not differ significantly between years. Assessments administered during the first two years were printed on paper, while assessments administered during the third and fourth year of the study were all electronic, submitted through a classroom management system. Following institutional review board approval, the change in mean overall examination scores between paper and electronic-based assessments were analyzed. RESULTS: Of the 948 students enrolled in this study, there was no difference in overall mean scores between paper and electronic-based assessments (74.8% vs. 73.8%). In addition, there was no difference in mean examination scores between overall individual paper and electronic Exam 1 through 4 or overall multiple-choice or free response scores between paper and electronic-based assessments. CONCLUSIONS: Scores did not differ between paper and electronic-based assessments. From this study, test method does not appear to impact exam results.

Assessment of structured classroom debate to teach an antimicrobial stewardship elective course.

BACKGROUND AND PURPOSE: The main aim of this study was to evaluate the effect of a structured classroom debate format on teaching antimicrobial stewardship. EDUCATIONAL ACTIVITY AND SETTING: An active learning approach using a debate format was implemented to engage students in infectious diseases concepts to further develop critical thinking skills. This was a one-group, pre- and posttest design conducted in third year pharmacy students enrolled at the Philadelphia College of Osteopathic Medicine School of Pharmacy Georgia campus. A ten-item assessment survey was used prior to and after the course to evaluate student knowledge. Student perception of skill development was assessed by a survey using a five-point Likert scale. The skills assessed included critical thinking, communication, public speaking, research/drug information, and teamwork. FINDINGS: Thirty-three students participated in the six debates over the course of the semester. There was a statistically significant increase in post-knowledge assessment mean score (75%) compared to pre-knowledge assessment mean score (45%). The post-course survey showed improved confidence in perception of skills in all of the areas assessed. SUMMARY: The structured classroom debate format has a positive association with increasing students' knowledge level and perception of skills assessed.

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance.

Construct: This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. Background: In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. Approach: A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Results: Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). Conclusions: The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.

Mind the Gap: Improving Care in Pediatric-to-Adult Rheumatology Transition Through Education.

Adolescence and young adulthood represents a vulnerable period in the life of patients with rheumatic diseases, many of whom struggle with self-management of their chronic disease and adjustment to care with adult rheumatologists. Many adult rheumatologists are uncomfortable caring for young adults, especially those with pediatric-onset diagnoses. The recent development of evidence-based best practices, expert opinion recommendations, and validated tools can assist pediatric and adult rheumatologists in caring for adolescent and young adult patients. Uptake of these guidelines and resources remains low, underscoring the need for rheumatologists and trainees to receive explicit training in the application of transition best practices.

Education and Professional Development in Rheumatology: Translating Quality Improvement and Education to Clinical Practice.

Academic institutions play an essential role in providing physicians with the necessary skills needed to improve the quality and value of care provided. The Accreditation Council for Graduate Medical Education has several milestones in the curriculum of fellowship programs that prioritize involvement in quality improvement (QI) and patient safety efforts. This article reviews the unique benefits and challenges of involving and teaching fellows to conduct QI initiatives. Strategies and resources available to overcome these challenges and develop a successful quality driven training environment are outlined in this article.

Interactive pedagogic tools: evaluation of three assessment systems in medical education.

Training in biology, pharmacy and medicine are essential in laboratory medicine in faculty and especially with recent residency modifications. Active learning improves critical thinking and is an essential component of health education. Interactive assessment systems for the interactive participation of students have emerged. Recently, many offers of audience response system (ARS) accessible by personal electronic devices such as smartphone, tablet or computer are available. These systems seem to be an effective teaching innovation according to students. We aimed to evaluate three pedagogical tools during real school lectures in order to be able to select them according to the needs: Votar, Socrative and Wooclap. Methods: Three connected participation tools will be tested during teaching at Lille University, faculty of pharmacy by 3 different teachers. 75 fifth-year pharmacy students divided into 2 groups of students will have attended at least one session using each of the systems studied. After lessons, an online questionnaire with 9 questions was submitted to students on their interest in each system. Questions measured student perception using a 1 to 10 scale. Results and discussion: 62 of 75 students completed online surveys and were included in the study. According students, ARS by smartphone or computer improve their education. Favorite application seems to be Socrative and Wooclap. This study provides student perception comparison of ARS. To complete, additional studies are needed to establish their efficacy after several month.

Incorporation of a multi-modal interactive exercise on student pharmacists' perceptions of lifestyle modification in weight loss promotion.

BACKGROUND AND PURPOSE: The purpose of this project was to evaluate the impact of a calorie tracking and dietary modification activity on students' perceptions of the challenges associated with weight loss. EDUCATIONAL ACTIVITY AND SETTING: Students maintained a two-day paper calorie log, then received a weight management lecture, followed by four additional days of calorie tracking using a mobile application. Demographics, self-evaluation of obesity-related knowledge, perceptions on weight loss difficulties, utility of mobile technology, and overall program value were evaluated. FINDINGS: A total of 79% of students completed all three surveys (n = 150). Students' perceptions regarding their knowledge and comfort with respect to weight loss counseling and dietary modification strategies increased following the lecture and mobile technology-based calorie counting activity. Seventy-eight percent of students either agreed or strongly agreed that this activity was valuable to their professional development while 86% of students agreed or strongly agreed this activity helped them understand the challenges faced by patients trying to lose weight. DISCUSSION: Student attitudes regarding weight loss, including nutritional tracking, and comfort in establishing a weight loss goal, as well as student's perceived knowledge of the types of dietary fats and empathy, all significantly improved over the course of activity participation. SUMMARY: This activity improved student pharmacists' understanding of lifestyle modifications necessary for weight loss, exposed them to novel technologies available to help support patient's weight loss attempts, and provided them with first-hand experience in tracking calories.

Tutor assessment of PBL process: does tutor variability affect objectivity and reliability?

BACKGROUND: Ensuring objectivity and maintaining reliability are necessary in order to consider any form of assessment valid. Evaluation of students in Problem-Based Learning (PBL) tutorials by the tutors has drawn the attention of critiques citing many challenges and limitations. The aim of this study was to determine the extent of tutor variability in assessing the PBL process in the Faculty of Medical Sciences, The University of the West Indies, St Augustine Campus, Trinidad and Tobago. METHOD: All 181 students of year 3 MBBS were assigned randomly to 14 PBL groups. Out of 18 tutors, 12 had an opportunity to assess three groups: one assessed 2 groups and 4 tutors assessed one group each; at the end each group had been assessed three times by different tutors. The tutors used a PBL assessment rating scale of 12 different criteria on a six-point scale to assess each PBL Group. To test the stated hypotheses, independent t-test, one-way ANOVA followed by post-hoc Bonferroni test, Intra Class Correlation, and Pearson product moment correlations were performed. RESULT: The analysis revealed significant differences between the highest- and lowest-rated groups (t-ratio = 12.64; p < 0.05) and between the most lenient and most stringent raters (t-ratio = 27.96; p < 0.05). ANOVA and post-hoc analysis for highest and lowest rated groups revealed that lenient- and stringent-raters significantly contribute (p < 0.01) in diluting the score in their respective category. The intra class correlations (ICC) among rating of different tutors for different groups showed low agreement among various ratings except three groups (Groups 6, 8 and 13) (r = 0.40). The correlation between tutors' PBL experiences and their mean ratings was found to be moderately significant (r = 0.52; p > 0.05). CONCLUSION: Leniency and stringency factors amongst raters affect objectivity and reliability to a great extent as is evident from the present study. Thus, more rigorous training in the areas of principles of assessment for the tutors are recommended. Moreover, putting that knowledge into practice to overcome the leniency and stringency factors is essential.

Re-using questions in classroom-based assessment: An exploratory study at the undergraduate medical education level.

INTRODUCTION: To alleviate some of the burden associated with the development of novel quality questions on a regular basis, medical education programs may favour the use of item banks. This practice answers the real pragmatic need of having to create exams de novo at each administration while benefiting from using psychometrically sound questions to assess students. Unfortunately, programs cannot prevent trainees from engaging in cheating behaviours such as content sharing, and little is known about the impact of re-using items. METHODS: We conducted an exploratory descriptive study to assess the effect of repeated use of banked items within an in-house assessment context. The difficulty and discrimination coefficients for the 16-unit exams of the past 5 years (1,629 questions) were analyzed using repeated measure ANOVAs. RESULTS: Difficulty coefficients increased significantly (M = 79.8% for the first use of an item, to a mean difficulty coefficient of 85.2% for the fourth use) and discrimination coefficients decreased significantly with repeated uses (M = 0.17, 0.16, 0.14, 0.14 for the first, second, third and fourth uses respectively). DISCUSSION: The results from our study suggest that using an item three times or more within a short time span may cause a significant risk to its psychometric properties and consequently to the quality of the examination. Pooling items from different institutions or the recourse to automatic generated items could offer a greater pool of questions to administrators and faculty members while limiting the re-use of questions within a short time span.

Implementation and lessons learned from a mock trial as a teaching-learning and assessment activity.

BACKGROUND AND PURPOSE: To describe implementation and lessons learned from use of a mock trial as a teaching-learning and assessment activity in a required evidence-based practice course. This innovative self-directed learning strategy reinforced evidence-based skills and affective domain competencies in Center for the Advancement of Pharmaceutical Education (CAPE) 2013 Outcomes and Accreditation Council for Pharmacy Education (ACPE) Standards 2016. EDUCATIONAL ACTIVITY AND SETTING: During spring semesters 2015 and 2016, first professional year (P1) cohorts were divided into teams and assigned controversial topics to research and debate in mock trials. The activity provided opportunities for teaching-learning and assessment. Statistical analysis included inter-rater reliability (IRR), comparison of faculty-judge and student-juror evaluation of trial performance. FINDINGS: Two cohorts were divided into eight teams (four per cohort) to debate four issues. Students within each team were assigned individual scores. Mean individual scores are reported by trial/topic. Faculty-judges and student-jurors rated eight criteria including content/knowledge, critical thinking, application/discussion of federal/state law, citations/references, visual aids, delivery/style, and active listening. Analysis indicated students met competency expectations with overall judges' and jurors' mean scores above 54/60 points (90%) in 2015 and above 51/60 points (85%) in 2016. Judges' scores showed a wider distribution than jurors' scores. Peer evaluation mean scores were above 55/60 points in all four trials. Intra-class correlation was calculated. Judges' scores had excellent IRR in two trials, and good IRR in a third trial, whereas jurors had good IRR in one trial. SUMMARY: Evaluation scores demonstrated students successfully applied knowledge and skills from this and prior P1 courses, and met competency expectations for the mock trial.

Progress and assessment of a contextualized teaching session of pharmacology.

BACKGROUND: Learning transfer, in medical teaching, remains an essential question and optimizing it is the main preoccupation of every trainer in medical sciences. Some learning methods showed their efficacy as the contextualized learning in the framework of a professional activity or in a situation recalling it in a realistic manner. AIM: To describe steps of planning and progress of a session of clarification, illustration, application et participation (CIAP) of pharmacology teaching students from second cycle of medical studies (DCEM) and to assess the session. METHODS: We performed a descriptive transversal study in April 2017 in the Faculty of Medicine of Tunis. Our work was composed of two parts. The first part consisted in a description of the preparation and the progress of the CIAP session entitled antiepileptic drugs, which is comprised in the pharmacology teaching of the certificate of Neurology to the students of DCEM. The second part consisted in an assessment of knowledge acquisition and the progress of the session by the students. RESULTS: We proceeded to a planning of the session which resulted in a contextualized teaching and induced an active participation and an interactivity of the students. Comparison of the results of the pretest and the posttest showed a statistically significant difference in terms of good responses. The assessment of the session progress was good. CONCLUSION: Our study demonstrated the feasibility of a session of contextualized teaching session or CIAP of pharmacology and its input in terms of knowledge to the students.

Medical certificate education: controlled study between lectures and flipped classroom.

BACKGROUND: Finnish permanent residents are covered by social security insurance administered by the Social Insurance Institution of Finland. The procedure of insurance is initiated with medical certificate written by the treating doctor. Thus, the doctor must have certificate writing skills accompanied with the knowledge of the content and goals for insurance. Quality certificates are important part of doctors' professional skills worldwide and most effective teaching methods for learning these should be investigated. METHODS: Medical certificate data were collected from two independent courses of fourth-year student taught in autumn 2015 (N = 141) and 2016 (N = 142) in the medical faculty of the University of Eastern Finland. A random sample of 40 students per course was drawn for the analysis. All certificates were analyzed as one sample. This was done to obtain reliable results with internal control group on the differences between two teaching methods, the traditional approach and the flipped classroom (FC) approach, in 2015 and 2016, respectively. The medical certificates were evaluated and scored with a rubric (range: - 4.00-14.25) by two independent experienced specialists. RESULTS: Compared to students in the traditional classroom, students involved in the FC received significantly higher scores in all relevant sections of the assessed certificates. The mean of the total scores was 8.87 (SD = 1.70) for the traditional group and 10.97 (SD = 1.25) for the FC group. Based on the common language effect size, a randomly selected student from the FC group had an 85% probability of receiving a higher total score than a student from the traditional group. CONCLUSION: In this study, the FC approach resulted in a statistical significant improvement in the content and technical quality of the certificates. The results suggest that the FC approach can be applied in the teaching of medical certificate writing.

Formatting an experiential learning education module to encourage dysphagia assessment in apheresis patients.

BACKGROUND: Dysphagia screening is oftentimes a focus of hospitalized patients, but dysphagia can also occur in outpatient settings. Dysphagia can be overlooked by nurses and clinicians, and it is therefore important to educate nurses on the importance of dysphagia screenings. METHODS: This was a randomized prospective pilot study to compare the effect of experiential learning versus traditional PowerPoint learning regarding nurses' attitudes towards performing dysphagia screening in an outpatient setting. Twelve pre and post-test surveys were collected from nurses working in outpatient apheresis about their attitudes towards dysphagia screening. Additionally, 128 electronic medical records (EMR) were reviewed to determine if education increased the occurrence of dysphagia screening. RESULTS: There was a statistically significant difference in the pre vs. post-test group scores (P < .001), but due to small sample size, there was insufficient evidence to reject the null hypothesis that nurses had changed their attitudes towards dysphagia screening. Comparing documentation of dysphagia assessment in the EMR, there was not a significant difference in practice before or after the educational intervention (P = 0.18). CONCLUSIONS: The study results showed that the both types of teaching strategies are possible with nurses and they were receptive to both. Although the results of this study did not show a significant difference in practice, more research is needed to determine how to raise awareness and put this into practice.

Design and activity evaluation of an Advanced-Introductory Pharmacy Practice Experience (aIPPE) course for assessment of student APPE-readiness.

BACKGROUND AND PURPOSE: The newly implemented Advanced-Introductory Pharmacy Practice Experience (aIPPE) course at the University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences was designed to assess student readiness for Advanced Pharmacy Practice Experience (APPE) and provide formative feedback regarding skills and abilities needed to be successful during subsequent coursework and practice experiences. EDUCATIONAL ACTIVITY AND SETTING: The aIPPE is a full-time, six week (240hours) direct patient care rotation occurring in the spring semester of the third-professional year following a longitudinal integrated IPPE program. Required aIPPE course elements mimic the activities and expectations students should anticipate encountering during APPE training. FINDINGS: All students participating in the initial aIPPE course offering were described as APPE-Ready. Students and preceptors described the aIPPE course as successful in achieving the primary outcome of preparing students for improved downstream performance. SUMMARY: The aIPPE provided students opportunities to demonstrate readiness to enter advanced pharmacy practice experience.

Incorporating patient partner scores into high stakes assessment: an observational study into opinions and attitudes.

BACKGROUND: Volunteer patients (also known as patient partners (PPs)) play a vital role in undergraduate healthcare curricula. They frequently take part in objective structured clinical examinations (OSCE) and rate aspects of students' performance. However, the inclusion and weighting of PP marks varies, while attitudes and opinions regarding how (and if) they should contribute towards the pass/fail outcome are uncertain. METHODS: A prospective observational study was conducted to explore beliefs of PPs regarding inclusion of their scores in a high stakes undergraduate OSCE in a single UK medical school. All PPs delivering components of the local MBChB curriculum were asked to participate in the questionnaire study. Quantitative and qualitative data were analysed using descriptive statistics and framework analysis respectively. RESULTS: Fifty out of 160 (31% response rate) PPs completed the questionnaire; 70% had participated in a final year OSCE. Thirty (60%) felt their marks should be incorporated into a student's overall score, while 28% were uncertain. The main reasons for inclusion were recognition of the patient perspective (31%) and their ability to assess attitudes and professionalism (27%), while reasons against inclusion included lack of PP qualification/training (18%) and concerns relating to consistency (14%). The majority of PPs were uncertain what proportion of the total mark they should contribute, although many felt that 5-10% of the total score was reasonable. Most respondents (70%) felt that globally low PP scores should not result in an automatic fail and many (62%) acknowledged that prior to mark inclusion, further training was required. CONCLUSION: These data show that most respondents considered it reasonable to "formalise their expertise" by contributing marks in the overall assessment of students in a high stakes OSCE, although what proportion they believe this should represent was variable. Some expressed concerns that using marks towards progress decisions may alter PP response patterns. It would therefore seem reasonable to compare outcomes (i.e. pass/fail status) using historical data both incorporating and not incorporating PP marks to evaluate the effects of doing so. Further attention to existing PP training programmes is also required in order to provide clear instruction on how to globally rate students to ensure validity and consistency.

Development of a Holistic Assessment Plan to Evaluate a Four-Semester Laboratory Course Series.

Objective. To implement a holistic assessment plan to evaluate the impact of a four-semester laboratory course series entitled Integrated Pharmacotherapy Skills on students' readiness to begin advanced pharmacy practice experiences (APPEs) following separation of the laboratory component from the Pharmacotherapy lecture component. Design. Faculty prospectively selected and employed a variety of course assessment methods including student self- and preceptor evaluation during APPEs, course evaluations, and a student confidence survey to evaluate student readiness for APPEs and ensure a quality learning experience for students. Assessment. APPE students' self-perceived confidence to perform skills increased after completion of the redesigned curriculum and after experiencing two APPE rotations. APPE preceptors did not report a change in student performance. Results from course evaluations suggest that separating the laboratory course from the lecture course created a positive learning experience for students. Conclusion. Students completing the new laboratory curriculum were equally prepared to begin APPE rotations as students who had completed the old curriculum. A similar multidimensional, holistic assessment plan could be used at other institutions to evaluate skills-based courses as part of continuous quality improvement.

Validity evidence for two objective structured clinical examination stations to evaluate core skills of the shoulder and knee assessment.

BACKGROUND: We developed two objective structured clinical examinations (OSCEs) to educate and evaluate trainees in the evaluation and management of shoulder and knee pain. Our objective was to examine the evidence for validity of these OSCEs. METHODS: A multidisciplinary team of content experts developed checklists of exam maneuvers and criteria to guide rater observations. Content was proposed by faculty, supplemented by literature review, and finalized using a Delphi process. One faculty simulated the patient, another rated examinee performance. Two faculty independently rated a portion of cases. Percent agreement was calculated and Cohen's kappa corrected for chance agreement on binary outcomes. Examinees' self-assessment was explored by written surveys. Responses were stratified into 3 categories and compared with similarly stratified OSCE scores using Pearson's coefficient. RESULTS: A multi-disciplinary cohort of 69 examinees participated. Examinees correctly identified rotator cuff and meniscal disease 88% and 89% of the time, respectively. Inter-rater agreement was moderate for the knee (87%; k = 0.61) and near perfect for the shoulder (97%; k = 0.88). No correlation between stratified self-assessment and OSCE scores were found for either shoulder (0.02) or knee (-0.07). CONCLUSIONS: Validity evidence supports the continuing use of these OSCEs in educational programs addressing the evaluation and management of shoulder and knee pain. Evidence for validity includes the systematic development of content, rigorous control of the response process, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessment suggests that these OSCEs measure a construct different from learners' self-confidence.