Primary prevention for resettled refugees from Burma: where to begin?

Developing effective primary prevention initiatives may help recently arrived refugees retain some of their own healthy cultural habits and reduce the tendency to adopt detrimental ones. This research explores recent arrivals' knowledge regarding eating behaviors, physical activity and sleep habits. Working collaboratively with community members, a healthy living curriculum was adapted and pilot tested in focus groups. A community-engaged approach to revising and implementing a health promotion tool was effective in beginning dialogue about primary prevention among a group of recently arrived refugees from Burma. Seven themes were identified as particularly relevant: food choices, living environment, health information, financial stress, mobility/transportation, social interaction and recreation, and hopes and dreams. Refugees desire more specific information about nutrition and exercise, and they find community health workers an effective medium for delivering this information. The outcomes of this study may inform future targeted interventions for health promotion with refugees from Burma.

Comparison of blogged and written reflections in two medicine clerkships.

CONTEXT: academic medical centres may adopt new learning technologies with little data on their effectiveness or on how they compare with traditional methodologies. We conducted a comparative study of student reflective writings produced using either an electronic (blog) format or a traditional written (essay) format to assess differences in content, depth of reflection and student preference. METHODS: students in internal medicine clerkships at two US medical schools during the 2008-2009 academic year were quasi-randomly assigned to one of two study arms according to which they were asked to either write a traditional reflective essay and subsequently join in faculty-moderated, small-group discussion (n = 45), or post two writings to a faculty-moderated group blog and provide at least one comment on a peer's posts (n = 50). Examples from a pilot block were used to refine coding methods and determine inter-rater reliability. Writings were coded for theme and level of reflection by two blinded authors; these coding processes reached inter-rater reliabilities of 91% and 80%, respectively. Anonymous pre- and post-clerkship surveys assessed student perceptions and preferences. RESULTS: student writing addressed seven main themes: (i) being humanistic; (ii) professional behaviour; (iii) understanding caregiving relationships; (iv) being a student; (v) clinical learning; (vi) dealing with death and dying, and (vii) the health care system, quality, safety and public health. The distribution of themes was similar across institutions and study arms. The level of reflection did not differ between study arms. Post-clerkship surveys showed that student preferences for blogging or essay writing were predicted by experience, with the majority favouring the method they had used. CONCLUSIONS: our study suggests there is no significant difference in themes addressed or levels of reflection achieved when students complete a similar assignment via online blogging or traditional essay writing. Given this, faculty staff should feel comfortable in utilising the blog format for reflective exercises. Faculty members could consider the option of using either format to address different learning styles of students.

Correctional health curriculum enhancement through focus groups.

BACKGROUND: Little has been published describing curricular experiences in correctional health (CH). PURPOSES: Our goal is to articulate a curriculum cognizant of the special needs of the correctional health care worker. METHODS: We conducted focus groups with nurses, nurse practitioners, physician assistants, and physicians focused on content crucial to prepare competent medical professionals committed to careers in correctional health. RESULTS: Six main themes emerged from the data, which were used to confirm and add to the growing curriculum on correctional health used in our university's correctional health electives. The themes are (a) characteristics of the population being served; (b) prevalent conditions requiring clinical expertise; (c) public health opportunities in correctional facilities; (d) ethical considerations; (e) medical-legal issues; and (d) the CH system, structure, and administration. CONCLUSIONS: The successful provision of health care in correctional settings requires specialized knowledge, skills, and awareness not typically available in other health care training settings.

Between two worlds: a multi-institutional qualitative analysis of students' reflections on joining the medical profession.

BACKGROUND: Recent changes in healthcare system and training mandates have altered the clinical learning environment. We incorporated reflective writing into Internal Medicine clerkships (IMcs) in multiple institutions so students could consider the impact of clerkship experiences on their personal and professional development. We analyzed student reflections to inform curricula and support learning. METHODS: We qualitatively analyzed the reflections of students at 3 US medical schools during IMcs (N = 292) to identify themes, tone, and reflective quality using an iterative approach. Chi-square tests assessed differences between these factors and across institutions. FINDINGS: Students openly described powerful experiences. Major themes focused on 4 categories: personal issues (PI), professional development (PD), relational issues (RI), and medical care (MC). Each major theme was represented at each institution, although with significant variability between institutions in many of the subcategories including student role (PI), development-as-a-physician (PD), professionalism (PD) (p < 0.001). Students used positive tones to describe student role, development-as-a-physician and physician-patient relationship (PD) (p < 0.01-0.001), and negative tones for quality and safety (MC) (p < 0.05). Only 4% of writings coded as professionalism had a positive tone. Students employed a "reporting" voice in writing about clinical problem-solving, healthcare systems, and quality/safety (MC). DISCUSSION: Reflection is considered important to professional development. Our analysis suggests that students at 3 institutions reflect on similar experiences. Theme variability across institutions implies curricula should be tailored to local culture. Reflective quality analysis suggests students are better equipped to reflect on certain experiences over others, which may impact learning. Student reflections can function as a mirror for our organizations, offer institutional feedback for support and improvement, and inform curricula for learners and faculty.

How patients perceive a doctor's caring attitude.

OBJECTIVE: Caring is closely associated with reduced malpractice litigation, adherence to treatment and even symptom relief. Caring also is included in pay for performance formulas as well as widely utilized for quality improvement purposes. Our objective in this prospective qualitative study was to define caring behaviors associated with three challenging encounters: discussing the transition from curative to palliative care, delivering bad news (cancer), and discussing a medical error (misplaced test result). The purpose was to lay the groundwork for the creation of a 'patient-centered' caring attitude checklist that could help the healthcare provider understand and ultimately enhance the patient's experience of care. METHODS: Groups of randomly selected lay people, henceforth referred to as patients: (1) engaged in 'think aloud' exercises to help create a 15-item caring behavior checklist; (2) used the checklist to rate videotapes of simulated challenging encounters conducted by twenty primary care physicians (total of 600 ratings sets); and (3) participated in 12 separate 1.5 h focus groups discussing the caring (and non-caring) behaviors exhibited in videotapes of the highest and lowest rated encounters. RESULTS: Thirteen behaviors emerged as focal for describing a doctor's caring attitude but with disagreement as to whether specific examples of these behaviors were 'caring' or 'uncaring.' For example, although the concept of empathic inquiry was considered important by most patients, the physician question, "Is there someone you can call or talk with" (about a cancer diagnosis) was interpreted by one patient as 'very caring' while another was 'impressed with how uncaring' the statement appeared. CONCLUSION: At the conceptual level there is a set of behaviors that represent caring, however, the manifestation of these behaviors is 'in the eye of the beholder.' The most important element of caring may not be the set of behaviors but a set of underlying abilities that include taking the patient's perspective and reflecting on the patient's responses. PRACTICE IMPLICATIONS: Medical education must focus on the underlying abilities of caring.

The role of relationships in the professional formation of physicians: case report and illustration of an elicitation technique.

OBJECTIVE: Studies of physicians' professional development highlight the important effect that the learning environment has in shaping student attitudes, behaviors, and values. The objective of this study was to better understand the interplay among relationships and experiences in mediating the effects of the learning environment. METHODS: We randomly recruited 2nd- and 4th-year students from among volunteers at each of five medical schools. One interviewer at each school conducted a face-to-face, open-ended, semi-structured interview with each student. The interviewers used a method called 'life-circle diagramming' to direct the student to draw a picture of all of the relationships in his/her life that had an influence on the sort of doctor that each student saw him/herself becoming. Interviews lasted between 60 and 120 min. Using a narrative framework that focuses on elements of students' stories (e.g., setting, characters, plot), we analyzed transcripts through an iterative process of individual reading and group discussion to derive themes and relationships among themes. RESULTS: Twenty students completed interviews. These students are embedded in complex webs of relationships with colleagues, friends, family, role models, patients, and others. Most students entered medical school with formed notions of what they wanted to 'be like' as physicians. While students generally gravitated toward relationships with like-minded people, their experiences varied, and some students could sense themselves changing as they moved through school. Such changes were often related to important events or issues. The relationships that students found themselves in during the context of these events had an important effect on students' beliefs about what kinds of behaviors and attitudes were possible and desirable in their future practice. CONCLUSIONS: Students proceed through medical school embedded in complex webs of relationships that exert a powerful influence (both positive and negative) on their formation as physicians. PRACTICE IMPLICATIONS: Educational interventions that foster adoption of professional values need to acknowledge the influence of relationships, and assist students to harness and shape relational effects on their growth and development. The life-circle diagramming activity holds potential to promote reflection and self-knowledge, and to provide a foundation for professional growth.

Factors influencing preceptors' responses to medical errors: a factorial survey.

BACKGROUND: Preceptors must respond to trainees' medical errors, but little is known about what factors influence their responses. METHOD: A total of 115 primary care preceptors from 16 medical schools responded to two medical error vignettes involving a trainee. Nine trainee-related factors were randomly varied. Preceptors indicated whether they would discuss what led to the error, provide reassurance, share responsibility, express disappointment, and adjust their written evaluation of the trainee. RESULTS: Almost all preceptors would discuss what led to the error; relatively few would express disappointment. The trainee's prior history of errors, knowledge level relative to peers, receptivity to feedback, training level, emotional reaction, offering to apologize, and offering an excuse were predictive of preceptors' responses; gender and time-in-office were not. CONCLUSION: This study identified seven trainee-related factors as predictive of preceptors' responses to medical errors. More research is needed to identify other influential factors, and to improve teaching from medical errors.

Primary care renewal: regional faculty development and organizational change.

BACKGROUND: Many reports, including the Future of Family Medicine, have called for change in primary care, but few have defined, implemented, and evaluated mechanisms to address such change. The regional, interdisciplinary Primary Care Renewal Project was designed to address problems in primary care practice and teaching related to practice management, compensation, increasing responsibility for teaching, and faculty development. METHODS: Twelve northeastern US medical schools assembled a conference attended by teams of key stakeholders representing both clinical and educational missions. Teams developed and implemented an institutional plan to address identified needs. Outcome data was collected during, and for 1 year after, the conference. RESULTS: Findings demonstrate novel ways of improving learning experiences, coordinating and centralizing planning efforts, and addressing faculty needs. The magnitude of organizational change ranged from establishing new administrative units with significant institutional authority (eg, restructuring dean's office) to enhancing the strategic planning process and refining mission statements to reflect emphasis on primary care. CONCLUSIONS: A well-planned, regional interdisciplinary effort that fosters the development of concrete plans can be associated with significant change in medical education. A central theme emerged--that primary care medicine will survive only if institutions align their educational and clinical missions and foster system-wide change.

Academic-correctional health partnerships: preparing the correctional health workforce for the changing landscape-focus group research results.

Providing health care in corrections is challenging. Attracting clinicians can be equally challenging. The future holds a shortage of nurses and primary care physicians. We have a unique opportunity, now, to develop and stabilize our workforce, create a positive image, and enhance quality before the health care landscape changes even more dramatically. Focus groups were conducted with 22 correctional health care professionals divided into three groups: physicians (6), nurses (4), and nurse practitioners/physician assistants (12). Content focused on curricular themes, but additional themes emerged related to recruitment and retention. This article describes recruitment challenges, strategic themes identified, and the proposed initiatives to support a stable, high-quality correctional health workforce.

Community preceptors' views of their identities as teachers.

PURPOSE: Community physicians are increasingly being recruited to teach medical students and residents, yet there has been little research about how they think of themselves as teachers or what factors contribute to "teacher identity." Physicians who think of themselves as teachers may be more likely to enjoy teaching, to teach more, and to be recognized by students and other faculty as good teachers. Identifying factors that enhance teacher identity may be helpful for the recruitment and retention of high-quality community faculty. METHOD: Thirty-five experienced community preceptors were audiotaped in five structured focus groups in April 2001, answering a series of questions about their teacher identity. Responses were qualitatively analyzed for evidence of themes. RESULTS: "Feeling intrinsic satisfaction" was the most common theme that emerged from the tapes. Preceptors also identified that "having knowledge and skill about teaching" and "belonging to a group of teacher" enhanced their roles as teachers. "Being a physician means being a teacher," "feeling a responsibility to teach medicine," and "sharing clinical expertise" also emerged as important themes. Although a group of participants were interested in "receiving rewards for teaching," rewards did not need to be financial compensation. For some, genuine recognition for their efforts by the medical school, particularly in the form of faculty development opportunities, constituted reward and recognition for teaching. CONCLUSIONS: Community physicians described a variety of factors that contribute to their identity as teachers. Faculty development programs offer opportunities to strengthen teacher identity and foster relationships between teaching programs and community-based faculty.

Developing culturally competent community faculty: a model program.

The Liaison Committee on Medical Education recently set standards for cultural diversity training as part of the medical school curriculum. To the authors' knowledge, this is the first description of a faculty-development program designed to develop the capacity of the clinical faculty to integrate culture and advocacy education into clinical training. The paper describes the first two years of the development of an ongoing cultural competence curriculum that has been integrated into the training of community preceptors from 13 medical schools in New England and New York. The training, entitled"Teaching the Culture of the Community," consists of four 2.5-hour modules that include interactive lectures and small-group role-play exercises on cultural needs assessment, patient-centered interviewing, feedback on cultural issues and use of the community to enhance cultural understanding. The 137 participants in the first two years of the program (1999-00 and 2000-01) reported a high level of acceptance of the curriculum. In the second year, the program began to document participants' self-reported"intention to change" in relation to the cultural competence curriculum. Many participants reported plans to change aspects of their clinical care and their teaching practices. Intentions to change were most frequently expressed in the context of content on effective communication skills. In summary, cultural competency training has been successfully integrated into an existing faculty-development program for community-based preceptors.

Weaving public health education into the fabric of a family medicine residency.

Policymakers and accrediting bodies have recognized the importance of integrating public health, population health, and prevention into graduate medical education programs. The high prevalence of chronic illness, coupled with the impact of behavioral and societal determinants of health, necessitate an urgent call for family medicine residencies to prepare future leaders to meet these challenges. The University of Massachusetts Worcester Family Medicine Residency recently developed an integrated curriculum that strives to develop a culture of incorporating fundamental public health principles into everyday practice. This public health curriculum was designed to integrate new topics within the current residency structure through longitudinal and concentrated experiences. This strategy has substantially improved public health and prevention education without substantial impact on the already strained residency curricular structure. This paper describes the integration of public health and prevention education into a family medicine residency to help residents acquire the fundamental skills necessary to improve a population's health.

Expert communication training for providers in community health centers.

BACKGROUND: Community health center (CHC) clinicians describe significant challenges in delivering care for populations with complex health needs. INTERVENTION: A three-workshop series was presented to 102 providers working in CHCs. Training focused on four areas identified through online needs assessment: challenging interactions; patient perspective; health literacy; and motivational interviewing. EVALUATION METHODS: A retrospective pre-post evaluation measured self-perceived change in content knowledge in all four areas. Participants documented commitments to change behaviors across workshops, which were analyzed for recurring themes. RESULTS: Paired t-tests documented improvement in all four content areas. Content analysis of commitments yielded four themes: empowering patients, structuring care, understanding patients, and reflecting purposefully. Of the sixty-eight percent of participants responding to post-workshop queries about their commitments (n=70), 94% report having fully implemented changes in practice behavior or planning to do so. CONCLUSIONS: Providers at CHCs benefit from opportunities to learn and reflect together about communication challenges in practice.

Modern moulage: evaluating the use of 3-dimensional prosthetic mimics in a dermatology teaching program for second-year medical students.

OBJECTIVES: To evaluate the effectiveness of a teaching method that uses 3-dimensional (3D) silicone-based prosthetic mimics of common serious lesions and eruptions and to compare learning outcomes with those achieved through the conventional method of lectures with 2-dimensional (2D) images. DESIGN: Prospective and comparative. SETTING: University of Massachusetts Medical School. PARTICIPANTS: Ninety second-year medical students. INTERVENTION: A 1-hour teaching intervention using a lecture with 2D images (2D group) or using 3D prosthetic mimics of lesions and eruptions (3D group). MAIN OUTCOME MEASURES: Mean scores in the domains of morphology, lesion and rash recognition, lesion and rash management, and overall performance assessed at baseline, immediately after, and 3 months after each group's respective teaching intervention. RESULTS: Immediately after the teaching intervention, the 3D group had significantly higher mean percentage scores than did the 2D group for overall performance (71 vs 65, P = .03), lesion recognition (65 vs 56, P = .02), and rash management (80 vs 67, P = .01). Three months later, the 3D group still had significantly higher mean percentage scores than did the 2D group for lesion recognition (47 vs 40, P = .03). The 3D group better recognized lesions at 3 months compared with at baseline, whereas the 2D group was no better at recognizing lesions at 3 months compared with at baseline. CONCLUSIONS: Despite limited curricular time, the novel teaching method using 3D prosthetic mimics of lesions and eruptions improves immediate and long-term learning outcomes, in particular, lesion recognition. It is also a preferred teaching format among second-year medical students.

A human factors curriculum for surgical clerkship students.

HYPOTHESIS: Early introduction of a full-day human factors training experience into the surgical clerkship curriculum will teach effective communication skills and strategies to gain professional satisfaction from a career in surgery. DESIGN: In pilot 1, which took place between July 1, 2007, and December 31, 2008, 50 students received training and 50 did not; all received testing at the end of the rotation for comparison of control vs intervention group performance. In pilot 2, a total of 50 students were trained and received testing before and after rotation to examine individual change over time. SETTING: University of Massachusetts Medical School. PARTICIPANTS: A total of 148 third-year medical students in required 12-week surgical clerkship rotations. INTERVENTIONS: Full-day training with lecture and small-group exercises, cotaught by surgeons and educators, with focus on empathetic communication, time management, and teamwork skills. MAIN OUTCOME MEASURES: Empathetic communication skill, teamwork, and patient safety attitudes and self-reported use of time management strategies. RESULTS: Empathy scores were not higher for trained vs untrained groups in pilot 1 but improved from 2.32 to 3.45 on a 5-point scale (P < .001) in pilot 2. Students also were more likely to ask for the nurse's perspective and to seek agreement on an action plan after team communication training (pilot 1, f = 7.52, P = .007; pilot 2, t = 2.65, P = .01). Results were mixed for work-life balance, with some trained groups scoring significantly lower than untrained groups in pilot 1 and no significant improvement shown in pilot 2. CONCLUSIONS: The significant increase in student-patient communication scores suggests that a brief focused presentation followed by simulation of difficult patient encounters can be successful. A video demonstration can improve interdisciplinary teamwork.

Human Emotion and Response in Surgery (HEARS): a simulation-based curriculum for communication skills, systems-based practice, and professionalism in surgical residency training.

BACKGROUND: This study examines the development and implementation of a pilot human factors curriculum during a 2-year period. It is one component of a comprehensive 5-year human factors curriculum spanning core competencies of interpersonal and communication skills, systems-based practice, and professionalism and using low-and high-fidelity simulation techniques. STUDY DESIGN: Members of the Department of Surgery and the Center for Clinical Communication and Performance Outcomes jointly constructed a curriculum for PGY1 and PGY2 residents on topics ranging from challenging communication to time and stress management. Video demonstrations, triggers, and simulated scenarios involving acting patients were created by surgeons and medical educators. Pre- and postintervention measures were obtained for communication skills, perceived stress level, and teamwork. Communication skills were evaluated using a series of video vignettes. The validated Perceived Stress Scale and Teamwork and Patient Safety Attitudes survey were used. Residents' perceptions of the program were also measured. RESULTS: Twenty-seven PGY1 residents and 15 PGY2 residents participated during 2 years. Analyses of video vignette tests indicated significant improvement in empathic communication for PGY1 (t = 3.62, p = 0.001) and PGY2 (t = 5.00, p = 0.004). There were no significant changes to teamwork attitudes. Perceived levels of stress became considerably higher. PGY1 residents reported trying 1 to 3 strategies taught in the time management session, with 60% to 75% reporting improvement post-training. CONCLUSIONS: This unique and comprehensive human factors curriculum is shown to be effective in building communication competency for junior-level residents in the human and emotional aspects of surgical training and practice. Continued refinement and ongoing data acquisition and analyses are underway.

The video-based test of communication skills: description, development, and preliminary findings.

BACKGROUND: The importance of assessing physician-patient communication skills is widely recognized, but assessment methods are limited. Objective structured clinical examinations are time-consuming and resource intensive. For practicing physicians, patient surveys may be useful, but these also require substantial resources. Clearly, it would be advantageous to develop alternative or supplemental methods for assessing communication skills of medical students, residents, and physicians. DESCRIPTION: The Video-based Test of Communication Skills (VTCS) is an innovative, computer-administered test, consisting of 20 very short video vignettes. In each vignette, a patient makes a statement or asks a question. The examinee responds verbally, as if it was a real encounter and he or she were the physician. Responses are recorded for later scoring. Test administration takes approximately 1 h. EVALUATION: Generalizability studies were conducted, and scores for two groups of physicians predicted to differ in their communication skills were compared. Preliminary results are encouraging; the estimated g coefficient for the communication score for 20-vignette test (scored by five raters) is 0.79; g for the personal/affective score under the same conditions is 0.62. Differences between physicians were in the predicted direction, with physicians considered "at risk" for communication difficulties scoring lower than those not so identified. CONCLUSIONS: The VTCS is a short, portable test of communication skills. Results reported here suggest that scores reflect differences in skill levels and are generalizable. However, these findings are based on very small sample sizes and must be considered preliminary. Additional work is required before it will be possible to argue confidently that this test in particular, and this approach to testing communication skills in general, is valuable and likely to make a substantial contribution to assessment in medical education.

Initial testing of an instrument to measure teacher identity in physicians.

BACKGROUND: A previous study described 7 elements of teacher identity: intrinsic satisfaction from teaching, knowledge and skill about teaching, belonging to a community of teachers, receiving rewards for teaching, believing that being a doctor means being a teacher, feeling a responsibility to teach, and sharing clinical expertise. PURPOSE: To conduct the initial testing of an instrument to measure the 7 elements of teacher identity in clinical educators and to consider the potential applications of such an instrument. METHODS: A 37-item questionnaire was mailed to 153 preceptors of preclinical students. Categories reflected the elements of teacher identity listed here. Demographic data were collected. Means, alphas, ANOVAs, and paired t tests were calculated. RESULTS: Of 153 preceptors, 127 (83%) completed the questionnaire. Cronbach's alpha for the overall scale and several subscales were high. Salaried physicians and those who had completed a faculty development program scored significantly higher on several subscales than physicians who volunteered to teach or who did not have faculty development. CONCLUSIONS: This study provides preliminary evidence that teacher identity can be measured and that preceptors do not respond as a homogeneous group. Assessing teacher identity may be helpful to medical schools looking to identify and support physicians who teach.

Teaching and medical errors: primary care preceptors' views.

PURPOSE: To assess primary care preceptors' perceptions of the issues involved in teaching when medical errors occur. In particular, we examined preceptors' responses to trainees involved in medical errors, factors influencing their response, and their perceptions of barriers to teaching from medical errors. METHODS: A total of 38 primary care preceptors participated in 7 focus groups on teaching and medical errors. Participants were drawn from medical schools throughout the northeastern USA. Content analysis of transcripts identified major themes. RESULTS: We developed a framework describing how preceptors and learners respond to medical errors, the factors that influence these responses, and the relationships between these. We also identified barriers to teaching from medical errors. Preceptors are especially sensitive to learners' distress as a result of errors. Emotional distress and self-doubt are seen as inimical to learning, possibly causing more attention to be directed to emotional support than to correction and instruction. At the same time, accepting responsibility for errors was seen as prerequisite to learning. For many preceptors, vivid recollections of their own errors during training were influential in determining how they in turn responded as preceptors; none reported having received training in this area. CONCLUSION: This study describes preceptors' experiences of responding to trainees' medical errors, and identifies barriers to teaching from errors. The intersection of patient safety and medical education is a critical area for future research. We propose a framework that may help guide future research efforts, which should focus on identifying factors that promote faculty development to optimise learning and reduce the likelihood of future errors.

Reliability and validity of checklists and global ratings by standardized students, trained raters, and faculty raters in an objective structured teaching environment.

BACKGROUND: Objective structured teaching exercises (OSTEs) are relatively new in medical education, with few studies that have reported reliability and validity. PURPOSE: To systematically examine the impact of OSTE design decisions, including number of cases, choice of raters, and type of scoring systems used. METHODS: We examined the impact of number of cases and raters using generalizability theory. We also compared scores from standardized students (SS), faculty raters (FR) and trained graduate student raters (TR), and examined the relation between behavior checklist ratings and global perception scores. RESULTS: Generalizability (g) coefficients for checklist scores were higher for SSs than TRs. The g estimates based on SSs' global scores were higher than g estimates for FRs. SSs' checklist scores were higher than TRs' checklist scores, and SSs' global evaluations were higher than FRs' and TRs' global scores. TRs' relative to SSs' global perceptions correlated more highly with checklist scores. CONCLUSIONS: SSs provide more generalizable checklist scores than TRs. Generalizability estimates for global scores from SSs and FRs were comparable. SSs are lenient raters compared to TRs and FRs.