Using standardized patients to assess professionalism: a generalizability study.

BACKGROUND: Assessment of professionalism in undergraduate medical education is challenging. One approach that has not been well studied in this context is performance-based examinations. PURPOSE: This study sought to investigate the reliability of standardized patients' scores of students' professionalism in performance-based examinations. METHODS: Twenty students were observed on 4 simulated cases involving professional challenges; 9 raters evaluated each encounter on 21 professionalism items. Correlational and multivariate generalizability (G) analyses were conducted. RESULTS: G coefficients were .75, .53, and .68 for physicians, standardized patients (SPs), and lay raters, respectively. Composite G coefficient for all raters reached acceptable level of .86. Results indicated SP raters were more variable than other rater types in severity with which they rated students, although rank ordering of students was consistent among SPs. CONCLUSIONS: SPs' ratings were less reliable and consistent than physician or lay ratings, although the SPs rank ordered students more consistently than the other rater types.

The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP).

In this paper, we define the Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP) for those working with human role players who interact with learners in a wide range of experiential learning and assessment contexts. These human role players are variously described by such terms as standardized/simulated patients or simulated participants (SP or SPs). ASPE is a global organization whose mission is to share advances in SP-based pedagogy, assessment, research, and scholarship as well as support the professional development of its members. The SOBP are intended to be used in conjunction with the International Nursing Association for Clinical Simulation and Learning (INACSL) Standards of Best Practice: SimulationSM, which address broader simulation practices. We begin by providing a rationale for the creation of the ASPE SOBP, noting that with the increasing use of simulation in healthcare training, it is incumbent on ASPE to establish SOBP that ensure the growth, integrity, and safe application of SP-based educational endeavors. We then describe the three and a half year process through which these standards were developed by a consensus of international experts in the field. Key terms used throughout the document are defined. Five underlying values inform the SOBP: safety, quality, professionalism, accountability, and collaboration. Finally, we describe five domains of best practice: safe work environment; case development; SP training for role portrayal, feedback, and completion of assessment instruments; program management; and professional development. Each domain is divided into principles with accompanying key practices that provide clear and practical guidelines for achieving desired outcomes and creating simulations that are safe for all stakeholders. Failure to follow the ASPE SOBP could compromise the safety of participants and the effectiveness of a simulation session. Care has been taken to make these guidelines precise yet flexible enough to address the diversity of varying contexts of SP practice. As a living document, these SOBP will be reviewed and modified periodically under the direction of the ASPE Standards of Practice Committee as SP methodology grows and adapts to evolving simulation practices.

Characteristics of simulation activities at North American medical schools and teaching hospitals: an AAMC-SSH-ASPE-AACN collaboration.

SUMMARY STATEMENT: In September 2011, the Association of American Medical Colleges released the results of a survey conducted in 2010 on simulation activities at its member medical schools and teaching hospitals. In this commentary, we offer a synthesis of data and conclude that (1) simulation is used broadly at Association of American Medical Colleges member institutions, for many types of learners, including other health care professionals; (2) it addresses core training competencies and has many educational purposes; (3) its use in learner assessment is more prevalent at medical schools but is still significant at teaching hospitals; and (4) it requires a considerable investment of money, space, personnel, and time. These data confirm general perceptions about the state of simulation in North America for physician training. Future endeavors should include a more granular examination of how simulation is integrated into curricula, a similar survey of other health care-related institutions and professions, and a periodic assessment to characterize trends over time.

An evaluation of tobacco treatment specialist counseling performance using standardized patient interviews.

In the United States, tobacco treatment specialists are professionals from a variety of backgrounds trained to deliver moderate to intensive evidence-based tobacco treatment in a variety of settings across the country. This paper reports the results of a study that examined the extent to which specialists participating in a larger study adhered to clinical practice guidelines for tobacco dependence using standardized patient assessments. A total of 64 tobacco treatment specialists completed a survey and two audiotaped standardized patient interviews. Overall, 41% and 31% of tobacco treatment specialists demonstrated 80% or more of session content and interviewing skills assessed, respectively, when required to demonstrate the skill in both standardized patient interactions. These rates increased to 85% and 56%, respectively, when using the less stringent criteria of demonstrating the skill in at least one of the two standardized patient interactions. Tobacco treatment specialists who had attended a greater number of types of tobacco treatment training exhibited both greater coverage of session content (p<.06) and greater interviewing skills (p<.02). Those who had achieved certification as a tobacco treatment specialist exhibited greater coverage of session content (p<.02), and those perceiving more positive support from their agency for their services exhibited greater interviewing skills (p=.02). Although the tobacco treatment specialists evaluated appear to have the necessary skills to deliver guideline-based intervention, they demonstrated only moderate adherence to the guidelines when expected to do so consistently across multiple assessments. Findings suggest specific components of tobacco treatment and interviewing skills that would benefit from strengthening through training and support.

Assessing professionalism in the context of an objective structured clinical examination: an in-depth study of the rating process.

INTRODUCTION: Professionalism is fundamental to the practice of medicine. Objective structured clinical examinations (OSCEs) have been proposed as appropriate for assessing some aspects of professionalism. This study investigated how raters assign professionalism ratings to medical students' performances in OSCE encounters. METHODS: Three standardised patients, 3 doctor preceptors, and 3 lay people viewed and rated 20 videotaped encounters between 3rd-year medical students and standardised patients. Raters recorded their thoughts while rating. Qualitative and quantitative analyses were conducted. Comments about observable behaviours were coded, and relative frequencies were computed. Correlations between counts of categorised comments and overall professionalism ratings were also computed. RESULTS: Raters varied in which behaviours they attended to, and how behaviours were evaluated. This was true within and between rater type. Raters also differed in the behaviours they consider when providing global evaluations of professionalism. CONCLUSIONS: This study highlights the complexity of the processes involved in assigning ratings to doctor-patient encounters. Greater emphasis on behavioural definitions of specific behaviours may not be a sufficient solution, as raters appear to vary in both attention to and evaluation of behaviours. Reliance on global ratings is also problematic, especially if relatively few raters are used, for similar reasons. We propose a model highlighting the multiple points where raters viewing the same encounter may diverge, resulting in different ratings of the same performance. Progress in assessment of professionalism will require further dialogue about what constitutes professional behaviour in the medical encounter, with input from multiple constituencies and multiple representatives within each constituency.