Validity considerations in the assessment of professionalism.

During the last decade, interest in assessing professionalism in medical education has increased exponentially and has led to the development of many new assessment tools. Efforts to validate the scores produced by tools designed to assess professionalism have lagged well behind the development of these tools. This paper provides a structured framework for collecting evidence to support the validity of assessments of professionalism. The paper begins with a short history of the concept of validity in the context of psychological assessment. It then describes Michael Kane's approach to validity as a structured argument. The majority of the paper then focuses on how Kane's framework can be applied to assessments of professionalism. Examples are provided from the literature, and recommendations for future investigation are made in areas where the literature is deficient.

Paradoxes of professionalism and error in complex systems.

Professionalism is at the heart of risk management in complex, dangerous work such as medicine, aviation, and military operations. Professionalism is closely connected to expertise and is therefore closely connected to the ability to prevent and mitigate errors. But there are two paradoxes in this connection. First, professionalism can increase, rather than reduce, the risk of errors and accidents by promoting practitioners' tendency to break procedural rules. This is because professional expertise tends to favor adaptation to local circumstances over standardized approaches to problem-solving. Second, professionalism can create blind spots within organizations, blocking the flow of critical information about unsafe conditions. This is because professional groups develop unique subcultures, specialized language, and communication habits that tend to separate them from other professional groups, even when those groups work within the same organization. I illustrate these paradoxes using case studies from several different professional domains. I then outline some methodological challenges for research on safety and professionalism. Finally, I argue that the kind of professionalism that can prevent errors is rooted in organizational practices that reduce the social separation between professional groups and promote the maintenance of adequate margins of safety. This requires the acceptance of safety as a central value that is at least as important as productivity.

Assessment of professionalism: recommendations from the Ottawa 2010 Conference.

Over the past 25 years, professionalism has emerged as a substantive and sustained theme, the operationalization and measurement of which has become a major concern for those involved in medical education. However, how to go about establishing the elements that constitute appropriate professionalism in order to assess them is difficult. Using a discourse analysis approach, the International Ottawa Conference Working Group on Professionalism studied some of the dominant notions of professionalism, and in particular the implications for its assessment. The results presented here reveal different ways of thinking about professionalism that can lead towards a multi-dimensional, multi-paradigmatic approach to assessing professionalism at different levels: individual, inter-personal, societal-institutional. Recommendations for research about professionalism assessment are also presented.

Assessment of medical professionalism: who, what, when, where, how, and ... why?

Medical professionalism is increasingly recognized as a core competence of medical trainees and practitioners. Although the general and specific domains of professionalism are thoroughly characterized, procedures for assessing them are not well-developed. This article outlines an approach to designing and implementing an assessment program for medical professionalism that begins and ends with asking and answering a series of critical questions about the purpose and nature of the program. The process of exposing an assessment program to a series of interrogatives that comprise an integrated and iterative framework for thinking about the assessment process should lead to continued improvement in the quality and defensibility of that program.

Variability in Institutional Board Review for a Multisite Assessment of Resident Professionalism.

Residents serve as both trainees and employees and can be considered potentially vulnerable research participants. This can lead to variation in the institutional review board (IRB) review. We studied sites participating in the Assessment of Professional Behaviors Study sponsored by the National Board of Medical Examiners (2009-2011). Of the 19 sites, all but one were university affiliated. IRB review varied; 2/19 did not submit to a local IRB, 4/17 (23%) were exempt, 11/17 (65%) were expedited, and 2/17 (12%) required full Board review; 12/17 (71%) required written informed consent. The interval from submission to approval was 1 to 2 months (8/17); the range was 1 to 7 months. Although most stated there were no major barriers to approval, the most common concern was resident coercion and loss of confidentiality. Local IRB review of this educational research study varied.

Evaluation of missing data in an assessment of professional behaviors.

BACKGROUND: The National Board of Medical Examiners is currently developing the Assessment of Professional Behaviors, a multisource feedback (MSF) tool intended for formative use with medical students and residents. This study investigated whether missing responses on this tool can be considered random; evidence that missing values are not random would suggest response bias, a significant threat to score validity. METHOD: Correlational analyses of pilot data (N = 2,149) investigated whether missing values were systematically related to global evaluations of observees. RESULTS: The percentage of missing items was correlated with global evaluations of observees; observers answered more items for preferred observees compared with nonpreferred observees. CONCLUSIONS: Missing responses on this MSF tool seem to be nonrandom and are instead systematically related to global perceptions of observees. Further research is needed to determine whether modifications to the items, the instructions, or other components of the assessment process can reduce this effect.

Disciplinary careers of drug-impaired physicians.

Alcohol and drug abuse are among the leading reasons for disciplinary action against physicians by state licensing authorities in the United States. I use event history models to describe the longitudinal patterns in disciplinary actions taken against physicians' licenses by state medical boards in the United States, 1990-2000. Adverse licensure action episodes that included discipline for drug or alcohol abuse were more likely to be followed by license restoration than episodes that did not. However, those restorations were also more likely to be followed by subsequent disciplinary action than episodes that did not include discipline for drug abuse. Furthermore, disciplinary licensure actions for drug abuse were the category most likely to be followed by a subsequent action for the same reason over the longer term (4-11 years). The increased risk of repeat disciplinary action associated with drug abuse may result in part from intensive surveillance of physicians who complete impaired physician programs, through mechanisms that include urine screening. However, it is also likely that the chronic nature of addiction leads to continued risk of relapse even among physicians receiving appropriate treatment.

A multilevel analysis of the relationships between selected examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge performance: revisiting old findings and asking new questions.

BACKGROUND: This study examines: (1) the relationships between examinee characteristics and United States Medical Licensing Examination Step 2 Clinical Knowledge (CK) performance; (2) the effect of gender and examination timing (time per item) on the relationship between Steps 1 and 2 CK; and (3) the effect of school characteristics on the relationships between examinee characteristics and Step 2 CK performance. METHOD: A series of hierarchical linear models (examinees-nested-in-schools) predicting Step 2 CK scores was fit to the data set. The sample included 54,487 examinees from 114 U.S. Liaison Committee on Medical Education-accredited medical schools. RESULTS: Consistent with past examinee-level research, women generally outperformed men on Step 2 CK, and examinees who received more time per item generally outperformed examinees who received less time per item. Step 1 score was generally more strongly associated with Step 2 CK performance for men and for examinees who received less time per item. School-level characteristics (size, average Step 1 performance) influenced the relationship between Steps 1 and 2 CK. CONCLUSION: Both examinee-level and school-level characteristics are important for understanding Step 2 CK performance.

Defining and measuring construct of interprofessional professionalism.

The Interprofessional Professionalism Collaborative (IPC), convened in 2006, currently consists of 11 national organizations representing health professions programs at the doctoral entry level, and is developing a framework of "interprofessional professionalism" (IPP) around observable behaviors that illustrate what professionalism looks like in the context of interprofessional collaborations focused on patient-, client-, and family-centered care. IPC's goal is to create tools to foster and measure these behaviors in health professionals and students. This paper describes the work of IPC to date and its future plans.

The relationship between direct observation, knowledge, and feedback: results of a national survey.

BACKGROUND: Multisource feedback can provide a comprehensive picture of a medical trainee's performance. The utility of a multisource feedback system could be undermined by lack of direct observation and accurate knowledge. METHOD: The National Board of Medical Examiners conducted a national survey of medical students, interns, residents, chief residents, and fellows to learn the extent to which certain behaviors were observed, to examine beliefs about knowledge of each other's performance, and to assess feedback. RESULTS: Increased direct observation is associated with the perception of more accurate knowledge, which is associated with increased feedback. Some evaluators provide feedback in the absence of accurate knowledge of a trainee's performance, and others who have accurate knowledge miss opportunities for feedback. CONCLUSIONS: Direct observation is a key component of an effective multisource feedback system. Medical educators and residency directors may be well advised to establish explicit criteria specifying a minimum number of observations for evaluations.

Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment.

BACKGROUND: Multisource feedback (MSF) is emerging as a central assessment method for several medical education competencies. Planning and resource requirements for a successful implementation can be significant. Our goal is to examine barriers and challenges to a successful multisite MSF implementation, and identify the benefits of MSF as perceived by participants. METHODS: We analyzed the 2007-2008 field trial implementation of the Assessment of Professional Behaviors, an MSF program of the National Board of Medical Examiners, conducted with 8 residency and fellowship programs at 4 institutions. We use a multimethod analysis that draws on quantitative process indicators and qualitative participant experience data. Process indicators include program attrition, completion of implementation milestones, number of participants at each site, number of MSF surveys assigned and completed, and adherence to an experimental rater training protocol. Qualitative data include communications with each program and semistructured interviews conducted with key field trial staff to elicit their experiences with implementation. RESULTS: Several implementation challenges are identified, including communication gaps and difficulty scheduling implementation and training workshops. Participant interviews indicate several program changes that should enhance feasibility, including increasing communication and streamlining the training process. CONCLUSIONS: Multisource feedback is a complex educational intervention that has the potential to provide users with a better understanding of performance expectations in the graduate medical education environment. Standardization of the implementation processes and tools should reduce the burden on program administrators and participants. Further study is warranted to broaden our understanding of the resource requirements for a successful MSF implementation and to show how outcomes change as MSF gains broader acceptance.

Which experiences in the hidden curriculum teach students about professionalism?

PURPOSE: To examine the relationship between learner experience in the "hidden curriculum" and student attribution of such experiences to professionalism categories. METHOD: Using the output of a thematic analysis of 272 consecutive narratives recorded by 135 students on a medical clerkship from June through November 2007, the authors describe the frequency of these experiences within and across student-designated Association of American Medical Colleges-National Board of Medical Examiners professionalism categories and employ logistic regression to link varieties of experience to specific professionalism categories. RESULTS: Thematic analysis uncovered two main domains of student experience: medical-clinical interaction and teaching-and-learning experiences. From a student perspective the critical incident stories evoked all professionalism categories. Most frequently checked off categories were caring/compassion/communication (77%) and respect (69%). Logistic regression suggested that student experiences within the teaching-and-learning environment were associated with professionalism categories of excellence, leadership, and knowledge and skills, whereas those involving medical-clinical interactions were associated with respect, responsibility and accountability, altruism, and honor and integrity. Experiences of communicating and working within teams had the broadest association with learning about professionalism. CONCLUSIONS: Student narratives touched on all major professionalism categories as well as illuminating the contexts in which critical experiences emerged. Linked qualitative and quantitative analysis identified those experiences that were associated with learning about particular aspects of professionalism. Experiences of teamwork were especially relevant to student learning about professionalism in action.

Lost in transition: the experience and impact of frequent changes in the inpatient learning environment.

PURPOSE: The traditional "rotating" model of inpatient training remains the gold standard of residency, moving residents through different systems every two to four weeks. The authors studied the experience and impact of frequent transitions on residents. METHOD: This was a qualitative study. Ninety-seven individuals participated in 12 focus groups at three academic medical centers purposefully chosen to represent a range of geographic locations and structural characteristics. Four groups were held at each site: residents only, faculty only, nurses and ancillary staff only, and a mixed group. Grounded theory was used to analyze data. RESULTS: Perceived benefits of transitions included the ability to adapt to new environments and practice styles, improved organization and triage skills, increased comfort with stressful situations, and flexibility. Residents primarily relied on each other to cope with and prepare for transitions, with little support from the program or faculty level. Several potentially problematic workarounds were described within the context of transitions, including shortened progress notes, avoiding pages, hiding information, and sidestepping critical situations. Nearly all residents acknowledged that frequent transitions contributed to a lack of ownership and other potentially harmful effects for patient care. CONCLUSIONS: These findings challenge the value of the traditional "rotating" model in residency. As residents adapt to frequent transitioning, they implicitly learn to value flexibility and efficiency over relationship building and deep system knowledge. These findings raise significant implications for professional development and patient care and highlight an important element of the hidden curriculum embedded within the current training model.

The quality of written comments on professional behaviors in a developmental multisource feedback program.

BACKGROUND: Written feedback on professional behaviors is an important part of medical training, but little attention has been paid to the quality of written feedback and its expected impact on learning. A large body of research on feedback suggests that feedback is most beneficial when it is specific, clear, and behavioral. Analysis of feedback comments may reveal opportunities to improve the value of feedback. METHOD: Using a directed content analysis, the authors coded and analyzed feedback phrases collected as part of a pilot of a developmental multisource feedback program. The authors coded feedback on various dimensions, including valence (positive or negative) and whether feedback was directed at the level of the self or behavioral performance. RESULTS: Most feedback comments were positive, self-oriented, and lacked actionable information that would make them useful to learners. CONCLUSIONS: Comments often lack effective feedback characteristics. Opportunities exist to improve the quality of comments provided in multisource feedback.

Medical students' professionalism narratives: a window on the informal and hidden curriculum.

PURPOSE: The aim of this study was to use medical students' critical incident narratives to deepen understanding of the informal and hidden curricula. METHOD: The authors conducted a thematic analysis of 272 stories of events recorded by 135 third-year medical students that "taught them something about professionalism and professional values." Students wrote these narratives in a "professionalism journal" during their internal medicine clerkships at Indiana University School of Medicine, June through November 2007. RESULTS: The majority of students' recorded experiences involved witnessing positive embodiment of professional values, rather than breaches. Attending physicians and residents were the central figures in the incidents. Analyses revealed two main thematic categories. The first focused on medical-clinical interactions, especially on persons who were role models interacting with patients, families, coworkers, and colleagues. The second focused on events in the teaching-and-learning environment, particularly on students' experiences as learners in the clinical setting. CONCLUSIONS: The findings strongly suggest that students' reflective narratives are a rich source of information about the elements of both the informal and hidden curricula, in which medical students learn to become physicians. Experiences with both positive and negative behaviors shaped the students' perceptions of the profession and its values. In particular, interactions that manifest respect and other qualities of good communication with patients, families, and colleagues taught powerfully.

A theoretical sketch of medical professionalism as a normative complex.

Validity arguments for assessment tools intended to measure medical professionalism suffer for lack of a clear theoretical statement of what professionalism is and how it should behave. Drawing on several decades of field research addressing deviance and informal social control among physicians, a theoretical sketch of professionalism is presented that can be used to predict how individual adjustment to professional norms should co-vary with other social and psychological phenomena. Physicians may understand and value professional norms but fail to enact them in practice because of conflicting normative demands. Physicians' social networks are predicted to act as conduits of social learning and social pressure, driving the resolution of normative conflicts in specific directions. A valid assessment of professionalism requires an adequate accounting of the social reaction to an individual's professional conduct, because the reaction and the conduct itself are inseparable.

Analysis of count data using poisson regression.

Nurses and other health researchers are often concerned with infrequently occurring, repeatable, health-related events such as number of hospitalizations, pregnancies, or visits to a health care provider. Reports on the occurrence of such discrete events take the form of non-negative integer or count data. Because the counts of infrequently occurring events tend to be non-normally distributed and highly positively skewed, the use of ordinary least squares (OLS) regression with non-transformed data has several shortcomings. Techniques such as Poisson regression and negative binomial regression may provide more appropriate alternatives for analyzing these data. The purpose of this article is to compare and contrast the use of these three methods for the analysis of infrequently occurring count data. The strengths, limitations, and special considerations of each approach are discussed. Data from the National Longitudinal Survey of Adolescent Health (AddHealth) are used for illustrative purposes.