A survey of student assessment in U.S. medical schools: the balance of breadth versus fidelity.

BACKGROUND: Faced with the challenge to develop models of assessment relevant to work of physicians, medical schools have broadened their assessment of medical student competency. PURPOSE: U.S. medical schools were surveyed to determine the extent to which student assessments have broadened beyond multiple-choice question (MCQ) examinations and preceptor ratings. METHODS: A survey mailed to 126 accredited U.S. medical schools asked respondents to indicate the frequency with which a variety of assessment methods were used in each year of the curriculum. RESULTS: Examinations dominated preclinical assessments. Year 3 relied heavily on faculty ratings, live observations, and MCQs. Preceptor ratings were used most in year 4. CONCLUSIONS: A variety of competency assessments currently are used; MCQs remain a core assessment method. Year 3 had the greatest breadth of assessment strategies. The findings suggest that educators continue to be challenged to balance the breadth of competencies sampled with the fidelity of the assessment experience.

Pilot test of family medicine faculty development fellowship accreditation guidelines.

BACKGROUND: We conducted a pilot test of accreditation guidelines for family medicine faculty development fellowship programs from September 1997 to March 1999. The accreditation guidelines included 8 application categories with 27 requirements and 5 self-study criteria. The process included completion of the accreditation application and self-study and a site visit. We selected 6 sites for participation in the pilot test, and 5 sites completed all steps. The results indicated that, while fellowship faculty felt that the requirements and criteria were valid for determining quality of faculty development fellowship programs, the process was time-consuming and could be shortened. Redundancy between information supplied on the application and on the self-study was also noted. Six recommendations were included in the final report, including streamlining the accreditation process, developing guidelines for probationary status, and considering alternatives to accreditation, such as peer review.

Does studying for an objective structured clinical examination make a difference?

PURPOSE: This study examines the extent to which second-year medical students studied for an objective structured clinical examination (OSCE), how they studied, and the impact of self-reported studying on OSCE performance. METHOD: One class of 113 medical students completed an end-of-second-year OSCE, held on two consecutive evenings. The OSCE was comprised of eight stations, each of which was of 20 minutes' duration. The OSCE was formative: students received performance feedback but were not graded. Prior to the OSCE, students completed a brief survey regarding their preparation for the OSCE and their perceptions of confidence, anxiety and preparedness. Only 78 students returned surveys with names, comprising the data for these analyses. RESULTS: Mean studying time was 3.3 h, ranging from 0 to 19 h. Studying time was positively associated with age and negatively associated with basic science examination scores. The most study time was dedicated to reviewing the physical examination textbook, class notes and supplemental course readings. The breadth of study strategies increased as more time was spent in OSCE preparation. OSCE performance was related to study time and to achievement on pre-clinical basic science examinations. DISCUSSION: The students whose performance was above average seemed to be the talented students whose records indicated a history of academic success. The amount of time they reported for OSCE preparation was comparable to that reported by students with below average performance. It appears that prior academic performance rather than preparatory studying time is a better predictor of OSCE outcomes.

Recruitment behavior and program directors: how ethical are their perspectives about the match process?

OBJECTIVE: This study examined family practice residency directors' perspectives on the 1999 National Resident Matching Program (NRMP) process and identified directors' expectations for students' recruitment behavior. METHODS: Subjects were the family practice residency program directors. A 22-item written questionnaire was mailed to each director. The questions related to the directors' perceptions of the following issues: applicants interviewing in more than one specialty, communication initiated by programs or applicants, commitments made to applicants and by applicants, ethical dilemmas faced by the program director, and the NRMP process itself. Descriptive statistics were reported. RESULTS: Only a few of the residency program directors (9.1%) felt that it was ethically wrong for an applicant to interview in more than one specialty. However, most program directors (83%) indicated that the knowledge of an applicant interviewing in more than one specialty had a "significant" negative or "some" negative effect on the applicant's rank order. Ninety-five percent of program directors indicated that they engage in follow-up communication with applicants following the formal interview. Almost all program directors (98%) reported that at least some applicants contact them following the formal interview to inform them that the program was a "high" or No. 1 rank-order choice. The majority of program directors (94%) felt that the NRMP process placed their program in the position of having to be dishonest with applicants to match their top choices. CONCLUSIONS: The results of the study indicate that the actions of many program directors and applicants may not be consistent with the written policies of the NRMP.

Feeling the stress: perceptions of burnout among general surgery program directors.

PURPOSE: To document the types and levels of stress experienced by general surgery program directors as they fulfill their education and administrative responsibilities. METHODS: This study consisted of a 3-part survey that incorporated 2 established instruments to help determine the presence of burnout in program directors. A personal projects analysis was used to help identify the tasks most relevant to the role of program director as well as to evaluate their perceptions of these tasks. The Maslach Burnout Inventory (MBI) was used to measure the degree of burnout among program directors. Demographic data were gathered to develop a picture of the background of the program directors and how they spent their time. RESULTS: A total of 71.8% of program directors responded. Of all tasks, teaching received the highest ratings for importance, enjoyment, and control, as well as the lowest ratings for stress. Emotional exhaustion was the most notable aspect of burnout in program directors on the MBI. Program directors scoring high in burnout were younger, had been in their current position fewer years, and had fewer years overall as a program director. CONCLUSIONS: Burnout is more related to age and experience of program director than to features of the program itself.

The influence of recovery status and education on addiction counselors' approach to treatment.

The goal of this study is to examine the influence of education and recovery status on substance abuse treatment counselors' approach to patients. Three hundred forty-four drug abuse and alcoholism treatment counselors were questioned about treatment goals. A subgroup of 197 were also questioned about treatment practices. The influences of education and recovery status on the choice of treatment goals and treatment practices were examined through structural modeling procedures. Level of education influenced neither treatment goals nor techniques. Being in recovery, however, was associated with more varied treatment techniques and a broader range of treatment goals. The other variable related to treatment goals and practices was treatment modality. Working in residential programs was linked to a wider range of treatment goals and treatment practices. Implications of these findings for counselor training and the movement to professionalize substance abuse treatment are discussed.

Substance abuse treatment staff: recovery status and approaches to treatment.

Recovery status is an important characteristic of staff members working within substance abuse treatment. Recovering and nonrecovering staff members were contrasted previously, however there is a third group: Individuals who are not recovering themselves but are part of families with recovering or addicted members. The purpose of the present study is to compare background, roles within program and approaches to treatment of these three groups. Six hundred and thirty-four staff members of fifty-one treatment programs completed questionnaires. Five hundred and seventy-five completed an item indicating their recovery status. Forty-four percent identified themselves as nonrecovering, 30 percent as recovering, and 26 percent as nonrecovering but part of families with an addicted or recovering member. Nonrecovering staff with addicted or recovering family members differed from the other two groups on gender, more of them were female, but were similar to nonrecovering staff in their approach to treatment but fell between recovering and nonrecovering staff on measures of roles within programs and background. Recovering counselors reported to pursue a wider range of treatment goals and to use more varied treatment techniques than nonrecovering counselors. The implication of these findings for training and licensure of paraprofessionals in the field of substance abuse treatment is discussed.

Postal surveys versus electronic mail surveys. The tortoise and the hare revisited.

The advent of computer-based technology has led to a consideration of change in research methods that exploit the advantages of computer-mediated communications. In survey research, electronic mail (e-mail) has anecdotally shown particular promise as a data collection tool. This article compares traditional postal and nontraditional e-mail surveys within the context of a larger listserv evaluation project in terms of overall return rate, distribution of survey returns over time, response to initial and follow-up mailings, representativeness of respondent groups, thoroughness of survey completion, and the likelihood of respondents to include additional written comments. In summary, whereas postal surveys were shown to be superior to e-mail surveys with regard to response rate, all things being equal, the decision of which to use may be situation-specific, dependent on issues such as survey cost, desire for convenience and timeliness in data collection, and need for higher response rates, among others.

The emperor's new clothes: the OSCE reassessed.

In response to increasing concerns about the prevalence of knowledge- based assessments of medical student competency, leaders in medical education have emphasized the importance of methods that quantify student performance. As a result, the use of objective structured clinical examinations (OSCEs) is viewed by many as the newest and most promising technique for assessing students' abilities. In considering the implementation of a fourth-year OSCE, faculty at the College of Human Medicine at Michigan State University became uncomfortable with some of the technical limitations of the method (limited generalizability; weak linkages to the curriculum; little opportunity provided for improvement in examinees' skills; and others), as well as the possible ramifications of such an innovation within their school's specific curricular and organizational contexts. This essay is offered as a reflection of the challenges and possible alternatives that have emerged as the faculty have considered how best to design and implement performance-based assessment within their institution. Rather than using the OSCE as a milestone marker of student performance, they consider the possibility of smaller assessment events, closely tied to the curriculum and consistent with the guiding principles of the medical school.

Sources of coming out self-efficacy for lesbians.

The empirical literature on disclosing a lesbian sexual orientation has explored the circumstantial and demographic variables related to this act. This exploratory study utilized self-efficacy theory (Bandura, 1986) to investigate the extent to which each of the four sources of efficacy information (e.g., performance accomplishments, vicarious experience, verbal persuasion, or emotional arousal) contributed to the coming out self-efficacy of lesbians, that is, the sense of confidence possessed by a lesbian to disclose her sexual orientation to others. Anonymous survey packets were completed by 134 lesbians. Results of regression analyses indicated that emotional arousal was the most potent predictor of coming out self-efficacy. Verbal persuasion and vicarious experience also were significant. The most theoretically salient source of self-efficacy information, performance accomplishments (Bandura, 1986), was not a significant predictor of coming out self-efficacy. Further, significant correlations were found between coming out self-efficacy and outness and life-style satisfaction, which were also significantly correlated to measures of psychological adjustment.

The relationship between career satisfaction and fellowship training in academic surgeons.

BACKGROUND: Academic surgeons make various important decisions about their careers; however, little is known about the relationships between fellowship training, career development issues, and academic responsibilities. METHODS: Surgeon members of the Association for Surgical Education were surveyed about career development issues. Three hundred ninety-two (75.2%) surgeons responded. RESULTS: An exploratory factor analysis of the career development issues revealed four career development factors. Statistically significant differences were found between types of fellowship training and the career development factors. Nonfellowship-trained and clinical-fellowship-trained surgeons spend their time similarly to physicians in other specialties. Research-fellowship-trained surgeons spent significantly more time doing research, had fewer concerns about professional confidence, and expressed greater satisfaction with their careers. CONCLUSION: There is a relationship between career development issues, fellowship training, and type of fellowship training. Attention to these issues may be important in recruiting and retaining academic surgeons.

The longitudinal stability of the Addiction Severity Index.

The Addiction Severity Index (ASI) is a structured interview widely used by substance abuse clinicians and researchers for client screening, determining treatment needs, and assessing treatment outcomes. Previous researchers have evaluated inter-rater agreement, test-retest reliability, and concurrent validity. The present report describes the stability of ASI scores in longitudinal work. In the context of an ongoing treatment outcome evaluation study involving seven assessors, inter-rater agreement, inter-rater reliability, as well as intra- and inter-rater accuracy were assessed repeatedly during a 2-year period. The results show the scores derived from the ASI to be stable across assessors and over time. The relationship between stable scores and resources required for training are discussed.

Program factors influencing client satisfaction in alcohol treatment.

There is little documentation about how the union of self-help and professional treatment services influences client treatment satisfaction. This study examines the relationship of treatment characteristics indicative of program size, staffing patterns, Alcoholics Anonymous (AA) influence, and staff recovery status to client satisfaction. Thirty-six public substance abuse treatment programs participated in this study. At outpatient programs, satisfaction was related to program size, and the number of paraprofessional and medical staff; satisfaction was unrelated to AA influence on treatment. For residential clients, AA influence on treatment and AA beliefs held by staff were consistently related to satisfaction; factors related to program size and staffing patterns were independent of satisfaction. The results question the appropriateness of self-help interventions in all settings, and emphasize contextual differences in outpatient and residential programs.

A work-site weight management program to reinforce behavior.

This pilot weight management project addresses the efficacy of reinforcing dietary behavior change versus weight loss. This 6-month program served professional and support staff participating in the work-site wellness program at a midwestern university. Behavior-contingent program data were compared with data from the previous model where contracts were made for weight loss. In the behavior-contingent program, dropout rate and satisfaction with the program compared favorably with the old model weight loss-contingent program. Contract adherence was 93% compared with 74% in the weight loss-contingent program. Actual pounds of weight lost were lower in the behavior-contingent program, however, long-term weight management must still be studied with this population.

Issues related to participation in worksite health promotion: a preliminary study.

PURPOSE: The purpose of this study was to identify characteristics associated with participation in worksite-based health promotion activities. DESIGN: Follow-up interviews were used to identify demographic, attitudinal, and behavioral differences among three employee groups. Reasons employees chose not to participate in health promotion activities were also explored. SETTING: All respondents were employed at a large midwestern university and were eligible to participate in free onsite health fairs and health promotion programs. SUBJECTS: A stratified random sample of 89 nonparticipants, health fair participants, and behavior change program participants was interviewed. MEASURES: The interview was comprised of questions related to demographic information, personal health habits, physical activity, perceived health status, perceived self-efficacy, worksite norms, health promoting lifestyle factors, and knowledge about health promotion activities. RESULTS: ANOVA and chi-squared comparisons revealed few group differences. Graduate students and employees with advanced degrees were most likely to take part in health fairs. Behavior change program participants were older, clerical-technical staff members, and women. Faculty members were least likely to participate. A lack of time was the most often cited reason for nonparticipation. CONCLUSIONS: The study was retrospective and the analyses limited due to low statistical power. The results suggest that different groups of employees are attracted to different types of health promotion activities.

A program model to enhance adherence in work-site-based fitness programs.

We describe a program model designed to achieve high adherence, a major problem for work-site exercise/fitness programs. Our model is a 6-month program consisting of 15 1-hour program meetings, with participants exercising on their own time four times per week. Procedures employed to enhance adherence are contracting, group competition, monitoring, and social support. This program model has been applied nine times. One hundred fifty-nine university employees took part in the initial test with a dropout rate of 9% (15 persons). The average adherence rate for nondropouts was 98%, which is higher than rates usually reported in the literature. Adherence was defined as exercising four times a week.

Effects of a 6-month incentive-based exercise program on adherence and work capacity.

The purpose of the study was to evaluate the effect of behavioral management techniques on exercise adherence linked to improvements in work capacity and maximal oxygen consumption (VO2max). One hundred thirty-seven participants in six different worksites on a university campus (five experimental and one comparison site) completed 6 months of a minimally supervised, incentive-based endurance exercise program. All participants in the experimental group contracted to engage in at least four bouts of 30 min of verified aerobic exercise within a prescribed target heart rate range each week for the duration of the program. Forty dollars deposited at the beginning of the program served as a response cost that could be lost as a result of failure to fulfill the weekly contracts. Individuals in the comparison group participated in a similar 6-month program but without the contracts and response cost strategies. Weekly adherence for both groups was strictly defined as verified fulfillment of all four bouts of exercise. Adherence for the experimental group was 97% by this definition, and adherence for the comparison group was 19% (P less than 0.01). VO2max increased 2.6% (P less than 0.01), and treadmill test time increased 16% (P less than 0.01) in the experimental group after the 6-month program, with no significant changes in the comparison group. Recovery heart rates at 2 and 4 min post-exercise were significantly lower at 6 months in the experimental group but not in the comparison group. These data provide evidence that adherence to a 6-month endurance exercise program can be improved significantly through the use of well conceived behavior management strategies.(ABSTRACT TRUNCATED AT 250 WORDS)

Gender differences in depression and anxiety among alcoholics.

Recent research suggests that psychopathology, in particular depression and anxiety, differentially affects the substance abuse treatment response of men and women. This study explores the relationship between global psychopathology, depression, anxiety, and alcoholism treatment outcome. These variables were assessed in a sample of 507 (373 men; 134 women) substance abuse clients at intake and at a 6-month follow-up. With the exception of alcohol dependence, there were significant differences in the levels of alcohol problems, depression, anxiety, and global psychopathology for men and women at both intake and follow-up. For the whole sample and for men, initial levels of alcohol problems and alcohol dependence were the best predictors of alcohol problems at follow-up. For women, the initial levels of alcohol dependence and a global measure of psychological functioning were predictive of outcome at follow-up. These findings are compared with past research, and suggestions for further investigation are proposed.

The perceptions of program directors and clients regarding the efficacy of methadone treatment.

A statewide sample of regional coordinators, program directors, and clients associated with opiate addiction treatment programs revealed their biases and expectations regarding the efficacy of methadone as a form of treatment. Methadone and drug-free program directors held consistent beliefs about treatment except the efficacy of methadone. Prior methadone clients currently in drug-free programs were skeptical of the methadone treatment, unlike clients currently in methadone treatment. The data reveal differences of opinions across the levels of the treatment system represented in the survey. In policy decisions, the need to consider empirical evidence in addition to personal opinions is emphasized, if consensus is ever to be realized.