When enough is enough: a conceptual basis for fair and defensible practice performance assessment.

INTRODUCTION: An essential element of practice performance assessment involves combining the results of various procedures in order to see the whole picture. This must be derived from both objective and subjective assessment, as well as a combination of quantitative and qualitative assessment procedures. Because of the severe consequences an assessment of practice performance may have, it is essential that the procedure is both defensible to the stakeholders and fair in that it distinguishes well between good performers and underperformers. LESSONS FROM COMPETENCE ASSESSMENT: Large samples of behaviour are always necessary because of the domain specificity of competence and performance. The test content is considerably more important in determining which competency is being measured than the test format, and it is important to recognise that the process of problem-solving process is more idiosyncratic than its outcome. It is advisable to add some structure to the assessment but to refrain from over-structuring, as this tends to trivialise the measurement. IMPLICATIONS FOR PRACTICE PERFORMANCE ASSESSMENT: A practice performance assessment should use multiple instruments. The reproducibility of subjective parts should not be increased by over-structuring, but by sampling through sources of bias. As many sources of bias may exist, sampling through all of them may not prove feasible. Therefore, a more project-orientated approach is suggested using a range of instruments. At various timepoints during any assessment with a particular instrument, questions should be raised as to whether the sampling is sufficient with respect to the quantity and quality of the observations, and whether the totality of assessments across instruments is sufficient to see 'the whole picture'. This policy is embedded within a larger organisational and health care context.

Association between licensing examination scores and resource use and quality of care in primary care practice.

CONTEXT: Clinical competence is a determinant of the quality of care delivered, and may be associated with use of health care resources by primary care physicians. Clinical competence is assumed to be assessed by licensing examinations, yet there is a paucity of information on whether scores achieved predict subsequent practice. OBJECTIVE: To determine if licensing examination scores were associated with selected aspects of quality of care and resource use in initial primary care practice. DESIGN: Prospective cohort study of recently licensed family physicians, followed up for the first 18 months of practice. SETTING: The Quebec health care system. PARTICIPANTS: A total of 614 family physicians who passed the licensing examination between 1991 and 1993 and entered fee-for-service practice in Quebec. MAIN OUTCOME MEASURES: All patients seen by physicians were identified by the universal health insurance board and all health services provided to these patients were retrieved for the 18 months prior to (baseline) and after (follow-up) the physicians' entry into practice. Medical service and prescription claims files were used to measure rates of resource use (specialty consultation, symptom-relief prescribing compared with disease-specific prescribing) and quality of care (inappropriate prescribing, mammography screening). Baseline data were used to adjust for differences in practice population. RESULTS: Study physicians saw a total of 1116389 patients, of whom 113535 (10.2%) were elderly and 83391 (7.5%) were women aged 50 to 69 years. Physicians with higher licensing examination scores referred more of their patients for consultation (3.8/1000 patients per SD increase in score; 95% confidence interval [CI], 1.2-7.0; P = .005), prescribed to elderly patients fewer inappropriate medications (-2.7/1000 patients per SD increase in score; 95% CI, -4.8 to -0.7; P=.009) and more disease-specific medications relative to symptom-relief medications (3.9/1000 patients per SD increase in score; 95% CI, 0.3 to 7.4; P= .03), and referred more women aged 50 to 69 years (6.6/1000 patients per SD increase in score; 95% CI, 1.2-11.9; P = .02) for mammography screening. If patients of physicians with the lowest scores had experienced the same rates of consultation, prescribing, and screening as patients of physicians with the highest scores, an additional 3027 patients would have been referred, 179 fewer elderly patients would have been prescribed symptom-relief medication, 912 more elderly patients would have been prescribed disease-specific medication, 189 fewer patients would have received inappropriate medication, and 121 more women would have received mammography screening. CONCLUSIONS: Licensing examination scores are significant predictors of consultation, prescribing, and mammography screening rates in initial primary care practice.

Using standardized patients in licensing/certification examinations: comparison of two tests in Canada.

BACKGROUND: A new licensing examination for family physicians was introduced in the Province of Quebec, Canada, in 1990. It contains a newly developed, standardized, patient-based objective structured clinical examination (OSCE) used in complementarity with two other instruments (Short Answers Management Problems [SAMPs] and Simulated Office Orals [SOOs]) that have been used for many years by the College of Family Physicians of Canada (CFPC) for certification purposes. This research studied the criterion validity of the OSCE using the last instruments as criteria. METHODS: The results of the 172 spring 1994 candidates were analyzed. Cronbach alpha reliability coefficients were computed. Pearson's correlation coefficients and regression analysis measured the extent to which scores on the new Quebec OSCE correlated with the scores on the CFPC instruments. RESULTS: Correlation coefficients were significant between the Quebec OSCE scores and the CFPC examination total scores (.700), the SAMPs scores (.634), and the SOOs scores (.514). Multiple regression analysis confirmed that scores on the Quebec OSCE explained a significant but still limited proportion of the variation in scores of the CFPC examination. DISCUSSION: This study supports the criterion validity of the Quebec OSCE used in the licensing examination. However, differences in the skills assessed by each instrument and in the clinical situations presented exist between the Quebec instrument and the two national instruments. These differences, plus the variation in the scores obtained by the candidates on the three instruments, support the decision of the Quebec licensing organization to use them in complementarity.

Content validity of the Quebec licensing examination (OSCE). Assessed by practising physicians.

OBJECTIVE: To assess content validity of the objective structured clinical examination (OSCE), which uses multiple encounters with standardized patients and which is included in the Quebec licensing examination (QLEx). DESIGN: Descriptive study using written questionnaires based on Likert-type scales. SETTING: Examination centres of the QLEx. PARTICIPANTS: Thirteen practising family physicians with 3 to 6 years' practice experience who passed the QLEx OSCE in May 1993. MAIN OUTCOME MEASURES: Opinion whether the QLEx OSCE globally measured the competence of family physicians; opinion whether OSCE cases and stations made it possible to assess the main clinical abilities required of a family physician; opinion on specific statements about each case, such as how frequently the case was seen in the participants' practice, if the participants knew how to deal with the problem, if the simulated patient was believable, if the clinical situation was realistic, and if the duration of the station was adequate; and participants' self-assessment of their performance on the case. RESULTS: Eleven participants (84.6%) agreed that the OSCE globally measured the competence of a family physician. All agreed that the OSCE cases and stations made it possible to assess the main clinical skills required of a family physician. For most of the cases, participants believed that they knew how to deal with the problem, that the clinical situation was realistic, that the simulated patient was believable, and that the case duration was adequate. CONCLUSIONS: The results of this study support the content validity of the QLEx OSCE and confirm that it assesses the main skills needed for practising family medicine and adequately samples possible encounters or cases seen in family practice.

[MRI aspects of progressive multifocal leukoencephalopathy]. / Aspects IRM de la leucoencéphalopathie multifocale progressive.

Progressive Multifocal Leukoencephalopathy is a demyelinating disease. MRI shows high signal intensity areas on T2w sequence and low intensity aeras on T1w sequence, without enhancement after intravenous contrast injection. The involvement of arcuate fiber (U fibers) creates a sharp border with the cortex. There is no mass effect. Involvement of parieto-occipital areas is frequent. The lesions may be uni or bilateral, single or multiple; bilateral lesions are asymmetric. This typical appearance on MR images occurs in 90% of the patients with PML. Some atypical patterns may occur: focal hemorrhage, atrophy, faint peripheral enhancement and involvement of deep gray matter (basal ganglia). In most cases, the clinical and MR features provide the diagnosis. The main differential diagnosis, in MRI, is HIV-leukoencephalitis, but lesions are diffuse, less intense on T2w sequence and not visible on T1, without involvement of the arcuate fibers. Stereotactic biopsy should be performed only for atypical lesions, particularly in case of predominant involvement of deep gray structures.

Residency directors' predictions of candidates' performances on a licensing examination.

PURPOSE: To look at how well residency directors in family medicine predict the performances of their candidates on the Quebec Licensing Examination (QLEx) for family physicians. METHOD: The four family medicine program directors in Quebec were asked in the spring of 1992 to identify the ten residents from their own programs who would get the highest marks on the QLEx and the ten who would get the lowest marks. From the results of these candidates, and those of the intermediate groups defined by default, the prediction abilities of the program directors were assessed. RESULTS: Descriptive statistics showed that the program directors had difficulties discriminating among the different groups and tended to collapse the predicted scores toward the total mean. Analysis-of-variance studies confirmed the absence of difference between the predicted weak and intermediate groups as well as between the predicted intermediate and strong groups for each program and for each QLEx component. Regression analysis as well as eta 2 studies showed that the program directors' prediction abilities were low for all components and represented less than 25% of the explained variance of the QLEx scores. CONCLUSION: The residency directors did not accurately categorize their residents' performances on the QLEx. Both the evaluations of program directors and terminal examination results are complementary approaches to clinical competence assessment and should be used for licensure.

[Diffuse toxoplasmic encephalitis in a non-immunosuppressed patient]. / Encéphalite toxoplasmique diffuse chez un sujet non immunodéprimé.

Diffuse encephalitic toxoplasmosis is an unusual presentation of toxoplasmosis, in which neuroradiological investigations may not show focal abcesses. Until now it was only reported in immunocompromised patients. In immunocompetent patients, cerebral toxoplasmosis is very unusual, and appears as multiple abcesses, like the classic form in immunocompromised patients. We report the case of an immunocompetent patient who presented a diffuse encephalitic toxoplasmosis; CT and MR examinations showed only nonspecific features of brain swelling and cortical infarct due to vasculitis.

[Myxoid liposarcoma. MRI imaging]. / Liposarcome myxoïde. Aspects en IRM.

PURPOSE: Myxoid liposarcoma is the most common type of liposarcoma (approximately 40 to 50% of all liposarcomas). The main tissue component is a myxoid matrix present primarily in extracellular compartments; proliferating lipoblasts account for less 10% of the tumor: MRI appearances are not typical for lipomatous tumor. Nevertheless histological features may permit understanding MRI findings and identifying patients with myxoid liposarcoma. MATERIALS AND METHODS: Clinical history and radiologic images of 7 patients with histologically verified myxoid liposarcoma were retrospectively studied. In all patients the tumor presented in a lower extremity as a painless, slowly growing mass. MR images were available for review in all cases (T1- and T2-weighted images); in addition fat-suppression before and after gadolinium enhancement were assessed with T1-weighted sequences. RESULTS: MRI examination revealed an "encapsulated" tumor, non infiltrating and septated. On T1-weighted sequences all the lesions show lacy, amorphous and linear foci of high signal within a low signal of the tumor due to the predominance of a myxoid matrix. The high sensitivity of MRI demonstrates the presence of small areas of high signal and fat-suppression technique is valuable for characterizing soft-tissue tumors: suppression of high signal intensity on fat-saturated T1-weighted images indicates the presence of fatty tissue components. CONCLUSION: Clinical correlation with MRI appearances on T1-weighted sequences (in addition to fat-suppression technique) may suggest the possibility of myxoid liposarcoma.

Canadian experience with structured clinical examinations.

The use of structured clinical examinations to improve the evaluation of medical students and graduates has become significantly more common in the past 25 years. Many Canadian medical educators have contributed to the development of this technique. The Canadian experience is reviewed from the introduction of simulated-standardized patients and objective-structured clinical examinations to more recent developments and the use of such examinations for licensure and certification.

Large-scale use of an objective, structured clinical examination for licensing family physicians.

Since 1988 in Quebec the completion of a residency training program in family medicine or a specialty and of a comprehensive examination has been necessary to obtain a licence. An objective, structured clinical examination (OSCE) was designed by the Corporation professionnelle des médecins du Québec and Quebec's four medical schools to evaluate the clinical competence of newly trained family physicians. The certification examination of the College of Family Physicians of Canada was added to the OSCE. More than 500 candidates have been assessed: 262 in the spring of 1990, 42 in the fall of 1990 and 235 in the spring of 1991. The spring session occurs in four centres, three offering it in French and one in English, and the fall session takes place in one bilingual centre. In each centre 25 standardized patients and 25 examiners are required on each day of the 2-day OSCE. The scores obtained by the candidates who completed the OSCE in the first three sessions showed a normal distribution. No more than 5% failed the OSCE, the pass level having been set at two standard deviations below the mean. Equivalence was shown among the OSCE tracks, and reliability coefficients of 0.644, 0.723 and 0.736 were obtained for the three sessions respectively. The overall success rate for the licensing examination was 92%. The integration of such a large-scale OSCE into a licensing examination and the results obtained show that assessment of clinical competence for licensing purposes is feasible. The Quebec experience may help other organizations that are developing OSCEs for summative purposes.

Quebec's Compulsory 2-Year Training Program: Effects on family medicine teaching.

The Government of Quebec promulgated regulations in June 1987 that precipitately expanded the number of family medicine residencies in Quebec while effectively abolishing the rotating internship. The authors describe the context in which the expansion took place, the problems posed for the four Quebec programs, the solutions attempted, the outcomes envisaged, and some remaining difficulties.

Referral continuity between private and public health care for the patient under 18 in the Montreal metropolitan region.

One of the objectives of the health and social services reform launched in Quebec in the early 70s was to achieve continuity between the public and the private sector. This study focuses on specific aspects of continuity, namely referral continuity as measured by the degree to which physicians refer to public health resources in the care they provide to patients under 18, as well as their perception of these resources. A telephone survey conducted in February, 1984 gathered information on the way Montreal general practitioners and pediatricians use other medical, paramedical and public health resources. The findings indicate that the objective of continuity has not been fully achieved. Among the factors which explain such results are the lack of communication between the two sectors and the perception by physicians that these other resources, especially CLSCs, offer competitive rather than complementary services. Such a view constitutes a major constraint to the effective and efficient coexistence of the private and public sectors in a medical care system.

Influence of cultural and environmental factors on breast-feeding.

A study was undertaken at hôpital Sainte-Justine, Montreal, to determine the factors that influence the choice of feeding method for infants and the duration of breast-feeding. The factors that most clearly distinguished the mothers who breast-fed were lifestyle, education and conditioning to the parental role; those who nursed successfully for a long time were well motivated, well educated, supported by their family, especially the husband, and the La Leche League, and were of an economic status such that it was not necessary for them to return to work soon after delivery. Although the breast-fed babies were more fussy, they were healthier and received fewer drugs.