Constructing validity evidence from a pilot key-features assessment of clinical decision-making in cerebral palsy diagnosis: application of Kane's validity framework to implementation evaluations.

BACKGROUND: Physician decision-making skills training is a priority to improve adoption of the cerebral palsy (CP) clinical guideline and, through this, lower the age of CP diagnosis. Clinical guideline implementation aims to improve physician practice, but evaluating meaningful change is complex. Limitations in the validity evidence of evaluation instruments impact the evidence base. Validity frameworks, such as Kane's, enable a targeted process to gather evidence for instrument scores, congruent to context and purpose. Yet, application of argument-based methodology to implementation validation is rare. Key-features examination methodology has established validity evidence supporting its use to measure decision-making skills, with potential to predict performance. We aimed to apply Kane's framework to evaluate a pilot key-features examination on physician decision-making in early CP diagnosis. METHODS: Following Kane's framework, we evaluated evidence across inferences of scoring, generalisation, extrapolation and implications in a study design describing the development and pilot of a CP diagnosis key-features examination for practising physicians. If found to be valid, we proposed to use the key-feature scores as an outcome measure of decision-making post education intervention to expedite CP diagnosis and to correlate with real-world performance data to predict physician practice. RESULTS: Supporting evidence for acceptance of scoring inferences was achieved through examination development with an expert group (n = 10) and pilot results (n = 10): (1) high internal consistency (0.82); (2) acceptable mean item-discrimination (0.34); and (3) acceptable reliability of examination scorers (95.2% congruence). Decreased physician acceptance of examination time (70%) was identified as a threat and prioritised in case reduction processes. Partial acceptance of generalisation, extrapolation and implications inferences were defensible with: (1) accumulated development evidence following established key-features methodology; (2) high pilot acceptance for authenticity (90%); and (3) plausibility of assumptions of score correlation with population register data. CONCLUSIONS: Kane's approach is beneficial for prioritising sources of validity evidence alongside the iterative development of a key-features examination in the CP field. The validity argument supports scoring assumptions and use of scores as an outcome measure of physician decision-making for CP guideline education implementation interventions. Scoring evidence provides the foundation to direct future studies exploring association of key-feature scores with real-world performance.

ABLE - assessment based learning.

Over the past decade the Australian health care system has moved rapidly toward a greater emphasis on medical care being provided within the community. This trend can only continue as our population ages and levels of chronic and complex illness continue to rise. Primary care now includes: a higher proportion of general practitioners working in group practices supported by practice nurses and allied health professionals- both on site and in the community, increased patient presentations for chronic and complex disease - often compounded by mental health and social issues, and, more hospital in the home, early discharge and similar programmes enabling shared management of sicker patients in the community.

Standards and performance--attainment and maintenance of professional capabilities.

The nature of The Royal Australian College of General Practitioners (RACGP) examination came under scrutiny in a recent debate among RACGP members, some of who suggested exploring an alternative pathway for assessment linked to continuing medical education. This article outlines key issues underpinning the examination that is part of the requirements for attaining Fellowship of the RACGP (FRACGP). It provides an overview of the theory and practice of assessment for general practice. The RACGP examination has an international reputation for quality, validity and reliability, a reason why the RACGP has been asked to assist many others in establishing and/or reviewing their own examination processes.

Selecting performance assessment methods for experienced physicians.

BACKGROUND: While much is now known about how to assess the competence of medical practitioners in a controlled environment, less is known about how to measure the performance in practice of experienced doctors working in their own environments. The performance of doctors depends increasingly on how well they function in teams and how well the health care system around them functions. METHODS: This paper reflects the combined experiences of a group of experienced education researchers and the results of literature searches on performance assessment methods. CONCLUSION: Measurement of competence is different to measurement of performance. Components of performance could be re-conceptualised within a different domain structure. Assessment methods may be of a different utility to that in competence assessment and, indeed, of different utility according to the purpose of the assessment. An exploration of the utility of potential performance assessment methods suggests significant gaps that indicate priority areas for research and development.

The teaching of occupational and environmental medicine to medical students in Australia and New Zealand.

All the medical schools in Australia and New Zealand were surveyed in order to determine the amount of teaching devoted to occupational and environmental medicine in the medical courses in 1998. A 100% response rate was achieved. The results showed that the number of hours devoted to these topics varied widely, but averaged 12.8 h and 10.5 topics. The most significant factor accounting for the variability was the presence on the universities' teaching staff of individuals trained in the practice of occupational medicine. While our findings show a greater time devoted to these topics than those of similar studies in the United States and Britain, the absolute time remains small when compared with the prevalence of occupational medicine problems in the community. There is little congruence in terms of both content and assessment processes between schools.

The usefulness of distinguishing different types of general practice consultation, or are needed skills always the same?

We hypothesized that it might be important to distinguish different types of consultation in order to understand better how help is delivered in general practice. This paper provides preliminary data on the ways that consultations can differ in how they unfold, and on how such differences might affect the helping process and the outcomes of each consultation. Several types of general practice consultation were noticed amongst 210 transcripts. Having established adequate inter-rater reliability of the classification system, we explored the differences between consultations of three broad types: Psychosocial, Complex, and Straightforward. Some of the questions asked in this preliminary study included: 1) do certain sorts of doctor, or patient, engage in certain sorts of consultation? 2) are there any relationships between the type of consultation and its length, patient-centredness or outcomes? 3) are the correlations between process and outcome clearer within categories of consultation than they are if consultations are treated as homogeneous? Findings provide a foundation for further investigations.