The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement.

BACKGROUND: Little is known about the technical adequacy of portfolios in reporting multiple complex academic and performance-based assessments. We explored, first, the influencing factors on the precision of scoring within a programmatic assessment of student learning outcomes within an integrated clinical placement. Second, the degree to which validity evidence supported interpretation of student scores. METHODS: Within generalisability theory, we estimated the contribution that each wanted factor (i.e. student capability) and unwanted factors (e.g. the impact of assessors) made to the variation in portfolio task scores. Relative and absolute standard errors of measurement provided a confidence interval around a pre-determined pass/fail standard for all six tasks. Validity evidence was sought through demonstrating the internal consistency of the portfolio and exploring the relationship of student scores with clinical experience. RESULTS: The mean portfolio mark for 257 students, across 372 raters, based on six tasks, was 75.56 (SD, 6.68). For a single student on one assessment task, 11% of the variance in scores was due to true differences in student capability. The most significant interaction was context specificity (49%), the tendency for one student to engage with one task and not engage with another task. Rater subjectivity was 29%. An absolute standard error of measurement of 4.74%, gave a 95% CI of +/- 9.30%, and a 68% CI of +/- 4.74% around a pass/fail score of 57%. Construct validity was supported by demonstration of an assessment framework, the internal consistency of the portfolio tasks, and higher scores for students who did the clinical placement later in the academic year. CONCLUSION: A portfolio designed as a programmatic assessment of an integrated clinical placement has sufficient evidence of validity to support a specific interpretation of student scores around passing a clinical placement. It has modest precision in assessing students' achievement of a competency standard. There were identifiable areas for reducing measurement error and providing more certainty around decision-making. Reducing the measurement error would require engaging with the student body on the value of the tasks, more focussed academic and clinical supervisor training, and revisiting the rubric of the assessment in the light of feedback.

Psychiatrists and GPs: diagnostic decision making, personality profiles and attitudes toward depression and anxiety.

OBJECTIVES: The objective of this article is to explore diagnostic decision making around psychological symptoms presenting to general practitioners (GPs) and psychiatrists, identify attitudinal and personality factors of possible relevance in these decisions, and compare GPs and psychiatrists to help identify potential educational targets. METHODS: GPs and psychiatrists attended separate peer-facilitated workshops in which two case presentations were discussed. Decision making was explored by structured questions embedded in the workshop, with responses recorded by electronic keypad technology. Participants completed demographic questionnaires and measures of personality and attitudes to depression. RESULTS: GPs and psychiatrists accorded emphasis to different elements of the history, and assigned different diagnoses based on the same set of symptoms. Both groups relied on non-pharmacological management for milder psychological symptoms; GPs were less likely to make a diagnosis of bipolar disorder. Traits of Extraversion and Agreeableness were associated with greater ease in treating depression. CONCLUSIONS: Differences in diagnostic decision making likely reflect the different contexts of specialist and generalist practice. Educational targets may include information about key symptoms to assist in diagnostic precision, but further information is needed to determine the best match between diagnostic processes, context and outcome. An awareness of the role of personality factors may help when designing education and support programs.

Diagnostic processes in mental health: GPs and psychiatrists reading from the same book but on a different page.

OBJECTIVE: To explore the clinical reasoning processes underpinning diagnostic and management decision-making in treating patients presenting with psychological distress in general practice. METHOD: Practising GPs were invited to attend small-group workshops in which two case histories were presented. Discussion was GP-facilitated and recorded for thematic analysis. GPs provided demographic data, completed personality and attitudinal questionnaires, and answered a series of multiple-choice questions embedded in the cases. RESULTS: GPs recognize the possibility of psychiatric disorders early in the clinical reasoning process, but are cautious about applying definitive diagnoses. GPs perceive that patients may be resistant to a psychiatric diagnosis and instead emphasize the need to build rapport and explore and exclude physical comorbidities. GPs see patients with a broad spectrum of distress, illness and impairment, in whom the initial presentation of psychological symptoms is often poorly differentiated and somatically focused, requiring elucidation over time. GPs therefore adopt a longitudinal strategy for diagnosis rather than investing heavily in cross-sectional assessment. CONCLUSION: GPs appear cognizant of possible psychiatric disorders and management strategies, but employ diagnostic strategies and decision-making processes that, in addition to experience and expertise, likely reflect key differences between the primary care and specialist practice settings.

Choosing general practice as a career - the influences of education and training.

BACKGROUND: In Australia, most medical students graduate without a firm career choice, with this decision being made during their early postgraduate years. Strategies addressing the current lack of meaningful exposure to general practice during these formative prevocational years are likely to be the most effective in increasing the proportion and number of entrants to general practice. OBJECTIVE: This review summarises the influences of medical student selection criteria, curriculum, geographical location, timing and duration of general practice exposure and experience, prevocational experience, and vocational training, on an eventual choice of general practice as a career. DISCUSSION: These are important influences on the complex process of career choice. Much research has focused on isolated interventions at one point along the pipeline. Varied and conflicting conclusions emerge from individual studies. In complex systems it is hard to understand the influence of an isolated intervention without looking at the system as a whole.

Choosing general practice - a review of career choice determinants.

This review takes a narrative synthesis approach - a systematic approach to interpreting complex evidence - to a broad literature review of the complex issue of medical career decision making. Addressing the Australian general practice workforce crisis requires an understanding of career choice motivators.

Shared decision making in Australia in 2017.

Shared decision making (SDM) is now firmly established within national clinical standards for accrediting hospitals, day procedure services, public dental services and medical education in Australia, with plans to align general practice, aged care and disability service. Implementation of these standards and training of health professionals is a key challenge for the Australian health sector at this time. Consumer involvement in health research, policy and clinical service governance has also increased, with a major focus on encouraging patients to ask questions during their clinical care. Tools to support shared decision making are increasingly used but there is a need for more systemic approaches to their development, cultural adaptation and implementation. Sustainable solutions to ensure tools are kept up-to-date with the best available evidence will be important for the future.

Evaluating peer teaching about chronic disease.

BACKGROUND: The primary care areas of priority (PCAP) activity was developed to engage medical students in learning about chronic disease management in a clinical context from their peers. It is one of several summative assessment tasks that occur during a primary care community term rotation in a graduate-entry medical programme. We evaluated the acceptability and effectiveness of the PCAP activity as a combined teaching, learning and assessment innovation. METHODS: Evaluation and assessment data from students who rotated through the four community term blocks during the 2011-2012 academic year was analysed using both qualitative and quantitative methods. PCAP peer teaching activity is an effective format for teaching about the management of chronic conditions in the community RESULTS: Analysis indicated that 89 per cent (n = 148/166) of responding students rated the PCAP as satisfactory, good or excellent. The marking rubric contained 11 assessable components, including teaching skills, mastery of clinical knowledge, developing a management plan, disease prevention and health promotion opportunities, identifying patient safety issues, the impact of the clinical presentation on the patient and family, and consideration of health equity factors. Analysis of the assessment scores indicated that the majority of students achieved the specified learning outcomes. DISCUSSION: The PCAP peer teaching activity was an acceptable and effective format for teaching about the management of chronic conditions in the community, and is adaptable to other teaching contexts. Students enjoyed teaching and being taught by their peers, and assessment results indicated that they developed their clinical knowledge as well as their teaching ability regarding chronic disease management.