Evaluation of an ARF diagnosis calculator: a survey and content analysis.

BACKGROUND: Acute Rheumatic Fever (ARF) is a critically important condition for which there is no diagnostic test. Diagnosis requires the use of a set of criteria comprising clinical, laboratory, electrocardiographic and echocardiographic findings. The complexity of the algorithm and the fact that clinicians lack familiarity with ARF, make ARF diagnosis ideally suited to an electronic decision support tool. The ARF Diagnosis Calculator was developed to assist clinicians in diagnosing ARF and correctly assign categories of 'possible, 'probable' or 'definite' ARF. This research aimed to evaluate the acceptability, accuracy, and test performance of the ARF Diagnosis Calculator. METHODS: Three strategies were used to provide triangulation of data. Users of the calculator employed at Top End Health Service, Northern Territory, Australia were invited to participate in an online survey, and clinicians with ARF expertise were invited to participate in semi-structured interviews. Qualitative data were analysed using inductive analysis. Performance of the calculator in correctly diagnosing ARF was assessed using clinical data from 35 patients presenting with suspected ARF. Diagnoses obtained from the calculator were compared using the Kappa statistic with those obtained from a panel of expert clinicians. RESULTS: Survey responses were available from 23 Top End Health Service medical practitioners, and interview data were available from five expert clinicians. Using a 6-point Likert scale, participants highly recommended the ARF Diagnosis Calculator (median 6, IQR 1), found it easy to use (median 5, IQR 1) and believed the calculator helped them diagnose ARF (median 5, IQR 1). Clinicians with ARF expertise noted that electronic decision making is not a substitute for clinical experience. There was high agreement between the ARF Diagnosis Calculator and the 'gold standard' ARF diagnostic process (κ = 0.767, 95% CI: 0.568-0.967). Incorrect assignment of diagnosis occurred in 4/35 (11%) patients highlighting the greater accuracy of expert clinical input for ambiguous presentations. Sixteen changes were incorporated into a revised version of the calculator. CONCLUSIONS: The ARF Diagnosis Calculator is an easy-to-use, accessible tool, but it does not replace clinical expertise. The calculator performed well amongst clinicians and is an acceptable tool for use within the clinical setting with a high level of accuracy in comparison to the gold standard diagnostic process. Effective resources to support clinicians are critically important for improving the quality of care of ARF.

Monitoring and assessing the quality of care for youth: developing an audit tool using an expert consensus approach.

BACKGROUND: The mental health needs of young people are often inadequately met by health services. Quality improvement approaches provide a framework for measuring, assessing and improving the quality of healthcare. However, a lack of performance standards and measurement tools are an impediment to their implementation. This paper reports on the initial stages of development of a clinical audit tool for assessing the quality of primary healthcare for Australian Indigenous youth aged 12-24 including mental health services provided within primary care. METHODS: Audit items were determined through review of relevant guidelines, expert reference group consensus opinion and specific inclusion criteria. Pilot testing was undertaken at four Indigenous primary healthcare services. A focus group discussion involving five staff from a health service participating in pilot testing explored user experiences of the tool. RESULTS: Audit items comprise key measures of processes and outcomes of care for Indigenous youth, as determined by the expert reference group. Gaps and conflicts in relevant guidelines and a lack of agreed performance indicators necessitated a tool development process that relied heavily on expert reference group advice and audit item inclusion criteria. Pilot testing and user feedback highlighted the importance of feasibility and context-specific considerations in tool development and design. CONCLUSIONS: The youth health audit tool provides a first step in monitoring, assessing and improving the way Indigenous primary healthcare services engage with and respond to the needs of youth. Our approach offers a way forward for further development of quality measures in the absence of clearly articulated standards of care.