Clinical indicators for routine use in the evaluation of early psychosis intervention: development, training support and inter-rater reliability.

AIM: Clinical practice improvement carried out in a quality assurance framework relies on routinely collected data using clinical indicators. Herein we describe the development, minimum training requirements, and inter-rater agreement of indicators that were used in an Australian multi-site evaluation of the effectiveness of early psychosis (EP) teams. METHODS: Surveys of clinician opinion and face-to-face consensus-building meetings were used to select and conceptually define indicators. Operationalization of definitions was achieved by iterative refinement until clinicians could be quickly trained to code indicators reliably. Calculation of percentage agreement with expert consensus coding was based on ratings of paper-based clinical vignettes embedded in a 2-h clinician training package. RESULTS: Consensually agreed upon conceptual definitions for seven clinical indicators judged most relevant to evaluating EP teams were operationalized for ease-of-training. Brief training enabled typical clinicians to code indicators with acceptable percentage agreement (60% to 86%). For indicators of suicide risk, psychosocial function, and family functioning this level of agreement was only possible with less precise 'broad range' expert consensus scores. Estimated kappa values indicated fair to good inter-rater reliability (kappa > 0.65). Inspection of contingency tables (coding category by health service) and modal scores across services suggested consistent, unbiased coding across services. CONCLUSIONS: Clinicians are able to agree upon what information is essential to routinely evaluate clinical practice. Simple indicators of this information can be designed and coding rules can be reliably applied to written vignettes after brief training. The real world feasibility of the indicators remains to be tested in field trials.

Appraising evidence for intervention effectiveness in early psychosis: conceptual framework and review of evaluation approaches.

The literature that is relevant to evaluation of treatment effectiveness is large, scattered and difficult to assemble for appraisal. This scoping review first develops a conceptual framework to help organize the field, and second, uses the framework to appraise early psychosis intervention (EPI) studies. Literature searches were used to identify representative study designs, which were then sorted according to evaluation approach. The groupings provided a conceptual framework upon which a map of the field could be drawn. Key words were cross-checked against definitions in dictionaries of scientific terms and the National Library of Medicine Medical Subject Headings (MeSH) browser. Using the final list of key words as search terms, the EPI evaluation literature was appraised. Experimental studies could be grouped into two classes: efficacy and effectiveness randomized controlled trials. Non-experimental studies could be subgrouped into at least four overlapping categories: clinical epidemiological; health service evaluations; quality assurance studies; and, quasi-experimental assessments of treatment effects. Applying this framework to appraise EPI studies indicated promising evidence for the effectiveness of EPI irrespective of study design type, and a clearer picture of where future evaluation efforts should be focused. Reliance on clinical trials alone will restrict the type of information that can inform clinical practice. There is convergent evidence for the benefits of specialized EPI service functions across a range of study designs. Greater investment in health services research and quality assurance approaches in evaluating EPI effectiveness should be made, which will involve scaling up of study sizes and development of an EPI programme fidelity rating template. The degree of complexity of the evaluation field suggests that greater focus on research methodology in the training of Australasian psychiatrists is urgently needed.

Is a national framework for implementing early psychosis services necessary? Results of a survey of Australian mental health service directors.

AIM: Expert opinion holds that the rate of implementation of specialist services for first presentation psychosis in Australia is much too slow. We aimed to collect evidence regarding this view from the first national survey of adult public mental health services about their self-reported efforts to implement specialist early psychosis intervention (EPI). METHODS: Using a purpose-designed Census form for assessing EPI implementation, adult public mental health service directors throughout Australia were asked about EPI-relevant local service activities. RESULTS: Sixty Census forms were returned (response rate = 61%), representing a total catchment population of 12.5 million people. A minority of services reported high levels of EPI implementation, which varied widely between area services and across state and territory jurisdictions. Rural and remote services were overrepresented in the lowest levels of reported EPI implementation. Only one service characteristic, the value of identifiable funding committed specifically to EPI, was predictive of level of reported EPI implementation. CONCLUSIONS: The disturbingly high levels of variability in EPI implementation across jurisdictions suggest a pressing need for a set of nationally agreed uniform EPI implementation standards. Additional specific strategies for rural and remote mental health services may be needed for these services to implement EPI.