Knowledge translation for realist reviews: a participatory approach for a review on scaling up complex interventions.

BACKGROUND: Knowledge syntheses that use a realist methodology are gaining popularity. Yet, there are few reports in the literature that describe how results are summarised, shared and used. This paper aims to inform knowledge translation (KT) for realist reviews by describing the process of developing a KT strategy for a review on pathways for scaling up complex public health interventions. METHODS: The participatory approach used for the realist review was also used to develop the KT strategy. The approach included three main steps, namely (1) an international meeting focused on interpreting preliminary findings from the realist review and seeking input on KT activities; (2) a targeted literature review on KT for realist reviews; and (3) consultations with primary knowledge users of the review. RESULTS: The international meeting identified a general preference among knowledge users for findings from the review that are action oriented. A need was also identified for understanding how to tailor findings for specific knowledge user groups in relation to their needs. The literature review identified four papers that included brief descriptions of planned or actual KT activities for specific research studies; however, information was minimal on what KT activities or products work for whom, under what conditions and why. The consultations revealed that KT for realist reviews should consider the following: (1) activities closely aligned with the preferences of specific knowledge user groups; (2) key findings that are sensitive to factors within the knowledge user's context; and (3) actionable statements that can advance KT goals, activities or products. The KT strategy derived from the three activities includes a planning framework and tailored KT activities that address preferences of knowledge users for findings that are action oriented and context relevant. CONCLUSIONS: This paper provides an example of a KT strategy for realist reviews that blends theoretical and practical insights. Evaluation of the strategy's implementation will provide useful insights on its effectiveness and potential for broader application.

Using systems thinking and the Intervention Level Framework to analyse public health planning for complex problems: Otitis media in Aboriginal and Torres Strait Islander children.

BACKGROUND: Middle ear disease (otitis media) is endemic among Aboriginal and Torres Strait Islander children in Australia and represents an important cause of hearing loss. The disease is the result of a mix of biological, environmental and host risk factors that interact in complex, non-linear ways along a dynamic continuum. As such, it is generally recognised that a holistic, systems approach is required to reverse the high rates of otitis media in Aboriginal and Torres Strait Islander children. The objective of this paper is to examine the alignment between efforts designed to address otitis media in Aboriginal and Torres Strait Islander children in Queensland, Australia and core concepts of systems thinking. This paper's overall purpose is to identify which combination of activities, and at which level, hold the potential to facilitate systems changes to better support ear health among Aboriginal and Torres Strait Islander children. METHODS: We began with a review of documents identified in consultation with stakeholders and an online search. In addition, key informants were invited to participate in an online survey and a face-to-face or phone interview. Qualitative interviews using a semi-structured interview guide were used to explore survey responses in more depth. We also undertook interviews at the community level to elicit a diverse range of views. Ideas, statements or activities reported in the documents and interviews as being performed under the Intervention Level Framework were identified using qualitative thematic and content analysis. A quantitative descriptive analysis was also undertaken, whereby data was extracted into an Excel spreadsheet and coded under the relevant strategic directions and performance indicators of the Framework. Subsequently, we coded activities against the five-level intervention framework developed by Malhi and colleagues, that is: 1) paradigm; 2) goals; 3) system structure; 4) feedback and delays; and 5) structural elements. RESULTS: Overall, twenty documents were reviewed. We examined surveys and interviews with six key informants. Twenty-four individual and 3 group interviews were conducted across central and community level informants. One hundred and four items were coded from the 20 documents and 156 items from interview data. For both data sets, the majority of activities were coded at the structural elements level. The results suggested three key areas where further work is needed to drive sustained improvements: 1) build the governance structures needed for paradigm shift to achieve a multi-sectoral approach; 2) develop shared system level goals; 3) develop system-wide feedback processes. CONCLUSIONS: Sustained progress in improving ear health within Aboriginal and Torres Strait Islander children requires a holistic, system-wide approach. To advance such work, governance structures for multi-sectoral collaboration including the development of joint goals and monitoring and feedback are required. Intervening at these higher leverage points could have a profound effect on persistent public health issues.

Collaborative Value Creation Framework: The Potential for Strengthening Intersectoral Partnerships in Chronic Disease Prevention.

BACKGROUND: There is considerable theoretical appeal and investment in intersectoral partnerships for chronic disease prevention, yet there is limited understanding of pathways for creating collaborative value. OBJECTIVES: The purpose of this paper was to discuss the Collaborative Value Creation Framework in terms of its potential to understand how value is created in prevention partnerships. RESULTS: The framework holds promise for a) studying the significance of all types of collaborative value and in what contexts, b) examining how sources of collaborative value can be enhanced, c) surfacing how mindsets advance or hinder collaborative efforts, and d) establishing interdependent links with and between other elements of the framework such as collaborative processes, stages, and outcomes to increase value creation potential. CONCLUSIONS: Mapping the interdependence of all elements would be an innovation to the application of the framework to discover leverage points to increase collaborative value and strengthen prevention partnerships.

Scaling up complex interventions: insights from a realist synthesis.

Preventing chronic diseases, such as cancer, cardiovascular disease and diabetes, requires complex interventions, involving multi-component and multi-level efforts that are tailored to the contexts in which they are delivered. Despite an increasing number of complex interventions in public health, many fail to be 'scaled up'. This study aimed to increase understanding of how and under what conditions complex public health interventions may be scaled up to benefit more people and populations.A realist synthesis was conducted and discussed at an in-person workshop involving practitioners responsible for scaling up activities. Realist approaches view causality through the linkages between changes in contexts (C) that activate mechanisms (M), leading to specific outcomes (O) (CMO configurations). To focus this review, three cases of complex interventions that had been successfully scaled up were included: Vibrant Communities, Youth Build USA and Pathways to Education. A search strategy of published and grey literature related to each case was developed, involving searches of relevant databases and nominations from experts. Data extracted from included documents were classified according to CMO configurations within strategic themes. Findings were compared and contrasted with guidance from diffusion theory, and interpreted with knowledge users to identify practical implications and potential directions for future research.Four core mechanisms were identified, namely awareness, commitment, confidence and trust. These mechanisms were activated within two broad scaling up strategies, those of renewing and regenerating, and documenting success. Within each strategy, specific actions to change contexts included building partnerships, conducting evaluations, engaging political support and adapting funding models. These modified contexts triggered the identified mechanisms, leading to a range of scaling up outcomes, such as commitment of new communities, changes in relevant legislation, or agreements with new funding partners.This synthesis applies and advances theory, realist methods and the practice of scaling up complex interventions. Practitioners may benefit from a number of coordinated efforts, including conducting or commissioning evaluations at strategic moments, mobilising local and political support through relevant partnerships, and promoting ongoing knowledge exchange in peer learning networks. Action research studies guided by these findings, and studies on knowledge translation for realist syntheses are promising future directions.

Improving the performance of interorganizational networks for preventing chronic disease: identifying and acting on research needs.

This article describes the role of interorganizational networks in chronic disease prevention and an action research agenda for promoting understanding and improvement. Through a model of engaged scholarship, leaders with expertise and experience in chronic disease prevention networks helped shape research directions focused on network value, governance, and evolution. The guiding principles for facilitating this research include applying existing knowledge, developing network-appropriate methods and measures, creating structural change, promoting an impact orientation, and fostering cultural change.

Content analysis of Australian direct-to-consumer websites for emerging breast cancer imaging devices.

OBJECTIVE: To describe the nature and frequency of information presented on direct-to-consumer websites for emerging breast cancer imaging devices. DESIGN: Content analysis of Australian website advertisements from 2 March 2011 to 30 March 2012, for three emerging breast cancer imaging devices: digital infrared thermal imaging, electrical impedance scanning and electronic palpation imaging. MAIN OUTCOME MEASURES: Type of imaging offered, device safety, device performance, application of device, target population, supporting evidence and comparator tests. RESULTS: Thirty-nine unique Australian websites promoting a direct-to-consumer breast imaging device were identified. Despite a lack of supporting evidence, 22 websites advertised devices for diagnosis, 20 advertised devices for screening, 13 advertised devices for prevention and 13 advertised devices for identifying breast cancer risk factors. Similarly, advertised ranges of diagnostic sensitivity (78%-99%) and specificity (44%-91%) were relatively high compared with published literature. Direct comparisons with conventional screening tools that favoured the new device were highly prominent (31 websites), and one-third of websites (12) explicitly promoted their device as a suitable alternative. CONCLUSIONS: Australian websites for emerging breast imaging devices, which are also available internationally, promote the use of such devices as safe and effective solutions for breast cancer screening and diagnosis in a range of target populations. Many of these claims are not supported by peer-reviewed evidence, raising questions about the manner in which these devices and their advertising material are regulated, particularly when they are promoted as direct alternatives to established screening interventions.

Advancing the application of systems thinking in health: advice seeking behavior among primary health care physicians in Pakistan.

BACKGROUND: Using measles and tuberculosis as case examples, with a systems thinking approach, this study examines the human advice-seeking behavior of primary health care (PHC) physicians in a rural district of Pakistan. This study analyzes the degree to which the existing PHC system supports their access to human advice, and explores in what ways this system might be strengthened to better meet provider needs. METHODS: The study was conducted in a rural district of Pakistan and, with a cross-sectional study design, it employed a range of research methods, namely extensive document review for mapping existing information systems, social network analysis of physicians' advice-seeking practice, and key stakeholder interviews for an in-depth understanding of the experience of physicians. Illustrations were prepared for information flow mechanism, sociographs were generated for analyzing social networks, and content analysis of qualitative findings was carried out for in-depth interpretation of underlying meanings. RESULTS: The findings of this study reveal that non-availability of competent supervisory staff, a focus on improving performance indicators rather than clinical guidance, and a lack of a functional referral system have collectively created an environment in which PHC physicians have developed their own strategies to overcome these constraints. They are well aware of the human expertise available within and outside the district. However, their advice-seeking behavior was dependent upon existence of informal social interaction with the senior specialists. Despite the limitations of the system, the physicians proactively used their professional linkages to seek advice and also to refer patients to the referral center based on their experience and the facilities that they trusted. CONCLUSIONS: The absence of functional referral systems, limited effective linkages between PHC and higher levels of care, and a focus on programmatic targets rather than clinical care have each contributed to the isolation of physicians and reactive information seeking behavior. The study findings underscore the need for a functional information system comprising context sensitive knowledge management and translation opportunities for physicians working in PHC centers. Such an information system needs to link people and resources in ways that transcend geography and discipline, and that builds on existing expertise, interpersonal relationships, and trust.

Networks to strengthen health systems for chronic disease prevention.

Interorganizational networks that harness the priorities, capacities, and skills of various agencies and individuals have emerged as useful approaches for strengthening preventive services in public health systems. We use examples from the Canadian Heart Health Initiative and Alberta's Primary Care Networks to illustrate characteristics of networks, describe the limitations of existing frameworks for assessing the performance of prevention-oriented networks, and propose a research agenda for guiding future efforts to improve the performance of these initiatives. Prevention-specific assessment strategies that capture relevant aspects of network performance need to be identified, and feedback mechanisms are needed that make better use of these data to drive change in network activities.

A time-responsive tool for informing policy making: rapid realist review.

BACKGROUND: A realist synthesis attempts to provide policy makers with a transferable theory that suggests a certain program is more or less likely to work in certain respects, for particular subjects, in specific kinds of situations. Yet realist reviews can require considerable and sustained investment over time, which does not always suit the time-sensitive demands of many policy decisions. 'Rapid Realist Review' methodology (RRR) has been developed as a tool for applying a realist approach to a knowledge synthesis process in order to produce a product that is useful to policy makers in responding to time-sensitive and/or emerging issues, while preserving the core elements of realist methodology. METHODS: Using examples from completed RRRs, we describe key features of the RRR methodology, the resources required, and the strengths and limitations of the process. All aspects of an RRR are guided by both a local reference group, and a group of content experts. Involvement of knowledge users and external experts ensures both the usability of the review products, as well as their links to current practice. RESULTS: RRRs have proven useful in providing evidence for and making explicit what is known on a given topic, as well as articulating where knowledge gaps may exist. From the RRRs completed to date, findings broadly adhere to four (often overlapping) classifications: guiding rules for policy-making; knowledge quantification (i.e., the amount of literature available that identifies context, mechanisms, and outcomes for a given topic); understanding tensions/paradoxes in the evidence base; and, reinforcing or refuting beliefs and decisions taken. CONCLUSIONS: 'Traditional' realist reviews and RRRs have some key differences, which allow policy makers to apply each type of methodology strategically to maximize its utility within a particular local constellation of history, goals, resources, politics and environment. In particular, the RRR methodology is explicitly designed to engage knowledge users and review stakeholders to define the research questions, and to streamline the review process. In addition, results are presented with a focus on context-specific explanations for what works within a particular set of parameters rather than producing explanations that are potentially transferrable across contexts and populations. For policy makers faced with making difficult decisions in short time frames for which there is sufficient (if limited) published/research and practice-based evidence available, RRR provides a practical, outcomes-focused knowledge synthesis method.

A systematic review of elastography, electrical impedance scanning, and digital infrared thermography for breast cancer screening and diagnosis.

The objective of this study aimed to systematically identify and evaluate all the available evidence of safety, effectiveness and diagnostic accuracy for three emerging classes of technology promoted for breast cancer screening and diagnosis: Digital infrared thermal imaging (DITI), electrical impedance scanning (EIS) and elastography. A systematic search of seven biomedical databases (EMBASE, PubMed, Web of Science, CRD, CINAHL, Cochrane Library, Current Contents Connect) was conducted through March 2011, along with a manual search of reference lists from relevant studies. The principal outcome measures were safety, effectiveness, and diagnostic accuracy. Data were extracted using a standardised form, and validated for accuracy by the secondary authors. Study quality was appraised using the quality assessment of diagnostic accuracy studies tool, while heterogeneity was assessed using forest plots, Cooks' distance and standardised residual scatter plots, and I (2) statistics. From 6,808 search results, 267 full-text articles were assessed, of which 60 satisfied the inclusion criteria. No effectiveness studies were identified. Only one EIS screening accuracy study was identified, while all other studies involved symptomatic populations. Significant heterogeneity was present among all device classes, limiting the potential for meta-analyses. Sensitivity and specificity varied greatly for DITI (Sens 0.25-0.97, Spec 0.12-0.85), EIS (Sens 0.26-0.98, Spec 0.08-0.81) and ultrasound elastography (Sens 0.35-1.00, Spec 0.21-0.99). It is concluded that there is currently insufficient evidence to recommend the use of these technologies for breast cancer screening. Moreover, the high level of heterogeneity among studies of symptomatic women limits inferences that may be drawn regarding their use as diagnostic tools. Future research employing standardised imaging, research and reporting methods is required.

Over 150 potentially low-value health care practices: an Australian study.

OBJECTIVE: To develop and apply a novel method for scanning a range of sources to identify existing health care services (excluding pharmaceuticals) that have questionable benefit, and produce a list of services that warrant further investigation. DESIGN AND SETTING: A multiplatform approach to identifying services listed on the Australian Medicare Benefits Schedule (MBS; fee-for-service) that comprised: (i) a broad search of peer-reviewed literature on the PubMed search platform; (ii) a targeted analysis of databases such as the Cochrane Library and National Institute for Health and Clinical Excellence (NICE) "do not do" recommendations; and (iii) opportunistic sampling, drawing on our previous and ongoing work in this area, and including nominations from clinical and non-clinical stakeholder groups. MAIN OUTCOME MEASURES: Non-pharmaceutical, MBS-listed health care services that were flagged as potentially unsafe, ineffective or otherwise inappropriately applied. RESULTS: A total of 5209 articles were screened for eligibility, resulting in 156 potentially ineffective and/or unsafe services being identified for consideration. The list includes examples where practice optimisation (ie, assessing relative value of a service against comparators) might be required. CONCLUSION: The list of health care services produced provides a launchpad for expert clinical detailing. Exploring the dimensions of how, and under what circumstances, the appropriateness of certain services has fallen into question, will allow prioritisation within health technology reassessment initiatives.

Engaging clinicians in evidence-based disinvestment: role and perceptions of evidence.

OBJECTIVES: The aim of this study was to determine how evidence from systematic review (SR) is perceived and negotiated by expert stakeholders in considering a technology for potential disinvestment. METHODS: An evidence-informed stakeholder engagement examined results from a diagnostic accuracy SR of vitamin B12 and folate tests. Pathologists deliberated around the SR findings to generate an informed contribution to future policy for the funding of B12 and folate tests. Deliberations were transcribed and subject to qualitative analysis. RESULTS: Pathologists did not engage with findings from the SR in depth; rather they sought to contest the terms of the problem driving the review and attempted to reframe it. Pathologists questioned the usefulness of SR outcomes given the variable definitions of B12 deficiency and deferred addressing disinvestment options specifically pertaining to B12 testing. However, folate testing was proffered as a potential disinvestment candidate, based upon pathologists' definition of "appropriate" evidence beyond the bounds of the SR. CONCLUSIONS: The value of SR to informing disinvestment deliberations by expert stakeholders may be a function of timing as well as content. Engagement of stakeholders in co-produced evidence may be required at two levels: (i) Early in the synthesis phase to help shape the SR and harmonize expert views with the available evidence (including gaps); (ii) Collaboration in primary research to fill evidence-gaps thus supporting evidence-based disinvestment. Without this, information asymmetry between clinically engaged experts and decision makers may preclude the collaborative, informed, and technical discussions required to generate productive policy change.

Monitoring drug safety with registries: useful components of postmarketing pharmacovigilance systems.

OBJECTIVE: At the time of licensing by regulatory agencies, the full range of risks and possible adverse drug reactions associated with a medication is rarely fully realized. This commentary aims to describe the role of registries as useful components of postmarketing pharmacovigilance systems for monitoring highly specialized medications associated with significant financial costs. STUDY DESIGN AND SETTING: We consider the limitations of traditional pharmacovigilance programs and discuss the strengths, limitations, and uses of registries in postmarketing pharmacovigilance systems. RESULTS: Registries have become increasingly appealing in postmarketing surveillance of medications; however, their exact role continues to evolve. Key registry projects, including the Prospective Immunogenicity Surveillance Registry, British Society for Rheumatology Biologics Register, Australian Rheumatology Association Database, the Haemostasis Registry, and the Bosentan Patient Registry highlight the value of registries for monitoring the incidence of rare adverse events. CONCLUSION: Although often limited by lack of a control group and the need for complete case ascertainment to maintain data integrity, registries are a useful component of postmarketing pharmacovigilance systems for monitoring highly specialized medications associated with significant financial costs.

System tools for system change.

BACKGROUND: Health system transformations are influenced by dynamic relationships within and between individuals and institutions, as well as political, educational and legislative factors. This article aims to promote awareness of five tools that recognise this complexity and that are proposed to have value for decision makers: concept mapping, social network analysis, system dynamics modelling, programme budgeting and marginal analysis, and the tools for knowledge management and translation. METHODS: The authors briefly describe the methodological approach of each tool, provide a commentary on the conditions in which these tools have been employed, and discuss their impact on the processes and outcomes of system transformation. An international advisory panel was convened based on a combination of experience, expertise and perspective. The panel assisted in synthesising the evidence relating to each tool and, in partnership with the authors, refined the interpretation of the role and value of each tool for system transformation. FINDINGS: The tools discussed may impact the structural and procedural outcomes of transformation as well as the values, behaviours and attitudes of people undergoing change. The techniques described provide those undertaking transformation with methods to negotiate clinical, academic, political, organisational and cultural perspectives, and recognise the pivotal role of context in transformation. CONCLUSIONS: This review offers a novel synthesis of how these tools may add value to decision making for health policy. The tools discussed, while not a panacea to the challenges of large system change, provide methods that acknowledge the complexity of the transformative challenge and present innovative paths to co-produced solutions.

Assisted reproductive technologies: a systematic review of safety and effectiveness to inform disinvestment policy.

OBJECTIVE: Health policy relating to assisted reproductive technologies (ART) has been variably informed by clinical evidence, social values, political and fiscal considerations. This systematic review examined key factors associated with specific benefits and harms of ART to inform the development of a model for generating policy recommendations within an Australian disinvestment research agenda. METHODS: Six databases were searched from 1994 to 2009. Included articles contained data on safety and/or effectiveness of in vitro fertilisation (IVF) or IVF with intracytoplasmic sperm injection with reference to female age, male age or cycle rank. Narrative descriptions of key outcomes (live birth, miscarriage) were constructed alongside tabular summaries. RESULTS: Sixty-eight studies and one registry report were included. There was substantial heterogeneity present within the evidence-base which limited the strength and scope of conclusions that could be drawn. However, this review does affirm the differential effectiveness associated with the ageing of ART patients with regard to live birth and miscarriage. CONCLUSION: From the available evidence, it was not possible to determine an explicit age or cycle rank that could be used to formulate defensible policy responsive to identified differential effectiveness. Stakeholder interpretation of this evidence-base may assist in developing policy that can incorporate uncertainty and reflect social values.

Diagnostic performance of serum cobalamin tests: a systematic review and meta-analysis.

AIMS: Serum cobalamin (cbl, vitamin B(12)) tests are routinely ordered for investigating conditions potentially amenable to cbl supplementation. This study aimed to systematically assess the evidence of diagnostic accuracy for serum cbl tests across patient subgroups. METHODS: Seven medical databases were searched (1990 to November 2009). Studies were included that compared serum cbl to a reference standard (all reference standards employed). Study quality was assessed using QUADAS. Summary estimates of test performance were determined using the bivariate model and hierarchical summary receiver operating characteristic curves (HSROC). RESULTS: Of 2878 identified studies, 54 were included. Studies rated poorly against QUADAS criteria. Positive (PLR) and negative likelihood ratios (NLR) were 2.72 [95% confidence interval (CI) 1.95, 3.81] and 0.59 (0.49, 0.72), respectively (studies employing methylmalonic acid as the referent). In studies employing a clinical reference standard, PLR was 3.33 (0.92, 12.10) and NLR 0.34 (0.13, 0.89). Test performance did not vary by clinical indication, test method or age. CONCLUSION: This review was limited by the quality of the evidence base and lack of a gold standard. From the available evidence, diagnosis of conditions amenable to cbl supplementation on the basis of serum cbl level alone cannot be considered a reliable approach to investigating suspected vitamin deficiency.

Identifying and improving unreliable items in registries through data auditing.

OBJECTIVE: Assessing the reliability of clinical registries is important for ensuring the availability of credible data. Therefore, this study aimed to investigate the reliability of data collected by the Australian and New Zealand Haemostasis Registry (the registry). DESIGN: Data from 5% of randomly selected registry cases were re-abstracted by an independent data auditor who was blinded to the results of the original data abstraction. Categorical data were investigated for agreement between original and re-abstracted data. The mean difference and standard deviations (SD) of differences were calculated for continuous variables. We estimated a 'prediction interval' as the mean difference ± twice the SD of differences. We computed a coefficient of variation as the SD of differences. SETTING: The registry records all cases of off-licence use of recombinant activated factor VII (rFVIIa) at participating institutions (on-licence use of rFVIIa is not recorded). RESULTS: Data on 76 registry cases (6% of registry) were re-abstracted. Various parameters demonstrated high levels of inter-rater reliability, including age, gender and intensive care unit admission (88, 99 and 99% agreement, respectively). Other variables were highly unreliable, including crystalloid infusion volumes (coefficient of variation 123.01%), red blood cell units (92.05%) and time from bleeding onset to administration of rFVIIa (153.06%). CONCLUSIONS: Registry audits are useful for identifying variables with poor reliability. Repeated audits will not improve data reliability; however, they can assist in identifying and evaluating the impact of modified data collection processes on improving data reliability.

Predicting trauma patient mortality: ICD [or ICD-10-AM] versus AIS based approaches.

BACKGROUND: The International Classification of Diseases Injury Severity Score (ICISS) has been proposed as an International Classification of Diseases (ICD)-10-based alternative to mortality prediction tools that use Abbreviated Injury Scale (AIS) data, including the Trauma and Injury Severity Score (TRISS). To date, studies have not examined the performance of ICISS using Australian trauma registry data. This study aimed to compare the performance of ICISS with other mortality prediction tools in an Australian trauma registry. METHODS: This was a retrospective review of prospectively collected data from the Victorian State Trauma Registry. A training dataset was created for model development and a validation dataset for evaluation. The multiplicative ICISS model was compared with a worst injury ICISS approach, Victorian TRISS (V-TRISS, using local coefficients), maximum AIS severity and a multivariable model including ICD-10-AM codes as predictors. Models were investigated for discrimination (C-statistic) and calibration (Hosmer-Lemeshow statistic). RESULTS: The multivariable approach had the highest level of discrimination (C-statistic 0.90) and calibration (H-L 7.65, P= 0.468). Worst injury ICISS, V-TRISS and maximum AIS had similar performance. The multiplicative ICISS produced the lowest level of discrimination (C-statistic 0.80) and poorest calibration (H-L 50.23, P < 0.001). CONCLUSIONS: The performance of ICISS may be affected by the data used to develop estimates, the ICD version employed, the methods for deriving estimates and the inclusion of covariates. In this analysis, a multivariable approach using ICD-10-AM codes was the best-performing method. A multivariable ICISS approach may therefore be a useful alternative to AIS-based methods and may have comparable predictive performance to locally derived TRISS models.