Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI.

Artificial intelligence (AI) systems hold great promise as decision-support tools, but we must be able to identify and understand their inevitable mistakes if they are to fulfill this potential. This is particularly true in domains where the decisions are high-stakes, such as law, medicine, and the military. In this Perspective, we describe the particular challenges for AI decision support posed in military coalition operations. These include having to deal with limited, low-quality data, which inevitably compromises AI performance. We suggest that these problems can be mitigated by taking steps that allow rapid trust calibration so that decision makers understand the AI system's limitations and likely failures and can calibrate their trust in its outputs appropriately. We propose that AI services can achieve this by being both interpretable and uncertainty-aware. Creating such AI systems poses various technical and human factors challenges. We review these challenges and recommend directions for future research.

A genetic programming approach to development of clinical prediction models: A case study in symptomatic cardiovascular disease.

BACKGROUND: Genetic programming (GP) is an evolutionary computing methodology capable of identifying complex, non-linear patterns in large data sets. Despite the potential advantages of GP over more typical, frequentist statistical approach methods, its applications to survival analyses are rare, at best. The aim of this study was to determine the utility of GP for the automatic development of clinical prediction models. METHODS: We compared GP against the commonly used Cox regression technique in terms of the development and performance of a cardiovascular risk score using data from the SMART study, a prospective cohort study of patients with symptomatic cardiovascular disease. The composite endpoint was cardiovascular death, non-fatal stroke, and myocardial infarction. A total of 3,873 patients aged 19-82 years were enrolled in the study 1996-2006. The cohort was split 70:30 into derivation and validation sets. The derivation set was used for development of both GP and Cox regression models. These models were then used to predict the discrete hazards at t = 1, 3, and 5 years. The predictive ability of both models was evaluated in terms of their risk discrimination and calibration using the validation set. RESULTS: The discrimination of both models was comparable. At time points t = 1, 3, and 5 years the C-index was 0.59, 0.69, 0.64 and 0.66, 0.70, 0.70 for the GP and Cox regression models respectively. At the same time points, the calibration of both models, which was assessed using calibration plots and the generalization of the Hosmer-Lemeshow test statistic, was also comparable, but with the Cox model being better calibrated to the validation data. CONCLUSION: Using empirical data, we demonstrated that a prediction model developed automatically by GP has predictive ability comparable to that of manually tuned Cox regression. The GP model was more complex, but it was developed in a fully automated way and comprised fewer covariates. Furthermore, it did not require the expertise normally needed for its derivation, thereby alleviating the knowledge elicitation bottleneck. Overall, GP demonstrated considerable potential as a method for the automated development of clinical prediction models for diagnostic and prognostic purposes.

TRAK App Suite: A Web-Based Intervention for Delivering Standard Care for the Rehabilitation of Knee Conditions.

BACKGROUND: Standard care for the rehabilitation of knee conditions involves exercise programs and information provision. Current methods of rehabilitation delivery struggle to keep up with large volumes of patients and the length of treatment required to maximize the recovery. Therefore, the development of novel interventions to support self-management is strongly recommended. Such interventions need to include information provision, goal setting, monitoring, feedback, and support groups, but the most effective methods of their delivery are poorly understood. The Internet provides a medium for intervention delivery with considerable potential for meeting these needs. OBJECTIVE: The objective of this study was to demonstrate the feasibility of a Web-based app and to conduct a preliminary review of its practicability as part of a complex medical intervention in the rehabilitation of knee disorders. This paper describes the development, implementation, and usability of such an app. METHODS: An interdisciplinary team of health care professionals and researchers, computer scientists, and app developers developed the TRAK app suite. The key functionality of the app includes information provision, a three-step exercise program based on a standard care for the rehabilitation of knee conditions, self-monitoring with visual feedback, and a virtual support group. There were two types of stakeholders (patients and physiotherapists) that were recruited for the usability study. The usability questionnaire was used to collect both qualitative and quantitative information on computer and Internet usage, task completion, and subjective user preferences. RESULTS: A total of 16 patients and 15 physiotherapists participated in the usability study. Based on the System Usability Scale, the TRAK app has higher perceived usability than 70% of systems. Both patients and physiotherapists agreed that the given Web-based approach would facilitate communication, provide information, help recall information, improve understanding, enable exercise progression, and support self-management in general. The Web app was found to be easy to use and user satisfaction was very high. The TRAK app suite can be accessed at http://apps.facebook.com/kneetrak/. CONCLUSIONS: The usability study suggests that a Web-based intervention is feasible and acceptable in supporting self-management of knee conditions.

FlexiTerm: a flexible term recognition method.

BACKGROUND: The increasing amount of textual information in biomedicine requires effective term recognition methods to identify textual representations of domain-specific concepts as the first step toward automating its semantic interpretation. The dictionary look-up approaches may not always be suitable for dynamic domains such as biomedicine or the newly emerging types of media such as patient blogs, the main obstacles being the use of non-standardised terminology and high degree of term variation. RESULTS: In this paper, we describe FlexiTerm, a method for automatic term recognition from a domain-specific corpus, and evaluate its performance against five manually annotated corpora. FlexiTerm performs term recognition in two steps: linguistic filtering is used to select term candidates followed by calculation of termhood, a frequency-based measure used as evidence to qualify a candidate as a term. In order to improve the quality of termhood calculation, which may be affected by the term variation phenomena, FlexiTerm uses a range of methods to neutralise the main sources of variation in biomedical terms. It manages syntactic variation by processing candidates using a bag-of-words approach. Orthographic and morphological variations are dealt with using stemming in combination with lexical and phonetic similarity measures. The method was evaluated on five biomedical corpora. The highest values for precision (94.56%), recall (71.31%) and F-measure (81.31%) were achieved on a corpus of clinical notes. CONCLUSIONS: FlexiTerm is an open-source software tool for automatic term recognition. It incorporates a simple term variant normalisation method. The method proved to be more robust than the baseline against less formally structured texts, such as those found in patient blogs or medical notes. The software can be downloaded freely at http://www.cs.cf.ac.uk/flexiterm.

Detecting deterioration in patients with chronic disease using telemonitoring: navigating the 'trough of disillusionment'.

OBJECTIVES: To examine the evidence base for telemonitoring designed for patients who have chronic obstructive pulmonary disease and heart failure, and to assess whether telemonitoring fulfils the principles of monitoring and is ready for implementation into routine settings. DESIGN: Qualitative data collection using interviews and participation in a multi-path mapping process. PARTICIPANTS: Twenty-six purposively selected informants completed semi-structured interviews and 24 individuals with expertise in the relevant clinical and informatics domains from academia, industry, policy and provider organizations and participated in a multi-path mapping workshop. RESULTS: The evidence base for the effectiveness of telemonitoring is weak and inconsistent, with insufficient cost-effectiveness studies. When considered against an accepted definition of monitoring, telemonitoring is found wanting. Telemonitoring has not been able so far to ensure that the technologies fit into the life world of the patient and into the clinical and organizational milieu of health service delivery systems. CONCLUSIONS: To develop effective telemonitoring for patients with chronic disease, more attention needs to be given to agreeing the central aim of early detection and, to ensure potential implementation, engaging a wide range of stakeholders in the design process, especially patients and clinicians.

Designing and implementing telemonitoring for early detection of deterioration in chronic disease: defining the requirements.

Patients with chronic disease may suffer frequent acute deteriorations and associated increased risk of hospitalisation. Earlier detection of these could enable successful intervention, improving patients' well-being and reducing costs; however, current telemonitoring systems do not achieve this effectively. We conducted a qualitative study using stakeholder interviews to define current standards of care and user requirements for improved early detection telemonitoring. We determined that early detection is not a concept that has informed technology or service design and that telemonitoring is driven by the available technology rather than by users' needs. We have described a set of requirements questions to inform the design and implementation of telemonitoring systems and suggested the research needed to develop successful early detection telemonitoring. User-centred design and genuine interdisciplinary approaches are needed to create solutions that are fit for purpose, sustainable and address the real needs of patients, clinicians and healthcare organisations.

Bridging two translation gaps: a new informatics research agenda for telemonitoring of chronic disease.

OBJECTIVE: To propose a research agenda that addresses technological and other knowledge gaps in developing telemonitoring solutions for patients with chronic diseases, with particular focus on detecting deterioration early enough to intervene effectively. DESIGN: A mixed methods approach incorporating literature review, key informant, and focus group interviews to gain an in-depth, multidisciplinary understanding of current approaches, and a roadmapping process to synthesise a research agenda. RESULTS: Counter to intuition, the research agenda for early detection of deterioration in patients with chronic diseases is not only primarily about advances in sensor technology but also much more about the problems of clinical specification, translation, and interfacing. The ultimate aim of telemonitoring is not fully agreed between the actors (patients, clinicians, technologists, and service providers). This leads to unresolved issues such as: (1) How are sensors used by patients as part of daily routines? (2) What are the indicators of early deterioration and how might they be used to trigger alerts? (3) How should alerts lead to appropriate levels of responses across different agencies and sectors? CONCLUSION: Attempts to use telemonitoring to improve the care of patients with chronic diseases over the last two decades have so far failed to lead to systems that are embedded in routine clinical practice. Attempts at implementation have paid insufficient attention to understanding patient and clinical needs and the complex dynamics and accountabilities that arise at the level of service models. A suggested way ahead is to co-design technology and services collaboratively with all stakeholders.

Information quality in proteomics.

Proteomics, the study of the protein complement of a biological system, is generating increasing quantities of data from rapidly developing technologies employed in a variety of different experimental workflows. Experimental processes, e.g. for comparative 2D gel studies or LC-MS/MS analyses of complex protein mixtures, involve a number of steps: from experimental design, through wet and dry lab operations, to publication of data in repositories and finally to data annotation and maintenance. The presence of inaccuracies throughout the processing pipeline, however, results in data that can be untrustworthy, thus offsetting the benefits of high-throughput technology. While researchers and practitioners are generally aware of some of the information quality issues associated with public proteomics data, there are few accepted criteria and guidelines for dealing with them. In this article, we highlight factors that impact on the quality of experimental data and review current approaches to information quality management in proteomics. Data quality issues are considered throughout the lifecycle of a proteomics experiment, from experiment design and technique selection, through data analysis, to archiving and sharing.

Universal metrics for quality assessment of protein identifications by mass spectrometry.

Increasing numbers of large proteomic datasets are becoming available. As attempts are made to interpret these datasets and integrate them with other forms of genomic data, researchers are becoming more aware of the importance of data quality with respect to protein identification. We present three simple and universal metrics that describe different aspects of the quality of protein identifications by peptide mass fingerprinting. Hit ratio gives an indication of the signal-to-noise ratio in a mass spectrum, mass coverage measures the amount of protein sequence matched, and excess of limit-digested peptides reflects the completeness of the digestion that precedes the peptide mass fingerprinting. Receiver-operating characteristic plots show that the novel metric, excess of limit-digested peptides, can discriminate between correct and random matches more accurately than search score when validating the results from a state-of-the-art protein identification software system (Mascot) especially when combined with the two other metrics, hit ratio and mass coverage. Recommendations are made regarding the use of the metrics when reporting protein identification experiments.