Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion.

BACKGROUND: Managing critical bleeding with massive transfusion (MT) requires a multidisciplinary team, often physically separated, to perform several simultaneous tasks at short notice. This places a significant cognitive load on team members, who must maintain situational awareness in rapidly changing scenarios. Similar resuscitation scenarios have benefited from the use of clinical decision support (CDS) tools. STUDY DESIGN AND METHODS: A multicenter, multidisciplinary, user-centered design (UCD) study was conducted to design a computerized CDS for MT. This study included analysis of the problem context with a cognitive walkthrough, development of a user requirement statement, and co-design with users of prototypes for testing. The final prototype was evaluated using qualitative assessment and the System Usability Scale (SUS). RESULTS: Eighteen participants were recruited across four institutions. The first UCD cycle resulted in the development of four prototype interfaces that addressed the user requirements and context of implementation. Of these, the preferred interface was further developed in the second UCD cycle to create a high-fidelity web-based CDS for MT. This prototype was evaluated by 15 participants using a simulated bleeding scenario and demonstrated an average SUS of 69.3 (above average, SD 16) and a clear interface with easy-to-follow blood product tracking. DISCUSSION: We used a UCD process to explore a highly complex clinical scenario and develop a prototype CDS for MT that incorporates distributive situational awareness, supports multiple user roles, and allows simulated MT training. Evaluation of the impact of this prototype on the efficacy and efficiency of managing MT is currently underway.

Massive transfusion experience, current practice and decision support: A survey of Australian and New Zealand anaesthetists.

Massive transfusions guided by massive transfusion protocols are commonly used to manage critical bleeding, when the patient is at significant risk of morbidity and mortality, and multiple timely decisions must be made by clinicians. Clinical decision support systems are increasingly used to provide patient-specific recommendations by comparing patient information to a knowledge base, and have been shown to improve patient outcomes. To investigate current massive transfusion practice and the experiences and attitudes of anaesthetists towards massive transfusion and clinical decision support systems, we anonymously surveyed 1000 anaesthetists and anaesthesia trainees across Australia and New Zealand. A total of 228 surveys (23.6%) were successfully completed and 227 were analysed for a 23.3% response rate. Most respondents were involved in massive transfusions infrequently (88.1% managed five or fewer massive transfusion protocols per year) and worked at hospitals which have massive transfusion protocols (89.4%). Massive transfusion management was predominantly limited by timely access to point-of-care coagulation assessment and by competition with other tasks, with trainees reporting more significant limitations compared to specialists. The majority of respondents reported that they were likely, or very likely, both to use (73.1%) and to trust (85%) a clinical decision support system for massive transfusions, with no significant difference between anaesthesia trainees and specialists (P = 0.375 and P = 0.73, respectively). While the response rate to our survey was poor, there was still a wide range of massive transfusion experience among respondents, with multiple subjective factors identified limiting massive transfusion practice. We identified several potential design features and barriers to implementation to assist with the future development of a clinical decision support system for massive transfusion, and overall wide support for a clinical decision support system for massive transfusion among respondents.

CareTrack: assessing the appropriateness of health care delivery in Australia.

OBJECTIVE: To determine the percentage of health care encounters at which a sample of adult Australians received appropriate care (ie, care in line with evidence-based or consensus-based guidelines). DESIGN, SETTING AND PARTICIPANTS: Computer-assisted telephone interviews and retrospective review of the medical records (for 2009-2010) of a sample of at least 1000 Australian adults to measure compliance with 522 expert consensus indicators representing appropriate care for 22 common conditions. Participants were selected from households in areas of South Australia and New South Wales chosen to be representative of the socioeconomic profile of Australians. Health care encounters occurred in health care practices and hospitals with general practitioners, specialists, physiotherapists, chiropractors, psychologists and counsellors. MAIN OUTCOME MEASURE: Percentage of health care encounters at which the sample received appropriate care. RESULTS: From 15 292 households contacted by telephone, 7649 individuals agreed to participate, 3567 consented, 2638 proved eligible, and 1154 were included after gaining the consent of their health care providers. The adult Australians in this sample received appropriate care at 57% (95% CI, 54%-60%) of 35 573 eligible health care encounters. Compliance with indicators of appropriate care at condition level ranged from 13% (95% CI, 1%-43%) for alcohol dependence to 90% (95% CI, 85%-93%) for coronary artery disease. For health care providers with more than 300 eligible encounters each, overall compliance ranged from 32% to 86%. CONCLUSIONS: Although there were pockets of excellence and some aspects of care were well managed across health care providers, the consistent delivery of appropriate care needs improvement, and gaps in care should be addressed. There is a need for national agreement on clinical standards and better structuring of medical records to facilitate the delivery of more appropriate care.

A PubMed-wide associational study of infectious diseases.

BACKGROUND: Computational discovery is playing an ever-greater role in supporting the processes of knowledge synthesis. A significant proportion of the more than 18 million manuscripts indexed in the PubMed database describe infectious disease syndromes and various infectious agents. This study is the first attempt to integrate online repositories of text-based publications and microbial genome databases in order to explore the dynamics of relationships between pathogens and infectious diseases. METHODOLOGY/PRINCIPAL FINDINGS: Herein we demonstrate how the knowledge space of infectious diseases can be computationally represented and quantified, and tracked over time. The knowledge space is explored by mapping of the infectious disease literature, looking at dynamics of literature deposition, zooming in from pathogen to genome level and searching for new associations. Syndromic signatures for different pathogens can be created to enable a new and clinically focussed reclassification of the microbial world. Examples of syndrome and pathogen networks illustrate how multilevel network representations of the relationships between infectious syndromes, pathogens and pathogen genomes can illuminate unexpected biological similarities in disease pathogenesis and epidemiology. CONCLUSIONS/SIGNIFICANCE: This new approach based on text and data mining can support the discovery of previously hidden associations between diseases and microbial pathogens, clinically relevant reclassification of pathogenic microorganisms and accelerate the translational research enterprise.

Computational reasoning across multiple models.

Computational support of clinical decisions frequently requires the integration of data in a variety of formats and from multiple sources and domains. Some impressive multiscale computational models of biological phenomena have been developed as part of the study of disease and healthcare systems. One can now contemplate harnessing these models arising from computational biology and using highly interconnected clinical data to support clinical decision-making. Indeed, understanding how to build computational systems able to reason across heterogeneous models and datasets is one of the major and perhaps foundational challenges of translational biomedical informatics. In this paper, the authors examine the use of multimodels (models composed of several daughter models) and explore three major research challenges to reasoning across multiple models: model selection, model composition, and computer aided model construction.

Can cognitive biases during consumer health information searches be reduced to improve decision making?

OBJECTIVE: To test whether the anchoring and order cognitive biases experienced during search by consumers using information retrieval systems can be corrected to improve the accuracy of, and confidence in, answers to health-related questions. DESIGN: A prospective study was conducted on 227 undergraduate students who used an online search engine developed by the authors to find health information and then answer six randomly assigned consumer health questions. The search engine was fitted with a baseline user interface and two modified interfaces specifically designed to debias anchoring or order effect. Each subject used all three user interfaces, answering two questions with each. MEASUREMENTS: Frequencies of correct answers pre- and post- search and confidence in answers were collected. Time taken to search and then answer a question, the number of searches conducted and the number of links accessed in a search session were also recorded. User preferences for each interface were measured. Chi-square analyses tested for the presence of biases with each user interface. The Kolmogorov-Smirnov test checked for equality of distribution of the evidence analyzed for each user interface. The test for difference between proportions and the Wilcoxon signed ranks test were used when comparing interfaces. RESULTS: Anchoring and order effects were present amongst subjects using the baseline search interface (anchoring: p < 0.001; order: p = 0.026). With use of the order debiasing interface, the initial order effect was no longer present (p = 0.34) but there was no significant improvement in decision accuracy (p = 0.23). While the anchoring effect persisted when using the anchor debiasing interface (p < 0.001), its use was associated with a 10.3% increase in subjects who had answered incorrectly pre-search, answering correctly post-search (p = 0.10). Subjects using either debiasing user interface conducted fewer searches and accessed more documents compared to baseline (p < 0.001). In addition, the majority of subjects preferred using a debiasing interface over baseline. CONCLUSION: This study provides evidence that (i) debiasing strategies can be integrated into the user interface of a search engine; (ii) information interpretation behaviors can be to some extent debiased; and that (iii) attempts to debias information searching by consumers can influence their ability to answer health-related questions accurately, their confidence in these answers, as well as the strategies used to conduct searches and retrieve information.

Impact of an ultrabroadband emergency department telemedicine system on the care of acutely ill patients and clinicians' work.

OBJECTIVES: To evaluate whether introduction of an emergency department (ED) telemedicine system changed patient management and outcome indicators and to investigate clinicians' perceptions of the impact of the system on care provided and on their work. DESIGN: Before-and-after study of use of the Virtual Critical Care Unit (ViCCU), which uses an ultrabroadband connection allowing real-time audiovisual communication between clinicians at distant sites. Semi-structured interviews were conducted with medical and nursing staff at the end of the study. PARTICIPANTS AND SETTING: The ViCCU intervention commenced on 1 January 2004. Our study was conducted in the EDs of an 85-bed district hospital and a 420-bed metropolitan tertiary hospital. It involved all acutely ill patients requiring urgent care (defined by triage category and grouped into critical care, major trauma and moderate trauma) who were treated during the 12 months before (n=169) and 18 months after (n=181) the intervention at the district hospital. Thirty-one of 33 clinicians (doctors and nurses) participating at the two hospitals took part in interviews at the end of the study. MAIN OUTCOME MEASURES: Changes in patterns of management (disposition [admission, discharge or transfer], treatment times, number of procedures) and outcomes (rapid acute physiology scores, hours on ventilation or in intensive care, length of stay). RESULTS: Patient disposition remained unaltered for major trauma patients. For critical care patients, admissions fell significantly (54% to 30%), transfers increased (21% to 39%), and more procedures were performed. For moderate trauma patients, discharges increased significantly (45% to 63%), transfers decreased (48% to 25%) and treatment times were longer. No significant changes were found in outcome indicators. Clinicians reported that the ViCCU allowed greater support to remote clinicians. Specialists reported increased workloads and feelings of greater responsibility for patients at the district hospital. Nurses at the district site reported reduced stress, but district doctors reported some loss of autonomy. CONCLUSIONS: The ViCCU appears most effective for moderate trauma patients, with associated reductions in admissions and transfers. Large-scale trials of telemedicine systems that include measurements of both patient care and impact on clinicians' work are required.

Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes.

OBJECTIVE: A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as 'relevant'. This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine. DESIGN: 227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question. MEASUREMENTS: Participants were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question. RESULTS: User relevance rankings had little or no predictive power. Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users. Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power. CONCLUSIONS: User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting. Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters. A more natural metric of search engine performance may be a user's ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.

Impact of web searching and social feedback on consumer decision making: a prospective online experiment.

BACKGROUND: The World Wide Web has increasingly become an important source of information in health care consumer decision making. However, little is known about whether searching online resources actually improves consumers' understanding of health issues. OBJECTIVES: The aim was to study whether searching on the World Wide Web improves consumers' accuracy in answering health questions and whether consumers' understanding of health issues is subject to further change under social feedback. METHODS: This was a pre/post prospective online study. A convenience sample of 227 undergraduate students was recruited from the population of the University of New South Wales. Subjects used a search engine that retrieved online documents from PubMed, MedlinePlus, and HealthInsite and answered a set of six questions (before and after use of the search engine) designed for health care consumers. They were then presented with feedback consisting of a summary of the post-search answers provided by previous subjects for the same questions and were asked to answer the questions again. RESULTS: There was an improvement in the percentage of correct answers after searching (pre-search 61.2% vs post-search 82.0%, P <.001) and after feedback with other subjects' answers (pre-feedback 82.0% vs post-feedback 85.3%, P =.051). The proportion of subjects with highly confident correct answers (ie, confident or very confident) and the proportion with highly confident incorrect answers significantly increased after searching (correct pre-search 61.6% vs correct post-search 95.5%, P <.001; incorrect pre-search 55.3% vs incorrect post-search 82.0%, P <.001). Subjects who were not as confident in their post-search answers were 28.5% more likely than those who were confident or very confident to change their answer after feedback with other subjects' post-search answers (chi(2) (1)= 66.65, P <.001). CONCLUSIONS: Searching across quality health information sources on the Web can improve consumers' accuracy in answering health questions. However, a consumer's confidence in an answer is not a good indicator of the answer being correct. Consumers who are not confident in their answers after searching are more likely to be influenced to change their views when provided with feedback from other consumers.

Do people experience cognitive biases while searching for information?

OBJECTIVE: To test whether individuals experience cognitive biases whilst searching using information retrieval systems. Biases investigated are anchoring, order, exposure and reinforcement. DESIGN: A retrospective analysis and a prospective experiment were conducted to investigate whether cognitive biases affect the way that documentary evidence is interpreted while searching online. The retrospective analysis was conducted on the search and decision behaviors of 75 clinicians (44 doctors, 31 nurses), answering questions for 8 clinical scenarios within 80 minutes in a controlled setting. The prospective study was conducted on 227 undergraduate students, who used the same search engine to answer two of six randomly assigned consumer health questions. MEASUREMENTS: Frequencies of correct answers pre- and post- search, and confidence in answers were collected. The impact of reading a document on the final decision was measured by the population likelihood ratio (LR) of the frequency of reading the document and the frequency of obtaining a correct answer. Documents with a LR > 1 were most likely to be associated with a correct answer, and those with a LR < 1 were most likely to be associated with an incorrect answer to a question. Agreement between a subject and the evidence they read was estimated by a concurrence rate, which measured the frequency that subjects' answers agreed with the likelihood ratios of a group of documents, normalized for document order, time exposure or reinforcement through repeated access. Serial position curves were plotted for the relationship between subjects' pre-search confidence, document order, the number of times and length of time a document was accessed, and concurrence with post-search answers. Chi-square analyses tested for the presence of biases, and the Kolmogorov-Smirnov test checked for equality of distribution of evidence in the comparison populations. RESULTS: A person's prior belief (anchoring) has a significant impact on their post-search answer (retrospective: P < 0.001; prospective: P < 0.001). Documents accessed at different positions in a search session (order effect [retrospective: P = 0.76; prospective: P = 0.026]), and documents processed for different lengths of time (exposure effect [retrospective: P = 0.27; prospective: P = 0.0081]) also influenced decision post-search more than expected in the prospective experiment but not in the retrospective analysis. Reinforcement through repeated exposure to a document did not yield statistical differences in decision outcome post-search (retrospective: P = 0.31; prospective: P = 0.81). CONCLUSION: People may experience anchoring, exposure and order biases while searching for information, and these biases may influence the quality of decision making during and after the use of information retrieval systems.

Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes.

BACKGROUND: Online evidence retrieval systems are a potential tool in supporting evidence-based practice. Effective and tested techniques for assessing the impact of these systems on care delivery and patient outcomes are limited. PURPOSE AND METHODS: In this study we applied the critical incident (CI) and journey mapping (JM) techniques to assess the integration of an online evidence system into everyday clinical practice and its impact on decision making and patient care. To elicit incidents semi-structured interviews were conducted with 29 clinicians (13 hospital physician specialists, 16 clinical nurse consultants (CNCs)) who were experienced users of the online evidence system. Clinicians were also asked questions about how they had first used the system and how their use and experiences had changed over time. These narrative accounts were then mapped and scored using the journey mapping technique. RESULTS: Clinicians generated 85 critical incidents. Three categories of impact were identified: impact on clinical practice, impact on individual clinicians and impact on colleagues through the dissemination of information gained from the online evidence system. One quarter of these included specific examples of system use leading to improvements in patient care. Clinicians obtained an average journey mapping score of 22 out of a possible score of 36, demonstrating a good level of system integration. Average scores of doctors and CNCs were similar. However individuals with the same scores often had very different journeys in system integration. CONCLUSIONS: The CI technique provided clear examples of the way in which system use had influenced practice and care delivery. The JM technique was found to be a useful method for providing a quantification of the different ways and extent to which, clinicians had integrated system use into practice, and insights into how system use can influence organisational culture. The development of the journey mapping stages provides a structure by which the program logic of a clinical information system and its desired outcomes can be made explicit and be based upon users' experiences in everyday practice. Further work is required using this technique to assess its value as an evaluation method.