Multicenter, multidisciplinary user-centered design of a clinical decision-support and simulation system for massive transfusion.

BACKGROUND: Managing critical bleeding with massive transfusion (MT) requires a multidisciplinary team, often physically separated, to perform several simultaneous tasks at short notice. This places a significant cognitive load on team members, who must maintain situational awareness in rapidly changing scenarios. Similar resuscitation scenarios have benefited from the use of clinical decision support (CDS) tools. STUDY DESIGN AND METHODS: A multicenter, multidisciplinary, user-centered design (UCD) study was conducted to design a computerized CDS for MT. This study included analysis of the problem context with a cognitive walkthrough, development of a user requirement statement, and co-design with users of prototypes for testing. The final prototype was evaluated using qualitative assessment and the System Usability Scale (SUS). RESULTS: Eighteen participants were recruited across four institutions. The first UCD cycle resulted in the development of four prototype interfaces that addressed the user requirements and context of implementation. Of these, the preferred interface was further developed in the second UCD cycle to create a high-fidelity web-based CDS for MT. This prototype was evaluated by 15 participants using a simulated bleeding scenario and demonstrated an average SUS of 69.3 (above average, SD 16) and a clear interface with easy-to-follow blood product tracking. DISCUSSION: We used a UCD process to explore a highly complex clinical scenario and develop a prototype CDS for MT that incorporates distributive situational awareness, supports multiple user roles, and allows simulated MT training. Evaluation of the impact of this prototype on the efficacy and efficiency of managing MT is currently underway.

CareTrack: assessing the appropriateness of health care delivery in Australia.

OBJECTIVE: To determine the percentage of health care encounters at which a sample of adult Australians received appropriate care (ie, care in line with evidence-based or consensus-based guidelines). DESIGN, SETTING AND PARTICIPANTS: Computer-assisted telephone interviews and retrospective review of the medical records (for 2009-2010) of a sample of at least 1000 Australian adults to measure compliance with 522 expert consensus indicators representing appropriate care for 22 common conditions. Participants were selected from households in areas of South Australia and New South Wales chosen to be representative of the socioeconomic profile of Australians. Health care encounters occurred in health care practices and hospitals with general practitioners, specialists, physiotherapists, chiropractors, psychologists and counsellors. MAIN OUTCOME MEASURE: Percentage of health care encounters at which the sample received appropriate care. RESULTS: From 15 292 households contacted by telephone, 7649 individuals agreed to participate, 3567 consented, 2638 proved eligible, and 1154 were included after gaining the consent of their health care providers. The adult Australians in this sample received appropriate care at 57% (95% CI, 54%-60%) of 35 573 eligible health care encounters. Compliance with indicators of appropriate care at condition level ranged from 13% (95% CI, 1%-43%) for alcohol dependence to 90% (95% CI, 85%-93%) for coronary artery disease. For health care providers with more than 300 eligible encounters each, overall compliance ranged from 32% to 86%. CONCLUSIONS: Although there were pockets of excellence and some aspects of care were well managed across health care providers, the consistent delivery of appropriate care needs improvement, and gaps in care should be addressed. There is a need for national agreement on clinical standards and better structuring of medical records to facilitate the delivery of more appropriate care.

Massive transfusion experience, current practice and decision support: A survey of Australian and New Zealand anaesthetists.

Massive transfusions guided by massive transfusion protocols are commonly used to manage critical bleeding, when the patient is at significant risk of morbidity and mortality, and multiple timely decisions must be made by clinicians. Clinical decision support systems are increasingly used to provide patient-specific recommendations by comparing patient information to a knowledge base, and have been shown to improve patient outcomes. To investigate current massive transfusion practice and the experiences and attitudes of anaesthetists towards massive transfusion and clinical decision support systems, we anonymously surveyed 1000 anaesthetists and anaesthesia trainees across Australia and New Zealand. A total of 228 surveys (23.6%) were successfully completed and 227 were analysed for a 23.3% response rate. Most respondents were involved in massive transfusions infrequently (88.1% managed five or fewer massive transfusion protocols per year) and worked at hospitals which have massive transfusion protocols (89.4%). Massive transfusion management was predominantly limited by timely access to point-of-care coagulation assessment and by competition with other tasks, with trainees reporting more significant limitations compared to specialists. The majority of respondents reported that they were likely, or very likely, both to use (73.1%) and to trust (85%) a clinical decision support system for massive transfusions, with no significant difference between anaesthesia trainees and specialists (P = 0.375 and P = 0.73, respectively). While the response rate to our survey was poor, there was still a wide range of massive transfusion experience among respondents, with multiple subjective factors identified limiting massive transfusion practice. We identified several potential design features and barriers to implementation to assist with the future development of a clinical decision support system for massive transfusion, and overall wide support for a clinical decision support system for massive transfusion among respondents.

Can cognitive biases during consumer health information searches be reduced to improve decision making?

OBJECTIVE: To test whether the anchoring and order cognitive biases experienced during search by consumers using information retrieval systems can be corrected to improve the accuracy of, and confidence in, answers to health-related questions. DESIGN: A prospective study was conducted on 227 undergraduate students who used an online search engine developed by the authors to find health information and then answer six randomly assigned consumer health questions. The search engine was fitted with a baseline user interface and two modified interfaces specifically designed to debias anchoring or order effect. Each subject used all three user interfaces, answering two questions with each. MEASUREMENTS: Frequencies of correct answers pre- and post- search and confidence in answers were collected. Time taken to search and then answer a question, the number of searches conducted and the number of links accessed in a search session were also recorded. User preferences for each interface were measured. Chi-square analyses tested for the presence of biases with each user interface. The Kolmogorov-Smirnov test checked for equality of distribution of the evidence analyzed for each user interface. The test for difference between proportions and the Wilcoxon signed ranks test were used when comparing interfaces. RESULTS: Anchoring and order effects were present amongst subjects using the baseline search interface (anchoring: p < 0.001; order: p = 0.026). With use of the order debiasing interface, the initial order effect was no longer present (p = 0.34) but there was no significant improvement in decision accuracy (p = 0.23). While the anchoring effect persisted when using the anchor debiasing interface (p < 0.001), its use was associated with a 10.3% increase in subjects who had answered incorrectly pre-search, answering correctly post-search (p = 0.10). Subjects using either debiasing user interface conducted fewer searches and accessed more documents compared to baseline (p < 0.001). In addition, the majority of subjects preferred using a debiasing interface over baseline. CONCLUSION: This study provides evidence that (i) debiasing strategies can be integrated into the user interface of a search engine; (ii) information interpretation behaviors can be to some extent debiased; and that (iii) attempts to debias information searching by consumers can influence their ability to answer health-related questions accurately, their confidence in these answers, as well as the strategies used to conduct searches and retrieve information.

Is relevance relevant? User relevance ratings may not predict the impact of Internet search on decision outcomes.

OBJECTIVE: A common measure of Internet search engine effectiveness is its ability to find documents that a user perceives as 'relevant'. This study sought to test whether user provided relevance ratings for documents retrieved by an Internet search engine correlate with the decision outcome after use of a search engine. DESIGN: 227 university students were asked to answer four randomly assigned consumer health questions, then to conduct an Internet search on one of two randomly assigned search engines of different performance, and to again answer the question. MEASUREMENTS: Participants were asked to provide a relevance score for each document retrieved as well as a pre and post search answer to each question. RESULTS: User relevance rankings had little or no predictive power. Relevance rankings were unable to predict whether the user of a search engine could correctly answer a question after search and could not differentiate between two search engines with statistically different performance in the hands of users. Only when users had strong prior knowledge of the questions, and the decision task was of low complexity, did relevance appear to have modest predictive power. CONCLUSIONS: User provided relevance rankings taken in isolation seem to be of limited to no value when designing a search engine that will be used in a general-purpose setting. Relevance rankings may have a place in situations in which experts provide rankings, and decision tasks are of complexity commensurate with the abilities of the raters. A more natural metric of search engine performance may be a user's ability to accurately complete a task, as this removes the inherent subjectivity of relevance rankings, and provides a direct and repeatable outcome measure which directly correlates with the performance of the search technology in the hands of users.

Impact of web searching and social feedback on consumer decision making: a prospective online experiment.

BACKGROUND: The World Wide Web has increasingly become an important source of information in health care consumer decision making. However, little is known about whether searching online resources actually improves consumers' understanding of health issues. OBJECTIVES: The aim was to study whether searching on the World Wide Web improves consumers' accuracy in answering health questions and whether consumers' understanding of health issues is subject to further change under social feedback. METHODS: This was a pre/post prospective online study. A convenience sample of 227 undergraduate students was recruited from the population of the University of New South Wales. Subjects used a search engine that retrieved online documents from PubMed, MedlinePlus, and HealthInsite and answered a set of six questions (before and after use of the search engine) designed for health care consumers. They were then presented with feedback consisting of a summary of the post-search answers provided by previous subjects for the same questions and were asked to answer the questions again. RESULTS: There was an improvement in the percentage of correct answers after searching (pre-search 61.2% vs post-search 82.0%, P <.001) and after feedback with other subjects' answers (pre-feedback 82.0% vs post-feedback 85.3%, P =.051). The proportion of subjects with highly confident correct answers (ie, confident or very confident) and the proportion with highly confident incorrect answers significantly increased after searching (correct pre-search 61.6% vs correct post-search 95.5%, P <.001; incorrect pre-search 55.3% vs incorrect post-search 82.0%, P <.001). Subjects who were not as confident in their post-search answers were 28.5% more likely than those who were confident or very confident to change their answer after feedback with other subjects' post-search answers (chi(2) (1)= 66.65, P <.001). CONCLUSIONS: Searching across quality health information sources on the Web can improve consumers' accuracy in answering health questions. However, a consumer's confidence in an answer is not a good indicator of the answer being correct. Consumers who are not confident in their answers after searching are more likely to be influenced to change their views when provided with feedback from other consumers.

Do people experience cognitive biases while searching for information?

OBJECTIVE: To test whether individuals experience cognitive biases whilst searching using information retrieval systems. Biases investigated are anchoring, order, exposure and reinforcement. DESIGN: A retrospective analysis and a prospective experiment were conducted to investigate whether cognitive biases affect the way that documentary evidence is interpreted while searching online. The retrospective analysis was conducted on the search and decision behaviors of 75 clinicians (44 doctors, 31 nurses), answering questions for 8 clinical scenarios within 80 minutes in a controlled setting. The prospective study was conducted on 227 undergraduate students, who used the same search engine to answer two of six randomly assigned consumer health questions. MEASUREMENTS: Frequencies of correct answers pre- and post- search, and confidence in answers were collected. The impact of reading a document on the final decision was measured by the population likelihood ratio (LR) of the frequency of reading the document and the frequency of obtaining a correct answer. Documents with a LR > 1 were most likely to be associated with a correct answer, and those with a LR < 1 were most likely to be associated with an incorrect answer to a question. Agreement between a subject and the evidence they read was estimated by a concurrence rate, which measured the frequency that subjects' answers agreed with the likelihood ratios of a group of documents, normalized for document order, time exposure or reinforcement through repeated access. Serial position curves were plotted for the relationship between subjects' pre-search confidence, document order, the number of times and length of time a document was accessed, and concurrence with post-search answers. Chi-square analyses tested for the presence of biases, and the Kolmogorov-Smirnov test checked for equality of distribution of evidence in the comparison populations. RESULTS: A person's prior belief (anchoring) has a significant impact on their post-search answer (retrospective: P < 0.001; prospective: P < 0.001). Documents accessed at different positions in a search session (order effect [retrospective: P = 0.76; prospective: P = 0.026]), and documents processed for different lengths of time (exposure effect [retrospective: P = 0.27; prospective: P = 0.0081]) also influenced decision post-search more than expected in the prospective experiment but not in the retrospective analysis. Reinforcement through repeated exposure to a document did not yield statistical differences in decision outcome post-search (retrospective: P = 0.31; prospective: P = 0.81). CONCLUSION: People may experience anchoring, exposure and order biases while searching for information, and these biases may influence the quality of decision making during and after the use of information retrieval systems.

What factors are associated with the integration of evidence retrieval technology into routine general practice settings?

BACKGROUND: Information retrieval systems have the potential to improve patient care but little is known about the variables which influence clinicians' uptake and use of systems in routine work. AIM: To determine which factors influenced use of an online evidence retrieval system. DESIGN OF STUDY: Computer logs and pre- and post-system survey analysis of a 4-week clinical trial of the Quick Clinical online evidence system involving 227 general practitioners across Australia. RESULTS: Online evidence use was not linked to general practice training or clinical experience but female clinicians conducted more searches than their male counterparts (mean use=14.38 searches, S.D.=11.68 versus mean use=8.50 searches, S.D.=9.99; t=2.67, d.f.=157, P=0.008). Practice characteristics such as hours worked, type and geographic location of clinic were not associated with search activity. Information seeking was also not related to participants' perceived information needs, computer skills, training nor Internet connection speed. Clinicians who reported direct improvements in patient care as a result of system use had significantly higher rates of system use than other users (mean use=12.55 searches, S.D.=13.18 versus mean use=8.15 searches, S.D.=9.18; t=2.322, d.f.=154 P=0.022). Comparison of participants' views pre- and post- the trial, showed that post-trial clinicians expressed more positive views about searching for information during a consultation (chi(2)=27.40, d.f.=4, P< or =0.001) and a significantly greater number reported seeking information between consultations as a result of having access to an online evidence system in their consulting rooms (chi(2)=9.818, d.f.=2, P=0.010). CONCLUSION: Clinicians' use of an online evidence system was directly related to their reported experiences of improvements in patient care. Post-trial clinicians positively changed their views about having time to search for information and pursued more questions during clinic hours.

Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes.

BACKGROUND: Online evidence retrieval systems are a potential tool in supporting evidence-based practice. Effective and tested techniques for assessing the impact of these systems on care delivery and patient outcomes are limited. PURPOSE AND METHODS: In this study we applied the critical incident (CI) and journey mapping (JM) techniques to assess the integration of an online evidence system into everyday clinical practice and its impact on decision making and patient care. To elicit incidents semi-structured interviews were conducted with 29 clinicians (13 hospital physician specialists, 16 clinical nurse consultants (CNCs)) who were experienced users of the online evidence system. Clinicians were also asked questions about how they had first used the system and how their use and experiences had changed over time. These narrative accounts were then mapped and scored using the journey mapping technique. RESULTS: Clinicians generated 85 critical incidents. Three categories of impact were identified: impact on clinical practice, impact on individual clinicians and impact on colleagues through the dissemination of information gained from the online evidence system. One quarter of these included specific examples of system use leading to improvements in patient care. Clinicians obtained an average journey mapping score of 22 out of a possible score of 36, demonstrating a good level of system integration. Average scores of doctors and CNCs were similar. However individuals with the same scores often had very different journeys in system integration. CONCLUSIONS: The CI technique provided clear examples of the way in which system use had influenced practice and care delivery. The JM technique was found to be a useful method for providing a quantification of the different ways and extent to which, clinicians had integrated system use into practice, and insights into how system use can influence organisational culture. The development of the journey mapping stages provides a structure by which the program logic of a clinical information system and its desired outcomes can be made explicit and be based upon users' experiences in everyday practice. Further work is required using this technique to assess its value as an evaluation method.

Do online information retrieval systems help experienced clinicians answer clinical questions?

OBJECTIVE: To assess the impact of clinicians' use of an online information retrieval system on their performance in answering clinical questions. DESIGN: Pre-/post-intervention experimental design. MEASUREMENTS: In a computer laboratory, 75 clinicians (26 hospital-based doctors, 18 family practitioners, and 31 clinical nurse consultants) provided 600 answers to eight clinical scenarios before and after the use of an online information retrieval system. We examined the proportion of correct answers pre- and post-intervention, direction of change in answers, and differences between professional groups. RESULTS: System use resulted in a 21% improvement in clinicians' answers, from 29% (95% confidence interval [CI] 25.4-32.6) correct pre- to 50% (95% CI 46.0-54.0) post-system use. In 33% (95% CI 29.1-36.9) answers were changed from incorrect to correct. In 21% (95% CI 17.1-23.9) correct pre-test answers were supported by evidence found using the system, and in 7% (95% CI 4.9-9.1) correct pre-test answers were changed incorrectly. For 40% (35.4-43.6) of scenarios, incorrect pre-test answers were not rectified following system use. Despite significant differences in professional groups' pre-test scores [family practitioners: 41% (95% CI 33.0-49.0), hospital doctors: 35% (95% CI 28.5-41.2), and clinical nurse consultants: 17% (95% CI 12.3-21.7; chi(2) = 29.0, df = 2, p < 0.01)], there was no difference in post-test scores. (chi(2) = 2.6, df = 2, p = 0.73). CONCLUSIONS: The use of an online information retrieval system was associated with a significant improvement in the quality of answers provided by clinicians to typical clinical problems. In a small proportion of cases, use of the system produced errors. While there was variation in the performance of clinical groups when answering questions unaided, performance did not differ significantly following system use. Online information retrieval systems can be an effective tool in improving the accuracy of clinicians' answers to clinical questions.

The impact of an online evidence system on confidence in decision making in a controlled setting.

OBJECTIVE: To examine the impact of online evidence retrieval on clinicians' decision-making confidence and to determine if this differs for experienced doctors and nurses. METHODS: A sample of 44 doctors and 31 clinical nurse consultants (CNCs) answered 8 clinical scenarios (600 scenario answers) before and after the use of online evidence resources. Clinicians rated their confidence in scenario answers and in the evidence they found using the information system. RESULTS: Prior to using online evidence, 37% of doctors and 18% of CNCs answered the scenarios correctly. These clinicians were more confident (56% very confident or confident) in their answers than those with incorrect (34%) answers. Doctors with incorrect answers prior to searching rated their confidence significantly higher than did nurses who were incorrect. After searching, both groups answered 50% of scenarios correctly. Clinicians with correct answers had greater confidence in the evidence found compared to those with incorrect answers. Doctors were more confident in evidence found confirming an initially correct answer than were nurses. More than 50% of clinicians who persisted with an incorrect answer after searching reported that they were confident or very confident in the evidence found. Clinicians who did not know scenario answers before searching placed equal confidence in evidence that led them to a correct or incorrect answer. CONCLUSIONS: The information obtained from an online evidence system influenced clinicians' confidence in their answers to the clinical scenarios. The relationship between confidence in answers and correctness is complex. Both existing knowledge and professional role were mediating factors. The finding that many clinicians placed confidence in information that led them to incorrect answers warrants further investigation.

General practitioners' use of online evidence during consultations.

BACKGROUND: Clinicians have many unanswered questions during clinical encounters which may impact on the quality and outcomes of decisions made. Provision of online evidence at the point of care is one strategy that provides clinicians with easy access to up-to-date evidence in clinical settings to support evidence-based decision-making. AIM: To determine if and when general practitioners use an online evidence system in routine clinical practice, the type of questions for which clinicians seek evidence and the extent to which the system provides clinically useful answers. DESIGN OF STUDY: A prospective cohort study which involved a 4-week clinical trial of Quick Clinical, an online evidence system specifically designed around the needs of general practitioners. SETTING: Two hundred and twenty-seven clinicians who had a computer with Internet access in their consulting rooms. METHODS: Computer logs and survey analysis. RESULTS: One hundred and ninety-three general practitioners used the online evidence system to conduct on average 8.7 searches/month. The majority of these (81%) were conducted from consulting rooms and carried out between 9a.m. and 7p.m. (83%). The most frequent searches conducted related to diagnosis (40%) and treatment (35%). 83% of clinicians believed that Quick Clinical (QC) had the potential to improve patient care, and one in four users reported direct experience of improvements in care. In 73% of queries with clinician feedback participants reported that they were able to find clinically useful information during their routine work. CONCLUSION: General practitioners will use an online evidence retrieval system in routine practice, and report that its use improves the quality of patient care.

Which clinical decisions benefit from automation? A task complexity approach.

OBJECTIVE: To describe a model for analysing complex medical decision making tasks and for evaluating their suitability for automation. METHOD: Assessment of a decision task's complexity in terms of the number of elementary information processes (EIPs) and the potential for cognitive effort reduction through EIP minimisation using an automated decision aid. RESULTS: The model consists of five steps: (1) selection of the domain and relevant tasks; (2) evaluation of the knowledge complexity for tasks selected; (3) identification of cognitively demanding tasks; (4) assessment of unaided and aided effort requirements for this task accomplishment; and (5) selection of computational tools to achieve this complexity reduction. The model is applied to the task of antibiotic prescribing in critical care and the most complex components of the task identified. Decision aids to support these components can provide a significant reduction of cognitive effort suggesting this is a decision task worth automating. CONCLUSION: We view the role of decision support for complex decision to be one of task complexity reduction, and the model described allows for task automation without lowering decision quality and can assist decision support systems developers.

Variation in the use of online clinical evidence: a qualitative analysis.

OBJECTIVE: To investigate factors influencing variations in clinicians' use of an online evidence retrieval system. SETTING: Public hospitals in New South Wales, Australia. METHOD: Web log analysis demonstrated considerable variation in rates of evidence use by clinicians at different hospitals. Focus groups and interviews were held with 61 staff from three hospitals, two with high rates of use and one with a low rate of use, to explore variation in evidence use. RESULTS: Differences between hospitals' and professional groups' (doctors, nurses and allied health) use of online evidence could be explained by organizational, professional and cultural factors. These included the presence of champions, organizational cultures which supported evidence-based practice (EBP), and database searching skills of individual clinicians. Staff shortages, ease of access and time taken to use the online evidence system were cited as barriers to use at the low use site, but no objective differences in these measures were found between the high and low use sites. CONCLUSION: Social and cultural factors were found to be better discriminators of high and low evidence use than technical factors.