Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study.

BACKGROUND: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE: The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS: Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide." RESULTS: The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle." CONCLUSIONS: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map.

BACKGROUND: Recruiting and retaining participants in randomised controlled trials (RCTs) is challenging. Digital tools, such as social media, data mining, email or text-messaging, could improve recruitment or retention, but an overview of this research area is lacking. We aimed to systematically map the characteristics of digital recruitment and retention tools for RCTs, and the features of the comparative studies that have evaluated the effectiveness of these tools during the past 10 years. METHODS: We searched Medline, Embase, other databases, the Internet, and relevant web sites in July 2018 to identify comparative studies of digital tools for recruiting and/or retaining participants in health RCTs. Two reviewers independently screened references against protocol-specified eligibility criteria. Included studies were coded by one reviewer with 20% checked by a second reviewer, using pre-defined keywords to describe characteristics of the studies, populations and digital tools evaluated. RESULTS: We identified 9163 potentially relevant references, of which 104 articles reporting 105 comparative studies were included in the systematic map. The number of published studies on digital tools has doubled in the past decade, but most studies evaluated digital tools for recruitment rather than retention. The key health areas investigated were health promotion, cancers, circulatory system diseases and mental health. Few studies focussed on minority or under-served populations, and most studies were observational. The most frequently-studied digital tools were social media, Internet sites, email and tv/radio for recruitment; and email and text-messaging for retention. One quarter of the studies measured efficiency (cost per recruited or retained participant) but few studies have evaluated people's attitudes towards the use of digital tools. CONCLUSIONS: This systematic map highlights a number of evidence gaps and may help stakeholders to identify and prioritise further research needs. In particular, there is a need for rigorous research on the efficiency of the digital tools and their impact on RCT participants and investigators, perhaps as studies-within-a-trial (SWAT) research. There is also a need for research into how digital tools may improve participant retention in RCTs which is currently underrepresented relative to recruitment research. REGISTRATION: Not registered; based on a pre-specified protocol, peer-reviewed by the project's Advisory Board.

Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK Clinical Trial Units and a qualitative study.

BACKGROUND: Recruitment and retention of participants in randomised controlled trials (RCTs) is a key determinant of success but is challenging. Trialists and UK Clinical Research Collaboration (UKCRC) Clinical Trials Units (CTUs) are increasingly exploring the use of digital tools to identify, recruit and retain participants. The aim of this UK National Institute for Health Research (NIHR) study was to identify what digital tools are currently used by CTUs and understand the performance characteristics required to be judged useful. METHODS: A scoping of searches (and a survey with NIHR funding staff), a survey with all 52 UKCRC CTUs and 16 qualitative interviews were conducted with five stakeholder groups including trialists within CTUs, funders and research participants. A purposive sampling approach was used to conduct the qualitative interviews during March-June 2018. Qualitative data were analysed using a content analysis and inductive approach. RESULTS: Responses from 24 (46%) CTUs identified that database-screening tools were the most widely used digital tool for recruitment, with the majority being considered effective. The reason (and to whom) these tools were considered effective was in identifying potential participants (for both Site staff and CTU staff) and reaching recruitment target (for CTU staff/CI). Fewer retention tools were used, with short message service (SMS) or email reminders to participants being the most reported. The qualitative interviews revealed five themes across all groups: 'security and transparency'; 'inclusivity and engagement'; 'human interaction'; 'obstacles and risks'; and 'potential benefits'. There was a high level of stakeholder acceptance of the use of digital tools to support trials, despite the lack of evidence to support them over more traditional techniques. Certain differences and similarities between stakeholder groups demonstrated the complexity and challenges of using digital tools for recruiting and retaining research participants. CONCLUSIONS: Our studies identified a range of digital tools in use in recruitment and retention of RCTs, despite the lack of high-quality evidence to support their use. Understanding the type of digital tools in use to support recruitment and retention will help to inform funders and the wider research community about their value and relevance for future RCTs. Consideration of further focused digital tool reviews and primary research will help to reduce gaps in the evidence base.

Influence of external peer reviewer scores for funding applications on funding board decisions: a retrospective analysis of 1561 reviews.

OBJECTIVES: To evaluate the influence of external peer reviewer scores on the National Institute for Health Research (NIHR) research funding board decisions by the number of reviewers and type of reviewer expertise. DESIGN: Retrospective analysis of external peer review scores for shortlisted full applications for funding (280 funding applications, 1236 individual reviewers, 1561 review scores). SETTING: Four applied health research funding programmes of NIHR, UK. MAIN OUTCOME MEASURES: Board decision to fund or not fund research applications. RESULTS: The mean score of reviewers predicted funding decisions better than individual reviewer scores (area under the receiver operating characteristic (ROC) curve 0.75, 95% CI 0.69 to 0.81 compared with 0.62, CI 0.59 to 0.65). There was no substantial improvement in how accurately mean reviewer scores predicted funding decisions when the number of reviewers increased above 4 (area under ROC curve 0.75, CI 0.59 to 0.91 for four reviewers; 0.80, CI 0.67 to 0.92 for seven or more). Reviewers with differing expertise influenced the board's decision equally, including public and patient reviewers (area under ROC curves from 0.57, CI 0.47 to 0.66 for health economists to 0.64, CI 0.57 to 0.70 for subject-matter experts). The areas under the ROC curves were quite low when using reviewers' scores, confirming that boards do not rely solely on those scores alone to make their funding decisions, which are best predicted by the mean board score. CONCLUSIONS: Boards value scores that originate from a diverse pool of reviewers. On the basis of independent reviewer score alone, there is no detectable benefit of using more than four reviewer scores in terms of their influence on board decisions, so to improve efficiency, it may be possible to avoid using larger numbers of reviewers. The funding decision is best predicted by the board score.

Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study.

OBJECTIVES: Innovations resulting from research have both national and global impact, so selecting the most promising research studies to fund is crucial. Peer review of research funding applications is part of the selection process, and requires considerable resources. This study aimed to elicit stakeholder opinions about which factors contribute to and influence effective peer review of funding applications to the UK National Institute for Health Research (NIHR), and to identify possible minor improvements to current processes and any major changes or potential innovations to achieve a more efficient peer review process. DESIGN: Qualitative interviews with 30 stakeholders involved in the peer review process. PARTICIPANTS: Participants were drawn from three NIHR coordinating centres and represented four types of stakeholders: board members with responsibility for making funding decisions, applicants, external peer reviewers and NIHR staff. METHODS: All interviews were conducted by telephone apart from three that were face to face with NIHR staff. Data were analysed using a thematic template method. RESULTS: The responses from NIHR staff, board members and reviewers differed from those received from applicants. The first three groups focused on how well the process of peer review did or did not function. The applicants mentioned these points but in addition often reflected on how their personal application was assessed. Process improvements suggested included: developing a more proportionate review process; providing greater guidance, feedback, training, acknowledgement or incentives for peer reviewers; reducing the time commitment and amount of paperwork; and asking reviewers to comment on the importance, strengths and weaknesses of applications and flaws which are potentially 'fixable'. CONCLUSIONS: Overall, participants were supportive of the need for peer review in evaluating applications for research funding. This study revealed which parts of the process are working well and are valued, and barriers, difficulties and potential areas for improvement and development.

The impact of three discharge coding methods on the accuracy of diagnostic coding and hospital reimbursement for inpatient medical care.

BACKGROUND: Coding of diagnoses is important for patient care, hospital management and research. However coding accuracy is often poor and may reflect methods of coding. This study investigates the impact of three alternative coding methods on the inaccuracy of diagnosis codes and hospital reimbursement. METHODS: Comparisons of coding inaccuracy were made between a list of coded diagnoses obtained by a coder using (i)the discharge summary alone, (ii)case notes and discharge summary, and (iii)discharge summary with the addition of medical input. For each method, inaccuracy was determined for the primary, secondary diagnoses, Healthcare Resource Group (HRG) and estimated hospital reimbursement. These data were then compared with a gold standard derived by a consultant and coder. RESULTS: 107 consecutive patient discharges were analysed. Inaccuracy of diagnosis codes was highest when a coder used the discharge summary alone, and decreased significantly when the coder used the case notes (70% vs 58% respectively, p < 0.0001) or coded from the discharge summary with medical support (70% vs 60% respectively, p < 0.0001). When compared with the gold standard, the percentage of incorrect HRGs was 42% for discharge summary alone, 31% for coding with case notes, and 35% for coding with medical support. The three coding methods resulted in an annual estimated loss of hospital remuneration of between £1.8 M and £16.5 M. CONCLUSION: The accuracy of diagnosis codes and percentage of correct HRGs improved when coders used either case notes or medical support in addition to the discharge summary. Further emphasis needs to be placed on improving the standard of information recorded in discharge summaries.

Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency.

OBJECTIVE: To investigate methods and processes for timely, efficient and good quality peer review of research funding proposals in health. METHODS: A two-stage evidence synthesis: (1) a systematic map to describe the key characteristics of the evidence base, followed by (2) a systematic review of the studies stakeholders prioritised as relevant from the map on the effectiveness and efficiency of peer review 'innovations'. Standard processes included literature searching, duplicate inclusion criteria screening, study keyword coding, data extraction, critical appraisal and study synthesis. RESULTS: A total of 83 studies from 15 countries were included in the systematic map. The evidence base is diverse, investigating many aspects of the systems for, and processes of, peer review. The systematic review included eight studies from Australia, Canada, and the USA, evaluating a broad range of peer review innovations. These studies showed that simplifying the process by shortening proposal forms, using smaller reviewer panels, or expediting processes can speed up the review process and reduce costs, but this might come at the expense of peer review quality, a key aspect that has not been assessed. Virtual peer review using videoconferencing or teleconferencing appears promising for reducing costs by avoiding the need for reviewers to travel, but again any consequences for quality have not been adequately assessed. CONCLUSIONS: There is increasing international research activity into the peer review of health research funding. The studies reviewed had methodological limitations and variable generalisability to research funders. Given these limitations it is not currently possible to recommend immediate implementation of these innovations. However, many appear promising based on existing evidence, and could be adapted as necessary by funders and evaluated. Where feasible, experimental evaluation, including randomised controlled trials, should be conducted, evaluating impact on effectiveness, efficiency and quality.

Direct improvement of quality of life in colorectal cancer patients using a tailored pathway with quality of life diagnosis and therapy (DIQOL): study protocol for a randomised controlled trial.

BACKGROUND: Medical treatment in patient-centred care in oncology is broadly managed and regulated in terms of guideline development, cancer centres, and quality assurance by cancer registries. In contrast to this quality management cycle (PDCA), there are no equal standards for patient-reported outcomes like quality of life (QoL). Therefore, the Tumour Centre Regensburg e.V., a population-based regional cancer registry covering a population of more than 2.2 million people in the Upper Palatinate and Lower Bavaria, Germany, designed and implemented a QoL pathway. In a complex intervention with QoL diagnosis and therapy (multidimensional therapeutic network), effectiveness for patients with breast cancer has been demonstrated. To provide local tailored QoL diagnosis and therapy to other cancer patients as well, external validity needs to be extended by adapting the QoL pathway to another tumour entity. METHODS/DESIGN: The QoL pathway will be tested for colorectal cancer patients in a pragmatic randomised controlled trial. Two hundred twenty primary colorectal cancer patients, surgically treated in one of four hospitals, will be included. QoL is measured in all patients 0, 3, 6, 12, and 18 months after surgery (European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30, QLQ-CR29). In the intervention group, QoL scores are transformed into a QoL profile. This is sent to the coordinating practitioner (general practitioner, internist, or oncologist) with an expert report including treatment recommendations for QoL deficits. The control group receives routine follow-up care attending the guideline recommendations for colorectal cancer without profile or expert report. At the primary endpoint (12 months), the rates of patients with diseased QoL in both groups are compared. DISCUSSION: This randomised trial is the first complex intervention investigating the effectiveness of an intervention with QoL diagnosis and tailored QoL therapy in colorectal cancer patients. The results will show if this QoL pathway improves the patients' QoL during follow-up care of their disease. TRIAL REGISTRATION: ClinicalTrials.gov, NCT02321813 (registered December 2014).

Patients' online access to their electronic health records and linked online services: a systematic review in primary care.

BACKGROUND: Online access to medical records by patients can potentially enhance provision of patient-centred care and improve satisfaction. However, online access and services may also prove to be an additional burden for the healthcare provider. AIM: To assess the impact of providing patients with access to their general practice electronic health records (EHR) and other EHR-linked online services on the provision, quality, and safety of health care. DESIGN AND SETTING: A systematic review was conducted that focused on all studies about online record access and transactional services in primary care. METHOD: Data sources included MEDLINE, Embase, CINAHL, Cochrane Library, EPOC, DARE, King's Fund, Nuffield Health, PsycINFO, OpenGrey (1999-2012). The literature was independently screened against detailed inclusion and exclusion criteria; independent dual data extraction was conducted, the risk of bias (RoB) assessed, and a narrative synthesis of the evidence conducted. RESULTS: A total of 176 studies were identified, 17 of which were randomised controlled trials, cohort, or cluster studies. Patients reported improved satisfaction with online access and services compared with standard provision, improved self-care, and better communication and engagement with clinicians. Safety improvements were patient-led through identifying medication errors and facilitating more use of preventive services. Provision of online record access and services resulted in a moderate increase of e-mail, no change on telephone contact, but there were variable effects on face-to-face contact. However, other tasks were necessary to sustain these services, which impacted on clinician time. There were no reports of harm or breaches in privacy. CONCLUSION: While the RoB scores suggest many of the studies were of low quality, patients using online services reported increased convenience and satisfaction. These services positively impacted on patient safety, although there were variations of record access and use by specific ethnic and socioeconomic groups. Professional concerns about privacy were unrealised and those about workload were only partly so.

Patients' online access to their electronic health records and linked online services: a systematic interpretative review.

OBJECTIVES: To investigate the effect of providing patients online access to their electronic health record (EHR) and linked transactional services on the provision, quality and safety of healthcare. The objectives are also to identify and understand: barriers and facilitators for providing online access to their records and services for primary care workers; and their association with organisational/IT system issues. SETTING: Primary care. PARTICIPANTS: A total of 143 studies were included. 17 were experimental in design and subject to risk of bias assessment, which is reported in a separate paper. Detailed inclusion and exclusion criteria have also been published elsewhere in the protocol. PRIMARY AND SECONDARY OUTCOME MEASURES: Our primary outcome measure was change in quality or safety as a result of implementation or utilisation of online records/transactional services. RESULTS: No studies reported changes in health outcomes; though eight detected medication errors and seven reported improved uptake of preventative care. Professional concerns over privacy were reported in 14 studies. 18 studies reported concern over potential increased workload; with some showing an increase workload in email or online messaging; telephone contact remaining unchanged, and face-to face contact staying the same or falling. Owing to heterogeneity in reporting overall workload change was hard to predict. 10 studies reported how online access offered convenience, primarily for more advantaged patients, who were largely highly satisfied with the process when clinician responses were prompt. CONCLUSIONS: Patient online access and services offer increased convenience and satisfaction. However, professionals were concerned about impact on workload and risk to privacy. Studies correcting medication errors may improve patient safety. There may need to be a redesign of the business process to engage health professionals in online access and of the EHR to make it friendlier and provide equity of access to a wider group of patients. A1 SYSTEMATIC REVIEW REGISTRATION NUMBER: PROSPERO CRD42012003091.

Computer decision support systems for asthma: a systematic review.

BACKGROUND: Increasing use of electronic health records offers the potential to incorporate computer decision support systems (CDSSs) to prompt evidence-based actions within routine consultations. AIM: To synthesise the evidence for the use of CDSSs by professionals managing people with asthma. MATERIALS AND METHODS: We systematically searched Medline, Embase, Health Technology Assessment, Cochrane and Inspec databases (1990 to April 2012, no language restrictions) for trials, and four online repositories for unpublished studies. We also wrote to authors. Eligible studies were randomised controlled trials of CDSSs supporting professional management of asthma. Studies were appraised (Cochrane Risk of Bias Tool) and findings synthesised narratively. RESULTS: A total of 5787 articles were screened, and eight trials were found eligible, with six at high risk of bias. Overall, CDSSs for professionals were ineffective. Usage of the systems was generally low: in the only trial at low risk of bias the CDSS was not used at all. When a CDSS was used, compliance with the advice offered was also low. However, if actually used, CDSSs could result in closer guideline adherence (improve investigating, prescribing and issuing of action plans) and could improve some clinical outcomes. The study at moderate risk of bias showed increased prescribing of inhaled steroids. CONCLUSIONS: The current generation of CDSSs is unlikely to result in improvements in outcomes for patients with asthma because they are rarely used and the advice is not followed. Future decision support systems need to align better with professional workflows so that pertinent and timely advice is easily accessible within the consultation.

From assessment to improvement of elderly care in general practice using decision support to increase adherence to ACOVE quality indicators: study protocol for randomized control trial.

BACKGROUND: Previous efforts such as Assessing Care of Vulnerable Elders (ACOVE) provide quality indicators for assessing the care of elderly patients, but thus far little has been done to leverage this knowledge to improve care for these patients. We describe a clinical decision support system to improve general practitioner (GP) adherence to ACOVE quality indicators and a protocol for investigating impact on GPs' adherence to the rules. DESIGN: We propose two randomized controlled trials among a group of Dutch GP teams on adherence to ACOVE quality indicators. In both trials a clinical decision support system provides un-intrusive feedback appearing as a color-coded, dynamically updated, list of items needing attention. The first trial pertains to real-time automatically verifiable rules. The second trial concerns non-automatically verifiable rules (adherence cannot be established by the clinical decision support system itself, but the GPs report whether they will adhere to the rules). In both trials we will randomize teams of GPs caring for the same patients into two groups, A and B. For the automatically verifiable rules, group A GPs receive support only for a specific inter-related subset of rules, and group B GPs receive support only for the remainder of the rules. For non-automatically verifiable rules, group A GPs receive feedback framed as actions with positive consequences, and group B GPs receive feedback framed as inaction with negative consequences. GPs indicate whether they adhere to non-automatically verifiable rules. In both trials, the main outcome measure is mean adherence, automatically derived or self-reported, to the rules. DISCUSSION: We relied on active end-user involvement in selecting the rules to support, and on a model for providing feedback displayed as color-coded real-time messages concerning the patient visiting the GP at that time, without interrupting the GP's workflow with pop-ups. While these aspects are believed to increase clinical decision support system acceptance and its impact on adherence to the selected clinical rules, systems with these properties have not yet been evaluated. TRIAL REGISTRATION: Controlled Trials NTR3566.

Introducing a nationally shared electronic patient record: case study comparison of Scotland, England, Wales and Northern Ireland.

AIM: To compare the experience of the four UK countries in introducing nationally accessible electronic summaries of patients' key medical details, intended for use in emergency and unscheduled care episodes, and generate transferable lessons for other countries. METHOD: Secondary analysis of data collected previously on all four schemes; cross-case comparison using a framework derived from diffusion of innovations theory. MAIN FINDINGS: Whilst all four programmes shared a similar vision, they differed widely in their strategy, budget, implementation plan, approach to clinical and public engagement and approach to evaluation and learning. They also differed, for various reasons, in stakeholder alignments, the nature and extent of resistance to the programme and the rate at which records were created. A nationally shared, widely accessible electronic record has powerful symbolic meaning; it may or may not be perceived as improving the quality and safety of care or (alternatively) as threatening patient confidentiality or the traditional role of the doctor or nurse. 'Hard' project management oriented to achieving specific milestones and deadlines sometimes appeared counterproductive when it cut across the 'softer' aspects of the programmes. CONCLUSION: When designing and implementing complex technologies with pervasive implications, policymakers must consider not only technical issues but also the personal, social and organisational aspects of the programme. A judicious blend of 'hard' and 'soft' management appears key to managing such programmes.

Computer-generated reminders delivered on paper to healthcare professionals; effects on professional practice and health care outcomes.

BACKGROUND: Clinical practice does not always reflect best practice and evidence, partly because of unconscious acts of omission, information overload, or inaccessible information. Reminders may help clinicians overcome these problems by prompting the doctor to recall information that they already know or would be expected to know and by providing information or guidance in a more accessible and relevant format, at a particularly appropriate time. OBJECTIVES: To evaluate the effects of reminders automatically generated through a computerized system and delivered on paper to healthcare professionals on processes of care (related to healthcare professionals' practice) and outcomes of care (related to patients' health condition). SEARCH METHODS: For this update the EPOC Trials Search Co-ordinator searched the following databases between June 11-19, 2012: The Cochrane Central Register of Controlled Trials (CENTRAL) and Cochrane Library (Economics, Methods, and Health Technology Assessment sections), Issue 6, 2012; MEDLINE, OVID (1946- ), Daily Update, and In-process; EMBASE, Ovid (1947- ); CINAHL, EbscoHost (1980- ); EPOC Specialised Register, Reference Manager, and INSPEC, Engineering Village. The authors reviewed reference lists of related reviews and studies.  SELECTION CRITERIA: We included individual or cluster-randomized controlled trials (RCTs) and non-randomized controlled trials (NRCTs) that evaluated the impact of computer-generated reminders delivered on paper to healthcare professionals on processes and/or outcomes of care. DATA COLLECTION AND ANALYSIS: Review authors working in pairs independently screened studies for eligibility and abstracted data. We contacted authors to obtain important missing information for studies that were published within the last 10 years. For each study, we extracted the primary outcome when it was defined or calculated the median effect size across all reported outcomes. We then calculated the median absolute improvement and interquartile range (IQR) in process adherence across included studies using the primary outcome or median outcome as representative outcome. MAIN RESULTS: In the 32 included studies, computer-generated reminders delivered on paper to healthcare professionals achieved moderate improvement in professional practices, with a median improvement of processes of care of 7.0% (IQR: 3.9% to 16.4%). Implementing reminders alone improved care by 11.2% (IQR 6.5% to 19.6%) compared with usual care, while implementing reminders in addition to another intervention improved care by 4.0% only (IQR 3.0% to 6.0%) compared with the other intervention. The quality of evidence for these comparisons was rated as moderate according to the GRADE approach. Two reminder features were associated with larger effect sizes: providing space on the reminder for provider to enter a response (median 13.7% versus 4.3% for no response, P value = 0.01) and providing an explanation of the content or advice on the reminder (median 12.0% versus 4.2% for no explanation, P value = 0.02). Median improvement in processes of care also differed according to the behaviour the reminder targeted: for instance, reminders to vaccinate improved processes of care by 13.1% (IQR 12.2% to 20.7%) compared with other targeted behaviours. In the only study that had sufficient power to detect a clinically significant effect on outcomes of care, reminders were not associated with significant improvements. AUTHORS' CONCLUSIONS: There is moderate quality evidence that computer-generated reminders delivered on paper to healthcare professionals achieve moderate improvement in process of care. Two characteristics emerged as significant predictors of improvement: providing space on the reminder for a response from the clinician and providing an explanation of the reminder's content or advice. The heterogeneity of the reminder interventions included in this review also suggests that reminders can improve care in various settings under various conditions.

Open source, open standards, and health care information systems.

Recognition of the improvements in patient safety, quality of patient care, and efficiency that health care information systems have the potential to bring has led to significant investment. Globally the sale of health care information systems now represents a multibillion dollar industry. As policy makers, health care professionals, and patients, we have a responsibility to maximize the return on this investment. To this end we analyze alternative licensing and software development models, as well as the role of standards. We describe how licensing affects development. We argue for the superiority of open source licensing to promote safer, more effective health care information systems. We claim that open source licensing in health care information systems is essential to rational procurement strategy.

The case for randomized controlled trials to assess the impact of clinical information systems.

There is a persistent view of a significant minority in the medical informatics community that the randomized controlled trial (RCT) has a limited role to play in evaluating clinical information systems. A common reason voiced by skeptics is that these systems are fundamentally different from drug interventions, so the RCT is irrelevant. There is an urgent need to promote the use of RCTs, given the shift to evidence-based policy and the need to demonstrate cost-effectiveness of these systems. The authors suggest returning to first principles and argue that what is required is clarity about how to match methods to evaluation questions. The authors address common concerns about RCTs, and the extent to which they are fallacious, and also discuss the challenges of conducting RCTs in informatics and alternative study designs when randomized trials are infeasible. While neither a perfect nor universal evaluation method, RCTs form an important part of an evaluator's toolkit.

National-scale clinical information exchange in the United Kingdom: lessons for the United States.

Over the last four decades, the UK has made large investments in healthcare information technology. The authors conducted interviews and reviewed published and unpublished documents to describe national-scale clinical information exchange in England, how it was achieved, and the problems experienced that the USA might avoid. Clinical information exchange in the UK was accomplished by establishing a foundation of policy, infrastructure, and systems of care, by creating and acquiring clinical computing applications and with strong use of financial and clinical incentives. Many software and hardware vendors played a part in this effort; they participated in a national framework created by the NHS in which standards for exchange are specified and their applications designed to make clinical information exchange part of normal practice. Great potential exists for cost reduction, increased safety, and greater patient involvement as a result of clinical information exchange.

Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study.

BACKGROUND: Computerized decision support systems (DSS) have mainly focused on improving clinicians' diagnostic accuracy in unusual and challenging cases. However, since diagnostic omission errors may predominantly result from incomplete workup in routine clinical practice, the provision of appropriate patient- and context-specific reminders may result in greater impact on patient safety. In this experimental study, a mix of easy and difficult simulated cases were used to assess the impact of a novel diagnostic reminder system (ISABEL) on the quality of clinical decisions made by various grades of clinicians during acute assessment. METHODS: Subjects of different grades (consultants, registrars, senior house officers and medical students), assessed a balanced set of 24 simulated cases on a trial website. Subjects recorded their clinical decisions for the cases (differential diagnosis, test-ordering and treatment), before and after system consultation. A panel of two pediatric consultants independently provided gold standard responses for each case, against which subjects' quality of decisions was measured. The primary outcome measure was change in the count of diagnostic errors of omission (DEO). A more sensitive assessment of the system's impact was achieved using specific quality scores; additional consultation time resulting from DSS use was also calculated. RESULTS: 76 subjects (18 consultants, 24 registrars, 19 senior house officers and 15 students) completed a total of 751 case episodes. The mean count of DEO fell from 5.5 to 5.0 across all subjects (repeated measures ANOVA, p < 0.001); no significant interaction was seen with subject grade. Mean diagnostic quality score increased after system consultation (0.044; 95% confidence interval 0.032, 0.054). ISABEL reminded subjects to consider at least one clinically important diagnosis in 1 in 8 case episodes, and prompted them to order an important test in 1 in 10 case episodes. Median extra time taken for DSS consultation was 1 min (IQR: 30 sec to 2 min). CONCLUSION: The provision of patient- and context-specific reminders has the potential to reduce diagnostic omissions across all subject grades for a range of cases. This study suggests a promising role for the use of future reminder-based DSS in the reduction of diagnostic error.

Decision tools in health care: focus on the problem, not the solution.

BACKGROUND: Systematic reviews or randomised-controlled trials usually help to establish the effectiveness of drugs and other health technologies, but are rarely sufficient by themselves to ensure actual clinical use of the technology. The process from innovation to routine clinical use is complex. Numerous computerised decision support systems (DSS) have been developed, but many fail to be taken up into actual use. Some developers construct technologically advanced systems with little relevance to the real world. Others did not determine whether a clinical need exists. With NHS investing 5 billion pounds sterling in computer systems, also occurring in other countries, there is an urgent need to shift from a technology-driven approach to one that identifies and employs the most cost-effective method to manage knowledge, regardless of the technology. The generic term, 'decision tool' (DT), is therefore suggested to demonstrate that these aids, which seem different technically, are conceptually the same from a clinical viewpoint. DISCUSSION: Many computerised DSSs failed for various reasons, for example, they were not based on best available knowledge; there was insufficient emphasis on their need for high quality clinical data; their development was technology-led; or evaluation methods were misapplied. We argue that DSSs and other computer-based, paper-based and even mechanical decision aids are members of a wider family of decision tools. A DT is an active knowledge resource that uses patient data to generate case specific advice, which supports decision making about individual patients by health professionals, the patients themselves or others concerned about them. The identification of DTs as a consistent and important category of health technology should encourage the sharing of lessons between DT developers and users and reduce the frequency of decision tool projects focusing only on technology. The focus of evaluation should become more clinical, with the impact of computer-based DTs being evaluated against other computer, paper- or mechanical tools, to identify the most cost effective tool for each clinical problem. SUMMARY: We suggested the generic term 'decision tool' to demonstrate that decision-making aids, such as computerised DSSs, paper algorithms, and reminders are conceptually the same, so the methods to evaluate them should be the same.