Improving the effectiveness of electronic health record-based referral processes.

Electronic health records are increasingly being used to facilitate referral communication in the outpatient setting. However, despite support by technology, referral communication between primary care providers and specialists is often unsatisfactory and is unable to eliminate care delays. This may be in part due to lack of attention to how information and communication technology fits within the social environment of health care. Making electronic referral communication effective requires a multifaceted "socio-technical" approach. Using an 8-dimensional socio-technical model for health information technology as a framework, we describe ten recommendations that represent good clinical practices to design, develop, implement, improve, and monitor electronic referral communication in the outpatient setting. These recommendations were developed on the basis of our previous work, current literature, sound clinical practice, and a systems-based approach to understanding and implementing health information technology solutions. Recommendations are relevant to system designers, practicing clinicians, and other stakeholders considering use of electronic health records to support referral communication.

Follow-up actions on electronic referral communication in a multispecialty outpatient setting.

OBJECTIVES: Electronic health records (EHR) enable transmission and tracking of referrals between primary-care practitioners (PCPs) and subspecialists. We used an EHR to examine follow-up actions on electronic referral communication in a large multispecialty VA facility. METHODS: We retrieved outpatient referrals to five subspecialties between October 2006 and December 2007, and queried the EHR to determine their status: completed, discontinued (returned to PCP), or unresolved (no action taken by subspecialist). All unresolved referrals, and random samples of discontinued and completed referrals were reviewed to determine whether subspecialists took follow-up actions (i.e., schedule appointments anytime in the future) within 30 days of referral-receipt. For referrals without timely follow-up, we determined whether inaction was supported by any predetermined justifiable reasons or associated with certain referral characteristics. We also reviewed if PCPs took the required action on returned information. RESULTS: Of 61,931 referrals, 22,535 were discontinued (36.4%), and 474 were unresolved (0.8%). We selected 412 discontinued referrals randomly for review. Of these, 52% lacked follow-up actions within 30 days. Appropriate justifications for inaction were documented in 69.8% (150/215) of those without action and included lack of prerequisite testing by the PCP and subspecialist opinion that no intervention was required despite referral. We estimated that at 30 days, 6.3% of all referrals were associated with an unexplained lack of follow-up actions by subspecialists. Conversely, 7.4% of discontinued referrals returned to PCPs were associated with an unexplained lack of follow-up. CONCLUSIONS: Although the EHR facilitates transmission of valuable information at the PCP-subspecialist interface, unexplained communication breakdowns in the referral process persist in a subset of cases.

Mosaics versus Early Treatment Diabetic Retinopathy seven standard fields for evaluation of diabetic retinopathy severity.

PURPOSE: To compare agreement between mosaicked and seven field photographs for classification of the diabetic retinopathy (DR) severity. METHODS: Mosaic digital (MosD) images were compared with seven field stereo film (7FF) and stereo digital (7FD) photographs from a 152-eye cohort with full-spectrum Early Treatment of Diabetic Retinopathy severity levels for agreement on severity level, DR presence with ascending severity thresholds, DR index lesion presence, and classification repeatability. RESULTS: There was a substantial agreement classifying the Early Treatment Diabetic Retinopathy Study DR severity level between MosD and 7FF (kunweighted = 0.59, klinear weighted = 0.83), MosD and 7FD (κ = 0.62, κ weighted = 0.86), and 7FD and 7FF (κ = 0.62, κ weighted = 0.86) images. Marginal homogeneity analyses found no significant difference between MosD and 7FF (P = 0.44, Bhapkar's test). Kappa between MosD and 7FF ranged from 0.75 to 0.91 for the presence or absence of DR at 8 ascending severity thresholds. Repeatability among readers using MosD images was similar to repeatability among those using 7FF or 7FD. Repeatability among readers using MosD and 7FF images at various severity thresholds was similar. Kappa between MosD and 7FF grading for identifying DR lesions ranged from 0.61 to 1.00. CONCLUSION: Mosaic images are generally comparable with standard seven-field photographs for classifying DR severity.

Understanding the management of electronic test result notifications in the outpatient setting.

BACKGROUND: Notifying clinicians about abnormal test results through electronic health record (EHR) -based "alert" notifications may not always lead to timely follow-up of patients. We sought to understand barriers, facilitators, and potential interventions for safe and effective management of abnormal test result delivery via electronic alerts. METHODS: We conducted a qualitative study consisting of six 6-8 member focus groups (N = 44) at two large, geographically dispersed Veterans Affairs facilities. Participants included full-time primary care providers, and personnel representing diagnostic services (radiology, laboratory) and information technology. We asked participants to discuss barriers, facilitators, and suggestions for improving timely management and follow-up of abnormal test result notifications and encouraged them to consider technological issues, as well as broader, human-factor-related aspects of EHR use such as organizational, personnel, and workflow. RESULTS: Providers reported receiving a large number of alerts containing information unrelated to abnormal test results, many of which were believed to be unnecessary. Some providers also reported lacking proficiency in use of certain EHR features that would enable them to manage alerts more efficiently. Suggestions for improvement included improving display and tracking processes for critical alerts in the EHR, redesigning clinical workflow, and streamlining policies and procedures related to test result notification. CONCLUSION: Providers perceive several challenges for fail-safe electronic communication and tracking of abnormal test results. A multi-dimensional approach that addresses technology as well as the many non-technological factors we elicited is essential to design interventions to reduce missed test results in EHRs.

Grading diabetic retinopathy severity from compressed digital retinal images compared with uncompressed images and film.

PURPOSE: To compare research grading of diabetic retinopathy (DR) severity level from compressed digital images versus uncompressed images and film. METHODS: Compressed (JPEG2000, 37:1) digital images (C) were compared with uncompressed digital (U) and film (F) stereoscopic photographs from a 152-eye cohort with full-spectrum Early Treatment Diabetic Retinopathy Study severity levels for agreement on severity level, DR presence with ascending severity threshold, presence of DR index lesions, and repeatability of grading. RESULTS: Classification of Early Treatment Diabetic Retinopathy Study severity levels from C images agreed substantially with results from F images (κ = 0.60, κ(w) [linear weighted] = 0.86) and uncompressed digital images (κ = 0.76, κ(w) = 0.92). For agreement of uncompressed digital versus F images, κ = 0.62 and κ(w) = 0.86. Distribution of Early Treatment Diabetic Retinopathy Study levels was not significantly different between C and F images (P = 0.09, Bhapkar's test for marginal homogeneity). For presence/absence of DR at 8 ascending severity thresholds, agreement between C and F was "almost perfect" (κ ≥ 0.8). Agreement on severity level between readers with C images was at least as good as that with uncompressed digital image or F. Repeatability of severity threshold grading between readers was similar using C or F images. For identifying individual DR lesions, agreement between C and F ranged from "moderate" to "perfect." Agreement of grading venous beading from C was slightly lower than from F. CONCLUSION: Full Early Treatment Diabetic Retinopathy Study scale DR severity level grading using C images is comparable to that using U images or film.

When your words count: a discriminative model to predict approval of referrals.

OBJECTIVE: To develop and test a statistical model which correctly predicts the approval of outpatient referrals when reviewed by a specialty service based on nine discriminating variables. DESIGN: Retrospective cross-sectional study. SETTING: Large public county hospital system in a southern US city. PARTICIPANTS: Written documents and associated data from 500 random adult referrals made by primary care providers to various specialty services during the course of one month. MAIN OUTCOME MEASURES: The resulting correct prediction rates obtained by the model. RESULTS: The model correctly predicted 78.6% of approved referrals using all nine discriminating variables, 75.3% of approved referrals using all variables in a stepwise manner and 74.7% of approved referrals using only the referral total word count as a single discriminating variable. CONCLUSIONS: Three iterations of the model correctly predicted at least 75% of the approved referrals in the validation set. A correct prediction of whether or not a referral will be approved can be made in three out of four cases.

Developing a taxonomy and an ontology of nurses' patient clinical summaries.

Nurses prepare a summary of patient information that they consult and update throughout the shift. This document is believed to be integral to cognition, working memory, and decision-making. While serving as a key support to nursing practice, this summary also represents risks to patient safety. Characterized as a PCCAT, or Personally Created Cognitive Artifact, studies of this document in the context of nursing practice have not been reported. The absence of reported research, the importance of the document to nurse cognition and practice, and related safety risks prompted the research that this paper discusses. A taxonomy was developed through the analysis and coding of 151 PCCATs. Further analysis and mapping provided an ontology of the PCCAT. Content differences were noted between nursing units and among nurses. This may reflect differences in unit-based culture and/or differences in the patient complexity. The interaction between culture and perceived complexity of practice is one of the great difficulties in generating automated information systems for clinical practice settings. This paper is part of a larger research protocol that explores meta-level knowledge structures and revision to the understanding of the granularity of nursing knowledge. Development of a taxonomy and ontology of the nurse PCCAT, an important component of the larger research protocol, is described in this paper.

Releasing test results directly to patients: A multisite survey of physician perspectives.

OBJECTIVE: To determine physician perspectives about direct notification of normal and abnormal test results. METHODS: We conducted a cross-sectional survey at five clinical sites in the US and Australia. The US-based study was conducted via web-based survey of primary care physicians and specialists between July and October 2012. An identical paper-based survey was self-administered between June and September 2012 with specialists in Australia. RESULTS: Of 1417 physicians invited, 315 (22.2%) completed the survey. Two-thirds (65.3%) believed that patients should be directly notified of normal results, but only 21.3% were comfortable with direct notification of clinically significant abnormal results. Physicians were more likely to endorse direct notification of abnormal results if they believed it would reduce the number of patients lost to follow-up (OR=4.98, 95%CI=2.21-1.21) or if they had personally missed an abnormal test result (OR=2.95, 95%CI=1.44-6.02). Conversely, physicians were less likely to endorse if they believed that direct notification interfered with the practice of medicine (OR=0.39, 95%CI=0.20-0.74). CONCLUSION: Physicians we surveyed generally favor direct notification of normal results but appear to have substantial concerns about direct notification of abnormal results. PRACTICE IMPLICATIONS: Widespread use of direct notification should be accompanied by strategies to help patients manage test result abnormalities they receive.

Electronic health record-based triggers to detect potential delays in cancer diagnosis.

BACKGROUND: Delayed diagnosis of cancer can lead to patient harm, and strategies are needed to proactively and efficiently detect such delays in care. We aimed to develop and evaluate 'trigger' algorithms to electronically flag medical records of patients with potential delays in prostate and colorectal cancer (CRC) diagnosis. METHODS: We mined retrospective data from two large integrated health systems with comprehensive electronic health records (EHR) to iteratively develop triggers. Data mining algorithms identified all patient records with specific demographics and a lack of appropriate and timely follow-up actions on four diagnostic clues that were newly documented in the EHR: abnormal prostate-specific antigen (PSA), positive faecal occult blood test (FOBT), iron-deficiency anaemia (IDA), and haematochezia. Triggers subsequently excluded patients not needing follow-up (eg, terminal illness) or who had already received appropriate and timely care. Each of the four final triggers was applied to a test cohort, and chart reviews of randomly selected records identified by the triggers were used to calculate positive predictive values (PPV). RESULTS: The PSA trigger was applied to records of 292 587 patients seen between 1 January 2009 and 31 December 2009, and the CRC triggers were applied to 291 773 patients seen between 1 March 2009 and 28 February 2010. Overall, 1564 trigger positive patients were identified (426 PSA, 355 FOBT, 610 IDA and 173 haematochezia). Record reviews revealed PPVs of 70.2%, 66.7%, 67.5%, and 58.3% for the PSA, FOBT, IDA and haematochezia triggers, respectively. Use of all four triggers at the study sites could detect an estimated 1048 instances of delayed or missed follow-up of abnormal findings annually and 47 high-grade cancers. CONCLUSIONS: EHR-based triggers can be used successfully to flag patient records lacking follow-up of abnormal clinical findings suspicious for cancer.

Comparability of digital photography with the ETDRS film protocol for evaluation of diabetic retinopathy severity.

PURPOSE: To evaluate digital photography parameters affecting comparability with the Early Treatment Diabetic Retinopathy Study (ETDRS) film protocol for diabetic retinopathy (DR) severity grading. METHODS: ETDRS protocol photographs and four variations of digital images (uncompressed stereoscopic, compressed stereoscopic, uncompressed monoscopic, and uncompressed monoscopic wide-angle mosaic) of 152 eyes were independently evaluated by using ETDRS classifications. Digital formats were compared to film and each other for agreement on severity level, DR presence at ascending threshold, presence of the DR index lesion, and repeatability of grading. Study parameters included image resolution sufficient to distinguish small lesions, color balancing of digital images to film, documenting essential ETDRS classification retinal regions, similar magnification, and supplementary green-channel viewing. RESULTS: The κ statistic was substantial or near substantial between all digital formats and film for classifying severity levels (κ = 0.59-0.62; κ(w) [linear weighted] = 0.83-0.87). The distribution of DR levels in all digital formats was not significantly different from that of the film (Bhapkar test, P = 0.09-0.44). The κ among digital formats for severity level was also substantial or near substantial (κ = 0.58-0.76, κ(w) = 0.82-0.92). Differences between digital formats and film for grading severity level, severity threshold, or index lesions were not significant. The repeatability of grading between readers using film and all digital formats was also similar. CONCLUSIONS: Digital format variations compared favorably with film for DR classification. Translating film characteristics (resolution, color/contrast) and protocol (magnification, retinal regions) to digital equivalents and augmentation of full color with green-channel viewing most likely contributed to the results.

Towards successful coordination of electronic health record based-referrals: a qualitative analysis.

BACKGROUND: Successful subspecialty referrals require considerable coordination and interactive communication among the primary care provider (PCP), the subspecialist, and the patient, which may be challenging in the outpatient setting. Even when referrals are facilitated by electronic health records (EHRs) (i.e., e-referrals), lapses in patient follow-up might occur. Although compelling reasons exist why referral coordination should be improved, little is known about which elements of the complex referral coordination process should be targeted for improvement. Using Okhuysen & Bechky's coordination framework, this paper aims to understand the barriers, facilitators, and suggestions for improving communication and coordination of EHR-based referrals in an integrated healthcare system. METHODS: We conducted a qualitative study to understand coordination breakdowns related to e-referrals in an integrated healthcare system and examined work-system factors that affect the timely receipt of subspecialty care. We conducted interviews with seven subject matter experts and six focus groups with a total of 30 PCPs and subspecialists at two tertiary care Department of Veterans Affairs (VA) medical centers. Using techniques from grounded theory and content analysis, we identified organizational themes that affected the referral process. RESULTS: Four themes emerged: lack of an institutional referral policy, lack of standardization in certain referral procedures, ambiguity in roles and responsibilities, and inadequate resources to adapt and respond to referral requests effectively. Marked differences in PCPs' and subspecialists' communication styles and individual mental models of the referral processes likely precluded the development of a shared mental model to facilitate coordination and successful referral completion. Notably, very few barriers related to the EHR were reported. CONCLUSIONS: Despite facilitating information transfer between PCPs and subspecialists, e-referrals remain prone to coordination breakdowns. Clear referral policies, well-defined roles and responsibilities for key personnel, standardized procedures and communication protocols, and adequate human resources must be in place before implementing an EHR to facilitate referrals.

Concept analysis of cognitive artifacts.

Cognitive artifacts--information displays that inform thought processes and increase knowledge--fulfill a fundamental role in distributed cognition. Cognitive work--the mental processes of selecting and evaluating data, reasoning, and making decisions--is guided and informed by cognitive artifacts, especially in clinical areas. The importance of cognitive artifacts to cognitive work suggests the need to study and comprehensively understand cognitive artifacts prepared and used by the clinical nurses and how these documents influence and guide nursing practice. This article identifies and describes the attributes of effectively constructed cognitive artifacts using the concept analysis process described by Walker and Avant.

Monoscopic versus stereoscopic retinal photography for grading diabetic retinopathy severity.

PURPOSE: To assess agreement between monoscopic and stereoscopic photography for research classification of the severity of diabetic retinopathy (DR). METHODS: Monoscopic digital (MD) images were compared with stereo digital (SD) and film (SF) photographs from a 152-eye cohort with full-spectrum Early Treatment Diabetic Retinopathy Study (ETDRS) severity levels for agreement on severity level, DR presence with ascending severity threshold, presence of DR index lesions, and repeatability of grading. RESULTS: There was substantial agreement classifying ETDRS DR severity levels between MD and SF (kappa = 0.65, kappa(w) [linear weighted] = 0.87), MD and SD (kappa = 0.66, kappa(w) = 0.87), and SD and SF (kappa = 0.62, kappa(w) = 0.86) images. Marginal homogeneity analyses found no significant difference between MD and SF images (P = 0.53, Bhapkar test). The kappa agreement between MD and SF ranged from 0.80 to 0.94 for the presence or absence of eight ascending DR severity thresholds. Repeatability between the readers of the MD images was equal to or better than that of the readers of SD or SF images. Severity threshold grading repeatability between readers was similar with the MD and SF images. The kappa agreement between MD and SF for identifying diabetic retinopathy lesions ranged from moderate to almost perfect. The kappa comparisons showed that performance of grading new vessels on the disc in MD images was slightly lower than that with the SF images. CONCLUSIONS: Monoscopic photography can equal the reliability of stereo photography for full ETDRS DR severity scale grading.

Comparison of multiple stereoscopic and monoscopic digital image formats to film for diabetic macular edema evaluation.

PURPOSE: To assess agreement between evaluations of monoscopic and stereoscopic digital images versus stereo film photographs in diabetic macular edema (DME). METHODS: A 152-eye group of digital monoscopic macular images (seven-field sets and wide-angle mosaics) were compared with digital stereoscopic images (uncompressed and compressed seven-field sets) and stereo 35-mm film photos (Early Treatment Diabetic Retinopathy Study protocol) for the presence of hard exudates (HE), retinal thickening (RT), clinically significant macular edema (CSME), and RT at the center of the macular (RTCM). RESULTS: Agreement, according to the κ statistic, was almost perfect in identifying HE and RT between all digital formats and stereo film (HE, κ = 0.81-0.87; RT, κ = 0.87-0.92). Distribution in all digital formats was not significantly different from that in film (Bhapkar test: HE, P = 0.20-0.40; RT, P = 0.06-1.0). CSME and RTCM grading differences were either significant or trended toward significance. The readers detected CSME and RTCM in film images more often than in digital formats. In identifying DME features, agreement between evaluations of monoscopic digital formats and film was similar to that between stereo digital formats and film, and the performance of uncompressed images versus film was similar to that of compressed images versus film. Repeatability between readers was similar in evaluations of film and all digital formats. Repeatability in identifying RTCM was lower than that of other DME components in film and all digital formats. CONCLUSIONS: Stereoscopic digital formats are equivalent to monoscopic for DME evaluation, but digital photography is not as sensitive as film in detecting CSME and RTCM.

Digital versus film Fundus photography for research grading of diabetic retinopathy severity.

PURPOSE: To assess agreement between digital and film photography for research classification of diabetic retinopathy severity. METHODS: Digital and film photographs from a 152-eye cohort with a full spectrum of Early Treatment Diabetic Retinopathy Study (ETDRS) severity levels were assessed for repeatability of grading within each image medium and for agreement on ETDRS discrete severity levels, ascending severity thresholds, and presence or absence of diabetic retinopathy index lesions, between digital and 35-mm slides (film). Digital photographs were color balanced to match film. RESULTS: There was substantial agreement (κ = 0.61, κ(w) [linear weighted] = 0.87) in classification of ETDRS diabetic retinopathy severity levels between digital images and film. Marginal homogeneity analyses found no significant difference in frequency distributions on the severity scale (P = 0.21, Bhapkar test). The κ results ranged from 0.72 to 0.95 for presence or absence of eight ascending diabetic retinopathy severity thresholds. Repeatability of grading between readers viewing digital images was equal to or better than that obtained with film (pair-wise interreader κ for digital images ranged from 0.47 to 0.57 and for film from 0.43 to 0.57. The κ results for identifying diabetic retinopathy lesions ranged from moderate to almost perfect. Moderate agreement of intraretinal microvascular abnormalities and venous beading between digital images and film accounted for slightly lower concordance for severity thresholds ≥47 and for slightly lower interreader agreement within digital and film images at severity thresholds ≥43 and ≥47. CONCLUSIONS: Under controlled circumstances, digital photography can equal the reliability of 35-mm slides for research classification of ETDRS severity level.

Improving outpatient safety through effective electronic communication: a study protocol.

BACKGROUND: Health information technology and electronic medical records (EMRs) are potentially powerful systems-based interventions to facilitate diagnosis and treatment because they ensure the delivery of key new findings and other health related information to the practitioner. However, effective communication involves more than just information transfer; despite a state of the art EMR system, communication breakdowns can still occur. [1-3] In this project, we will adapt a model developed by the Systems Engineering Initiative for Patient Safety (SEIPS) to understand and improve the relationship between work systems and processes of care involved with electronic communication in EMRs. We plan to study three communication activities in the Veterans Health Administration's (VA) EMR: electronic communication of abnormal imaging and laboratory test results via automated notifications (i.e., alerts); electronic referral requests; and provider-to-pharmacy communication via computerized provider order entry (CPOE). AIM: Our specific aim is to propose a protocol to evaluate the systems and processes affecting outcomes of electronic communication in the computerized patient record system (related to diagnostic test results, electronic referral requests, and CPOE prescriptions) using a human factors engineering approach, and hence guide the development of interventions for work system redesign. DESIGN: This research will consist of multiple qualitative methods of task analysis to identify potential sources of error related to diagnostic test result alerts, electronic referral requests, and CPOE; this will be followed by a series of focus groups to identify barriers, facilitators, and suggestions for improving the electronic communication system. Transcripts from all task analyses and focus groups will be analyzed using methods adapted from grounded theory and content analysis.

Accuracy and self correction of information received from an internet breast cancer list: content analysis.

OBJECTIVES: To determine the prevalence of false or misleading statements in messages posted by internet cancer support groups and whether these statements were identified as false or misleading and corrected by other participants in subsequent postings. DESIGN: Analysis of content of postings. SETTING: Internet cancer support group Breast Cancer Mailing List. MAIN OUTCOME MEASURES: Number of false or misleading statements posted from 1 January to 23 April 2005 and whether these were identified and corrected by participants in subsequent postings. RESULTS: 10 of 4600 postings (0.22%) were found to be false or misleading. Of these, seven were identified as false or misleading by other participants and corrected within an average of four hours and 33 minutes (maximum, nine hours and nine minutes). CONCLUSIONS: Most posted information on breast cancer was accurate. Most false or misleading statements were rapidly corrected by participants in subsequent postings.

Using technology to teach technology: design and evaluation of bilingual online physician education about electronic medical records.

The "EMR Tutorial" is designed to be a bilingual online physician education environment about electronic medical records. After iterative assessment and redesign, the tutorial was tested in two groups: U.S. physicians and Mexican medical students. Split-plot ANOVA revealed significantly different pre-test scores in the two groups, significant cognitive gains for the two groups overall, and no significant difference in the gains made by the two groups. Users rated the module positively on a satisfaction questionnaire.

Evaluation of variability and significance of fundus camera lens distortion.

Six digital fundus cameras were examined to determine degrees of distortion introduced by lenses. Results quantified lens distortion effects on fundus images and demonstrate intrinsic properties in similar camera systems may vary.