Understanding data requirements of retrospective studies.

BACKGROUND AND OBJECTIVE: Usage of data from electronic health records (EHRs) in clinical research is increasing, but there is little empirical knowledge of the data needed to support multiple types of research these sources support. This study seeks to characterize the types and patterns of data usage from EHRs for clinical research. MATERIALS AND METHODS: We analyzed the data requirements of over 100 retrospective studies by mapping the selection criteria and study variables to data elements of two standard data dictionaries, one from the healthcare domain and the other from the clinical research domain. We also contacted study authors to validate our results. RESULTS: The majority of variables mapped to one or to both of the two dictionaries. Studies used an average of 4.46 (range 1-12) data element types in the selection criteria and 6.44 (range 1-15) in the study variables. The most frequently used items (e.g., procedure, condition, medication) are often available in coded form in EHRs. Study criteria were frequently complex, with 49 of 104 studies involving relationships between data elements and 22 of the studies using aggregate operations for data variables. Author responses supported these findings. DISCUSSION AND CONCLUSION: The high proportion of mapped data elements demonstrates the significant potential for clinical data warehousing to facilitate clinical research. Unmapped data elements illustrate the difficulty in developing a complete data dictionary.

A patient-driven adaptive prediction technique to improve personalized risk estimation for clinical decision support.

OBJECTIVE: Competing tools are available online to assess the risk of developing certain conditions of interest, such as cardiovascular disease. While predictive models have been developed and validated on data from cohort studies, little attention has been paid to ensure the reliability of such predictions for individuals, which is critical for care decisions. The goal was to develop a patient-driven adaptive prediction technique to improve personalized risk estimation for clinical decision support. MATERIAL AND METHODS: A data-driven approach was proposed that utilizes individualized confidence intervals (CIs) to select the most 'appropriate' model from a pool of candidates to assess the individual patient's clinical condition. The method does not require access to the training dataset. This approach was compared with other strategies: the BEST model (the ideal model, which can only be achieved by access to data or knowledge of which population is most similar to the individual), CROSS model, and RANDOM model selection. RESULTS: When evaluated on clinical datasets, the approach significantly outperformed the CROSS model selection strategy in terms of discrimination (p<1e-14) and calibration (p<0.006). The method outperformed the RANDOM model selection strategy in terms of discrimination (p<1e-12), but the improvement did not achieve significance for calibration (p=0.1375). LIMITATIONS: The CI may not always offer enough information to rank the reliability of predictions, and this evaluation was done using aggregation. If a particular individual is very different from those represented in a training set of existing models, the CI may be somewhat misleading. CONCLUSION: This approach has the potential to offer more reliable predictions than those offered by other heuristics for disease risk estimation of individual patients.

Protecting count queries in study design.

OBJECTIVE: Today's clinical research institutions provide tools for researchers to query their data warehouses for counts of patients. To protect patient privacy, counts are perturbed before reporting; this compromises their utility for increased privacy. The goal of this study is to extend current query answer systems to guarantee a quantifiable level of privacy and allow users to tailor perturbations to maximize the usefulness according to their needs. METHODS: A perturbation mechanism was designed in which users are given options with respect to scale and direction of the perturbation. The mechanism translates the true count, user preferences, and a privacy level within administrator-specified bounds into a probability distribution from which the perturbed count is drawn. RESULTS: Users can significantly impact the scale and direction of the count perturbation and can receive more accurate final cohort estimates. Strong and semantically meaningful differential privacy is guaranteed, providing for a unified privacy accounting system that can support role-based trust levels. This study provides an open source web-enabled tool to investigate visually and numerically the interaction between system parameters, including required privacy level and user preference settings. CONCLUSIONS: Quantifying privacy allows system administrators to provide users with a privacy budget and to monitor its expenditure, enabling users to control the inevitable loss of utility. While current measures of privacy are conservative, this system can take advantage of future advances in privacy measurement. The system provides new ways of trading off privacy and utility that are not provided in current study design systems.

iDASH: integrating data for analysis, anonymization, and sharing.

iDASH (integrating data for analysis, anonymization, and sharing) is the newest National Center for Biomedical Computing funded by the NIH. It focuses on algorithms and tools for sharing data in a privacy-preserving manner. Foundational privacy technology research performed within iDASH is coupled with innovative engineering for collaborative tool development and data-sharing capabilities in a private Health Insurance Portability and Accountability Act (HIPAA)-certified cloud. Driving Biological Projects, which span different biological levels (from molecules to individuals to populations) and focus on various health conditions, help guide research and development within this Center. Furthermore, training and dissemination efforts connect the Center with its stakeholders and educate data owners and data consumers on how to share and use clinical and biological data. Through these various mechanisms, iDASH implements its goal of providing biomedical and behavioral researchers with access to data, software, and a high-performance computing environment, thus enabling them to generate and test new hypotheses.

Anomaly and signature filtering improve classifier performance for detection of suspicious access to EHRs.

Our objective is to facilitate semi-automated detection of suspicious access to EHRs. Previously we have shown that a machine learning method can play a role in identifying potentially inappropriate access to EHRs. However, the problem of sampling informative instances to build a classifier still remained. We developed an integrated filtering method leveraging both anomaly detection based on symbolic clustering and signature detection, a rule-based technique. We applied the integrated filtering to 25.5 million access records in an intervention arm, and compared this with 8.6 million access records in a control arm where no filtering was applied. On the training set with cross-validation, the AUC was 0.960 in the control arm and 0.998 in the intervention arm. The difference in false negative rates on the independent test set was significant, P=1.6×10(-6). Our study suggests that utilization of integrated filtering strategies to facilitate the construction of classifiers can be helpful.

A multi-layered framework for disseminating knowledge for computer-based decision support.

BACKGROUND: There are several challenges in encoding guideline knowledge in a form that is portable to different clinical sites, including the heterogeneity of clinical decision support (CDS) tools, of patient data representations, and of workflows. METHODS: We have developed a multi-layered knowledge representation framework for structuring guideline recommendations for implementation in a variety of CDS contexts. In this framework, guideline recommendations are increasingly structured through four layers, successively transforming a narrative text recommendation into input for a CDS system. We have used this framework to implement rules for a CDS service based on three guidelines. We also conducted a preliminary evaluation, where we asked CDS experts at four institutions to rate the implementability of six recommendations from the three guidelines. CONCLUSION: The experience in using the framework and the preliminary evaluation indicate that this approach has promise in creating structured knowledge, to implement in CDS systems, that is usable across organizations.

Using statistical and machine learning to help institutions detect suspicious access to electronic health records.

OBJECTIVE: To determine whether statistical and machine-learning methods, when applied to electronic health record (EHR) access data, could help identify suspicious (ie, potentially inappropriate) access to EHRs. METHODS: From EHR access logs and other organizational data collected over a 2-month period, the authors extracted 26 features likely to be useful in detecting suspicious accesses. Selected events were marked as either suspicious or appropriate by privacy officers, and served as the gold standard set for model evaluation. The authors trained logistic regression (LR) and support vector machine (SVM) models on 10-fold cross-validation sets of 1291 labeled events. The authors evaluated the sensitivity of final models on an external set of 58 events that were identified as truly inappropriate and investigated independently from this study using standard operating procedures. RESULTS: The area under the receiver operating characteristic curve of the models on the whole data set of 1291 events was 0.91 for LR, and 0.95 for SVM. The sensitivity of the baseline model on this set was 0.8. When the final models were evaluated on the set of 58 investigated events, all of which were determined as truly inappropriate, the sensitivity was 0 for the baseline method, 0.76 for LR, and 0.79 for SVM. LIMITATIONS: The LR and SVM models may not generalize because of interinstitutional differences in organizational structures, applications, and workflows. Nevertheless, our approach for constructing the models using statistical and machine-learning techniques can be generalized. An important limitation is the relatively small sample used for the training set due to the effort required for its construction. CONCLUSION: The results suggest that statistical and machine-learning methods can play an important role in helping privacy officers detect suspicious accesses to EHRs.

Design and implementation of the GLIF3 guideline execution engine.

We have developed the GLIF3 Guideline Execution Engine (GLEE) as a tool for executing guidelines encoded in the GLIF3 format. In addition to serving as an interface to the GLIF3 guideline representation model to support the specified functions, GLEE provides defined interfaces to electronic medical records (EMRs) and other clinical applications to facilitate its integration with the clinical information system at a local institution. The execution model of GLEE takes the "system suggests, user controls" approach. A tracing system is used to record an individual patient's state when a guideline is applied to that patient. GLEE can also support an event-driven execution model once it is linked to the clinical event monitor in a local environment. Evaluation has shown that GLEE can be used effectively for proper execution of guidelines encoded in the GLIF3 format. When using it to execute each guideline in the evaluation, GLEE's performance duplicated that of the reference systems implementing the same guideline but taking different approaches. The execution flexibility and generality provided by GLEE, and its integration with a local environment, need to be further evaluated in clinical settings. Integration of GLEE with a specific event-monitoring and order-entry environment is the next step of our work to demonstrate its use for clinical decision support. Potential uses of GLEE also include quality assurance, guideline development, and medical education.

A methodology for modular representation of guidelines.

Computer-based clinical practice guidelines often need to be modified when medical knowledge evolves or when guidelines are implemented in a local setting with specific constraints and preferences. To enable easy modifications to guidelines and maintain their integrity, we have developed a methodology for modular representation of guidelines. Using this approach, we create guidelines in a hierarchical and modular manner. We use the Axiomatic Design methodology to facilitate the development of independent modules. Design matrices capture the interactions among modules. The design matrices can be used during guideline modification to create a change process and to enable identification of other modules that are affected by a change to a module. We implemented this modular knowledge representation approach by incorporating it into the Guideline Interchange Format (GLIF) language. We applied this approach to encode parts of three outdated guidelines released during 2000-2001, and we revised these designs to model updated releases of the guideline. Qualitative and quantitative metrics were developed to assess the types of changes made to the encoded guidelines.

Description and status update on GELLO: a proposed standardized object-oriented expression language for clinical decision support.

A major obstacle to sharing computable clinical knowledge is the lack of a common language for specifying expressions and criteria. Such a language could be used to specify decision criteria, formulae, and constraints on data and action. Al-though the Arden Syntax addresses this problem for clinical rules, its generalization to HL7's object-oriented data model is limited. The GELLO Expression language is an object-oriented language used for expressing logical conditions and computations in the GLIF3 (GuideLine Interchange Format, v. 3) guideline modeling language. It has been further developed under the auspices of the HL7 Clinical Decision Support Technical Committee, as a proposed HL7 standard., GELLO is based on the Object Constraint Language (OCL), because it is vendor-independent, object-oriented, and side-effect-free. GELLO expects an object-oriented data model. Although choice of model is arbitrary, standardization is facilitated by ensuring that the data model is compatible with the HL7 Reference Information Model (RIM).

Organization and representation of patient safety data: current status and issues around generalizability and scalability.

Recent reports have identified medical errors as a significant cause of morbidity and mortality among patients. A variety of approaches have been implemented to identify errors and their causes. These approaches include retrospective reporting and investigation of errors and adverse events and prospective analyses for identifying hazardous situations. The above approaches, along with other sources, contribute to data that are used to analyze patient safety risks. A variety of data structures and terminologies have been created to represent the information contained in these sources of patient safety data. Whereas many representations may be well suited to the particular safety application for which they were developed, such application-specific and often organization-specific representations limit the sharability of patient safety data. The result is that aggregation and comparison of safety data across organizations, practice domains, and applications is difficult at best. A common reference data model and a broadly applicable terminology for patient safety data are needed to aggregate safety data at the regional and national level and conduct large-scale studies of patient safety risks and interventions.

GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines.

The Guideline Interchange Format (GLIF) is a model for representation of sharable computer-interpretable guidelines. The current version of GLIF (GLIF3) is a substantial update and enhancement of the model since the previous version (GLIF2). GLIF3 enables encoding of a guideline at three levels: a conceptual flowchart, a computable specification that can be verified for logical consistency and completeness, and an implementable specification that is intended to be incorporated into particular institutional information systems. The representation has been tested on a wide variety of guidelines that are typical of the range of guidelines in clinical use. It builds upon GLIF2 by adding several constructs that enable interpretation of encoded guidelines in computer-based decision-support systems. GLIF3 leverages standards being developed in Health Level 7 in order to allow integration of guidelines with clinical information systems. The GLIF3 specification consists of an extensible object-oriented model and a structured syntax based on the resource description framework (RDF). Empirical validation of the ability to generate appropriate recommendations using GLIF3 has been tested by executing encoded guidelines against actual patient data. GLIF3 is accordingly ready for broader experimentation and prototype use by organizations that wish to evaluate its ability to capture the logic of clinical guidelines, to implement them in clinical systems, and thereby to provide integrated decision support to assist clinicians.

The InterMed approach to sharable computer-interpretable guidelines: a review.

InterMed is a collaboration among research groups from Stanford, Harvard, and Columbia Universities. The primary goal of InterMed has been to develop a sharable language that could serve as a standard for modeling computer-interpretable guidelines (CIGs). This language, called GuideLine Interchange Format (GLIF), has been developed in a collaborative manner and in an open process that has welcomed input from the larger community. The goals and experiences of the InterMed project and lessons that the authors have learned may contribute to the work of other researchers who are developing medical knowledge-based tools. The lessons described include (1) a work process for multi-institutional research and development that considers different viewpoints, (2) an evolutionary lifecycle process for developing medical knowledge representation formats, (3) the role of cognitive methodology to evaluate and assist in the evolutionary development process, (4) development of an architecture and (5) design principles for sharable medical knowledge representation formats, and (6) a process for standardization of a CIG modeling language.

A method for subdividing clinical guidelines into process modules with associated triggers and objectives to facilitate implementation.

Representation of multi-step clinical guidelines (CG) and their implementation in computerized decision support (DS) systems are complex and logistically challenging tasks. However, many simple rules based on CGs (e.g., medical logic modules), have been successfully implemented through a few popular DS models (e.g., prevention reminders, order entry systems). To facilitate mapping of CGs to practical DS models, we propose an empirical method for sub-dividing CGs into modules according to the locus in a clinical process flow model where implementation would be most effective (e.g., post-encounter provider order entry). We further propose a classification of triggers and objectives for CG modules that provides a framework for a DS system to implement the module Successful application of the method to ten diverse CGs in the outpatient setting is described.

Building an application framework for integrative genomics.

The accelerated pace of biological research and the current availability of whole-genome data sets provides significant new sources of functional insight. We designed an architecture and framework for software to query and explore such data in an orderly and iterative fashion. The architecture is intended to provide an extensible platform for developing web based bioinformatics applications and to offer a flexible and end-user-extensible software environment to explore and integrate disparate biological data sources. This will enable the user to explore existing relationships and discover new functional relationships among these data.

Applying axiomatic design methodology for guideline revision.

We are investigating the use of axiomatic design (AD) as a principled approach to the revision of guidelines. AD models guidelines in a modular and hierarchical manner and captures interactions be-tween modules. To test this approach we applied AD to encode segments of three guidelines and their revised versions. Guideline encodings for the original versions were modified to incorporate changes made in the revised documents. The results indicate that AD is a promising approach for guideline modeling.

GELLO: an object-oriented query and expression language for clinical decision support.

GELLO is a purpose-specific, object-oriented (OO) query and expression language. GELLO is the result of a concerted effort of the Decision Systems Group (DSG) working with the HL7 Clinical Decision Support Technical Committee (CDSTC) to provide the HL7 community with a common format for data encoding and manipulation. GELLO will soon be submitted for ballot to the HL7 CDSTC for consideration as a standard.

Coverage of patient safety terms in the UMLS metathesaurus.

The integration and large-scale analyses of medical error databases would be greatly facilitated by the use of a standard terminology. We investigated the availability in the UMLS metathesaurus of concepts that are required for coding patient safety data. Terms from three proprietary patient safety terminologies were mapped to the concepts in UMLS by an automated mapping program developed by us. From these candidate mappings, the concept that matched its corresponding term was selected manually. The reliability of the mapping procedure was verified by manually searching for terms in the UMLS Knowledge Source Server. Matching concepts in UMLS were identified for less than 27% of the terms in the study dataset. The matching rates of terms that describe the type of error and the causes of errors were even lower. The lack of such terms in the existing standard terminologies underscores the need for development of a standard patient safety terminology.

A meta-data model for knowledge in decision support systems.

Clinical decision support such as alerts, reminders and guidance are driven by rules often distributed among a variety of applications in a healthcare information system. Due to the increasing size of rule bases, there is a growing need to manage this dispersed knowledge in an integrated environment. A system for management of executable clinical knowledge such as rules should (1) assist in the development and maintenance of rules throughout the rules' life-cycles, (2) support search and retrieval of rules in the knowledge base (e.g., rules for diabetes, rules created by a particular individual), and (3) facilitate the analyses of rules in the knowledge base (e.g., identify rules not updated in the last year). In order to create such a clinical knowledge management system it is necessary to model the meta-data of rules. There have been efforts to document meta-data about rules within the Arden Syntax Medical Logical Modules' project. However, the maintenance and library categories in that project allow mainly free-text information about a rule. We have created a comprehensive meta-data structure and taxonomy for describing clinical rules that supports the features of a knowledge management system. We also tested this model using a representative set of rules.

Providing context-sensitive decision-support based on WHO guidelines.

A decision support system was developed implementing the WHO guideline for diarrhea management. The decision-support system is integrated into a medical records application on a handheld computer. The system will be used by primary health care workers in rural India. The guideline was encoded as a set of chained rules in CLIPS format. To enhance adherence to guidelines, we use a model based on a context-adapted guideline to provide decision support at the point of care in a particular setting. The purpose of the system is to tailor the recommendations based on the patient's condition and the local factors such as resource availability in order to create feasible uniformity in a practice across different providers of care.