Empirical formulation of a generic query set for clinical information retrieval systems.

Information needs in clinical practice take the form of specific questions about a given clinical situation, and are best satisfied by concise and specific information retrieval. We sought to develop a comprehensive set of generic queries for information retrieval from electronic medical information resources. We collected one hundred and ten real-world questions asked at the point of care in a variety of settings, and from these developed a set of generic queries of which each of the real-world queries could be shown to be a special case. To provide allowed values for each of the concept terms in the queries, we defined generic nouns as unions of UMLS semantic types, and specified which of these were appropriate to each query. We have begun to use the set to index reference texts from general and subspecialty medicine, and found it capable of full text indexing in the clinical domain. We hypothesize that the query set can serve as a basis for more specialized query sets, and that it will remain generalizable to other electronic medical resources, indexing tasks, and non-UMLS controlled vocabularies.

Integration of textual guideline documents with formal guideline knowledge bases.

Numerous approaches have been proposed to integrate the text of guideline documents with guideline-based care systems. Current approaches range from serving marked up guideline text documents to generating advisories using complex guideline knowledge bases. These approaches have integration problems mainly because they tend to rigidly link the knowledge base with text. We are developing a bridge approach that uses an information retrieval technology. The new approach facilitates a versatile decision-support system by using flexible links between the formal structures of the knowledge base and the natural language style of the guideline text.

Knowledge requirements for automated inference of medical textbook markup.

Indexing medical text in journals or textbooks requires a tremendous amount of resources. We tested two algorithms for automatically indexing nouns, noun-modifiers, and noun phrases, and inferring selected binary relations between UMLS concepts in a textbook of infectious disease. Sixty-six percent of nouns and noun-modifiers and 81% of noun phrases were correctly matched to UMLS concepts. Semantic relations were identified with 100% specificity and 94% sensitivity. For some medical sub-domains, these algorithms could permit expeditious generation of more complex indexing.

Automation and integration of components for generalized semantic markup of electronic medical texts.

Our group has built an information retrieval system based on a complex semantic markup of medical textbooks. We describe the construction of a set of web-based knowledge-acquisition tools that expedites the collection and maintenance of the concepts required for text markup and the search interface required for information retrieval from the marked text. In the text markup system, domain experts (DEs) identify sections of text that contain one or more elements from a finite set of concepts. End users can then query the text using a predefined set of questions, each of which identifies a subset of complementary concepts. The search process matches that subset of concepts to relevant points in the text. The current process requires that the DE invest significant time to generate the required concepts and questions. We propose a new system--called ACQUIRE (Acquisition of Concepts and Queries in an Integrated Retrieval Environment)--that assists a DE in two essential tasks in the text-markup process. First, it helps her to develop, edit, and maintain the concept model: the set of concepts with which she marks the text. Second, ACQUIRE helps her to develop a query model: the set of specific questions that end users can later use to search the marked text. The DE incorporates concepts from the concept model when she creates the questions in the query model. The major benefit of the ACQUIRE system is a reduction in the time and effort required for the text-markup process. We compared the process of concept- and query-model creation using ACQUIRE to the process used in previous work by rebuilding two existing models that we previously constructed manually. We observed a significant decrease in the time required to build and maintain the concept and query models.

MYCIN II: design and implementation of a therapy reference with complex content-based indexing.

We describe the construction of MYCIN II, a prototype system that provides for content-based markup and search of a forthcoming clinical therapeutics textbook, Antimicrobial Therapy and Vaccines. Existing commercial search technology for digital references utilizes generic tools such as textword-based searches with geographical or statistical refinements. We suggest that the drawbacks of such systems significantly restrict their use in everyday clinical practice. This is in spite of the fact that there is a great need for the information contained within these same references. The system we describe is intended to supplement keyword searching so that certain important questions can be asked easily and can be answered reliably (in terms of precision and recall). Our method attacks this problem in a restricted domain of knowledge-clinical infectious disease. For example, we would like to be able to answer the class of questions exemplified by the following query: "What antimicrobial agents can be used to treat endocarditis caused by Eikenella corrodens?" We have compiled and analyzed a list of such questions to develop a concept-based markup scheme. This scheme was then applied within an HTML markup to electronically "highlight" passages from three textbook chapters. We constructed a functioning web-based search interface. Our system also provides semi-automated querying of PubMed using our concept markup and the user's actions as a guide.

Automated planning target volume generation: an evaluation pitting a computer-based tool against human experts.

PURPOSE: Software tools are seeing increased use in three-dimensional treatment planning. However, the development of these tools frequently omits careful evaluation before placing them in clinical use. This study demonstrates the application of a rigorous evaluation methodology using blinded peer review to an automated software tool that produces ICRU-50 planning target volumes (PTVs). METHODS AND MATERIALS: Seven physicians from three different institutions involved in three-dimensional treatment planning participated in the evaluation. Four physicians drew partial PTVs on nine test cases, consisting of four nasopharynx and five lung primaries. Using the same information provided to the human experts, the computer tool generated PTVs for comparison. The remaining three physicians, designated evaluators, individually reviewed the PTVs for acceptability. To exclude bias, the evaluators were blinded to the source (human or computer) of the PTVs they reviewed. Their scorings of the PTVs were statistically examined to determine if the computer tool performed as well as the human experts. RESULTS: The computer tool was as successful as the human experts in generating PTVs. Failures were primarily attributable to insufficient margins around the clinical target volume and to encroachment upon critical structures. In a qualitative analysis, the human and computer experts displayed similar types and distributions of errors. CONCLUSIONS: Rigorous evaluation of computer-based radiotherapy tools requires comparison to current practice and can reveal areas for improvement before the tool enters clinical practice.

Lightweight, mobile E-mail for intra-clinic communication.

We have developed a mobile messaging system designed for use in the clinic setting. The system is designed to facilitate quick, informal, interactions that occur in a clinical setting, e.g., requests for assistance or information. The system includes safeguards to make sure that the sender of a message is aware if a message is not read in a timely fashion. Evaluation of the system shows message delivery was about 50% slower than our target of 30 seconds. Although the mobile device used is fairly small when combined with a radio unit, it is too bulky and users did not necessarily carry the system with them. This led to delays (over eleven minutes on average) before messages were seen. We expect that improvements in hardware and clinical software will lead to more common use of such adjunct software systems.

The PEN-Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine.

OBJECTIVE: To explore different user-interface designs for structured progress note entry, with a long-term goal of developing design guidelines for user interfaces where users select items from large medical vocabularies. DESIGN: The authors created eight different prototypes of a pen-based progress-note-writing system called PEN-Ivory. Each prototype allows physicians to write patient progress notes using simple pen-based gestures such as circle, line-out, and scratch-out. The result of an interaction with PEN-Ivory is a progress note in English prose. The eight prototypes were designed in a principled way, so that they differ from one another in just one of three different user-interface characteristics. MEASUREMENTS: Five of the eight prototypes were tested by measuring the time it took 15 users, each using a distinct prototype, to document three patient cases consisting of a total of 63 medical findings. RESULTS: The prototype that allowed the fastest data entry had the following three user-interface characteristics: it used a paging rather than a scrolling form, it used a fixed palette of modifiers rather than a dynamic "pop-up" palette, and it made available all findings from the controlled vocabulary at once rather than displaying only a subset of findings generated by analyzing the patient's problem list. CONCLUSION: Even simple design changes to a user interface can make dramatic differences in user performance. The authors discuss possible influences on performance, such as positional constancy, user uncertainty and system anticipation, that may contribute significantly to the effectiveness of systems that display menus of items from large controlled vocabularies of medicine.

Toward reusable software components at the point of care.

An architecture built from five software components -a Router, Parser, Matcher, Mapper, and Server -fulfills key requirements common to several point-of-care information and knowledge processing tasks. The requirements include problem-list creation, exploiting the contents of the Electronic Medical Record for the patient at hand, knowledge access, and support for semantic visualization and software agents. The components use the National Library of Medicine Unified Medical Language System to create and exploit lexical closure-a state in which terms, text and reference models are represented explicitly and consistently. Preliminary versions of the components are in use in an oncology knowledge server.

Development of a controlled medical terminology: knowledge acquisition and knowledge representation.

The creation of controlled medical terminologies is a central challenge in the development of electronic patient records. In the T-Helper patient-record system, designed for the care of patients with HIV disease, the IVORY module allows health-care workers to compose textual progress notes by making selections from menus generated automatically from a controlled medical terminology. Construction of this IVORY terminology required extensive design sessions with a team of computer scientists and an expert physician. Refinement of the terminology was only possible when the design team could envision how the completed T-Helper system would be used in the context of clinical practice. Development of controlled medical terminologies is a significant problem in knowledge acquisition. Techniques used to acquire and represent clinical concepts for the purpose of building decision-support systems also are appropriate for the construction of controlled terminologies such as the one in T-Helper.

A continuous-speech interface to a decision support system: I. Techniques to accommodate for misrecognized input.

OBJECTIVE: Develop a continuous-speech interface that allows flexible input of clinical findings into a medical diagnostic application. DESIGN: The authors' program allows users to enter clinical findings using their own vernacular. It displays from the diagnostic program's controlled vocabulary a list of terms that most closely matches the input, and allows the user to select the single best term. The interface program includes two components: a speech-recognition component that converts utterances into text strings, and a language-processing component that matches recognized text strings with controlled-vocabulary terms. The speech-recognition component is composed of commercially available speech-recognition hardware and software, and developer-created grammars, which specify the language to be recognized. The language-processing component is composed of a translator, which extracts a canonical form from both recognized text strings and controlled-vocabulary terms, and a matcher, which measures the similarity between the two canonical forms. RESULTS: The authors discovered that grammars constructed by a physician, who could anticipate how users might speak findings, supported speech recognition better than did grammars constructed programmatically from the controlled vocabulary. However, this programmatic method of grammar construction was more time efficient and better supported long-term maintenance of the grammars. The authors also found that language-processing techniques recovered some of the information lost due to speech misrecognition, but were dependent on the completeness of supporting synonym dictionaries. CONCLUSIONS: The authors' program demonstrated the feasibility of using continuous speech to enter findings into a medical application. However, improvements in speech-recognition technology and language-processing techniques are needed before natural continuous speech becomes an acceptable input modality for clinical applications.

A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm.

OBJECTIVE: Evaluate the performance of a continuous-speech interface to a decision support system. DESIGN: The authors performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). The performance of the speech interface was assessed in two stages: in the real-time experiment, physician subjects viewed audiovisual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, the authors replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the "gold standard." MEASUREMENTS: The authors measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). RESULTS: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. CONCLUSIONS: Reasonable semantic accuracy was attained when language-processing techniques were used to accommodate for speech misrecognition. In addition, the Wizard-of-Oz experimental design offered many advantages for this evaluation. The authors believe that this technique may be useful to future evaluators of speech-input systems.

Semantic visualization of oncology knowledge sources.

Visualization of knowledge sources can have a substantial impact on the use of such sources at the point of care. This is because barriers to use at the point of care include hours required to master the electronic interfaces to those sources, and minutes required to master the electronic interfaces to those sources, and minutes required to accomplish any one retrieval. For a system to be used regularly at the point of care, therefore, it must be intuitive and fast. This paper presents a three dimensional interface to oncology knowledge sources that aims to meet this challenge.

Evaluation of an expert system producing geometric solids as output.

This paper reports the evaluation of an expert system whose output is a three-dimensional geometric solid. Evaluating such an output emphasizes the problems of establishing a comparison standard, and of identifying and classifying deviations from that standard. Our evaluation design used a panel of physicians for the first task and a separate panel of expert judges for the second. We found that multi-parameter or multi-dimensional expert system outputs, such as this one, may result in lower overall performance scores and increased variation in acceptability to different physicians. We surmise that these effects are a consequence of the higher number of factors which may be deemed unacceptable. The effects appear, however, to be equal for computer and human output. This evaluation design is thus applicable to other expert systems producing similarly complex output.

The development of a controlled medical terminology: identification, collaboration, and customization.

An increasing focus in health care is the development and use of electronic medical record systems to capture and store patient information. T-HELPER is an electronic medical record system that health care providers use to record ambulatory-care patient progress notes. These data are stored in an on-line database and analyzed by T-HELPER to provide users with decision support regarding patient eligibility for clinical trial protocols and assistance with continued protocol-based care. Our goal is to provide a system that enhances the process of identifying patients who are potentially eligible for clinical trials of experimental therapies in a clinic that is limited by the existence of a singular clinical trial coordinator. Effective implementation of such a system requires the development of a meaningful controlled medical terminology that satisfies the needs of a diverse community of providers all of who contribute to the health care process. The development of a controlled medical terminology is a process of identification, collaboration, and customization. We enlisted the help of collaborators familiar with the proposed work environment to identify user needs, to collaborate with our development team to construct the preliminary terminology, and to customize the controlled medical terminology to make it meaningful and acceptable to the clinic users.

Accessing oncology information at the point of care: experience using speech, pen, and 3-D interfaces with a knowledge server.

Oncologists' information needs arise at diverse times and settings. For example: "Is superior vena cava syndrome a medical emergency?" Our collaborative group is developing a system that supports an interface with combinations of spoken, gestural, and simulated three-dimensional manipulation to help an oncologist focus on the information need, not the system. The system requires a small amount of input from the oncologist, and then anticipates what information is pertinent to the patient at hand, based on the sources it has available. The system makes use of a "Knowledge Server" to find relevant information. The Knowledge Server uses selected data for the particular patient from a Computer-based Patient Record (CPR) to provide context for the information needs. The Knowledge Server leverages the Unified Medical Language System (UMLS) resources as well as relevant communications standards. A layered, interaction protocol is used to help manage the fulfillment of information needs. Each of the oncology knowledge sources is transformed into a uniform representation that utilizes both its formal schema (e.g., its table of contents) and its concepts and words indexed through the UMLS Metathesaurus. Our focus on the appropriate use of information from a CPR, and on anticipating oncologists' information needs, resulted from our study of several longitudinal patient scenarios. We believe that our use of scenario-based design techniques will help to ensure the system's success.

PEN-Ivory: the design and evaluation of a pen-based computer system for structured data entry.

PEN-Ivory is a pen-based computer system that uses structured data entry for creating patient progress notes. Users make simple gestures such as circles, lines, and scratch-outs to enter medical findings from a controlled vocabulary. The result of an interaction with PEN-Ivory is a computer-generated patient progress note in English prose. We designed PEN-Ivory's user interface in a principled way. We first created multiple working prototypes, each differing in one of three user-interface characteristics. Then we empirically evaluated the prototypes in a controlled, experimental setting for their efficiencies in enabling users to create patient progress notes. The prototype that allowed the fastest data entry had the following three user-interface characteristics: it used a paging form, used a fixed palette of modifiers, and made available all findings from the controlled vocabulary at once.

The design and implementation of a ventilator-management advisor.

VentPlan is an implementation of the architecture developed by the qualitative-quantitative (QQ) research group for combining qualitative and quantitative computation in a ventilator-management advisor (VMA). VentPlan calculates recommended settings for four controls of a ventilator by evaluating the predicted effects of alternative ventilator settings. A belief network converts clinical diagnoses to distributions on physiologic parameters. A mathematical-modeling module applies a patient-specific mathematical model of cardiopulmonary physiology to predict the effects of alternative ventilator settings. A decision-theoretic plan evaluator ranks the predicted effects of alternative ventilator settings according to a multiattribute-value model that specifies physician preferences for ventilator treatments. Our architecture allows VentPlan to interpret quantitative observations in light of the clinical context (such as the clinical diagnosis). We report a retrospective study of the ventilator-setting changes encountered in postoperative patients in a surgical intensive-care unit (ICU). We conclude that the QQ architecture allows VentPlan to apply a patient-specific physiologic model to calculate ventilator settings that are optimal with respect to a decision-theoretic value model describing physician preferences for setting the ventilator.

Graphical access to medical expert systems: IV. Experiments to determine the role of spoken input.

The goal of our research is to design improved interfaces for medical expert systems. Previously, the use of graphical techniques was explored to improve the acceptance by clinicians of the user interface. Now that devices that accept spoken input are available, we wish to design interfaces that take advantage of this potentially more natural modality for interaction. To understand how clinicians might want to speak to a medical decision-support system, we carried out an experiment that simulated the availability of a spoken interface to the ONCOCIN medical expert system. ONCOCIN provides therapy advice for patients on complex cancer therapy protocols based on a description of the patient's current medical status and laboratory-test values. In the experiment, we had oncologists present a clinical case while observing the ONCOCIN flowsheet display. A project member listened to the presentation and filled in values for the flowsheet, as well as introducing purposeful misunderstandings of the input. The results suggest that each individual developed a stereotypical grammar for communicating with the program. Our experience with the purposeful miscommunications suggests particular ways to tailor requests for repetition based on the part of the utterance that was not understood.