Automated planning target volume generation: an evaluation pitting a computer-based tool against human experts.

PURPOSE: Software tools are seeing increased use in three-dimensional treatment planning. However, the development of these tools frequently omits careful evaluation before placing them in clinical use. This study demonstrates the application of a rigorous evaluation methodology using blinded peer review to an automated software tool that produces ICRU-50 planning target volumes (PTVs). METHODS AND MATERIALS: Seven physicians from three different institutions involved in three-dimensional treatment planning participated in the evaluation. Four physicians drew partial PTVs on nine test cases, consisting of four nasopharynx and five lung primaries. Using the same information provided to the human experts, the computer tool generated PTVs for comparison. The remaining three physicians, designated evaluators, individually reviewed the PTVs for acceptability. To exclude bias, the evaluators were blinded to the source (human or computer) of the PTVs they reviewed. Their scorings of the PTVs were statistically examined to determine if the computer tool performed as well as the human experts. RESULTS: The computer tool was as successful as the human experts in generating PTVs. Failures were primarily attributable to insufficient margins around the clinical target volume and to encroachment upon critical structures. In a qualitative analysis, the human and computer experts displayed similar types and distributions of errors. CONCLUSIONS: Rigorous evaluation of computer-based radiotherapy tools requires comparison to current practice and can reveal areas for improvement before the tool enters clinical practice.

The PEN-Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine.

OBJECTIVE: To explore different user-interface designs for structured progress note entry, with a long-term goal of developing design guidelines for user interfaces where users select items from large medical vocabularies. DESIGN: The authors created eight different prototypes of a pen-based progress-note-writing system called PEN-Ivory. Each prototype allows physicians to write patient progress notes using simple pen-based gestures such as circle, line-out, and scratch-out. The result of an interaction with PEN-Ivory is a progress note in English prose. The eight prototypes were designed in a principled way, so that they differ from one another in just one of three different user-interface characteristics. MEASUREMENTS: Five of the eight prototypes were tested by measuring the time it took 15 users, each using a distinct prototype, to document three patient cases consisting of a total of 63 medical findings. RESULTS: The prototype that allowed the fastest data entry had the following three user-interface characteristics: it used a paging rather than a scrolling form, it used a fixed palette of modifiers rather than a dynamic "pop-up" palette, and it made available all findings from the controlled vocabulary at once rather than displaying only a subset of findings generated by analyzing the patient's problem list. CONCLUSION: Even simple design changes to a user interface can make dramatic differences in user performance. The authors discuss possible influences on performance, such as positional constancy, user uncertainty and system anticipation, that may contribute significantly to the effectiveness of systems that display menus of items from large controlled vocabularies of medicine.

A continuous-speech interface to a decision support system: I. Techniques to accommodate for misrecognized input.

OBJECTIVE: Develop a continuous-speech interface that allows flexible input of clinical findings into a medical diagnostic application. DESIGN: The authors' program allows users to enter clinical findings using their own vernacular. It displays from the diagnostic program's controlled vocabulary a list of terms that most closely matches the input, and allows the user to select the single best term. The interface program includes two components: a speech-recognition component that converts utterances into text strings, and a language-processing component that matches recognized text strings with controlled-vocabulary terms. The speech-recognition component is composed of commercially available speech-recognition hardware and software, and developer-created grammars, which specify the language to be recognized. The language-processing component is composed of a translator, which extracts a canonical form from both recognized text strings and controlled-vocabulary terms, and a matcher, which measures the similarity between the two canonical forms. RESULTS: The authors discovered that grammars constructed by a physician, who could anticipate how users might speak findings, supported speech recognition better than did grammars constructed programmatically from the controlled vocabulary. However, this programmatic method of grammar construction was more time efficient and better supported long-term maintenance of the grammars. The authors also found that language-processing techniques recovered some of the information lost due to speech misrecognition, but were dependent on the completeness of supporting synonym dictionaries. CONCLUSIONS: The authors' program demonstrated the feasibility of using continuous speech to enter findings into a medical application. However, improvements in speech-recognition technology and language-processing techniques are needed before natural continuous speech becomes an acceptable input modality for clinical applications.

A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm.

OBJECTIVE: Evaluate the performance of a continuous-speech interface to a decision support system. DESIGN: The authors performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). The performance of the speech interface was assessed in two stages: in the real-time experiment, physician subjects viewed audiovisual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, the authors replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the "gold standard." MEASUREMENTS: The authors measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). RESULTS: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. CONCLUSIONS: Reasonable semantic accuracy was attained when language-processing techniques were used to accommodate for speech misrecognition. In addition, the Wizard-of-Oz experimental design offered many advantages for this evaluation. The authors believe that this technique may be useful to future evaluators of speech-input systems.

A methodology for generating computer-based explanations of decision-theoretic advice.

Decision analysis is an appealing methodology with which to provide decision support to the practicing physician. However, its use in the clinical setting is impeded because computer-based explanations of decision-theoretic advice are difficult to generate without resorting to mathematical arguments. Nevertheless, human decision analysts generate useful and intuitive explanations based on decision trees. To facilitate the use of decision theory in a computer-based decision support system, the authors developed a computer program that uses symbolic reasoning techniques to generate nonquantitative explanations of the results of decision analyses. A combined approach has been implemented to explain the differences in expected utility among branches of a decision tree. First, the mathematical relationships inherent in the structure of the tree are used to find any asymmetries in tree structure or inequalities among analogous decision variables that are responsible for a difference in expected utility. Next, an explanation technique is selected and applied to the most significant variables, creating a symbolic expression that justifies the decision. Finally, the symbolic expression is converted to English-language text, thereby generating an explanation that justifies the desirability of the choice with the greater expected utility. The explanation does not refer to mathematical formulas, nor does it include probability or utility values. The results suggest that explanations produced by a combination of decision analysis and symbolic processing techniques may be more persuasive and acceptable to clinicians than those produced by either technique alone.

The design and implementation of a ventilator-management advisor.

VentPlan is an implementation of the architecture developed by the qualitative-quantitative (QQ) research group for combining qualitative and quantitative computation in a ventilator-management advisor (VMA). VentPlan calculates recommended settings for four controls of a ventilator by evaluating the predicted effects of alternative ventilator settings. A belief network converts clinical diagnoses to distributions on physiologic parameters. A mathematical-modeling module applies a patient-specific mathematical model of cardiopulmonary physiology to predict the effects of alternative ventilator settings. A decision-theoretic plan evaluator ranks the predicted effects of alternative ventilator settings according to a multiattribute-value model that specifies physician preferences for ventilator treatments. Our architecture allows VentPlan to interpret quantitative observations in light of the clinical context (such as the clinical diagnosis). We report a retrospective study of the ventilator-setting changes encountered in postoperative patients in a surgical intensive-care unit (ICU). We conclude that the QQ architecture allows VentPlan to apply a patient-specific physiologic model to calculate ventilator settings that are optimal with respect to a decision-theoretic value model describing physician preferences for setting the ventilator.

Extensions to the time-oriented database model to support temporal reasoning in medical expert systems.

Physicians faced with diagnostic and therapeutic decisions must reason about clinical features that change over time. Database-management systems (DBMS) can increase access to patient data, but most systems are limited in their ability to store and retrieve complex temporal information. The Time-Oriented Databank (TOD) model, the most widely used data model for medical database systems, associates a single time stamp with each observation. The proper analysis of most clinical data requires accounting for multiple concurrent clinical events that may alter the interpretation of the raw data. Most medical DBMSs cannot retrieve patient data indexed by multiple clinical events. We describe two logical extensions to TOD-based databases that solve a set of temporal reasoning problems we encountered in constructing medical expert systems. A key feature of both extensions is that stored data are partitioned into groupings, such as sequential clinical visits, clinical exacerbations, or other abstract events that have clinical decision-making relevance. The temporal network (TNET) is an object-oriented database that extends the temporal reasoning capabilities of ONCOCIN, a medical expert system that provides chemotherapy advice. TNET uses persistent objects to associate observations with intervals of time during which "an event of clinical interest" occurred. A second object-oriented system called the extended temporal network (ETNET), is both an extension and a simplification of TNET. Like TNET, ETNET uses persistent objects to represent relevant intervals; unlike the first system, however, ETNET contains reasoning methods (rules) that can be executed when an event "begins", and that are withdrawn when that event "concludes". TNET and ETNET capture temporal relationships among recorded information that are not represented in TOD-based databases. Although they do not solve all temporal reasoning problems found in medical decision making, these new structures enable patient database systems to encode complex temporal relationships, to store and retrieve patient data based on multiple clinical contexts and, in ETNET, to modify the reasoning methods available to an expert system based on the onset or conclusion of specific clinical events.

Validating the knowledge base of a therapy planning system.

Validation of expert system knowledge bases has proved to be difficult. This paper presents a description of a system called ScriptGen that generates test data for validating the knowledge base of the ONCOCIN cancer therapy planning system. Because of the size and complexity of the ONCOCIN knowledge base, we require tools for automated validation. ScriptGen, which applies techniques developed in testing both traditional software and expert systems, uses a parallel model of the ONCOCIN knowledge base and its own inference engine to generate test cases. We derived the limits of the system from a study that seeded errors into an existing knowledge base.

Combining physiologic models and symbolic methods to interpret time-varying patient data.

This paper describes a methodology for representing and using medical knowledge about temporal relationships to infer the presence of clinical events that evolve over time. The methodology consists of three steps: (1) the incorporation of patient observations into a generic physiologic model, (2) the conversion of model states and predictions into domain-specific temporal abstractions, and (3) the transformation of temporal abstractions into clinically meaningful descriptive text. The first step converts raw observations to underlying model concepts, the second step identifies temporal features of the fitted model that have clinical interest, and the third step replaces features represented by model parameters and predictions into concepts expressed in clinical language. We describe a program, called TOPAZ, that uses this three-step methodology. TOPAZ generates a narrative summary of the temporal events found in the electronic medical record of patients receiving cancer chemotherapy. A unique feature of TOPAZ is its use of numeric and symbolic techniques to perform different temporal reasoning tasks. Time is represented both as a continuous process and as a set of temporal intervals. These two temporal models differ in the temporal ontology they assume and in the temporal concepts they encode. Without multiple temporal models, this diversity of temporal knowledge could not be represented.

Development of a controlled medical terminology: knowledge acquisition and knowledge representation.

The creation of controlled medical terminologies is a central challenge in the development of electronic patient records. In the T-Helper patient-record system, designed for the care of patients with HIV disease, the IVORY module allows health-care workers to compose textual progress notes by making selections from menus generated automatically from a controlled medical terminology. Construction of this IVORY terminology required extensive design sessions with a team of computer scientists and an expert physician. Refinement of the terminology was only possible when the design team could envision how the completed T-Helper system would be used in the context of clinical practice. Development of controlled medical terminologies is a significant problem in knowledge acquisition. Techniques used to acquire and represent clinical concepts for the purpose of building decision-support systems also are appropriate for the construction of controlled terminologies such as the one in T-Helper.

Graphical access to medical expert systems: IV. Experiments to determine the role of spoken input.

The goal of our research is to design improved interfaces for medical expert systems. Previously, the use of graphical techniques was explored to improve the acceptance by clinicians of the user interface. Now that devices that accept spoken input are available, we wish to design interfaces that take advantage of this potentially more natural modality for interaction. To understand how clinicians might want to speak to a medical decision-support system, we carried out an experiment that simulated the availability of a spoken interface to the ONCOCIN medical expert system. ONCOCIN provides therapy advice for patients on complex cancer therapy protocols based on a description of the patient's current medical status and laboratory-test values. In the experiment, we had oncologists present a clinical case while observing the ONCOCIN flowsheet display. A project member listened to the presentation and filled in values for the flowsheet, as well as introducing purposeful misunderstandings of the input. The results suggest that each individual developed a stereotypical grammar for communicating with the program. Our experience with the purposeful miscommunications suggests particular ways to tailor requests for repetition based on the part of the utterance that was not understood.

Graphical access to medical expert systems: V. Integration with continuous-speech recognition.

This paper describes three prototypes of computer-based clinical record-keeping tools that use a combination of window-based graphics and continuous speech in their user interfaces. Although many of today's commercial speech-recognition products achieve high rates of accuracy for large grammars (vocabularies of words or collections of sentences and phrases), they can only "listen for" (and therefore recognize) a limited number of words or phrases at a time. When a speech application requires a grammar whose size exceeds a speech-recognition product's limits, the application designer must partition the large grammar into several smaller ones and develop control mechanisms that permit users to select the grammar that contains the words or phrases they wish to utter. Furthermore, the user interfaces they design must provide feedback mechanisms that show users the scope of the selected grammars. The three prototypes described were designed to explore the use of window-based graphics as control and feedback mechanisms for continuous-speech recognition in medical applications. Our experiments indicate that window-based graphics can be effectively used to provide control and feedback for certain classes of speech applications, but they suggest that the techniques we describe will not suffice for applications whose grammars are very complex.

Knowledge engineering for a clinical trial advice system: uncovering errors in protocol specification.

ONCOCIN is an expert system that provides advice to physicians who are treating cancer patients enrolled in clinical trials. The process of encoding oncology protocol knowledge for the system has revealed serious omissions and unintentional ambiguities in the protocol documents. We have also discovered that many protocols allow for significant latitude in treating patients and that even when protocol guidelines are explicit, physicians often choose to apply their own judgment on the assumption that the specifications are incomplete. Computer-based tools offer the possibility of insuring completeness and reproducibility in the definition of new protocols. One goal of our automated protocol authoring environment, called OPAL, is to help physicians develop protocols that are free of ambiguity and thus to assure better compliance and standardization of care.

Empirical formulation of a generic query set for clinical information retrieval systems.

Information needs in clinical practice take the form of specific questions about a given clinical situation, and are best satisfied by concise and specific information retrieval. We sought to develop a comprehensive set of generic queries for information retrieval from electronic medical information resources. We collected one hundred and ten real-world questions asked at the point of care in a variety of settings, and from these developed a set of generic queries of which each of the real-world queries could be shown to be a special case. To provide allowed values for each of the concept terms in the queries, we defined generic nouns as unions of UMLS semantic types, and specified which of these were appropriate to each query. We have begun to use the set to index reference texts from general and subspecialty medicine, and found it capable of full text indexing in the clinical domain. We hypothesize that the query set can serve as a basis for more specialized query sets, and that it will remain generalizable to other electronic medical resources, indexing tasks, and non-UMLS controlled vocabularies.

PEN-Ivory: the design and evaluation of a pen-based computer system for structured data entry.

PEN-Ivory is a pen-based computer system that uses structured data entry for creating patient progress notes. Users make simple gestures such as circles, lines, and scratch-outs to enter medical findings from a controlled vocabulary. The result of an interaction with PEN-Ivory is a computer-generated patient progress note in English prose. We designed PEN-Ivory's user interface in a principled way. We first created multiple working prototypes, each differing in one of three user-interface characteristics. Then we empirically evaluated the prototypes in a controlled, experimental setting for their efficiencies in enabling users to create patient progress notes. The prototype that allowed the fastest data entry had the following three user-interface characteristics: it used a paging form, used a fixed palette of modifiers, and made available all findings from the controlled vocabulary at once.

Design and evaluation of multimedia stimuli to evoke clinical concepts.

Continuous speech recognition systems have the potential to facilitate clinical data entry, but evaluating them rigorously is difficult. We describe a tool to aid evaluators of such systems. The tool is a HyperCard stack with stimuli consisting of pictures, sounds and the minimum of words to evoke 20 QMR physical findings. Despite using up to four different stimuli to communicate each finding and piloting the material on six subjects, eight test subjects made a total of 66 errors (42%) in interpreting the 20 sets of stimuli, of which 22 errors (14%) were serious. These results are relevant to those designing interfaces for decision-support, tutorial and student testing systems.

TQuery: a context-sensitive temporal query language.

Users of electronic medical databases request pertinent information by recasting their clinical questions into a formal database query language. Because the query language is the user's only access to the data, the query language must be powerful enough to enable users to express their data requirements. However, a competing need is for the query language to be restrictive enough so that queries can have unambiguous semantics and the query processor can generate correct answers. We describe a query language, called TQuery , that was designed specifically to formulate database queries that are dependent on temporal and contextual relationships. TQuery specifications express contextual constraints without the need to explicitly reference calendar dates. TQuery is the database query language used to retrieve patient data from an object-oriented electronic patient medical-record system called the temporal network (TNET). TNET and TQuery were developed to support the real-time temporal reasoning and representation needs of a LISP workstation-based medical expert system.