Watson for Oncology and breast cancer treatment recommendations: agreement with an expert multidisciplinary tumor board.

Background: Breast cancer oncologists are challenged to personalize care with rapidly changing scientific evidence, drug approvals, and treatment guidelines. Artificial intelligence (AI) clinical decision-support systems (CDSSs) have the potential to help address this challenge. We report here the results of examining the level of agreement (concordance) between treatment recommendations made by the AI CDSS Watson for Oncology (WFO) and a multidisciplinary tumor board for breast cancer. Patients and methods: Treatment recommendations were provided for 638 breast cancers between 2014 and 2016 at the Manipal Comprehensive Cancer Center, Bengaluru, India. WFO provided treatment recommendations for the identical cases in 2016. A blinded second review was carried out by the center's tumor board in 2016 for all cases in which there was not agreement, to account for treatments and guidelines not available before 2016. Treatment recommendations were considered concordant if the tumor board recommendations were designated 'recommended' or 'for consideration' by WFO. Results: Treatment concordance between WFO and the multidisciplinary tumor board occurred in 93% of breast cancer cases. Subgroup analysis found that patients with stage I or IV disease were less likely to be concordant than patients with stage II or III disease. Increasing age was found to have a major impact on concordance. Concordance declined significantly (P ≤ 0.02; P < 0.001) in all age groups compared with patients <45 years of age, except for the age group 55-64 years. Receptor status was not found to affect concordance. Conclusion: Treatment recommendations made by WFO and the tumor board were highly concordant for breast cancer cases examined. Breast cancer stage and patient age had significant influence on concordance, while receptor status alone did not. This study demonstrates that the AI clinical decision-support system WFO may be a helpful tool for breast cancer treatment decision making, especially at centers where expert breast cancer resources are limited.

Expert Systems Research.

Artificial intelligence, long a topic of basic computer science research, is now being applied to problems of scientific, technical, and commercial interest. Some consultation programs, although limited in versatility, have achieved levels of performance rivaling those of human experts. A collateral benefit of this work is the systematization of previously unformalized knowledge in areas such as medical diagnosis and geology.

On exemplary scientific conduct regarding submission of manuscripts to biomedical informatics journals.

As the Editors of leading international biomedical informatics journals, the authors report on a recent pattern of improper manuscript submissions to journals in our field. As a guide for future authors, we describe ethical and pragmatic issues related to submitting work for peer-reviewed journal publication. We propose a coordinated approach to the problem that our respective journals will follow. This Editorial is being jointly published in the following journals represented by the authors: Computer Methods and Programs in Biomedicine, International Journal of Medical Informatics, Journal of Biomedical Informatics, Journal of the American Medical Informatics Association, and Methods of Information in Medicine.

Improvements in data collection through physician use of a computer-based chemotherapy treatment consultant.

The impact of a computer-based data management system on the completeness of clinical trial data was studied before and after the system's introduction in an oncology clinic. Physicians use the system, termed ONCOCIN, to record data during patient visits and to receive advice about treatment and tests required by experimental cancer protocols. Although ONCOCIN does not force the user to enter all data expected by the protocol, after its introduction there was improvement in the recording frequency of such data. The percentage of expected physical findings recorded increased from 74% to 91% (P less than .05), toxicity history from less than 1% to 45% (P less than .01), general chemistry results from 36% to 82% (P less than .01), x-ray results from 44% to 73% (P less than .01), and physicians' assessments of overall disease activity and Karnofsky performance status from 73% to 91% (P less than .05). Analysis of the steps in data collection and their contribution to loss of data suggests that observations or test ordering which are dependent on the physician are most improved by the system. Furthermore, analysis of post-ONCOCIN visits when the system was unavailable suggests that the recording of physician-dependent data (physical findings and assessments of disease activity and performance status) is likely to revert to pre-ONCOCIN levels if the system is not used routinely. The results show that ONCOCIN can greatly enhance recovery of those data expected for chemotherapy protocol patients. The program's interaction with the physician is central to its effectiveness in data collection, especially for data that arise directly from the patient-physician encounter.

The PEN-Ivory project: exploring user-interface design for the selection of items from large controlled vocabularies of medicine.

OBJECTIVE: To explore different user-interface designs for structured progress note entry, with a long-term goal of developing design guidelines for user interfaces where users select items from large medical vocabularies. DESIGN: The authors created eight different prototypes of a pen-based progress-note-writing system called PEN-Ivory. Each prototype allows physicians to write patient progress notes using simple pen-based gestures such as circle, line-out, and scratch-out. The result of an interaction with PEN-Ivory is a progress note in English prose. The eight prototypes were designed in a principled way, so that they differ from one another in just one of three different user-interface characteristics. MEASUREMENTS: Five of the eight prototypes were tested by measuring the time it took 15 users, each using a distinct prototype, to document three patient cases consisting of a total of 63 medical findings. RESULTS: The prototype that allowed the fastest data entry had the following three user-interface characteristics: it used a paging rather than a scrolling form, it used a fixed palette of modifiers rather than a dynamic "pop-up" palette, and it made available all findings from the controlled vocabulary at once rather than displaying only a subset of findings generated by analyzing the patient's problem list. CONCLUSION: Even simple design changes to a user interface can make dramatic differences in user performance. The authors discuss possible influences on performance, such as positional constancy, user uncertainty and system anticipation, that may contribute significantly to the effectiveness of systems that display menus of items from large controlled vocabularies of medicine.

The Unified Medical Language System: toward a collaborative approach for solving terminologic problems.

The approach taken by the Unified Medical Language System (UMLS), in which disparate terminology systems are integrated, has allowed construction of an electronic thesaurus (the Metathesaurus) that avoids imposing any restrictions upon the content, structure, or semantics of the source terminologies. As such, the UMLS has served as a unifying paradigm by providing appropriate links among equivalent entities that are used in different contexts or for different purposes. It accordingly provides a vehicle through which possibly orthogonal semantic models can co-exist within a single framework. This framework provides a model for the collaborative evolution of biomedical terminology and allows a synergistic relationship between the UMLS and its source terminology systems.

Are medical informatics and nursing informatics distinct disciplines? The 1999 ACMI debate.

The 1999 debate of the American College of Medical Informatics focused on the proposition that medical informatics and nursing informatics are distinctive disciplines that require their own core curricula, training programs, and professional identities. Proponents of this position emphasized that informatics training, technology applications, and professional identities are closely tied to the activities of the health professionals they serve and that, as nursing and medicine differ, so do the corresponding efforts in information science and technology. Opponents of the proposition asserted that informatics is built on a re-usable and widely applicable set of methods that are common to all health science disciplines, and that "medical informatics" continues to be a useful name for a composite core discipline that should be studied by all students, regardless of their health profession orientation.

Representing thoughts, words, and things in the UMLS.

The authors describe a framework, based on the Ogden-Richards semiotic triangle, for understanding the relationship between the Unified Medical Language System (UMLS) and the source terminologies from which the UMLS derives its content. They pay particular attention to UMLS's Concept Unique Identifier (CUI) and the sense of "meaning" it represents as contrasted with the sense of "meaning" represented by the source terminologies. The CUI takes on emergent meaning through linkage to terms in different terminology systems. In some cases, a CUI's emergent meaning can differ significantly from the original sources' intended meanings of terms linked by that CUI. Identification of these different senses of meaning within the UMLS is consistent with historical themes of semantic interpretation of language. Examination of the UMLS within such a historical framework makes it possible to better understand the strengths and limitations of the UMLS approach for integrating disparate terminologic systems and to provide a model, or theoretic foundation, for evaluating the UMLS as a Possible World--that is, as a mathematical formalism that represents propositions about some perspective or interpretation of the physical world.

Representing clinical guidelines in GLIF: individual and collaborative expertise.

OBJECTIVE: An evaluation of the cognitive processes used in the translation of a clinical guideline from text into an encoded form so that it can be shared among medical institutions. DESIGN: A comparative study at three sites regarding the generation of individual and collaborative representations of a guideline for the management of encephalopathy using the GuideLine Interchange Format (GLIF) developed by members of the InterMed Collaboratory. MEASUREMENTS: Using theories and methods of cognitive science, the study involves a detailed analysis of the cognitive processes used in generating representations in GLIF. The resulting process-outcome measures are used to compare subjects with various types of computer science or clinical expertise and from different institutions. RESULTS: Consistent with prior studies of text comprehension and expertise, the variability in strategies was found to be dependent on the degree of prior experience and knowledge of the domain. Differing both in content and structure, the representations developed by physicians were found to have additional information and organization not explicitly stated in the guidelines, reflecting the physicians' understanding of the underlying pathophysiology. The computer scientists developed more literal representations of the guidelines; addition were mostly limited to specifications mandated by the logic of GLIF itself. Collaboration between physicians and computer scientists resulted in consistent representations that were more than the sum of the separate parts, in that both domain-specific knowledge of medicine and generic knowledge of guideline structure were seamlessly integrated. CONCLUSION: Because of the variable construction of guideline representations, understanding the processes and limitations involved in their generation is important in developing strategies to construct shared representations that are both accurate and efficient. The encoded guidelines developed by teams that include both clinicians and experts in computer-based representations are preferable to those developed by individuals of either type working alone.

The federal role in the health information infrastructure: a debate of the pros and cons of government intervention.

Some observers feel that the federal government should play a more active leadership role in educating the medical community and in coordinating and encouraging a more rapid and effective implementation of clinically relevant applications of wide-area networking. Other people argue that the private sector is recognizing the importance of these issues and will, when the market demands it, adopt and enhance the telecommunications systems that are needed to produce effective uses of the National Information Infrastructure (NII) by the healthcare community. This debate identifies five areas for possible government involvement: convening groups for the development of standards; providing funding for research and development; ensuring the equitable distribution of resources, particularly to places and people considered by private enterprise to provide low opportunities for profit; protecting rights of privacy, intellectual property, and security; and overcoming the jurisdictional barriers to cooperation, particularly when states offer conflicting regulations. Arguments against government involvement include the likely emergence of an adequate infrastructure under free market forces, the often stifling effect of regulation, and the need to avoid a common-and-control mentality in an infrastructure that is best promoted collaboratively.

The guideline interchange format: a model for representing guidelines.

OBJECTIVE: To allow exchange of clinical practice guidelines among institutions and computer-based applications. DESIGN: The GuideLine Interchange Format (GLIF) specification consists of GLIF model and the GLIF syntax. The GLIF model is an object-oriented representation that consists of a set of classes for guideline entities, attributes for those classes, and data types for the attribute values. The GLIF syntax specifies the format of the test file that contains the encoding. METHODS: Researchers from the InterMed Collaboratory at Columbia University, Harvard University (Brigham and Women's Hospital and Massachusetts General Hospital), and Stanford University analyzed four existing guideline systems to derive a set of requirements for guideline representation. The GLIF specification is a consensus representation developed through a brainstorming process. Four clinical guidelines were encoded in GLIF to assess its expressivity and to study the variability that occurs when two people from different sites encode the same guideline. RESULTS: The encoders reported that GLIF was adequately expressive. A comparison of the encodings revealed substantial variability. CONCLUSION: GLIF was sufficient to model the guidelines for the four conditions that were examined. GLIF needs improvement in standard representation of medical concepts, criterion logic, temporal information, and uncertainty.

Networking health: learning from others, taking the lead.

The Internet provides one of the most compelling examples of the way in which government research investments can, in time, lead to innovations of broad social and economic impact. This paper reviews the history of the Internet's evolution, emphasizing in particular its relationship to biomedical computing and to the nation's health care system. Here I summarize current national research programs, emphasizing the need for greater involvement by the medical research community and leadership from federal health care agencies.

The evolution of electronic medical records.

No clinical computing topic is being given more attention than that of electronic medical records. Health care organizations, finding that they do not have systems adequate for answering questions crucial to strategic planning and for remaining competitive with other provider groups, are looking to information technologies for help. Many institutions are developing integrated clinical workstations, which provide a single point of entry for access to patient-related, administrative, and research information. At the heart of the evolving clinical workstation lies the medical record in a new incarnation: electronic, accessible, confidential, secure, acceptable to clinicians and patients, and integrated with other, non-patient-specific information. The author describes the problems associated with paper-based record keeping and the promise of the electronic medical record, emphasizing the areas of clinical trials and decision support. He then discusses the issues that must be addressed and the requirements that must be met if electronic medical record systems are to move beyond intranet environments within single health systems or practices and to integrate with regional, national, and international resources via the Internet.

Medical informatics and clinical decision making: the science and the pragmatics.

There are important scientific and pragmatic synergies between the medical decision making field and the emerging discipline of medical informatics. In the 1970s, the field of medicine forced clinically oriented artificial intelligence (AI) researchers to develop ways to manage explicit statements of uncertainty in expert systems. Classic probability theory was considered and discussed, but it tended to be abandoned because of complexities that limited its use. In medical AI systems, uncertainty was handled by a variety of ad hoc models that simulated probabilistic considerations. To illustrate the scientific interactions between the fields, the author describes recent work in his laboratory that has attempted to show that formal normative models based on probability and decision theory can be practically melded with AI methods to deliver effective advisory tools. In addition, the practical needs of decision makers and health policy planners are increasingly necessitating collaborative efforts to develop a computing and communications infrastructure for the decision making and informatics communities. This point is illustrated with an example drawn from outcomes management research.

A methodology for generating computer-based explanations of decision-theoretic advice.

Decision analysis is an appealing methodology with which to provide decision support to the practicing physician. However, its use in the clinical setting is impeded because computer-based explanations of decision-theoretic advice are difficult to generate without resorting to mathematical arguments. Nevertheless, human decision analysts generate useful and intuitive explanations based on decision trees. To facilitate the use of decision theory in a computer-based decision support system, the authors developed a computer program that uses symbolic reasoning techniques to generate nonquantitative explanations of the results of decision analyses. A combined approach has been implemented to explain the differences in expected utility among branches of a decision tree. First, the mathematical relationships inherent in the structure of the tree are used to find any asymmetries in tree structure or inequalities among analogous decision variables that are responsible for a difference in expected utility. Next, an explanation technique is selected and applied to the most significant variables, creating a symbolic expression that justifies the decision. Finally, the symbolic expression is converted to English-language text, thereby generating an explanation that justifies the desirability of the choice with the greater expected utility. The explanation does not refer to mathematical formulas, nor does it include probability or utility values. The results suggest that explanations produced by a combination of decision analysis and symbolic processing techniques may be more persuasive and acceptable to clinicians than those produced by either technique alone.

Choice and explanation in medical management: a multiattribute model of artificial intelligence approaches.

This paper explores a model of choice and explanation in medical management and makes clear its advantages and limitations. The model is based on multiattribute decision making (MADM) and consists of four distinct strategies for choice and explanation, plus combinations of these four. Each strategy is a restricted form of the general MADM approach, and each makes restrictive assumptions about the nature of the domain. The advantage of tailoring a restricted form of a general technique to a particular domain is that such efforts may better capture the character of the domain and allow choice and explanation to be more naturally modelled. The uses of the strategies for both choice and explanation are illustrated with analyses of several existing medical management artificial intelligence (AI) systems, and also with examples from the management of primary breast cancer. Using the model it is possible to identify common underlying features of these AI systems, since each employs portions of this model in different ways. Thus the model enables better understanding and characterization of the seemingly ad hoc decision making of previous systems.

The adolescence of AI in medicine: will the field come of age in the '90s?

Artificial intelligence in medicine (AIM) has reached a period of adolescence in which interactions with the outside world are not only natural but mandatory. Although the basic research topics in AIM may be those of artificial intelligence, the applied issues touch more generally on the broad field of medical informatics. To the extent that AIM research is driven by performance goals for biomedicine, AIM is simply one component within a wide range of research and development activities. Furthermore, an adequate appraisal of AIM research requires an understanding of the research motivations, the complexity of the problems, and a suitable definition of the criteria for judging the field's success. Effective fielding of AIM systems will be dependent on the development of integrated environments for communication and computing that allow merging of knowledge-based tools with other patient data-management and information-retrieval applications. The creation of this kind of infrastructure will require vision and resources from leaders who realize that the practice of medicine is inherently an information-management task and that biomedicine must make the same kind of coordinated commitment to computing technologies as have other segments of our society in which the importance of information management is well understood.

Representation of change in controlled medical terminologies.

Computer-based systems that support health care require large controlled terminologies to manage names and meanings of data elements. These terminologies are not static, because change in health care is inevitable. To share data and applications in health care, we need standards not only for terminologies and concept representation, but also for representing change. To develop a principled approach to managing change, we analyze the requirements of controlled medical terminologies and consider features that frame knowledge-representation systems have to offer. Based on our analysis, we present a concept model, a set of change operations, and a change-documentation model that may be appropriate for controlled terminologies in health care. We are currently implementing our modeling approach within a computational architecture.

A study of collaboration among medical informatics research laboratories.

The InterMed Collaboratory involves five medical institutions (Stanford University, Columbia University, Brigham and Women's Hospital, Massachusetts General Hospital, and McGill University) whose mandate has been to join in the development of shared infrastructural software, tools, and system components that will facilitate and support the development of diverse, institution-specific applications. Collaboration among geographically distributed organizations with different goals and cultures provides significant challenges. One experimental question, underlying all that InterMed has set out to achieve, is whether modern communication technologies can effectively bridge such cultural and geographical gaps, allowing the development of shared visions and cooperative activities so that the end results are greater than any one group could have accomplished on its own. In this paper we summarize the InterMed philosophy and mission, describe our progress over 3 years of collaborative activities, and present study results regarding the nature of the evolving collaborative processes, the perceptions of the participants regarding those processes, and the role that telephone conference calls have played in furthering project goals. Both informal introspection and more formal evaluative work, in which project participants became subjects of study by our evaluation experts from McGill, helped to shift our activities from relatively unfocused to more focused efforts while allowing us to understand the facilitating roles that communications technologies could play in our activities. Our experience and study results suggest that occasional face-to-face meetings are crucial precursors to the effective use of distance communications technologies; that conference calls play an important role in both task-related activities and executive (project management) activities, especially when clarifications are required; and that collaborative productivity is highly dependent upon the gradual development of a shared commitment to a well-defined task that leverages the varying expertise of both local and distant colleagues in the creation of tools of broad utility across the participating sites.