SCALE-UP II: protocol for a pragmatic randomised trial examining population health management interventions to increase the uptake of at-home COVID-19 testing in community health centres.

INTRODUCTION: SCALE-UP II aims to investigate the effectiveness of population health management interventions using text messaging (TM), chatbots and patient navigation (PN) in increasing the uptake of at-home COVID-19 testing among patients in historically marginalised communities, specifically, those receiving care at community health centres (CHCs). METHODS AND ANALYSIS: The trial is a multisite, randomised pragmatic clinical trial. Eligible patients are >18 years old with a primary care visit in the last 3 years at one of the participating CHCs. Demographic data will be obtained from CHC electronic health records. Patients will be randomised to one of two factorial designs based on smartphone ownership. Patients who self-report replying to a text message that they have a smartphone will be randomised in a 2×2×2 factorial fashion to receive (1) chatbot or TM; (2) PN (yes or no); and (3) repeated offers to interact with the interventions every 10 or 30 days. Participants who do not self-report as having a smartphone will be randomised in a 2×2 factorial fashion to receive (1) TM with or without PN; and (2) repeated offers every 10 or 30 days. The interventions will be sent in English or Spanish, with an option to request at-home COVID-19 test kits. The primary outcome is the proportion of participants using at-home COVID-19 tests during a 90-day follow-up. The study will evaluate the main effects and interactions among interventions, implementation outcomes and predictors and moderators of study outcomes. Statistical analyses will include logistic regression, stratified subgroup analyses and adjustment for stratification factors. ETHICS AND DISSEMINATION: The protocol was approved by the University of Utah Institutional Review Board. On completion, study data will be made available in compliance with National Institutes of Health data sharing policies. Results will be disseminated through study partners and peer-reviewed publications. TRIAL REGISTRATION NUMBER: ClinicalTrials.gov: NCT05533918 and NCT05533359.

Enhanced family history-based algorithms increase the identification of individuals meeting criteria for genetic testing of hereditary cancer syndromes but would not reduce disparities on their own.

OBJECTIVE: This study aimed to 1) investigate algorithm enhancements for identifying patients eligible for genetic testing of hereditary cancer syndromes using family history data from electronic health records (EHRs); and 2) assess their impact on relative differences across sex, race, ethnicity, and language preference. MATERIALS AND METHODS: The study used EHR data from a tertiary academic medical center. A baseline rule-base algorithm, relying on structured family history data (structured data; SD), was enhanced using a natural language processing (NLP) component and a relaxed criteria algorithm (partial match [PM]). The identification rates and differences were analyzed considering sex, race, ethnicity, and language preference. RESULTS: Among 120,007 patients aged 25-60, detection rate differences were found across all groups using the SD (all P < 0.001). Both enhancements increased identification rates; NLP led to a 1.9 % increase and the relaxed criteria algorithm (PM) led to an 18.5 % increase (both P < 0.001). Combining SD with NLP and PM yielded a 20.4 % increase (P < 0.001). Similar increases were observed within subgroups. Relative differences persisted across most categories for the enhanced algorithms, with disproportionately higher identification of patients who are White, Female, non-Hispanic, and whose preferred language is English. CONCLUSION: Algorithm enhancements increased identification rates for patients eligible for genetic testing of hereditary cancer syndromes, regardless of sex, race, ethnicity, and language preference. However, differences in identification rates persisted, emphasizing the need for additional strategies to reduce disparities such as addressing underlying biases in EHR family health information and selectively applying algorithm enhancements for disadvantaged populations. Systematic assessment of differences in algorithm performance across population subgroups should be incorporated into algorithm development processes.

Evaluation of software impact designed for biomedical research: Are we measuring what's meaningful?

Software is vital for the advancement of biology and medicine. Through analysis of usage and impact metrics of software, developers can help determine user and community engagement. These metrics can be used to justify additional funding, encourage additional use, and identify unanticipated use cases. Such analyses can help define improvement areas and assist with managing project resources. However, there are challenges associated with assessing usage and impact, many of which vary widely depending on the type of software being evaluated. These challenges involve issues of distorted, exaggerated, understated, or misleading metrics, as well as ethical and security concerns. More attention to the nuances, challenges, and considerations involved in capturing impact across the diverse spectrum of biological software is needed. Furthermore, some tools may be especially beneficial to a small audience, yet may not have comparatively compelling metrics of high usage. Although some principles are generally applicable, there is not a single perfect metric or approach to effectively evaluate a software tool's impact, as this depends on aspects unique to each tool, how it is used, and how one wishes to evaluate engagement. We propose more broadly applicable guidelines (such as infrastructure that supports the usage of software and the collection of metrics about usage), as well as strategies for various types of software and resources. We also highlight outstanding issues in the field regarding how communities measure or evaluate software impact. To gain a deeper understanding of the issues hindering software evaluations, as well as to determine what appears to be helpful, we performed a survey of participants involved with scientific software projects for the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We also investigated software among this scientific community and others to assess how often infrastructure that supports such evaluations is implemented and how this impacts rates of papers describing usage of the software. We find that although developers recognize the utility of analyzing data related to the impact or usage of their software, they struggle to find the time or funding to support such analyses. We also find that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seem to be associated with increased usage rates. Our findings can help scientific software developers make the most out of the evaluations of their software so that they can more fully benefit from such assessments.

Rapid-cycle designs to adapt interventions for COVID-19 in safety-net healthcare systems.

Racial/ethnic minority, low socioeconomic status, and rural populations are disproportionately affected by COVID-19. Developing and evaluating interventions to address COVID-19 testing and vaccination among these populations are crucial to improving health inequities. The purpose of this paper is to describe the application of a rapid-cycle design and adaptation process from an ongoing trial to address COVID-19 among safety-net healthcare system patients. The rapid-cycle design and adaptation process included: (a) assessing context and determining relevant models/frameworks; (b) determining core and modifiable components of interventions; and (c) conducting iterative adaptations using Plan-Do-Study-Act (PDSA) cycles. PDSA cycles included: Plan. Gather information from potential adopters/implementers (e.g., Community Health Center [CHC] staff/patients) and design initial interventions; Do. Implement interventions in single CHC or patient cohort; Study. Examine process, outcome, and context data (e.g., infection rates); and, Act. If necessary, refine interventions based on process and outcome data, then disseminate interventions to other CHCs and patient cohorts. Seven CHC systems with 26 clinics participated in the trial. Rapid-cycle, PDSA-based adaptations were made to adapt to evolving COVID-19-related needs. Near real-time data used for adaptation included data on infection hot spots, CHC capacity, stakeholder priorities, local/national policies, and testing/vaccine availability. Adaptations included those to study design, intervention content, and intervention cohorts. Decision-making included multiple stakeholders (e.g., State Department of Health, Primary Care Association, CHCs, patients, researchers). Rapid-cycle designs may improve the relevance and timeliness of interventions for CHCs and other settings that provide care to populations experiencing health inequities, and for rapidly evolving healthcare challenges such as COVID-19.

Racial/ethnic minority, low socioeconomic status, and rural populations experience a disproportionate burden of COVID-19. Finding ways to address COVID-19 among these populations is crucial to improving health inequities. The purpose of this paper is to describe the rapid-cycle design process for a research project to address COVID-19 testing and vaccination among safety-net healthcare system patients. The project used real-time information on changes in COVID-19 policy (e.g., vaccination authorization), local case rates, and the capacity of safety-net healthcare systems to iteratively change interventions to ensure interventions were relevant and timely for patients. Key changes that were made to interventions included a change to the study design to include vaccination as a focus of the interventions after the vaccine was authorized; change in intervention content according to the capacity of local Community Health Centers to provide testing to patients; and changes to intervention cohorts such that priority groups of patients were selected for intervention based on characteristics including age, residency in an infection "hot spot," or race/ethnicity. Iteratively improving interventions based on real-time data collection may increase intervention relevance and timeliness, and rapid-cycle adaptions can be successfully implemented in resource constrained settings like safety-net healthcare systems.

Association of Disparities in Family History and Family Cancer History in the Electronic Health Record With Sex, Race, Hispanic or Latino Ethnicity, and Language Preference in 2 Large US Health Care Systems.

Importance: Clinical decision support (CDS) algorithms are increasingly being implemented in health care systems to identify patients for specialty care. However, systematic differences in missingness of electronic health record (EHR) data may lead to disparities in identification by CDS algorithms. Objective: To examine the availability and comprehensiveness of cancer family history information (FHI) in patients' EHRs by sex, race, Hispanic or Latino ethnicity, and language preference in 2 large health care systems in 2021. Design, Setting, and Participants: This retrospective EHR quality improvement study used EHR data from 2 health care systems: University of Utah Health (UHealth) and NYU Langone Health (NYULH). Participants included patients aged 25 to 60 years who had a primary care appointment in the previous 3 years. Data were collected or abstracted from the EHR from December 10, 2020, to October 31, 2021, and analyzed from June 15 to October 31, 2021. Exposures: Prior collection of cancer FHI in primary care settings. Main Outcomes and Measures: Availability was defined as having any FHI and any cancer FHI in the EHR and was examined at the patient level. Comprehensiveness was defined as whether a cancer family history observation in the EHR specified the type of cancer diagnosed in a family member, the relationship of the family member to the patient, and the age at onset for the family member and was examined at the observation level. Results: Among 144 484 patients in the UHealth system, 53.6% were women; 74.4% were non-Hispanic or non-Latino and 67.6% were White; and 83.0% had an English language preference. Among 377 621 patients in the NYULH system, 55.3% were women; 63.2% were non-Hispanic or non-Latino, and 55.3% were White; and 89.9% had an English language preference. Patients from historically medically undeserved groups-specifically, Black vs White patients (UHealth: 17.3% [95% CI, 16.1%-18.6%] vs 42.8% [95% CI, 42.5%-43.1%]; NYULH: 24.4% [95% CI, 24.0%-24.8%] vs 33.8% [95% CI, 33.6%-34.0%]), Hispanic or Latino vs non-Hispanic or non-Latino patients (UHealth: 27.2% [95% CI, 26.5%-27.8%] vs 40.2% [95% CI, 39.9%-40.5%]; NYULH: 24.4% [95% CI, 24.1%-24.7%] vs 31.6% [95% CI, 31.4%-31.8%]), Spanish-speaking vs English-speaking patients (UHealth: 18.4% [95% CI, 17.2%-19.1%] vs 40.0% [95% CI, 39.7%-40.3%]; NYULH: 15.1% [95% CI, 14.6%-15.6%] vs 31.1% [95% CI, 30.9%-31.2%), and men vs women (UHealth: 30.8% [95% CI, 30.4%-31.2%] vs 43.0% [95% CI, 42.6%-43.3%]; NYULH: 23.1% [95% CI, 22.9%-23.3%] vs 34.9% [95% CI, 34.7%-35.1%])-had significantly lower availability and comprehensiveness of cancer FHI (P < .001). Conclusions and Relevance: These findings suggest that systematic differences in the availability and comprehensiveness of FHI in the EHR may introduce informative presence bias as inputs to CDS algorithms. The observed differences may also exacerbate disparities for medically underserved groups. System-, clinician-, and patient-level efforts are needed to improve the collection of FHI.

Correction: Identifying Patients Who Meet Criteria for Genetic Testing of Hereditary Cancers Based on Structured and Unstructured Family Health History Data in the Electronic Health Record: Natural Language Processing Approach.

[This corrects the article DOI: 10.2196/37842.].

Identifying Patients Who Meet Criteria for Genetic Testing of Hereditary Cancers Based on Structured and Unstructured Family Health History Data in the Electronic Health Record: Natural Language Processing Approach.

BACKGROUND: Family health history has been recognized as an essential factor for cancer risk assessment and is an integral part of many cancer screening guidelines, including genetic testing for personalized clinical management strategies. However, manually identifying eligible candidates for genetic testing is labor intensive. OBJECTIVE: The aim of this study was to develop a natural language processing (NLP) pipeline and assess its contribution to identifying patients who meet genetic testing criteria for hereditary cancers based on family health history data in the electronic health record (EHR). We compared an algorithm that uses structured data alone with structured data augmented using NLP. METHODS: Algorithms were developed based on the National Comprehensive Cancer Network (NCCN) guidelines for genetic testing for hereditary breast, ovarian, pancreatic, and colorectal cancers. The NLP-augmented algorithm uses both structured family health history data and the associated unstructured free-text comments. The algorithms were compared with a reference standard of 100 patients with a family health history in the EHR. RESULTS: Regarding identifying the reference standard patients meeting the NCCN criteria, the NLP-augmented algorithm compared with the structured data algorithm yielded a significantly higher recall of 0.95 (95% CI 0.9-0.99) versus 0.29 (95% CI 0.19-0.40) and a precision of 0.99 (95% CI 0.96-1.00) versus 0.81 (95% CI 0.65-0.95). On the whole data set, the NLP-augmented algorithm extracted 33.6% more entities, resulting in 53.8% more patients meeting the NCCN criteria. CONCLUSIONS: Compared with the structured data algorithm, the NLP-augmented algorithm based on both structured and unstructured family health history data in the EHR increased the number of patients identified as meeting the NCCN criteria for genetic testing for hereditary breast or ovarian and colorectal cancers.

GARDE: a standards-based clinical decision support platform for identifying population health management cohorts.

Population health management (PHM) is an important approach to promote wellness and deliver health care to targeted individuals who meet criteria for preventive measures or treatment. A critical component for any PHM program is a data analytics platform that can target those eligible individuals. OBJECTIVE: The aim of this study was to design and implement a scalable standards-based clinical decision support (CDS) approach to identify patient cohorts for PHM and maximize opportunities for multi-site dissemination. MATERIALS AND METHODS: An architecture was established to support bidirectional data exchanges between heterogeneous electronic health record (EHR) data sources, PHM systems, and CDS components. HL7 Fast Healthcare Interoperability Resources and CDS Hooks were used to facilitate interoperability and dissemination. The approach was validated by deploying the platform at multiple sites to identify patients who meet the criteria for genetic evaluation of familial cancer. RESULTS: The Genetic Cancer Risk Detector (GARDE) platform was created and is comprised of four components: (1) an open-source CDS Hooks server for computing patient eligibility for PHM cohorts, (2) an open-source Population Coordinator that processes GARDE requests and communicates results to a PHM system, (3) an EHR Patient Data Repository, and (4) EHR PHM Tools to manage patients and perform outreach functions. Site-specific deployments were performed on onsite virtual machines and cloud-based Amazon Web Services. DISCUSSION: GARDE's component architecture establishes generalizable standards-based methods for computing PHM cohorts. Replicating deployments using one of the established deployment methods requires minimal local customization. Most of the deployment effort was related to obtaining site-specific information technology governance approvals.

Patient Interactions With an Automated Conversational Agent Delivering Pretest Genetics Education: Descriptive Study.

BACKGROUND: Cancer genetic testing to assess an individual's cancer risk and to enable genomics-informed cancer treatment has grown exponentially in the past decade. Because of this continued growth and a shortage of health care workers, there is a need for automated strategies that provide high-quality genetics services to patients to reduce the clinical demand for genetics providers. Conversational agents have shown promise in managing mental health, pain, and other chronic conditions and are increasingly being used in cancer genetic services. However, research on how patients interact with these agents to satisfy their information needs is limited. OBJECTIVE: Our primary aim is to assess user interactions with a conversational agent for pretest genetics education. METHODS: We conducted a feasibility study of user interactions with a conversational agent who delivers pretest genetics education to primary care patients without cancer who are eligible for cancer genetic evaluation. The conversational agent provided scripted content similar to that delivered in a pretest genetic counseling visit for cancer genetic testing. Outside of a core set of information delivered to all patients, users were able to navigate within the chat to request additional content in their areas of interest. An artificial intelligence-based preprogrammed library was also established to allow users to ask open-ended questions to the conversational agent. Transcripts of the interactions were recorded. Here, we describe the information selected, time spent to complete the chat, and use of the open-ended question feature. Descriptive statistics were used for quantitative measures, and thematic analyses were used for qualitative responses. RESULTS: We invited 103 patients to participate, of which 88.3% (91/103) were offered access to the conversational agent, 39% (36/91) started the chat, and 32% (30/91) completed the chat. Most users who completed the chat indicated that they wanted to continue with genetic testing (21/30, 70%), few were unsure (9/30, 30%), and no patient declined to move forward with testing. Those who decided to test spent an average of 10 (SD 2.57) minutes on the chat, selected an average of 1.87 (SD 1.2) additional pieces of information, and generally did not ask open-ended questions. Those who were unsure spent 4 more minutes on average (mean 14.1, SD 7.41; P=.03) on the chat, selected an average of 3.67 (SD 2.9) additional pieces of information, and asked at least one open-ended question. CONCLUSIONS: The pretest chat provided enough information for most patients to decide on cancer genetic testing, as indicated by the small number of open-ended questions. A subset of participants were still unsure about receiving genetic testing and may require additional education or interpersonal support before making a testing decision. Conversational agents have the potential to become a scalable alternative for pretest genetics education, reducing the clinical demand on genetics providers.

Establishing a multidisciplinary initiative for interoperable electronic health record innovations at an academic medical center.

OBJECTIVE: To establish an enterprise initiative for improving health and health care through interoperable electronic health record (EHR) innovations. MATERIALS AND METHODS: We developed a unifying mission and vision, established multidisciplinary governance, and formulated a strategic plan. Key elements of our strategy include establishing a world-class team; creating shared infrastructure to support individual innovations; developing and implementing innovations with high anticipated impact and a clear path to adoption; incorporating best practices such as the use of Fast Healthcare Interoperability Resources (FHIR) and related interoperability standards; and maximizing synergies across research and operations and with partner organizations. RESULTS: University of Utah Health launched the ReImagine EHR initiative in 2016. Supportive infrastructure developed by the initiative include various FHIR-related tooling and a systematic evaluation framework. More than 10 EHR-integrated digital innovations have been implemented to support preventive care, shared decision-making, chronic disease management, and acute clinical care. Initial evaluations of these innovations have demonstrated positive impact on user satisfaction, provider efficiency, and compliance with evidence-based guidelines. Return on investment has included improvements in care; over $35 million in external grant funding; commercial opportunities; and increased ability to adapt to a changing healthcare landscape. DISCUSSION: Key lessons learned include the value of investing in digital innovation initiatives leveraging FHIR; the importance of supportive infrastructure for accelerating innovation; and the critical role of user-centered design, implementation science, and evaluation. CONCLUSION: EHR-integrated digital innovation initiatives can be key assets for enhancing the EHR user experience, improving patient care, and reducing provider burnout.

Standards-Based Clinical Decision Support Platform to Manage Patients Who Meet Guideline-Based Criteria for Genetic Evaluation of Familial Cancer.

PURPOSE: The ubiquitous adoption of electronic health records (EHRs) with family health history (FHH) data provides opportunities for tailoring cancer screening strategies to individuals. We aimed to enable a standards-based clinical decision support (CDS) platform for identifying and managing patients who meet guidelines for genetic evaluation of hereditary cancer. METHODS: The CDS platform (www.opencds.org) was used to implement algorithms based on the 2018 National Comprehensive Cancer Network guidelines for genetic evaluation of hereditary breast/ovarian and colorectal cancer. The platform was designed to be interfaced with different EHR systems via the Health Level Seven International Fast Healthcare Interoperability Resources standard. The platform was integrated with the Epic EHR and evaluated in a pilot study at an academic health care system. RESULTS: The CDS platform was executed against a target population of 143,012 patients; 5,245 (3.7%) met criteria for genetic evaluation based on the FHH recorded in the EHR. In a clinical pilot study, genetic counselors attempted to reach out to 71 of the patients. Of those patients, 25 (35%) scheduled an appointment, 10 (14%) declined, 2 (3%) did not need genetic counseling, 7 (10%) said they would consider it in the future, and 27 (38%) were unreachable. To date, 13 (52%) of the scheduled patients completed visits, and 2 (15%) of those were found to have pathogenic variants in cancer predisposition genes. CONCLUSION: A standards-based CDS platform integrated with EHR systems is a promising population-based approach to identify patients who are appropriate candidates for genetic evaluation of hereditary cancers.

An Efficient, Clinically-Natural Electronic Medical Record System that Produces Computable Data.

Current commercially-available electronic medical record systems produce mainly text-based information focused on financial and regulatory performance. We combined an existing method for organizing complex computer systems-which we label activity-based design-with a proven approach for integrating clinical decision support into front-line care delivery-Care Process Models. The clinical decision support approach increased the structure of textual clinical documentation, to the point where established methods for converting text into computable data (natural language processing) worked efficiently. In a simple trial involving radiology reports for examinations performed to rule out pneumonia, more than 98 percent of all documentation generated was captured as computable data. Use cases across a broad range of other physician, nursing, and physical therapy clinical applications subjectively show similar effects. The resulting system is clinically natural, puts clinicians in direct, rapid control of clinical content without information technology intermediaries, and can generate complete clinical documentation. It supports embedded secondary functions such as the generation of granular activity-based costing data, and embedded generation of clinical coding (e.g., CPT, ICD-10 or SNOMED). Most important, widely-available computable data has the potential to greatly improve care delivery management and outcomes.

Architecture of a federated query engine for heterogeneous resources.

The Federated Utah Research and Translational Health e-Repository (FURTHeR) is a Utah statewide informatics platform for the new Center for Clinical and Translational Science at the University of Utah. We have been working on one of FURTHeR's key components, a federated query engine for heterogeneous resources, that we believe has the potential to meet some of the fundamental needs of translational science to access and integrate diverse biomedical data and promote discovery of new knowledge. The architecture of the federated query engine for heterogeneous resources is described and demonstrated.

Towards an on-demand peer feedback system for a clinical knowledge base: a case study with order sets.

OBJECTIVE: We have developed an automated knowledge base peer feedback system as part of an effort to facilitate the creation and refinement of sound clinical knowledge content within an enterprise-wide knowledge base. The program collects clinical data stored in our Clinical Data Repository during usage of a physician order entry program. It analyzes usage patterns of order sets relative to their templates and creates a report detailing the usage patterns of the order set template. This report includes a set of suggested modifications to the template. DESIGN: A quantitative analysis was performed to assess the validity of the program's suggested order set template modifications. MEASUREMENTS: We collected and de-identified 2951 instances of POE-based orders. Our program then identified and generated feedback reports for thirty different order set templates from this data set. These reports contained 500 suggested modifications. Five order set authors were then asked to 'accept' or 'reject' each suggestion contained in his/her respective order set templates. They were also asked to categorize their rationale for doing so ('clinical relevance' or 'user convenience'). RESULTS: In total, 62% (309/500) suggestions were accepted by clinical content authors. Of these, authors accepted 32% (36/114) of the suggested additions, 74% (123/167) of the suggested pre-selections, 76% (16/25) of the suggested de-selections, and 68% (131/194) of the suggested changes in combo box order. CONCLUSION: Overall, the feedback system generated suggestions that were deemed highly acceptable among order set authors. Future refinements and enhancements to the software will add to its utility.

Knowledge management strategies: Enhancing knowledge transfer to clinicians and patients.

At Intermountain Healthcare (Intermountain), executive clinical content experts are responsible for disseminating consistent evidence-based clinical content throughout the enterprise at the point-of-care. With a paper-based system it was difficult to ensure that current information was received and was being used in practice. With electronic information systems multiple applications were supplying similar, but different, vendor-licensed and locally-developed content. These issues influenced the consistency of clinical practice within the enterprise, jeopardized patient and clinician safety, and exposed the enterprise and its employees to potential financial penalties. In response to these issues Intermountain is developing a knowledge management infrastructure providing tools and services to support clinical content development, deployment, maintenance, and communication. The Intermountain knowledge management philosophy includes strategies guiding clinicians and consumers of health information to relevant best practice information with the intention of changing behaviors. This paper presents three case studies describing different information management problems identified within Intermountain, methods used to solve the problems, implementation challenges, and the current status of each project.

Towards ubiquitous peer review strategies to sustain and enhance a clinical knowledge management framework.

Widespread cooperation between domain experts and front-line clinicians is a key component of any successful clinical knowledge management framework. Peer review is an established form of cooperation that promotes the dissemination of new knowledge. The authors describe three peer collaboration scenarios that have been implemented using the knowledge management infrastructure available at Intermountain Healthcare. Utilization results illustrating the early adoption patterns of the proposed scenarios are presented and discussed, along with succinct descriptions of planned enhancements and future implementation efforts.

An XML model that enables the development of complex order sets by clinical experts.

Medication errors are significant and well-known problems in health care. Despite the evidence supporting the use of computerized physician order entering (CPOE) to help reduce medication errors, only a small number of hospitals in the U.S. have successfully implemented a CPOE system. Different authors have indicated that the utilization of order sets derived from best-practice standards can reduce medication errors and improve physicians' acceptance of CPOE systems. However, a variety of issues related to the development and continuous maintenance of best-practice order sets still need to be understood. This paper presents a model that supports an order set development process driven by clinical experts. Model requirements and details are presented and discussed.

KAT: a flexible XML-based knowledge authoring environment.

As part of an enterprise effort to develop new clinical information systems at Intermountain Health Care, the authors have built a knowledge authoring tool that facilitates the development and refinement of medical knowledge content. At present, users of the application can compose order sets and an assortment of other structured clinical knowledge documents based on XML schemas. The flexible nature of the application allows the immediate authoring of new types of documents once an appropriate XML schema and accompanying Web form have been developed and stored in a shared repository. The need for a knowledge acquisition tool stems largely from the desire for medical practitioners to be able to write their own content for use within clinical applications. We hypothesize that medical knowledge content for clinical use can be successfully created and maintained through XML-based document frameworks containing structured and coded knowledge.

Customized document validation to support a flexible XML-based knowledge management framework.

This paper describes a validation architecture used within Intermountain Health Care's Clinical Knowledge Repository (CKR). The architecture provides additional functionality that complements XML Schema validation, producing user-friendly error messages and enabling validation rules reuse. The validation architecture helps document authors to fix their own errors. As a result, less than 1% of all documents in the CKR are considered invalid.

Exploratory case method to determine the frequency of redundant orders within manually consolidated order lists.

A computerized provider order entry (CPOE) system can provide an efficient means of retrieving and consolidating order lists from multiple electronic clinical practice standards and protocols. However, the consolidated order list may contain exact duplicate or overlapping orders. Benner's framework for levels of nursing expertise can be used to explicate the variability of the nurse's responses to redundancies in order lists and the potential compromise to patient safety. An exploratory case method was performed to consolidate 74 orders from 11 sources. The consolidated order list contained 35% fewer orders after the redundant orders were removed. Our work has shown that many redundant orders may arise by consolidating order lists from multiple electronic standards. It is imperative that consolidated electronic order lists be manageable by the nurse according to their level of clinical and computer expertise, and that redundant orders are resolved before being displayed to the nurse.