NOBLE - Flexible concept recognition for large-scale biomedical natural language processing.

BACKGROUND: Natural language processing (NLP) applications are increasingly important in biomedical data analysis, knowledge engineering, and decision support. Concept recognition is an important component task for NLP pipelines, and can be either general-purpose or domain-specific. We describe a novel, flexible, and general-purpose concept recognition component for NLP pipelines, and compare its speed and accuracy against five commonly used alternatives on both a biological and clinical corpus. NOBLE Coder implements a general algorithm for matching terms to concepts from an arbitrary vocabulary set. The system's matching options can be configured individually or in combination to yield specific system behavior for a variety of NLP tasks. The software is open source, freely available, and easily integrated into UIMA or GATE. We benchmarked speed and accuracy of the system against the CRAFT and ShARe corpora as reference standards and compared it to MMTx, MGrep, Concept Mapper, cTAKES Dictionary Lookup Annotator, and cTAKES Fast Dictionary Lookup Annotator. RESULTS: We describe key advantages of the NOBLE Coder system and associated tools, including its greedy algorithm, configurable matching strategies, and multiple terminology input formats. These features provide unique functionality when compared with existing alternatives, including state-of-the-art systems. On two benchmarking tasks, NOBLE's performance exceeded commonly used alternatives, performing almost as well as the most advanced systems. Error analysis revealed differences in error profiles among systems. CONCLUSION: NOBLE Coder is comparable to other widely used concept recognition systems in terms of accuracy and speed. Advantages of NOBLE Coder include its interactive terminology builder tool, ease of configuration, and adaptability to various domains and tasks. NOBLE provides a term-to-concept matching system suitable for general concept recognition in biomedical NLP pipelines.

METACOGNITIVE SCAFFOLDS IMPROVE SELF-JUDGMENTS OF ACCURACY IN A MEDICAL INTELLIGENT TUTORING SYSTEM.

In this study, we examined the effect of two metacognitive scaffolds on the accuracy of confidence judgments made while diagnosing dermatopathology slides in SlideTutor. Thirty-one (N = 31) first- to fourth-year pathology and dermatology residents were randomly assigned to one of the two scaffolding conditions. The cases used in this study were selected from the domain of Nodular and Diffuse Dermatitides. Both groups worked with a version of SlideTutor that provided immediate feedback on their actions for two hours before proceeding to solve cases in either the Considering Alternatives or Playback condition. No immediate feedback was provided on actions performed by participants in the scaffolding mode. Measurements included learning gains (pre-test and post-test), as well as metacognitive performance, including Goodman-Kruskal Gamma correlation, bias, and discrimination. Results showed that participants in both conditions improved significantly in terms of their diagnostic scores from pre-test to post-test. More importantly, participants in the Considering Alternatives condition outperformed those in the Playback condition in the accuracy of their confidence judgments and the discrimination of the correctness of their assertions while solving cases. The results suggested that presenting participants with their diagnostic decision paths and highlighting correct and incorrect paths helps them to become more metacognitively accurate in their confidence judgments.

The Use of Rescue Insulin in the Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study (GRADE).

OBJECTIVE: To describe rescue insulin use and associated factors in the Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study (GRADE). RESEARCH DESIGN AND METHODS: GRADE participants (type 2 diabetes duration <10 years, baseline A1C 6.8%-8.5% on metformin monotherapy, N = 5,047) were randomly assigned to insulin glargine U-100, glimepiride, liraglutide, or sitagliptin and followed quarterly for a mean of 5 years. Rescue insulin (glargine or aspart) was to be started within 6 weeks of A1C >7.5%, confirmed. Reasons for delaying rescue insulin were reported by staff-completed survey. RESULTS: Nearly one-half of GRADE participants (N = 2,387 [47.3%]) met the threshold for rescue insulin. Among participants assigned to glimepiride, liraglutide, or sitagliptin, rescue glargine was added by 69% (39% within 6 weeks). Rescue aspart was added by 44% of glargine-assigned participants (19% within 6 weeks) and by 30% of non-glargine-assigned participants (14% within 6 weeks). Higher A1C values were associated with adding rescue insulin. Intention to change health behaviors (diet/lifestyle, adherence to current treatment) and not wanting to take insulin were among the most common reasons reported for not adding rescue insulin within 6 weeks. CONCLUSIONS: Proportionately, rescue glargine, when required, was more often used than rescue aspart, and higher A1C values were associated with greater rescue insulin use. Wanting to use noninsulin strategies to improve glycemia was commonly reported, although multiple factors likely contributed to not using rescue insulin. These findings highlight the persistent challenge of intensifying type 2 diabetes treatment with insulin, even in a clinical trial.

Emotional Distress Predicts Reduced Type 2 Diabetes Treatment Adherence in the Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study (GRADE).

OBJECTIVE: We examined longitudinal associations between emotional distress (specifically, depressive symptoms and diabetes distress) and medication adherence in Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study (GRADE), a large randomized controlled trial comparing four glucose-lowering medications added to metformin in adults with relatively recent-onset type 2 diabetes mellitus (T2DM). RESEARCH DESIGN AND METHODS: The Emotional Distress Substudy assessed medication adherence, depressive symptoms, and diabetes distress in 1,739 GRADE participants via self-completed questionnaires administered biannually up to 3 years. We examined baseline depressive symptoms and diabetes distress as predictors of medication adherence over 36 months. Bidirectional visit-to-visit relationships were also examined. Treatment satisfaction, beliefs about medication, diabetes care self-efficacy, and perceived control over diabetes were evaluated as mediators of longitudinal associations. RESULTS: At baseline, mean ± SD age of participants (56% of whom were White, 17% Hispanic/Latino, 18% Black, and 66% male) was 58.0 ± 10.2 years, diabetes duration 4.2 ± 2.8 years, HbA1c 7.5% ± 0.5%, and medication adherence 89.9% ± 11.1%. Higher baseline depressive symptoms and diabetes distress were independently associated with lower adherence over 36 months (P < 0.001). Higher depressive symptoms and diabetes distress at one visit predicted lower adherence at the subsequent 6-month visit (P < 0.0001) but not vice versa. Treatment assignment did not moderate relationships. Patient-reported concerns about diabetes medications mediated the largest percentage (11.9%-15.5%) of the longitudinal link between emotional distress and adherence. CONCLUSIONS: Depressive symptoms and diabetes distress both predict lower adherence to glucose-lowering medications over time among adults with T2DM. Addressing emotional distress and concerns about anticipated negative effects of taking these treatments may be important to support diabetes treatment adherence.

Automated detection of heuristics and biases among pathologists in a computer-based system.

The purpose of this study is threefold: (1) to develop an automated, computer-based method to detect heuristics and biases as pathologists examine virtual slide cases, (2) to measure the frequency and distribution of heuristics and errors across three levels of training, and (3) to examine relationships of heuristics to biases, and biases to diagnostic errors. The authors conducted the study using a computer-based system to view and diagnose virtual slide cases. The software recorded participant responses throughout the diagnostic process, and automatically classified participant actions based on definitions of eight common heuristics and/or biases. The authors measured frequency of heuristic use and bias across three levels of training. Biases studied were detected at varying frequencies, with availability and search satisficing observed most frequently. There were few significant differences by level of training. For representativeness and anchoring, the heuristic was used appropriately as often or more often than it was used in biased judgment. Approximately half of the diagnostic errors were associated with one or more biases. We conclude that heuristic use and biases were observed among physicians at all levels of training using the virtual slide system, although their frequencies varied. The system can be employed to detect heuristic use and to test methods for decreasing diagnostic errors resulting from cognitive biases.

Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold.

Previous studies in our laboratory have shown the benefits of immediate feedback on cognitive performance for pathology residents using an intelligent tutoring system (ITS) in pathology. In this study, we examined the effect of immediate feedback on metacognitive performance, and investigated whether other metacognitive scaffolds will support metacognitive gains when immediate feedback is faded. Twenty-three participants were randomized into intervention and control groups. For both groups, periods working with the ITS under varying conditions were alternated with independent computer-based assessments. On day 1, a within-subjects design was used to evaluate the effect of immediate feedback on cognitive and metacognitive performance. On day 2, a between-subjects design was used to compare the use of other metacognitive scaffolds (intervention group) against no metacognitive scaffolds (control group) on cognitive and metacognitive performance, as immediate feedback was faded. Measurements included learning gains (a measure of cognitive performance), as well as several measures of metacognitive performance, including Goodman-Kruskal gamma correlation (G), bias, and discrimination. For the intervention group, we also computed metacognitive measures during tutoring sessions. Results showed that immediate feedback in an intelligent tutoring system had a statistically significant positive effect on learning gains, G and discrimination. Removal of immediate feedback was associated with decreasing metacognitive performance, and this decline was not prevented when students used a version of the tutoring system that provided other metacognitive scaffolds. Results obtained directly from the ITS suggest that other metacognitive scaffolds do have a positive effect on G and discrimination, as immediate feedback is faded. We conclude that immediate feedback had a positive effect on both metacognitive and cognitive gains in a medical tutoring system. Other metacognitive scaffolds were not sufficient to replace immediate feedback in this study. However, results obtained directly from the tutoring system are not consistent with results obtained from assessments. In order to facilitate transfer to real-world tasks, further research will be needed to determine the optimum methods for supporting metacognition as immediate feedback is faded.

A natural language intelligent tutoring system for training pathologists: implementation and evaluation.

INTRODUCTION: We developed and evaluated a Natural Language Interface (NLI) for an Intelligent Tutoring System (ITS) in Diagnostic Pathology. The system teaches residents to examine pathologic slides and write accurate pathology reports while providing immediate feedback on errors they make in their slide review and diagnostic reports. Residents can ask for help at any point in the case, and will receive context-specific feedback. RESEARCH QUESTIONS: We evaluated (1) the performance of our natural language system, (2) the effect of the system on learning (3) the effect of feedback timing on learning gains and (4) the effect of ReportTutor on performance to self-assessment correlations. METHODS: The study uses a crossover 2 x 2 factorial design. We recruited 20 subjects from 4 academic programs. Subjects were randomly assigned to one of the four conditions--two conditions for the immediate interface, and two for the delayed interface. An expert dermatopathologist created a reference standard and 2 board certified AP/CP pathology fellows manually coded the residents' assessment reports. Subjects were given the opportunity to self grade their performance and we used a survey to determine student response to both interfaces. RESULTS: Our results show a highly significant improvement in report writing after one tutoring session with 4-fold increase in the learning gains with both interfaces but no effect of feedback timing on performance gains. Residents who used the immediate feedback interface first experienced a feature learning gain that is correlated with the number of cases they viewed. There was no correlation between performance and self-assessment in either condition.

Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance.

OBJECTIVE: Determine effects of computer-based tutoring on diagnostic performance gains, meta-cognition, and acceptance using two different problem representations. Describe impact of tutoring on spectrum of diagnostic skills required for task performance. Identify key features of student-tutor interaction contributing to learning gains. DESIGN: Prospective, between-subjects study, controlled for participant level of training. Resident physicians in two academic pathology programs spent four hours using one of two interfaces which differed mainly in external problem representation. The case-focused representation provided an open-learning environment in which students were free to explore evidence-hypothesis relationships within a case, but could not visualize the entire diagnostic space. The knowledge-focused representation provided an interactive representation of the entire diagnostic space, which more tightly constrained student actions. MEASUREMENTS: Metrics included results of pretest, post-test and retention-test for multiple choice and case diagnosis tests, ratios of performance to student reported certainty, results of participant survey, learning curves, and interaction behaviors during tutoring. RESULTS: Students had highly significant learning gains after one tutoring session. Learning was retained at one week. There were no differences between the two interfaces in learning gains on post-test or retention test. Only students in the knowledge-focused interface exhibited significant metacognitive gains from pretest to post-test and pretest to retention test. Students rated the knowledge-focused interface significantly higher than the case-focused interface. CONCLUSIONS: Cognitive tutoring is associated with improved diagnostic performance in a complex medical domain. The effect is retained at one-week post-training. Knowledge-focused external problem representation shows an advantage over case-focused representation for metacognitive effects and user acceptance.

Needs Assessment for Research Use of High-Throughput Sequencing at a Large Academic Medical Center.

Next Generation Sequencing (NGS) methods are driving profound changes in biomedical research, with a growing impact on patient care. Many academic medical centers are evaluating potential models to prepare for the rapid increase in NGS information needs. This study sought to investigate (1) how and where sequencing data is generated and analyzed, (2) research objectives and goals for NGS, (3) workforce capacity and unmet needs, (4) storage capacity and unmet needs, (5) available and anticipated funding resources, and (6) future challenges. As a precursor to informed decision making at our institution, we undertook a systematic needs assessment of investigators using survey methods. We recruited 331 investigators from over 60 departments and divisions at the University of Pittsburgh Schools of Health Sciences and had 140 respondents, or a 42% response rate. Results suggest that both sequencing and analysis bottlenecks currently exist. Significant educational needs were identified, including both investigator-focused needs, such as selection of NGS methods suitable for specific research objectives, and program-focused needs, such as support for training an analytic workforce. The absence of centralized infrastructure was identified as an important institutional gap. Key principles for organizations managing this change were formulated based on the survey responses. This needs assessment provides an in-depth case study which may be useful to other academic medical centers as they identify and plan for future needs.

A Federated Network for Translational Cancer Research Using Clinical Data and Biospecimens.

Advances in cancer research and personalized medicine will require significant new bridging infrastructures, including more robust biorepositories that link human tissue to clinical phenotypes and outcomes. In order to meet that challenge, four cancer centers formed the Text Information Extraction System (TIES) Cancer Research Network, a federated network that facilitates data and biospecimen sharing among member institutions. Member sites can access pathology data that are de-identified and processed with the TIES natural language processing system, which creates a repository of rich phenotype data linked to clinical biospecimens. TIES incorporates multiple security and privacy best practices that, combined with legal agreements, network policies, and procedures, enable regulatory compliance. The TIES Cancer Research Network now provides integrated access to investigators at all member institutions, where multiple investigator-driven pilot projects are underway. Examples of federated search across the network illustrate the potential impact on translational research, particularly for studies involving rare cancers, rare phenotypes, and specific biologic behaviors. The network satisfies several key desiderata including local control of data and credentialing, inclusion of rich phenotype information, and applicability to diverse research objectives. The TIES Cancer Research Network presents a model for a national data and biospecimen network.

Implementation and evaluation of a negation tagger in a pipeline-based system for information extract from pathology reports.

We have developed a pipeline-based system for automated annotation of Surgical Pathology Reports with UMLS terms that builds on GATE--an open-source architecture for language engineering. The system includes a module for detecting and annotating negated concepts, which implements the NegEx algorithm--an algorithm originally described for use in discharge summaries and radiology reports. We describe the implementation of the system, and early evaluation of the Negation Tagger. Our results are encouraging. In the key Final Diagnosis section, with almost no modification of the algorithm or phrase lists, the system performs with precision of 0.84 and recall of 0.80 against a gold-standard corpus of negation annotations, created by modified Delphi technique by a panel of pathologists. Further work will focus on refining the Negation Tagger and UMLS Tagger and adding additional processing resources for annotating free-text pathology reports.

Perceptual analysis of the reading of dermatopathology virtual slides by pathology residents.

CONTEXT: The process by which pathologists arrive at a given diagnosis-a combination of their slide exploration strategy, perceptual information gathering, and cognitive decision making-has not been thoroughly explored, and many questions remain unanswered. OBJECTIVE: To determine how pathology residents learn to diagnose inflammatory skin dermatoses, we contrasted the slide exploration strategy, perceptual capture of relevant histopathologic findings, and cognitive integration of identified features between 2 groups of residents, those who had and those who had not undergone their dermatopathology rotation. DESIGN: Residents read a case set of 20 virtual slides (10 depicting nodular and diffuse dermatitis and 10 depicting subepidermal vesicular dermatitis), using an in-house-developed interface. We recorded residents' reports of diagnostic findings, conjectured diagnostic hypotheses, and final (or differential) diagnosis for each case, and time stamped each interaction with the interface. We created search maps of residents' slide exploration strategy. RESULTS: No statistically significant differences were observed between the resident groups in the number of correctly or incorrectly reported diagnostic findings, but residents with dermatopathology training generated significantly more correct hypotheses (mean improvement of 88.5%) and correct diagnoses (70% of all correct diagnoses). CONCLUSIONS: Two types of slide exploration strategy were identified for both groups: (1) a focused and efficient search, observed when the final diagnosis was correct; and (2) a more dispersed, time-consuming strategy, observed when the final diagnosis was incorrect. This difference was statistically significant, and it suggests that initial interpretation of a slide may bias further slide exploration.

Effect of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths.

OBJECTIVES: Determine effects of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths. Determine if limited enforcement in a medical tutoring system inhibits students from learning the optimal and most efficient solution path. Describe the type of deviations from the optimal solution path that occur during tutoring, and how these deviations change over time. Determine if the size of the problem-space (domain scope), has an effect on learning gains when using a tutor with limited enforcement. METHODS: Analyzed data mined from 44 pathology residents using SlideTutor-a Medical Intelligent Tutoring System in Dermatopathology that teaches histopathologic diagnosis and reporting skills based on commonly used diagnostic algorithms. Two subdomains were included in the study representing sub-algorithms of different sizes and complexities. Effects of the tutoring system on student errors, goal states and solution paths were determined. RESULTS: Students gradually increase the frequency of steps that match the tutoring system's expectation of expert performance. Frequency of errors gradually declines in all categories of error significance. Student performance frequently differs from the tutor-defined optimal path. However, as students continue to be tutored, they approach the optimal solution path. Performance in both subdomains was similar for both errors and goal differences. However, the rate at which students progress toward the optimal solution path differs between the two domains. Tutoring in superficial perivascular dermatitis, the larger and more complex domain was associated with a slower rate of approximation towards the optimal solution path. CONCLUSIONS: Students benefit from a limited-enforcement tutoring system that leverages diagnostic algorithms but does not prevent alternative strategies. Even with limited enforcement, students converge toward the optimal solution path.

A method for automated detection of usability problems from client user interface events.

Think-aloud usability analysis provides extremely useful data but is very time-consuming and expensive to perform because of the extensive manual video analysis that is required. We describe a simple method for automated detection of usability problems from client user interface events for a developing medical intelligent tutoring system. The method incorporates (1) an agent-based method for communication that funnels all interface events and system responses to a centralized database, (2) a simple schema for representing interface events and higher order subgoals, and (3) an algorithm that reproduces the criteria used for manual coding of usability problems. A correction factor was empirically determining to account for the slower task performance of users when thinking aloud. We tested the validity of the method by simultaneously identifying usability problems using TAU and manually computing them from stored interface event data using the proposed algorithm. All usability problems that did not rely on verbal utterances were detectable with the proposed method.