Overview and Discussion of the Competition on Legal Information, Extraction/Entailment (COLIEE) 2023.

We summarize the 10th Competition on Legal Information Extraction and Entailment. In this tenth edition, the competition included four tasks on case law and statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and a selected unseen case (Task 2). The statute law component includes an information retrieval task (Task 3), and an entailment/question-answering task based on retrieved civil code statutes (Task 4). Participation was open to any group based on any approach. Ten different teams participated in the case law competition tasks, most of them in more than one task. We received results from 8 teams for Task 1 (22 runs) and seven teams for Task 2 (18 runs). On the statute law task, there were 9 different teams participating, most in more than one task. 6 teams submitted a total of 16 runs for Task 3, and 9 teams submitted a total of 26 runs for Task 4. We describe the variety of approaches, our official evaluation, and analysis of our data and submission results.

Understanding psychiatric illness through natural language processing (UNDERPIN): Rationale, design, and methodology.

Introduction: Psychiatric disorders are diagnosed through observations of psychiatrists according to diagnostic criteria such as the DSM-5. Such observations, however, are mainly based on each psychiatrist's level of experience and often lack objectivity, potentially leading to disagreements among psychiatrists. In contrast, specific linguistic features can be observed in some psychiatric disorders, such as a loosening of associations in schizophrenia. Some studies explored biomarkers, but biomarkers have yet to be used in clinical practice. Aim: The purposes of this study are to create a large dataset of Japanese speech data labeled with detailed information on psychiatric disorders and neurocognitive disorders to quantify the linguistic features of those disorders using natural language processing and, finally, to develop objective and easy-to-use biomarkers for diagnosing and assessing the severity of them. Methods: This study will have a multi-center prospective design. The DSM-5 or ICD-11 criteria for major depressive disorder, bipolar disorder, schizophrenia, and anxiety disorder and for major and minor neurocognitive disorders will be regarded as the inclusion criteria for the psychiatric disorder samples. For the healthy subjects, the absence of a history of psychiatric disorders will be confirmed using the Mini-International Neuropsychiatric Interview (M.I.N.I.). The absence of current cognitive decline will be confirmed using the Mini-Mental State Examination (MMSE). A psychiatrist or psychologist will conduct 30-to-60-min interviews with each participant; these interviews will include free conversation, picture-description task, and story-telling task, all of which will be recorded using a microphone headset. In addition, the severity of disorders will be assessed using clinical rating scales. Data will be collected from each participant at least twice during the study period and up to a maximum of five times at an interval of at least one month. Discussion: This study is unique in its large sample size and the novelty of its method, and has potential for applications in many fields. We have some challenges regarding inter-rater reliability and the linguistic peculiarities of Japanese. As of September 2022, we have collected a total of >1000 records from >400 participants. To the best of our knowledge, this data sample is one of the largest in this field. Clinical Trial Registration: Identifier: UMIN000032141.

Speech Analysis Using Artificial Intelligence as a Peri-Operative Evaluation: A Case Report of a Patient with Temporal Lobe Epilepsy Secondary to Tuberous Sclerosis Complex Who Underwent Epilepsy Surgery.

BACKGROUND: Improved conversational fluency is sometimes identified postoperatively in patients with epilepsy, but improvements can be difficult to assess using tests such as the intelligence quotient (IQ) test. Evaluation of pre- and postoperative differences might be considered subjective at present because of the lack of objective criteria. Artificial intelligence (AI) could possibly be used to make the evaluations more objective. The aim of this case report is thus to analyze the speech of a young female patient with epilepsy before and after surgery. METHOD: The speech of a nine-year-old girl with epilepsy secondary to tuberous sclerosis complex is recorded during interviews one month before and two months after surgery. The recorded speech is then manually transcribed and annotated, and subsequently automatically analyzed using AI software. IQ testing is also conducted on both occasions. The patient remains seizure-free for at least 13 months postoperatively. RESULTS: There are decreases in total interview time and subjective case markers per second, whereas there are increases in morphemes and objective case markers per second. Postoperatively, IQ scores improve, except for the Perceptual Reasoning Index. CONCLUSIONS: AI analysis is able to identify differences in speech before and after epilepsy surgery upon an epilepsy patient with tuberous sclerosis complex.

De-identifying free text of Japanese electronic health records.

BACKGROUND: Recently, more electronic data sources are becoming available in the healthcare domain. Electronic health records (EHRs), with their vast amounts of potentially available data, can greatly improve healthcare. Although EHR de-identification is necessary to protect personal information, automatic de-identification of Japanese language EHRs has not been studied sufficiently. This study was conducted to raise de-identification performance for Japanese EHRs through classic machine learning, deep learning, and rule-based methods, depending on the dataset. RESULTS: Using three datasets, we implemented de-identification systems for Japanese EHRs and compared the de-identification performances found for rule-based, Conditional Random Fields (CRF), and Long-Short Term Memory (LSTM)-based methods. Gold standard tags for de-identification are annotated manually for age, hospital, person, sex, and time. We used different combinations of our datasets to train and evaluate our three methods. Our best F1-scores were 84.23, 68.19, and 81.67 points, respectively, for evaluations of the MedNLP dataset, a dummy EHR dataset that was virtually written by a medical doctor, and a Pathology Report dataset. Our LSTM-based method was the best performing, except for the MedNLP dataset. The rule-based method was best for the MedNLP dataset. The LSTM-based method achieved a good score of 83.07 points for this MedNLP dataset, which differs by 1.16 points from the best score obtained using the rule-based method. Results suggest that LSTM adapted well to different characteristics of our datasets. Our LSTM-based method performed better than our CRF-based method, yielding a 7.41 point F1-score, when applied to our Pathology Report dataset. This report is the first of study applying this LSTM-based method to any de-identification task of a Japanese EHR. CONCLUSIONS: Our LSTM-based machine learning method was able to extract named entities to be de-identified with better performance, in general, than that of our rule-based methods. However, machine learning methods are inadequate for processing expressions with low occurrence. Our future work will specifically examine the combination of LSTM and rule-based methods to achieve better performance. Our currently achieved level of performance is sufficiently higher than that of publicly available Japanese de-identification tools. Therefore, our system will be applied to actual de-identification tasks in hospitals.

Tweet Classification Toward Twitter-Based Disease Surveillance: New Data, Methods, and Evaluations.

BACKGROUND: The amount of medical and clinical-related information on the Web is increasing. Among the different types of information available, social media-based data obtained directly from people are particularly valuable and are attracting significant attention. To encourage medical natural language processing (NLP) research exploiting social media data, the 13th NII Testbeds and Community for Information access Research (NTCIR-13) Medical natural language processing for Web document (MedWeb) provides pseudo-Twitter messages in a cross-language and multi-label corpus, covering 3 languages (Japanese, English, and Chinese) and annotated with 8 symptom labels (such as cold, fever, and flu). Then, participants classify each tweet into 1 of the 2 categories: those containing a patient's symptom and those that do not. OBJECTIVE: This study aimed to present the results of groups participating in a Japanese subtask, English subtask, and Chinese subtask along with discussions, to clarify the issues that need to be resolved in the field of medical NLP. METHODS: In summary, 8 groups (19 systems) participated in the Japanese subtask, 4 groups (12 systems) participated in the English subtask, and 2 groups (6 systems) participated in the Chinese subtask. In total, 2 baseline systems were constructed for each subtask. The performance of the participant and baseline systems was assessed using the exact match accuracy, F-measure based on precision and recall, and Hamming loss. RESULTS: The best system achieved exactly 0.880 match accuracy, 0.920 F-measure, and 0.019 Hamming loss. The averages of match accuracy, F-measure, and Hamming loss for the Japanese subtask were 0.720, 0.820, and 0.051; those for the English subtask were 0.770, 0.850, and 0.037; and those for the Chinese subtask were 0.810, 0.880, and 0.032, respectively. CONCLUSIONS: This paper presented and discussed the performance of systems participating in the NTCIR-13 MedWeb task. As the MedWeb task settings can be formalized as the factualization of text, the achievement of this task could be directly applied to practical clinical applications.

[Medical Natural Language Processing for Japanese Language].

Artificial Intelligence technologies are recently attracting attentions. More medical information is available including inspection results and health records of numbers, images and documents, also including audio and video information of subjects. I briefly explain natural language processing of written and spoken language, then introduce a couple of research issues in medical natural language processing for the Japanese language.

Natural language processing-based assessment of consistency in summaries of product characteristics of generic antimicrobials.

To investigate consistency in summaries of product characteristics (SmPCs) of generic antimicrobials, we used natural language processing (NLP) to analyze and compare large amounts of text quantifying consistency between original and generic SmPCs. We manually compared each section of generic and original SmPCs for antimicrobials listed in the electronic Medicines Compendium in the United Kingdom, focusing on omissions and additions of clinically significant information (CSI). Independently, we quantified differences between the original and generic SmPCs using Kachako, a fully automatic NLP platform. Among the 137 antimicrobials listed in the electronic Medicines Compendium, we identified 193 pairs of original and generic antimicrobial SmPCs for the 48 antimicrobials for which generic SmPCs existed. Of these 193 pairs, 157 (81%) were consistent and 36 were inconsistent with the original SmPC. When the cut-off value of RATE (the index of similarity between two SmPCs) was set at 0.860, our NLP system effectively discriminated consistent generic SmPCs with a specificity of 100% and a sensitivity of 61%. We observed CSI omissions but not additions in the SmPC subsection related to pharmacokinetic properties. CSI additions but not omissions were found in the subsections dealing with therapeutic indications and fertility, pregnancy and lactation. Despite regulatory guidance, we observed substantial inconsistencies in the information in the United Kingdom SmPCs for antimicrobials. NLP technology proved to be a useful tool for checking large numbers of SmPCs for consistency.

BioHackathon series in 2011 and 2012: penetration of ontology and linked data in life science domains.

The application of semantic technologies to the integration of biological data and the interoperability of bioinformatics analysis and visualization tools has been the common theme of a series of annual BioHackathons hosted in Japan for the past five years. Here we provide a review of the activities and outcomes from the BioHackathons held in 2011 in Kyoto and 2012 in Toyama. In order to efficiently implement semantic technologies in the life sciences, participants formed various sub-groups and worked on the following topics: Resource Description Framework (RDF) models for specific domains, text mining of the literature, ontology development, essential metadata for biological databases, platforms to enable efficient Semantic Web technology development and interoperability, and the development of applications for Semantic Web data. In this review, we briefly introduce the themes covered by these sub-groups. The observations made, conclusions drawn, and software development projects that emerged from these activities are discussed.

Functional evaluation of activation-dependent alterations in the sialoglycan composition of T cells.

Sialic acids (Sias) are often conjugated to the termini of cellular glycans and are key mediators of cellular recognition. Sias are nine-carbon acidic sugars, and, in vertebrates, the major species are N-acetylneuraminic acid (Neu5Ac) and N-glycolylneuraminic acid (Neu5Gc), differing in structure at the C5 position. Previously, we described a positive feedback loop involving regulation of Neu5Gc expression in mouse B cells. In this context, Neu5Gc negatively regulated B-cell proliferation, and Neu5Gc expression was suppressed upon activation. Similarly, resting mouse T cells expressed principally Neu5Gc, and Neu5Ac was induced upon activation. In the present work, we used various probes to examine sialoglycan expression by activated T cells in terms of the Sia species expressed and the linkages of Sias to glycans. Upon T-cell activation, sialoglycan expression shifted from Neu5Gc to Neu5Ac, and the linkage shifted from α2,6 to α2,3. These changes altered the expression levels of sialic acid-binding immunoglobulin-like lectin (siglec) ligands. Expression of sialoadhesin and Siglec-F ligands increased, and that of CD22 ligands decreased. Neu5Gc exerted a negative effect on T-cell activation, both in terms of the proliferative response and in the context of activation marker expression. Suppression of Neu5Gc expression in mouse T and B cells prevented the development of nonspecific CD22-mediated T cell-B cell interactions. Our results suggest that an activation-dependent shift from Neu5Gc to Neu5Ac and replacement of α2,6 by α2,3 linkages may regulate immune cell interactions at several levels.

U-Compare bio-event meta-service: compatible BioNLP event extraction services.

BACKGROUND: Bio-molecular event extraction from literature is recognized as an important task of bio text mining and, as such, many relevant systems have been developed and made available during the last decade. While such systems provide useful services individually, there is a need for a meta-service to enable comparison and ensemble of such services, offering optimal solutions for various purposes. RESULTS: We have integrated nine event extraction systems in the U-Compare framework, making them intercompatible and interoperable with other U-Compare components. The U-Compare event meta-service provides various meta-level features for comparison and ensemble of multiple event extraction systems. Experimental results show that the performance improvements achieved by the ensemble are significant. CONCLUSIONS: While individual event extraction systems themselves provide useful features for bio text mining, the U-Compare meta-service is expected to improve the accessibility to the individual systems, and to enable meta-level uses over multiple event extraction systems such as comparison and ensemble.

[Evaluation of patient dose from diagnostic X-ray using glass badges].

Radiography is used in medical practices based on the principles of justification and optimization. Patients' exposure doses should be kept as low as still allows for image quality that does not disturb the diagnostic processes. To optimize diagnostic radiological procedures, the international commission on radiological protection (ICRP) advocated the establishment of diagnosis reference levels (DRLs) in the new basic recommendation (Publication 103) in 2007 by stating that "The DRL should be expressed as a readily measurable patient-dose-related quantity for the specified procedure." In this context, a simple and standardized dosimetric method is needed to verify the adaptability of a radiation dose to the DRLs. As a measuring instrument that has good availability, high accuracy, and easy operability, we adopted the glass badge system, which has been used for individual exposure dose management. We evaluated the accuracy of the system as a tool of simplified dosimetry of diagnostic X-rays by comparing it to the standard dosimetry of an ionization chamber. In an energy range of 50 to 140 kV for X-ray exposure, the glass badge showed values within 7% of or closer to those measured by the standard ionization chamber. Moreover, the glass badge measurement was independent of the rectification modes of the X-ray tubes. In conclusion, glass badge measurement is feasible for verifying diagnostic X-ray doses in relation to DRLs and can be widely used in hospitals and clinics.

The DBCLS BioHackathon: standardization and interoperability for bioinformatics web services and workflows. The DBCLS BioHackathon Consortium*.

Web services have become a key technology for bioinformatics, since life science databases are globally decentralized and the exponential increase in the amount of available data demands for efficient systems without the need to transfer entire databases for every step of an analysis. However, various incompatibilities among database resources and analysis services make it difficult to connect and integrate these into interoperable workflows. To resolve this situation, we invited domain specialists from web service providers, client software developers, Open Bio* projects, the BioMoby project and researchers of emerging areas where a standard exchange data format is not well established, for an intensive collaboration entitled the BioHackathon 2008. The meeting was hosted by the Database Center for Life Science (DBCLS) and Computational Biology Research Center (CBRC) and was held in Tokyo from February 11th to 15th, 2008. In this report we highlight the work accomplished and the common issues arisen from this event, including the standardization of data exchange formats and services in the emerging fields of glycoinformatics, biological interaction networks, text mining, and phyloinformatics. In addition, common shared object development based on BioSQL, as well as technical challenges in large data management, asynchronous services, and security are discussed. Consequently, we improved interoperability of web services in several fields, however, further cooperation among major database centers and continued collaborative efforts between service providers and software developers are still necessary for an effective advance in bioinformatics web service technologies.

Text mining meets workflow: linking U-Compare with Taverna.

UNLABELLED: Text mining from the biomedical literature is of increasing importance, yet it is not easy for the bioinformatics community to create and run text mining workflows due to the lack of accessibility and interoperability of the text mining resources. The U-Compare system provides a wide range of bio text mining resources in a highly interoperable workflow environment where workflows can very easily be created, executed, evaluated and visualized without coding. We have linked U-Compare to Taverna, a generic workflow system, to expose text mining functionality to the bioinformatics community. AVAILABILITY: http://u-compare.org/taverna.html, http://u-compare.org.

Extracting protein interactions from text with the unified AkaneRE event extraction system.

Currently, relation extraction (RE) and event extraction (EE) are the two main streams of biological information extraction. In 2009, the majority of these RE and EE research efforts were centered around the BioCreative II.5 Protein-Protein Interaction (PPI) challenge and the "BioNLP event extraction shared task." Although these challenges took somewhat different approaches, they share the same ultimate goal of extracting bio-knowledge from the literature. This paper compares the two challenge task definitions, and presents a unified system that was successfully applied in both these and several other PPI extraction task settings. The AkaneRE system has three parts: A core engine for RE, a pool of modules for specific solutions, and a configuration language to adapt the system to different tasks. The core engine is based on machine learning, using either Support Vector Machines or Statistical Classifiers and features extracted from given training data. The specific modules solve tasks like sentence boundary detection, tokenization, stemming, part-of-speech tagging, parsing, named entity recognition, generation of potential relations, generation of machine learning features for each relation, and finally, assignment of confidence scores and ranking of candidate relations. With these components, the AkaneRE system produces state-of-the-art results, and the system is freely available for academic purposes at http://www-tsujii.is.s.u-tokyo.ac.jp/satre/akane/.

U-Compare: share and compare text mining tools with UIMA.

SUMMARY: Due to the increasing number of text mining resources (tools and corpora) available to biologists, interoperability issues between these resources are becoming significant obstacles to using them effectively. UIMA, the Unstructured Information Management Architecture, is an open framework designed to aid in the construction of more interoperable tools. U-Compare is built on top of the UIMA framework, and provides both a concrete framework for out-of-the-box text mining and a sophisticated evaluation platform allowing users to run specific tools on any target text, generating both detailed statistics and instance-based visualizations of outputs. U-Compare is a joint project, providing the world's largest, and still growing, collection of UIMA-compatible resources. These resources, originally developed by different groups for a variety of domains, include many famous tools and corpora. U-Compare can be launched straight from the web, without needing to be manually installed. All U-Compare components are provided ready-to-use and can be combined easily via a drag-and-drop interface without any programming. External UIMA components can also simply be mixed with U-Compare components, without distinguishing between locally and remotely deployed resources. AVAILABILITY: http://u-compare.org/

Filling the gaps between tools and users: a tool comparator, using protein-protein interaction as an example.

Recently, several text mining programs have reached a near-practical level of performance. Some systems are already being used by biologists and database curators. However, it has also been recognized that current Natural Language Processing (NLP) and Text Mining (TM) technology is not easy to deploy, since research groups tend to develop systems that cater specifically to their own requirements. One of the major reasons for the difficulty of deployment of NLP/TM technology is that re-usability and interoperability of software tools are typically not considered during development. While some effort has been invested in making interoperable NLP/TM toolkits, the developers of end-to-end systems still often struggle to reuse NLP/TM tools, and often opt to develop similar programs from scratch instead. This is particularly the case in BioNLP, since the requirements of biologists are so diverse that NLP tools have to be adapted and re-organized in a much more extensive manner than was originally expected. Although generic frameworks like UIMA (Unstructured Information Management Architecture) provide promising ways to solve this problem, the solution that they provide is only partial. In order for truly interoperable toolkits to become a reality, we also need sharable type systems and a developer-friendly environment for software integration that includes functionality for systematic comparisons of available tools, a simple I/O interface, and visualization tools. In this paper, we describe such an environment that was developed based on UIMA, and we show its feasibility through our experience in developing a protein-protein interaction (PPI) extraction system.