A C lassi H onk: a system framework to annotate and classify vehicular honk from road traffic.

Some recent studies highlight that vehicular traffic and honking contribute to more than 50% of noise pollution in urban or sub-urban areas in developing countries, including Indian cities. Frequent honking has an adverse effect on health and hampers road safety, the environment, etc. Therefore, recognizing the various vehicle honks and classifying the honk of different vehicles can provide good insights into environmental noise pollution. Moreover, classifying honks based on vehicle types allows for the inference of contextual information of a location, area, or traffic. So far, the researchers have done outdoor sound classification and honk detection, where vehicular honks are collected in a controlled environment or in the absence of ambient noise. Such classification models fail to classify honk based on vehicle types. Therefore, it becomes imperative to design a system that can detect and classify honks of different types of vehicles to infer some contextual information. This paper presents a novel framework A C lassi H onk that performs raw vehicular honk sensing, data labeling, and classifies the honk into three major groups, i.e., light-weight vehicles, medium-weight vehicles, and heavy-weight vehicles. Raw audio samples of different vehicular honking are collected based on spatio-temporal characteristics and converted them into spectrogram images. A deep learning-based multi-label autoencoder model (MAE) is proposed for automated labeling of the unlabeled data samples, which provides 97.64% accuracy in contrast to existing deep learning-based data labeling methods. Further, various pre-trained models, namely Inception V3, ResNet50, MobileNet, and ShuffleNet are used and proposed an Ensembled Transfer Learning model (EnTL) for vehicle honks classification and performed comparative analysis. Results reveal that EnTL exhibits the best performance compared to pre-trained models and achieves 96.72% accuracy in our dataset. In addition, context of a location is identified based on these classified honk signatures in a city.

Strong versus Weak Data Labeling for Artificial Intelligence Algorithms in the Measurement of Geographic Atrophy.

Purpose: To gain an understanding of data labeling requirements to train deep learning models for measurement of geographic atrophy (GA) with fundus autofluorescence (FAF) images. Design: Evaluation of artificial intelligence (AI) algorithms. Subjects: The Age-Related Eye Disease Study 2 (AREDS2) images were used for training and cross-validation, and GA clinical trial images were used for testing. Methods: Training data consisted of 2 sets of FAF images; 1 with area measurements only and no indication of GA location (Weakly labeled) and the second with GA segmentation masks (Strongly labeled). Main Outcome Measures: Bland-Altman plots and scatter plots were used to compare GA area measurement between ground truth and AI measurements. The Dice coefficient was used to compare accuracy of segmentation of the Strong model. Results: In the cross-validation AREDS2 data set (n = 601), the mean (standard deviation [SD]) area of GA measured by human grader, Weakly labeled AI model, and Strongly labeled AI model was 6.65 (6.3) mm2, 6.83 (6.29) mm2, and 6.58 (6.24) mm2, respectively. The mean difference between ground truth and AI was 0.18 mm2 (95% confidence interval, [CI], -7.57 to 7.92) for the Weakly labeled model and -0.07 mm2 (95% CI, -1.61 to 1.47) for the Strongly labeled model. With GlaxoSmithKline testing data (n = 156), the mean (SD) GA area was 9.79 (5.6) mm2, 8.82 (4.61) mm2, and 9.55 (5.66) mm2 for human grader, Strongly labeled AI model, and Weakly labeled AI model, respectively. The mean difference between ground truth and AI for the 2 models was -0.97 mm2 (95% CI, -4.36 to 2.41) and -0.24 mm2 (95% CI, -4.98 to 4.49), respectively. The Dice coefficient was 0.99 for intergrader agreement, 0.89 for the cross-validation data, and 0.92 for the testing data. Conclusions: Deep learning models can achieve reasonable accuracy even with Weakly labeled data. Training methods that integrate large volumes of Weakly labeled images with small number of Strongly labeled images offer a promising solution to overcome the burden of cost and time for data labeling. Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Application of a human-centered design for embedded machine learning model to develop data labeling software with nurses: Human-to-Artificial Intelligence (H2AI).

BACKGROUND: Nurses are essential for assessing and managing acute pain in hospitalized patients, especially those who are unable to self-report pain. Given their role and subject matter expertise (SME), nurses are also essential for the design and development of a supervised machine learning (ML) model for pain detection and clinical decision support software (CDSS) in a pain recognition automated monitoring system (PRAMS). Our first step for developing PRAMS with nurses was to create SME-friendly data labeling software. PURPOSE: To develop an intuitive and efficient data labeling software solution, Human-to-Artificial Intelligence (H2AI). METHOD: The Human-centered Design for Embedded Machine Learning Solutions (HCDe-MLS) model was used to engage nurses. In this paper, HCDe-MLS will be explained using H2AI and PRAMS as illustrative cases. FINDINGS: Using HCDe-MLS, H2AI was developed and facilitated labeling of 139 videos (mean = 29.83 min) with 3189 images labeled (mean = 75 s) by 6 nurses. OpenCV was used for video-to-image pre-processing; and MobileFaceNet was used for default landmark placement on images. H2AI randomly assigned videos to nurses for data labeling, tracked labelers' inter-rater reliability, and stored labeled data to train ML models. CONCLUSIONS: Nurses' engagement in CDSS development was critical for ensuring the end-product addressed nurses' priorities, reflected nurses' cognitive and decision-making processes, and garnered nurses' trust for technology adoption.

[Inter-rater Reliability and Cost in Pricing for Creating Dataset Focused on Mediolateral Oblique View in Mammography].

PURPOSE: The purpose of this study was to assess inter-rater reliability and workload for creating accurate training data in the clinical evaluation of mammographic positioning for deep learning. METHODS: A total of 107 mammographic images without lesions were labeled by two certified radiologic technologists in seven items: six clinical image evaluation criteria in positioning and breast tissue density. The kappa coefficient was calculated as an indicator of interrater reliability. Furthermore, the labeling cost per image was calculated based on labeling time and salary for the technologists. RESULTS: The kappa coefficients were 0.71 for inframammary fold, 0.43 for nipple in profile, 0.45 for great pectoral muscle, 0.10 for symmetrical images, and 0.61 for retromammary fat. No significant difference was found in the coefficients of spread of breast tissue. The cost per image was calculated at 11.0 yen. CONCLUSION: The inter-rater reliability for the inframammary fold, nipple in profile, great pectoral muscle, and retromammary fat ranged from "moderate" to "substantial." The reliability for symmetrical images was "slight," indicating the need for a consensus among evaluators during labeling. The labeling cost was equivalent to or higher than that of existing services.

Medical volume segmentation by overfitting sparsely annotated data.

Purpose: Semantic segmentation is one of the most significant tasks in medical image computing, whereby deep neural networks have shown great success. Unfortunately, supervised approaches are very data-intensive, and obtaining reliable annotations is time-consuming and expensive. Sparsely labeled approaches, such as bounding boxes, have shown some success in reducing the annotation time. However, in 3D volume data, each slice must still be manually labeled. Approach: We evaluate approaches that reduce the annotation effort by reducing the number of slices that need to be labeled in a 3D volume. In a two-step process, a similarity metric is used to select slices that should be annotated by a trained radiologist. In the second step, a predictor is used to predict the segmentation mask for the rest of the slices. We evaluate different combinations of selectors and predictors on medical CT and MRI volumes. Thus we can determine that combination works best, and how far slice annotations can be reduced. Results: Our results show that for instance for the Medical Segmentation Decathlon-heart dataset, some selector, and predictor combinations allow for a Dice score 0.969 when only annotating 20% of slices per volume. Experiments on other datasets show a similarly positive trend. Conclusions: We evaluate a method that supports experts during the labeling of 3D medical volumes. Our approach makes it possible to drastically reduce the number of slices that need to be manually labeled. We present a recommendation in which selector predictor combination to use for different tasks and goals.

Data-Centric AI for Healthcare Fraud Detection.

Automated methods for detecting fraudulent healthcare providers have the potential to save billions of dollars in healthcare costs and improve the overall quality of patient care. This study presents a data-centric approach to improve healthcare fraud classification performance and reliability using Medicare claims data. Publicly available data from the Centers for Medicare & Medicaid Services (CMS) are used to construct nine large-scale labeled data sets for supervised learning. First, we leverage CMS data to curate the 2013-2019 Part B, Part D, and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies (DMEPOS) Medicare fraud classification data sets. We provide a review of each data set and data preparation techniques to create Medicare data sets for supervised learning and we propose an improved data labeling process. Next, we enrich the original Medicare fraud data sets with up to 58 new provider summary features. Finally, we address a common model evaluation pitfall and propose an adjusted cross-validation technique that mitigates target leakage to provide reliable evaluation results. Each data set is evaluated on the Medicare fraud classification task using extreme gradient boosting and random forest learners, multiple complementary performance metrics, and 95% confidence intervals. Results show that the new enriched data sets consistently outperform the original Medicare data sets that are currently used in related works. Our results encourage the data-centric machine learning workflow and provide a strong foundation for data understanding and preparation techniques for machine learning applications in healthcare fraud.

A supervised machine learning approach to discriminate the effect of carcass leachate on shallow groundwater quality around on-farm livestock mortality burial sites.

The evaluation of leachate leakage at livestock mortality burial sites is challenging, particularly when groundwater is previously contaminated by agro-livestock farming. Supervised machine learning was applied to discriminate the impacts of carcass leachate from pervasive groundwater contamination in the following order: data labeling, feature selection, synthetic data generation, and classification. Physicochemical data of 359 water samples were collected from burial pits (LC), monitoring wells near pits (MW), pre-existing shallow household wells (HW), and background wells with pervasive contamination (BG). A linear classification model was built using two representative groups (LC and BG) affected by different pollution sources as labeled data. A classifier was then applied to assess the impact of leachate leakage in MW and HW. As a result, leachate impacts were observed in 40% of MW samples, which indicates improper construction and management of some burial pits. Leachate impacts were also detected in six HW samples, up to 120 m downgradient, within one year. The quantitative decision-making tool to diagnose groundwater contamination with leachate leakage can contribute to ensuring timely responses to leakage. The proposed machine learning approach can also be used to improve the environmental impact assessment of water pollution by improper disposal of organic waste.

A novel approach for automatic annotation of human actions in 3D point clouds for flexible collaborative tasks with industrial robots.

Manual annotation for human action recognition with content semantics using 3D Point Cloud (3D-PC) in industrial environments consumes a lot of time and resources. This work aims to recognize, analyze, and model human actions to develop a framework for automatically extracting content semantics. Main Contributions of this work: 1. design a multi-layer structure of various DNN classifiers to detect and extract humans and dynamic objects using 3D-PC preciously, 2. empirical experiments with over 10 subjects for collecting datasets of human actions and activities in one industrial setting, 3. development of an intuitive GUI to verify human actions and its interaction activities with the environment, 4. design and implement a methodology for automatic sequence matching of human actions in 3D-PC. All these procedures are merged in the proposed framework and evaluated in one industrial Use-Case with flexible patch sizes. Comparing the new approach with standard methods has shown that the annotation process can be accelerated by 5.2 times through automation.

Rule-Enhanced Active Learning for Semi-Automated Weak Supervision.

A major bottleneck preventing the extension of deep learning systems to new domains is the prohibitive cost of acquiring sufficient training labels. Alternatives such as weak supervision, active learning, and fine-tuning of pretrained models reduce this burden but require substantial human input to select a highly informative subset of instances or to curate labeling functions. REGAL (Rule-Enhanced Generative Active Learning) is an improved framework for weakly supervised text classification that performs active learning over labeling functions rather than individual instances. REGAL interactively creates high-quality labeling patterns from raw text, enabling a single annotator to accurately label an entire dataset after initialization with three keywords for each class. Experiments demonstrate that REGAL extracts up to 3 times as many high-accuracy labeling functions from text as current state-of-the-art methods for interactive weak supervision, enabling REGAL to dramatically reduce the annotation burden of writing labeling functions for weak supervision. Statistical analysis reveals REGAL performs equal or significantly better than interactive weak supervision for five of six commonly used natural language processing (NLP) baseline datasets.

Automatically Annotated Dataset of a Ground Mobile Robot in Natural Environments via Gazebo Simulations.

This paper presents a new synthetic dataset obtained from Gazebo simulations of an Unmanned Ground Vehicle (UGV) moving on different natural environments. To this end, a Husky mobile robot equipped with a tridimensional (3D) Light Detection and Ranging (LiDAR) sensor, a stereo camera, a Global Navigation Satellite System (GNSS) receiver, an Inertial Measurement Unit (IMU) and wheel tachometers has followed several paths using the Robot Operating System (ROS). Both points from LiDAR scans and pixels from camera images, have been automatically labeled into their corresponding object class. For this purpose, unique reflectivity values and flat colors have been assigned to each object present in the modeled environments. As a result, a public dataset, which also includes 3D pose ground-truth, is provided as ROS bag files and as human-readable data. Potential applications include supervised learning and benchmarking for UGV navigation on natural environments. Moreover, to allow researchers to easily modify the dataset or to directly use the simulations, the required code has also been released.

Herramienta de código abierto para la anotación de imágenes oculares / Open-source tool for eye image annotation / Ferramenta de anotação de imagem de olho de código aberto

RESUMEN Introducción: El seguimiento del centro de la pupila usando imágenes de videooculografía se usa ampliamente para el diagnóstico de enfermedades del sistema nervioso. La diferencia entre el valor detectado automáticamente del centro de la pupila y el valor de referencia marcado por un especialista (anotación) determina la exactitud del diagnóstico. El proceso de anotación manual es muy laborioso, tedioso y propenso a errores humanos. Las anotaciones son esenciales para desarrollar y evaluar los algoritmos en el área de visión artificial, principalmente aquellos basados en el aprendizaje supervisado, sin embargo, existen pocas herramientas interactivas para realizar la anotación fiable del centro de la pupila. Objetivo: Desarrollar una herramienta de código abierto para anotar el centro de la pupila. Método: Se definieron los requisitos funcionales y no funcionales de la herramienta y se implementaron dos algoritmos para la anotación semiautomática del centro de la pupila basados en los métodos de ajuste de elipse y de círculo, a partir de varios puntos marcados por el especialista. Resultados: La aplicación software, denominada PUPILA, fue desarrollada en Python, desde marzo de 2020 a septiembre de 2020, y proporciona varias funciones auxiliares que facilitan la tarea del anotador. Conclusiones: La nueva herramienta proporciona un ambiente agradable e interactivo para anotar el centro de la pupila garantizando comodidad, exactitud y reducción de subjetividades en el trabajo del especialista. Es de código abierto y multiplataforma, lo que permite su compatibilidad con diversos dispositivos y su uso gratuito. Ha permitido anotar imágenes de bases de datos públicas y otras adquiridas experimentalmente.

ABSTRACT Introduction: The tracking of the pupil center using videoculography images is widely used for the diagnosis of diseases of the nervous system. The difference between the automatically detected value of the pupil center and the reference value marked by a specialist (annotation) determines the accuracy of the diagnosis. The manual annotation process is very laborious, tedious, and prone to human error. Annotations are essential to develop and evaluate algorithms in the area of artificial vision, mainly those based on supervised learning, however, there are few interactive tools to perform reliable annotation of the center of the pupil. Objective: To develop an open source tool to annotate the center of the pupil. Method: The functional and non-functional requirements of the tool are defined and two algorithms are implemented for the semi-automatic annotation of the center of the pupil based on the ellipse and circle adjustment methods, from several points marked by the specialist. Results: The software application, called PUPILA, was developed in Python, from March 2020 to September 2020, and provides various auxiliary functions that facilitate the annotator's task. Conclusions: The new tool provides an agreeable and interactive environment to record the center of the pupil, guaranteeing comfort, accuracy and reduction of subjectivities in the specialist's work. It is open source and cross-platform, allowing it to be compatible with various devices and free to use. It has made it possible to annotate images from public databases and others acquired experimentally.

RESUMO Introdução: O rastreamento do centro pupilar por meio de imagens de vídeo-oculografia é amplamente utilizado para o diagnóstico de doenças do sistema nervoso. A diferença entre o valor detectado automaticamente do centro da pupila e o valor de referência marcado por um especialista (anotação) determina a precisão do diagnóstico. O processo de anotação manual é muito trabalhoso, tedioso e propenso a erros humanos. As anotações são essenciais para desenvolver e avaliar algoritmos na área de visão artificial, principalmente aqueles baseados em aprendizado supervisionado, porém, existem poucas ferramentas interativas para realizar anotação confiável do centro do aluno. Objetivo: Desenvolver uma ferramenta de código aberto para anotar o centro da pupila. Método: Foram definidos os requisitos funcionais e não funcionais da ferramenta e implementados dois algoritmos para a anotação semiautomática do centro da pupila com base nos métodos de ajuste de elipse e círculo, a partir de vários pontos marcados pelo especialista. Resultados: O aplicativo de software, denominado PUPILA, foi desenvolvido em Python, no período de março de 2020 a setembro de 2020, e disponibiliza diversas funções auxiliares que facilitam a tarefa do anotador. Conclusões: A nova ferramenta proporciona um ambiente legais e interativo para registrar o centro do aluno, garantindo conforto, precisão e redução de subjetividades no trabalho do especialista. É de código aberto e multiplataforma, permitindo que seja compatível com vários dispositivos e de uso gratuito. Tornou possível anotar imagens de bancos de dados públicos e outros adquiridos experimentalmente.

Labels in a haystack: Approaches beyond supervised learning in biomedical applications.

Recent advances in biomedical machine learning demonstrate great potential for data-driven techniques in health care and biomedical research. However, this potential has thus far been hampered by both the scarcity of annotated data in the biomedical domain and the diversity of the domain's subfields. While unsupervised learning is capable of finding unknown patterns in the data by design, supervised learning requires human annotation to achieve the desired performance through training. With the latter performing vastly better than the former, the need for annotated datasets is high, but they are costly and laborious to obtain. This review explores a family of approaches existing between the supervised and the unsupervised problem setting. The goal of these algorithms is to make more efficient use of the available labeled data. The advantages and limitations of each approach are addressed and perspectives are provided.

Deep Learning of Histopathology Images at the Single Cell Level.

The tumor immune microenvironment (TIME) encompasses many heterogeneous cell types that engage in extensive crosstalk among the cancer, immune, and stromal components. The spatial organization of these different cell types in TIME could be used as biomarkers for predicting drug responses, prognosis and metastasis. Recently, deep learning approaches have been widely used for digital histopathology images for cancer diagnoses and prognoses. Furthermore, some recent approaches have attempted to integrate spatial and molecular omics data to better characterize the TIME. In this review we focus on machine learning-based digital histopathology image analysis methods for characterizing tumor ecosystem. In this review, we will consider three different scales of histopathological analyses that machine learning can operate within: whole slide image (WSI)-level, region of interest (ROI)-level, and cell-level. We will systematically review the various machine learning methods in these three scales with a focus on cell-level analysis. We will provide a perspective of workflow on generating cell-level training data sets using immunohistochemistry markers to "weakly-label" the cell types. We will describe some common steps in the workflow of preparing the data, as well as some limitations of this approach. Finally, we will discuss future opportunities of integrating molecular omics data with digital histopathology images for characterizing tumor ecosystem.

The Ontologies Community of Practice: A CGIAR Initiative for Big Data in Agrifood Systems.

Heterogeneous and multidisciplinary data generated by research on sustainable global agriculture and agrifood systems requires quality data labeling or annotation in order to be interoperable. As recommended by the FAIR principles, data, labels, and metadata must use controlled vocabularies and ontologies that are popular in the knowledge domain and commonly used by the community. Despite the existence of robust ontologies in the Life Sciences, there is currently no comprehensive full set of ontologies recommended for data annotation across agricultural research disciplines. In this paper, we discuss the added value of the Ontologies Community of Practice (CoP) of the CGIAR Platform for Big Data in Agriculture for harnessing relevant expertise in ontology development and identifying innovative solutions that support quality data annotation. The Ontologies CoP stimulates knowledge sharing among stakeholders, such as researchers, data managers, domain experts, experts in ontology design, and platform development teams.

Semi-Automated Data Labeling for Activity Recognition in Pervasive Healthcare.

Activity recognition, a key component in pervasive healthcare monitoring, relies on classification algorithms that require labeled data of individuals performing the activity of interest to train accurate models. Labeling data can be performed in a lab setting where an individual enacts the activity under controlled conditions. The ubiquity of mobile and wearable sensors allows the collection of large datasets from individuals performing activities in naturalistic conditions. Gathering accurate data labels for activity recognition is typically an expensive and time-consuming process. In this paper we present two novel approaches for semi-automated online data labeling performed by the individual executing the activity of interest. The approaches have been designed to address two of the limitations of self-annotation: (i) The burden on the user performing and annotating the activity, and (ii) the lack of accuracy due to the user labeling the data minutes or hours after the completion of an activity. The first approach is based on the recognition of subtle finger gestures performed in response to a data-labeling query. The second approach focuses on labeling activities that have an auditory manifestation and uses a classifier to have an initial estimation of the activity, and a conversational agent to ask the participant for clarification or for additional data. Both approaches are described, evaluated in controlled experiments to assess their feasibility and their advantages and limitations are discussed. Results show that while both studies have limitations, they achieve 80% to 90% precision.

Forearm Motion Recognition With Noncontact Capacitive Sensing.

This study presents a noncontact capacitive sensing method for forearm motion recognition. A method is proposed to record upper limb motion information from muscle contractions without contact with human skin, compensating for the limitations of existing sEMG-based methods. The sensing front-ends are designed based on human forearm shapes, and the forearm limb shape changes caused by muscle contractions will be represented by capacitance signals. After implementation of the capacitive sensing system, experiments on healthy subjects are conducted to evaluate the effectiveness. Nine motion patterns combined with 16 motion transitions are investigated on seven participants. We also designed an automatic data labeling method based on inertial signals from the measured hand, which greatly accelerated the training procedure. With the capacitive sensing system and the designed recognition algorithm, the method produced an average recognition of over 92%. Correct decisions could be made with approximately a 347-ms delay from the relaxed state to the time point of motion initiation. The confounding factors that affect the performances are also analyzed, including the sliding window length, the motion types and the external disturbances. We found the average accuracy increased to 98.7% when five motion patterns were recognized. The results of the study proved the feasibility and revealed the problems of the noncontact capacitive sensing approach on upper-limb motion sensing and recognition. Future efforts in this direction could be worthwhile for achieving more promising outcomes.

Learning classification models with soft-label information.

OBJECTIVE: Learning of classification models in medicine often relies on data labeled by a human expert. Since labeling of clinical data may be time-consuming, finding ways of alleviating the labeling costs is critical for our ability to automatically learn such models. In this paper we propose a new machine learning approach that is able to learn improved binary classification models more efficiently by refining the binary class information in the training phase with soft labels that reflect how strongly the human expert feels about the original class labels. MATERIALS AND METHODS: Two types of methods that can learn improved binary classification models from soft labels are proposed. The first relies on probabilistic/numeric labels, the other on ordinal categorical labels. We study and demonstrate the benefits of these methods for learning an alerting model for heparin induced thrombocytopenia. The experiments are conducted on the data of 377 patient instances labeled by three different human experts. The methods are compared using the area under the receiver operating characteristic curve (AUC) score. RESULTS: Our AUC results show that the new approach is capable of learning classification models more efficiently compared to traditional learning methods. The improvement in AUC is most remarkable when the number of examples we learn from is small. CONCLUSIONS: A new classification learning framework that lets us learn from auxiliary soft-label information provided by a human expert is a promising new direction for learning classification models from expert labels, reducing the time and cost needed to label data.