Explaining predictors of discharge destination assessed along the patients' acute stroke journey.

INTRODUCTION: Accurate prediction of outcome destination at an early stage would help manage patients presenting with stroke. This study assessed the predictive ability of three machine learning (ML) algorithms to predict outcomes at four different stages as well as compared the predictive power of stroke scores. METHODS: Patients presenting with acute stroke to the Canberra Hospital between 2015 and 2019 were selected retrospectively. 16 potential predictors and one target variable (discharge destination) were obtained from the notes. k-Nearest Neighbour (kNN) and two ensemble-based classification algorithms (Adaptive Boosting and Bootstrap Aggregation) were employed to predict outcomes. Predictive accuracy was assessed at each of the four stages using both overall and per-class accuracy. The contribution of each variable to the prediction outcome was evaluated by the ensemble-based algorithm and using the Relief feature selection algorithm. Various combinations of stroke scores were tested using the aforementioned models. RESULTS: Of the three ML models, Adaptive Boosting demonstrated the highest accuracy (90%) at Stage 4 in predicting death while the highest overall accuracy (81.7%) was achieved by kNN (k=2/City-block distance). Feature importance analysis has shown that the most important features are the 24-hour Scandinavian Stroke Scale (SSS) and 24-hour National Institutes of Health Stroke Scale (NIHSS) scores, dyslipidaemia, hypertension and premorbid mRS score. For the initial and 24-hour scores, there was a higher correlation (0.93) between SSS scores than for NIHSS scores (0.81). Reducing the overall four scores to InitSSS/24hrNIHSS increased accuracy to 95% in predicting death (Adaptive Boosting) and overall accuracy to 85.4% (kNN). Accuracies at Stage 2 (pre-treatment, 11 predictors) were not far behind those at Stage 4. CONCLUSION: Our findings suggest that even in the early stages of management, a clinically useful prediction regarding discharge destination can be made. Adaptive Boosting might be the best ML model, especially when it comes to predicting death. The predictors' importance analysis also showed that dyslipidemia and hypertension contributed to the discharge outcome even more than expected. Further, surprisingly using mixed score systems might also lead to higher prediction accuracies.

Explainable discovery of disease biomarkers: The case of ovarian cancer to illustrate the best practice in machine learning and Shapley analysis.

OBJECTIVE: Ovarian cancer is a significant health issue with lasting impacts on the community. Despite recent advances in surgical, chemotherapeutic and radiotherapeutic interventions, they have had only marginal impacts due to an inability to identify biomarkers at an early stage. Biomarker discovery is challenging, yet essential for improving drug discovery and clinical care. Machine learning (ML) techniques are invaluable for recognising complex patterns in biomarkers compared to conventional methods, yet they can lack physical insights into diagnosis. eXplainable Artificial Intelligence (XAI) is capable of providing deeper insights into the decision-making of complex ML algorithms increasing their applicability. We aim to introduce best practice for combining ML and XAI techniques for biomarker validation tasks. METHODS: We focused on classification tasks and a game theoretic approach based on Shapley values to build and evaluate models and visualise results. We described the workflow and apply the pipeline in a case study using the CDAS PLCO Ovarian Biomarkers dataset to demonstrate the potential for accuracy and utility. RESULTS: The case study results demonstrate the efficacy of the ML pipeline, its consistency, and advantages compared to conventional statistical approaches. CONCLUSION: The resulting guidelines provide a general framework for practical application of XAI in medical research that can inform clinicians and validate and explain cancer biomarkers.

The Potential of Current Noninvasive Wearable Technology for the Monitoring of Physiological Signals in the Management of Type 1 Diabetes: Literature Survey.

BACKGROUND: Monitoring glucose and other parameters in persons with type 1 diabetes (T1D) can enhance acute glycemic management and the diagnosis of long-term complications of the disease. For most persons living with T1D, the determination of insulin delivery is based on a single measured parameter-glucose. To date, wearable sensors exist that enable the seamless, noninvasive, and low-cost monitoring of multiple physiological parameters. OBJECTIVE: The objective of this literature survey is to explore whether some of the physiological parameters that can be monitored with noninvasive, wearable sensors may be used to enhance T1D management. METHODS: A list of physiological parameters, which can be monitored by using wearable sensors available in 2020, was compiled by a thorough review of the devices available in the market. A literature survey was performed using search terms related to T1D combined with the identified physiological parameters. The selected publications were restricted to human studies, which had at least their abstracts available. The PubMed and Scopus databases were interrogated. In total, 77 articles were retained and analyzed based on the following two axes: the reported relations between these parameters and T1D, which were found by comparing persons with T1D and healthy control participants, and the potential areas for T1D enhancement via the further analysis of the found relationships in studies working within T1D cohorts. RESULTS: On the basis of our search methodology, 626 articles were returned, and after applying our exclusion criteria, 77 (12.3%) articles were retained. Physiological parameters with potential for monitoring by using noninvasive wearable devices in persons with T1D included those related to cardiac autonomic function, cardiorespiratory control balance and fitness, sudomotor function, and skin temperature. Cardiac autonomic function measures, particularly the indices of heart rate and heart rate variability, have been shown to be valuable in diagnosing and monitoring cardiac autonomic neuropathy and, potentially, predicting and detecting hypoglycemia. All identified physiological parameters were shown to be associated with some aspects of diabetes complications, such as retinopathy, neuropathy, and nephropathy, as well as macrovascular disease, with capacity for early risk prediction. However, although they can be monitored by available wearable sensors, most studies have yet to adopt them, as opposed to using more conventional devices. CONCLUSIONS: Wearable sensors have the potential to augment T1D sensing with additional, informative biomarkers, which can be monitored noninvasively, seamlessly, and continuously. However, significant challenges associated with measurement accuracy, removal of noise and motion artifacts, and smart decision-making exist. Consequently, research should focus on harvesting the information hidden in the complex data generated by wearable sensors and on developing models and smart decision strategies to optimize the incorporation of these novel inputs into T1D interventions.

The role of machine learning in developing non-magnetic resonance imaging based biomarkers for multiple sclerosis: a systematic review.

BACKGROUND: Multiple sclerosis (MS) is a neurological condition whose symptoms, severity, and progression over time vary enormously among individuals. Ideally, each person living with MS should be provided with an accurate prognosis at the time of diagnosis, precision in initial and subsequent treatment decisions, and improved timeliness in detecting the need to reassess treatment regimens. To manage these three components, discovering an accurate, objective measure of overall disease severity is essential. Machine learning (ML) algorithms can contribute to finding such a clinically useful biomarker of MS through their ability to search and analyze datasets about potential biomarkers at scale. Our aim was to conduct a systematic review to determine how, and in what way, ML has been applied to the study of MS biomarkers on data from sources other than magnetic resonance imaging. METHODS: Systematic searches through eight databases were conducted for literature published in 2014-2020 on MS and specified ML algorithms. RESULTS: Of the 1, 052 returned papers, 66 met the inclusion criteria. All included papers addressed developing classifiers for MS identification or measuring its progression, typically, using hold-out evaluation on subsets of fewer than 200 participants with MS. These classifiers focused on biomarkers of MS, ranging from those derived from omics and phenotypical data (34.5% clinical, 33.3% biological, 23.0% physiological, and 9.2% drug response). Algorithmic choices were dependent on both the amount of data available for supervised ML (91.5%; 49.2% classification and 42.3% regression) and the requirement to be able to justify the resulting decision-making principles in healthcare settings. Therefore, algorithms based on decision trees and support vector machines were commonly used, and the maximum average performance of 89.9% AUC was found in random forests comparing with other ML algorithms. CONCLUSIONS: ML is applicable to determining how candidate biomarkers perform in the assessment of disease severity. However, applying ML research to develop decision aids to help clinicians optimize treatment strategies and analyze treatment responses in individual patients calls for creating appropriate data resources and shared experimental protocols. They should target proceeding from segregated classification of signals or natural language to both holistic analyses across data modalities and clinically-meaningful differentiation of disease.

'It struck at the heart of who I thought I was': A meta-synthesis of the qualitative literature examining the experiences of people with multiple sclerosis.

BACKGROUND: People with multiple sclerosis (MS) have varied experiences and approaches to self-management. This review aimed to explore the experiences of people with MS, and consider the implications of these experiences for clinical practice and research. METHODS: A meta-synthesis of the qualitative literature examining experiences of people with MS was conducted using systematic searches of ProQuest, PubMed, CINAHL and PsycINFO. We incorporated feedback from team members with MS as expert patient knowledge-users to capture the complex subjectivities of persons with lived experience responding to research on lived experience of the same disease. RESULTS: Of 1680 unique articles, 77 met the inclusion criteria. We identified five experiential themes: (a) the quest for knowledge, expertise and understanding, (b) uncertain trajectories (c) loss of valued roles and activities, and the threat of a changing identity, (d) managing fatigue and its impacts on life and relationships, and (f) adapting to life with MS. These themes were distributed across three domains related to disease (symptoms; diagnosis; progression and relapse) and two contexts (the health-care sector; and work, social and family life). CONCLUSION: The majority of people in the studies included in this review expressed a determination to adapt to MS, indicating a strong motivation for people with MS and clinicians to collaborate in the quest for knowledge. Clinicians caring for people with MS need to consider the experiential and social outcomes of this disease such as fatigue and the preservation of valued social roles, and incorporate this into case management and clinical planning.

Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances.

The importance of incorporating Natural Language Processing (NLP) methods in clinical informatics research has been increasingly recognized over the past years, and has led to transformative advances. Typically, clinical NLP systems are developed and evaluated on word, sentence, or document level annotations that model specific attributes and features, such as document content (e.g., patient status, or report type), document section types (e.g., current medications, past medical history, or discharge summary), named entities and concepts (e.g., diagnoses, symptoms, or treatments) or semantic attributes (e.g., negation, severity, or temporality). From a clinical perspective, on the other hand, research studies are typically modelled and evaluated on a patient- or population-level, such as predicting how a patient group might respond to specific treatments or patient monitoring over time. While some NLP tasks consider predictions at the individual or group user level, these tasks still constitute a minority. Owing to the discrepancy between scientific objectives of each field, and because of differences in methodological evaluation priorities, there is no clear alignment between these evaluation approaches. Here we provide a broad summary and outline of the challenging issues involved in defining appropriate intrinsic and extrinsic evaluation methods for NLP research that is to be used for clinical outcomes research, and vice versa. A particular focus is placed on mental health research, an area still relatively understudied by the clinical NLP research community, but where NLP methods are of notable relevance. Recent advances in clinical NLP method development have been significant, but we propose more emphasis needs to be placed on rigorous evaluation for the field to advance further. To enable this, we provide actionable suggestions, including a minimal protocol that could be used when reporting clinical NLP method development and its evaluation.

Automatic detection of patients with invasive fungal disease from free-text computed tomography (CT) scans.

BACKGROUND: Invasive fungal diseases (IFDs) are associated with considerable health and economic costs. Surveillance of the more diagnostically challenging invasive fungal diseases, specifically of the sino-pulmonary system, is not feasible for many hospitals because case finding is a costly and labour intensive exercise. We developed text classifiers for detecting such IFDs from free-text radiology (CT) reports, using machine-learning techniques. METHOD: We obtained free-text reports of CT scans performed over a specific hospitalisation period (2003-2011), for 264 IFD and 289 control patients from three tertiary hospitals. We analysed IFD evidence at patient, report, and sentence levels. Three infectious disease experts annotated the reports of 73 IFD-positive patients for language suggestive of IFD at sentence level, and graded the sentences as to whether they suggested or excluded the presence of IFD. Reliable agreement between annotators was obtained and this was used as training data for our classifiers. We tested a variety of Machine Learning (ML), rule based, and hybrid systems, with feature types including bags of words, bags of phrases, and bags of concepts, as well as report-level structured features. Evaluation was carried out over a robust framework with separate Development and Held-Out datasets. RESULTS: The best systems (using Support Vector Machines) achieved very high recall at report- and patient-levels over unseen data: 95% and 100% respectively. Precision at report-level over held-out data was 71%; however, most of the associated false-positive reports (53%) belonged to patients who had a previous positive report appropriately flagged by the classifier, reducing negative impact in practice. CONCLUSIONS: Our machine learning application holds the potential for developing systematic IFD surveillance systems for hospital populations.

A systematic review of speech recognition technology in health care.

BACKGROUND: To undertake a systematic review of existing literature relating to speech recognition technology and its application within health care. METHODS: A systematic review of existing literature from 2000 was undertaken. Inclusion criteria were: all papers that referred to speech recognition (SR) in health care settings, used by health professionals (allied health, medicine, nursing, technical or support staff), with an evaluation or patient or staff outcomes. Experimental and non-experimental designs were considered. Six databases (Ebscohost including CINAHL, EMBASE, MEDLINE including the Cochrane Database of Systematic Reviews, OVID Technologies, PreMED-LINE, PsycINFO) were searched by a qualified health librarian trained in systematic review searches initially capturing 1,730 references. Fourteen studies met the inclusion criteria and were retained. RESULTS: The heterogeneity of the studies made comparative analysis and synthesis of the data challenging resulting in a narrative presentation of the results. SR, although not as accurate as human transcription, does deliver reduced turnaround times for reporting and cost-effective reporting, although equivocal evidence of improved workflow processes. CONCLUSIONS: SR systems have substantial benefits and should be considered in light of the cost and selection of the SR system, training requirements, length of the transcription task, potential use of macros and templates, the presence of accented voices or experienced and in-experienced typists, and workflow patterns.

A usability framework for speech recognition technologies in clinical handover: a pre-implementation study.

A multi-disciplinary research team is undertaking a trial of speech-to-text (STT) technology for clinical handover management. Speech-to-text technologies allow for the capture of handover data from voice recordings using speech recognition software and systems. The text documents created from this system can be used together with traditional handover notes and checklists to enhance the depth and breadth of data available for clinical decision-making at the point of care and so improve patient care and reduce medical errors. This paper reports on a preliminary study of perceived usability by nurses of speech-to-text technology based on interviews at a "test day" and using a user-task-technology usability framework to explore expectations of nurses of the use of speech-to-text (STT) technology for clinical handover. The results of this study will be used to design field studies to test the use of speech-to-text (STT) technologies at the point of care in several hospital settings.

Evaluating Effects of Resting-State Electroencephalography Data Pre-Processing on a Machine Learning Task for Parkinson's Disease.

Resting-state electroencephalography pre-processing methods in machine learning studies into Parkinson's disease classification vary widely. Here three separate data sets were pre-processed to four different stages to investigate the effects on evaluation metrics, using power features from six regions-of-interest, Random Forest Classifiers for feature selection, and Support Vector Machines for classification. This showed muscle artefact inflated evaluation metrics, and alpha and theta band features produced the best results when fully pre-processing data.

Early diabetic eye damage: Comparing detection methods using diagnostic power.

It is now clear that retinal neuropathy precedes classical microvascular retinopathy in diabetes. Therefore, tests that underpin useful new endpoints must provide high diagnostic power well before the onset of moderate diabetic retinopathy. Hence, we compare detection methods of early diabetic eye damage. We reviewed data from a range of functional and structural studies of early diabetic eye disease and computed standardized effect size as a measure of diagnostic power, allowing the studies to be compared quantitatively. We then derived minimum performance criteria for tests to provide useful clinical endpoints. This included the criteria that tests should be rapid and easy so that children with type 1 diabetes can be followed into adulthood with the same tests. We also defined attributes that lend test data to further improve performance using Machine/Deep Learning. Data from a new form of objective perimetry suggested that the criteria are achievable.

The use of event-related potentials in the investigation of cognitive performance in people with Multiple Sclerosis: Systematic review.

A biomarker of cognition in Multiple Sclerosis (MS) that is independent from the response of people with MS (PwMS) to test questions would provide a more holistic assessment of cognitive decline. One suggested method involves event-related potentials (ERPs). This systematic review tried to answer five questions about the use of ERPs in distinguishing PwMS from controls: which stimulus modality, which experimental paradigm, which electrodes, and which ERP components are most discriminatory, and whether amplitude or latency is a better measure. Our results show larger pooled effect sizes for visual stimuli than auditory stimuli, and larger pooled effect sizes for latency measurements than amplitude measurements. We observed great heterogeneity in methods and suggest that future research would benefit from more uniformity in methods and that results should be reported for the individual subtypes of PwMS. With more standardised methods, ERPs have the potential to be developed into a clinical tool in MS.

Has machine learning over-promised in healthcare?: A critical analysis and a proposal for improved evaluation, with evidence from Parkinson's disease.

Adoption of artificial intelligence (AI) by the medical community has long been anticipated, endorsed by a stream of machine learning literature showcasing AI systems that yield extraordinary performance. However, many of these systems are likely over-promising and will under-deliver in practice. One key reason is the community's failure to acknowledge and address the presence of inflationary effects in the data. These simultaneously inflate evaluation performance and prevent a model from learning the underlying task, thus severely misrepresenting how that model would perform in the real world. This paper investigated the impact of these inflationary effects on healthcare tasks, as well as how these effects can be addressed. Specifically, we defined three inflationary effects that occur in medical data sets and allow models to easily reach small training losses and prevent skillful learning. We investigated two data sets of sustained vowel phonation from participants with and without Parkinson's disease, and revealed that published models which have achieved high classification performances on these were artificially enhanced due to the inflationary effects. Our experiments showed that removing each inflationary effect corresponded with a decrease in classification accuracy, and that removing all inflationary effects reduced the evaluated performance by up to 30%. Additionally, the performance on a more realistic test set increased, suggesting that the removal of these inflationary effects enabled the model to better learn the underlying task and generalize. Source code is available at https://github.com/Wenbo-G/pd-phonation-analysis under the MIT license.

Jointly Predicting Postprandial Hypoglycemia and Hyperglycemia Using Continuous Glucose Monitoring Data in Type 1 Diabetes.

The development of continuous glucose monitoring (CGM) systems has enabled people with type 1 diabetes mellitus (T1DM) to track their glucose trajectory in real-time and inspired research in personalised glucose prediction. In this paper, our aim is to predict postprandial abnormal-glycemia events. Different from prior research which focuses on hypoglycemia only, we make the first attempt to establish our problem as the joint prediction of hyperglycemia and hypoglycemia. On this basis, we propose a machine learning model that learns from the pattern of 1 hour past glucose and makes predictions for the two tasks simultaneously using a unified backbone. Key benefits of our methodology include 1) requiring only the CGM sequence as the input, thus making it more widely applicable than other counterparts using extra inputs such as the nutrition details, and 2) minimising the computational cost as the two tasks are unified into a single model. Our experiments on the openly available OhioT1DM dataset achieve state-of-the-art performance (Matthew's correlation coefficient of 0.61 for hyperglycemia and 0.48 for hypoglycemia). To encourage further study, we release our codes at https://github.com/r-cui/PostprandialHyperHypoPrediction under the MIT license.

Evaluating methods of oversampling and averaging resting-state electroencephalography data in classifying Parkinson's disease.

Collecting resting-state electroencephalography (RSEEG) data is time-consuming and data sets are therefore often small. Because many machine learning (ML) algorithms work better with ample data, researchers looking to use RSEEG and ML to develop diagnostic models have used oversampling methods that may seem to contradict averaging methods used in conventional electroencephalography (EEG) research to improve the signal-to-noise ratio. Using eyes open (EO) and eyes closed (EC) recordings from 3 different research groups, we investigated the effect of different averaging and oversampling methods on classification metrics when classifying people with Parkinson's disease (PD) and controls. Both EC and EO recordings were used due to differences found between these methods. Our results indicated that grouping 58 electrodes into regions-of-interest (ROI) based on anatomical location is preferable to using single electrodes. Furthermore, although recording EO data led to slightly better classification, the number of data points for each participant was reduced and recordings for three participants entirely lost during pre-processing due to a higher level of artefacts than in the EC data.Clinical relevance- RSEEG is a potential biomarker for the diagnosis and prognostication of PD, but for RSEEG to have clinical relevance, it is necessary to establish which averaging and oversampling of data most reliably segregates the classes for people with PD and controls. We found that using of ROIs and EC data performed the best, as EO data was often contaminated with artefacts.

High Accuracy Protein Identification: Fusion of Solid-State Nanopore Sensing and Machine Learning.

Proteins are arguably one of the most important class of biomarkers for health diagnostic purposes. Label-free solid-state nanopore sensing is a versatile technique for sensing and analyzing biomolecules such as proteins at single-molecule level. While molecular-level information on size, shape, and charge of proteins can be assessed by nanopores, the identification of proteins with comparable sizes remains a challenge. Here, solid-state nanopore sensing is combined with machine learning to address this challenge. The translocations of four similarly sized proteins is assessed using amplifiers with bandwidths (BWs) of 100 kHz and 10 MHz, the highest bandwidth reported for protein sensing, using nanopores fabricated in <10 nm thick silicon nitride membranes. F-values of up to 65.9% and 83.2% (without clustering of the protein signals) are achieved with 100 kHz and 10 MHz BW measurements, respectively, for identification of the four proteins. The accuracy of protein identification is further enhanced by classifying the signals into different clusters based on signal attributes, with F-value and specificity of up to 88.7% and 96.4%, respectively, for combinations of four proteins. The combined use of high bandwidth instruments, advanced clustering and machine learning methods allows label-free identification of proteins with high accuracy.

Toward Diabetes Device Development That Is Mindful to the Needs of Young People Living With Type 1 Diabetes: A Data- and Theory-Driven Qualitative Study.

BACKGROUND: An important strategy to understand young people's needs regarding technologies for type 1 diabetes mellitus (T1DM) management is to examine their day-to-day experiences with these technologies. OBJECTIVE: This study aimed to examine young people's and their caregivers' experiences with diabetes technologies in an exploratory way and relate the findings to the existing technology acceptance and technology design theories. On the basis of this procedure, we aimed to develop device characteristics that meet young people's needs. METHODS: Overall, 16 in-person and web-based face-to-face interviews were conducted with 7 female and 9 male young people with T1DM (aged between 12 and 17 years) and their parents between December 2019 and July 2020. The participants were recruited through a pediatric diabetes clinic based at Canberra Hospital. Data-driven thematic analysis was performed before theory-driven analysis to incorporate empirical data results into the unified theory of acceptance and use of technology (UTAUT) and value-sensitive design (VSD). We used the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist for reporting our research procedure and findings. In this paper, we summarize the key device characteristics that meet young people's needs. RESULTS: Summarized interview themes from the data-driven analysis included aspects of self-management, device use, technological characteristics, and feelings associated with device types. In the subsequent theory-driven analysis, the interview themes aligned with all UTAUT and VSD factors except for one (privacy). Privacy concerns or related aspects were not reported throughout the interviews, and none of the participants made any mention of data privacy. Discussions around ideal device characteristics focused on reliability, flexibility, and automated closed loop systems that enable young people with T1DM to lead an independent life and alleviate parental anxiety. However, in line with a previous systematic review by Brew-Sam et al, the analysis showed that reality deviated from these expectations, with inaccuracy problems reported in continuous glucose monitoring devices and technical failures occurring in both continuous glucose monitoring devices and insulin pumps. CONCLUSIONS: Our research highlights the benefits of the transdisciplinary use of exploratory and theory-informed methods for designing improved technologies. Technologies for diabetes self-management require continual advancement to meet the needs and expectations of young people with T1DM and their caregivers. The UTAUT and VSD approaches were found useful as a combined foundation for structuring the findings of our study.

Towards an international electronic repository and virtual laboratory of open data and open-source software for telehealth research: comparison of international, Australian and Finnish privacy policies.

Health data includes all content related to health in all data formats, document types, information systems, publication media and languages from all specialties, organisations, regions, states and countries. Capabilities to share, integrate and compare these data contents, clinical trial results and other evaluation outcomes together with telehealth applications for data processing are critical to accelerate discovery and its diffusion to clinical practice. However, the same ethical and legal frameworks that protect privacy hinder this open data and open-source code approach and the issues accumulate if moving data across national, regional or organisational borders. This can be seen as one of the reasons why many telehealth applications and health-research findings tend to be limited to very narrow domains and global results are lacking. The aim of this paper is to take steps towards establishing an international electronic repository and virtual laboratory of open data and open-source code for research purposes by comparing international, Australian and Finnish frameworks. The frameworks seem to be fundamentally similar; they apply the principles of accountability and adequacy to using and disclosing personal data. Their requirements to inform data subjects about the purposes of data collection and use before the dataset is collected, assure that individuals are no longer identifiable and to destruct data when the research activities are finished make sharing data and even secondary data difficult. Using the Internet or cloud services for sharing without proper approvals by ethics committees is technically not allowed if the data are stored in another country. The research community needs to overcome these barriers and develop a virtual laboratory, which operates on distributed data repositories. This empowers the community by enabling systematic evaluations of new technologies and research hypotheses on a rich variety of data and against existing applications, and subsequent tracking of quality improvements in time.

Analyzing Suicide Risk From Linguistic Features in Social Media: Evaluation Study.

BACKGROUND: Effective suicide risk assessments and interventions are vital for suicide prevention. Although assessing such risks is best done by health care professionals, people experiencing suicidal ideation may not seek help. Hence, machine learning (ML) and computational linguistics can provide analytical tools for understanding and analyzing risks. This, therefore, facilitates suicide intervention and prevention. OBJECTIVE: This study aims to explore, using statistical analyses and ML, whether computerized language analysis could be applied to assess and better understand a person's suicide risk on social media. METHODS: We used the University of Maryland Suicidality Dataset comprising text posts written by users (N=866) of mental health-related forums on Reddit. Each user was classified with a suicide risk rating (no, low, moderate, or severe) by either medical experts or crowdsourced annotators, denoting their estimated likelihood of dying by suicide. In language analysis, the Linguistic Inquiry and Word Count lexicon assessed sentiment, thinking styles, and part of speech, whereas readability was explored using the TextStat library. The Mann-Whitney U test identified differences between at-risk (low, moderate, and severe risk) and no-risk users. Meanwhile, the Kruskal-Wallis test and Spearman correlation coefficient were used for granular analysis between risk levels and to identify redundancy, respectively. In the ML experiments, gradient boost, random forest, and support vector machine models were trained using 10-fold cross validation. The area under the receiver operator curve and F1-score were the primary measures. Finally, permutation importance uncovered the features that contributed the most to each model's decision-making. RESULTS: Statistically significant differences (P<.05) were identified between the at-risk (671/866, 77.5%) and no-risk groups (195/866, 22.5%). This was true for both the crowd- and expert-annotated samples. Overall, at-risk users had higher median values for most variables (authenticity, first-person pronouns, and negation), with a notable exception of clout, which indicated that at-risk users were less likely to engage in social posturing. A high positive correlation (ρ>0.84) was present between the part of speech variables, which implied redundancy and demonstrated the utility of aggregate features. All ML models performed similarly in their area under the curve (0.66-0.68); however, the random forest and gradient boost models were noticeably better in their F1-score (0.65 and 0.62) than the support vector machine (0.52). The features that contributed the most to the ML models were authenticity, clout, and negative emotions. CONCLUSIONS: In summary, our statistical analyses found linguistic features associated with suicide risk, such as social posturing (eg, authenticity and clout), first-person singular pronouns, and negation. This increased our understanding of the behavioral and thought patterns of social media users and provided insights into the mechanisms behind ML models. We also demonstrated the applicative potential of ML in assisting health care professionals to assess and manage individuals experiencing suicide risk.