Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

BACKGROUND AND PURPOSE: Artificial intelligence (AI) models in radiology are frequently developed and validated using datasets from a single institution and are rarely tested on independent, external datasets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multi-center AI competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency. MATERIALS AND METHODS: In total, 1201 anonymized, full-head NCCT clinical scans from five institutions were pooled to form the dataset. The dataset encompassed normal studies as well as pathologies including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these-task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed each model's performance. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each AI model, along with the area under the ROC curve (AUROC). RESULTS: 1177 studies were processed by four teams. The median age of patients was 62, with an interquartile range of 33. 19 teams from various academic institutions registered for the competition. Of these, four teams submitted their final results. No commercial entities participated in the competition. For task 1, AUROCs ranged from 0.49 to 0.59. For task 2, two teams completed the task with AUROC values of 0.57 and 0.52. For task 3, teams had little to no agreement with the ground truth. CONCLUSIONS: To assess the performance of AI models in real-world clinical scenarios, we analyzed their performance in the ASFNR AI Competition. The first ASFNR Competition underscored the gap between expectation and reality; the models largely fell short in their assessments. As the integration of AI tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.ABBREVIATIONS: AI = artificial intelligence; ASFNR = American Society of Functional Neuroradiology; AUROC = area under the receiver operating characteristic curve; DICOM = Digital Imaging and Communications in Medicine; GEE = generalized estimation equation; IQR = interquartile range; NPV = negative predictive value; PPV = positive predictive value; ROC = receiver operating characteristic; TBI = traumatic brain injury.

Automated extraction of heart rate variability from magnetoencephalography signals.

Magnetoencephalography (MEG) measures magnetic fluctuations in the brain generated by neural processes, some of which, such as cardiac signals, are generally removed as artifacts and discarded. However, heart rate variability (HRV) has long been regarded as a biomarker related to autonomic function, suggesting the cardiac signal in MEG contains valuable information that can provide supplemental health information about a patient. To enable access to these ancillary HRV data, we created an automated extraction tool capable of capturing HRV directly from raw MEG data with artificial intelligence. Five scans were conducted with simultaneous MEG and electrocardiogram (ECG) acquisition, which provides a ground truth metric for assessing our algorithms and data processing pipeline. In addition to directly comparing R-peaks between the MEG and ECG signals, this work explores the variation of the corresponding HRV output in time, frequency, and non-linear domains. After removing outlier intervals and aligning the ECG and derived cardiac MEG signals, the RMSE between the RR-intervals of each was RMSE1 = 2 ms, RMSE2 = 2 ms, RMSE3 = 8 ms, RMSE4 = 4 ms, RMSE5 = 13 ms. The findings indicate that cardiac artifacts from MEG data carry sufficient signal to approximate an individual's HRV metrics.

Strengths-weaknesses-opportunities-threats analysis of artificial intelligence in anesthesiology and perioperative medicine.

The use of artificial intelligence (AI) and machine learning (ML) in anesthesiology and perioperative medicine is quickly becoming a mainstay of clinical practice. Anesthesiology is a data-rich medical specialty that integrates multitudes of patient-specific information. Perioperative medicine is ripe for applications of AI and ML to facilitate data synthesis for precision medicine and predictive assessments. Examples of emergent AI models include those that assist in assessing depth and modulating control of anesthetic delivery, event and risk prediction, ultrasound guidance, pain management, and operating room logistics. AI and ML support analyzing integrated perioperative data at scale and can assess patterns to deliver optimal patient-specific care. By exploring the benefits and limitations of this technology, we provide a basis of considerations for evaluating the adoption of AI models into various anesthesiology workflows. This analysis of AI and ML in anesthesiology and perioperative medicine explores the current landscape to understand better the strengths, weaknesses, opportunities, and threats (SWOT) these tools offer.

Radiomics Outperforms Clinical and Radiologic Signs in Predicting Spontaneous Basal Ganglia Hematoma Expansion: A Pilot Study.

Prediction of the hematoma expansion (HE) of spontaneous basal ganglia hematoma (SBH) from the first non-contrast CT can result in better management, which has the potential of improving outcomes. This study has been designed to compare the performance of "Radiomics analysis," "radiology signs," and "clinical-laboratory data" for this task. We retrospectively reviewed the electronic medical records for clinical, demographic, and laboratory data in patients with SBH. CT images were reviewed for the presence of radiologic signs, including black-hole, blend, swirl, satellite, and island signs. Radiomic features from the SBH on the first brain CT were extracted, and the most predictive features were selected. Different machine learning models were developed based on clinical, laboratory, and radiology signs and selected Radiomic features to predict hematoma expansion (HE). The dataset used for this analysis included 116 patients with SBH. Among different models and different thresholds to define hematoma expansion (10%, 20%, 25%, 33%, 40%, and 50% volume enlargement thresholds), the Random Forest based on 10 selected Radiomic features achieved the best performance (for 25% hematoma enlargement) with an area under the curve (AUC) of 0.9 on the training dataset and 0.89 on the test dataset. The models based on clinical-laboratory and radiology signs had low performance (AUCs about 0.5-0.6).

MEGnet: Automatic ICA-based artifact removal for MEG using spatiotemporal convolutional neural networks.

Magnetoencephalography (MEG) is a functional neuroimaging tool that records the magnetic fields induced by neuronal activity; however, signal from non-neuronal sources can corrupt the data. Eye-blinks, saccades, and cardiac activity are three of the most common sources of non-neuronal artifacts. They can be measured by affixing eye proximal electrodes, as in electrooculography (EOG), and chest electrodes, as in electrocardiography (ECG), however this complicates imaging setup, decreases patient comfort, and can induce further artifacts from movement. This work proposes an EOG- and ECG-free approach to identify eye-blinks, saccades, and cardiac activity signals for automated artifact suppression. The contribution of this work is three-fold. First, using a data driven, multivariate decomposition approach based on Independent Component Analysis (ICA), a highly accurate artifact classifier is constructed as an amalgam of deep 1-D and 2-D Convolutional Neural Networks (CNNs) to automate the identification and removal of ubiquitous whole brain artifacts including eye-blink, saccade, and cardiac artifacts. The specific architecture of this network is optimized through an unbiased, computer-based hyperparameter random search. Second, visualization methods are applied to the learned abstraction to reveal what features the model uses and to bolster user confidence in the model's training and potential for generalization. Finally, the model is trained and tested on both resting-state and task MEG data from 217 subjects, and achieves a new state-of-the-art in artifact detection accuracy of 98.95% including 96.74% sensitivity and 99.34% specificity on the held out test-set. This work automates MEG processing for both clinical and research use, adapts to the acquired acquisition time, and can obviate the need for EOG or ECG electrodes for artifact detection.

The one-inflated positive Poisson mixture model for use in population size estimation.

The one-inflated positive Poisson mixture model (OIPPMM) is presented, for use as the truncated count model in Horvitz-Thompson estimation of an unknown population size. The OIPPMM offers a way to address two important features of some capture-recapture data: one-inflation and unobserved heterogeneity. The OIPPMM provides markedly different results than some other popular estimators, and these other estimators can appear to be quite biased, or utterly fail due to the boundary problem, when the OIPPMM is the true data-generating process. In addition, the OIPPMM provides a solution to the boundary problem, by labelling any mixture components on the boundary instead as one-inflation.

Structure and Dynamics of tRNAMet Containing Core Substitutions.

The fidelity of protein synthesis is largely dominated by the accurate recognition of transfer RNAs (tRNAs) by their cognate aminoacyl-tRNA synthetases. Aminoacylation of each tRNA with its cognate amino acid is necessary to maintain the accuracy of genetic code input. Aminoacylated tRNAMet functions in both initiation and elongation steps during protein synthesis. As a precursor to the investigation of a methionyl-tRNA synthetase-tRNAMet complex, presented here are the results of molecular dynamics (MD) for single nucleotide substitutions in the D-loop of tRNAMet (G15A, G18A, and G19A) probing structure/function relationships. The core of tRNAMet likely mediates an effective communication between the tRNA anticodon and acceptor ends, contributing an acceptor stem rearrangement to fit into the enzyme-active site. Simulations of Escherichia coli tRNAMet were performed for 1 µs four times each. The MD simulations showed changes in tRNA flexibility and long-range communication most prominently in the G18A variant. The results indicate that the overall tertiary structure of tRNAMet remains unchanged with these substitutions; yet, there are perturbations to the secondary structure. Network-based analysis of the hydrogen bond structure and correlated motion indicates that the secondary structure elements of the tRNA are highly intraconnected, but loosely interconnected. Specific nucleotides, including U8 and G22, stabilize the mutated structures and are candidates for substitution in future studies.

Using correlated motions to determine sufficient sampling times for molecular dynamics.

Here we present a time-dependent correlation method that provides insight into how long a system takes to grow into its equal-time (Pearson) correlation. We also show a usage of an extant time-lagged correlation method that indicates the time for parts of a system to become decorrelated, relative to equal-time correlation. Given a completed simulation (or set of simulations), these tools estimate (i) how long of a simulation of the same system would be sufficient to observe the same correlated motions, (ii) if patterns of observed correlated motions indicate events beyond the timescale of the simulation, and (iii) how long of a simulation is needed to observe these longer timescale events. We view this method as a decision-support tool that will aid researchers in determining necessary sampling times. In principle, this tool is extendable to any multidimensional time series data with a notion of correlated fluctuations; however, here we limit our discussion to data from molecular-dynamics simulations.

All-atom molecular dynamics comparison of disease-associated zinc fingers.

An important regulatory domain of NF-[Formula: see text]B Essential Modulator (NEMO) is a ubiquitin-binding zinc finger, with a tetrahedral CYS3HIS1 zinc-coordinating binding site. Two variations of NEMO's zinc finger are implicated in various disease states including ectodermal dysplasia and adult-onset glaucoma. To discern structural and dynamical differences between these disease states, we present results of 48-[Formula: see text]s of molecular dynamics simulations for three zinc finger systems each in two states, with and without zinc-bound and correspondingly appropriate cysteine thiol/thiolate configurations. The wild-type protein, often studied for its role in cancer, maintains the most rigid and conformationally stable zinc-bound configuration compared with the diseased counterparts. The glaucoma-related protein has persistent loss of secondary structure except within the dominant conformation. Conformational overlap between wild-type and glaucoma isoforms indicate a competitive binding mechanism may be substantial in the malfunctioning configuration, while the alpha-helical disruption of the ectodermal dysplasia suggests a loss of binding selectivity is responsible for aberrant function.

Visualizing correlated motion with HDBSCAN clustering.

Correlated motion analysis provides a method for understanding communication between and dynamic similarities of biopolymer residues and domains. The typical equal-time correlation matrices-frequently visualized with pseudo-colorings or heat maps-quickly convey large regions of highly correlated motion but hide more subtle similarities of motion. Here we propose a complementary method for visualizing correlations within proteins (or general biopolymers) that quickly conveys intuition about which residues have a similar dynamic behavior. For grouping residues, we use the recently developed non-parametric clustering algorithm HDBSCAN. Although the method we propose here can be used to group residues using correlation as a similarity matrix-the most straightforward and intuitive method-it can also be used to more generally determine groups of residues which have similar dynamic properties. We term these latter groups "Dynamic Domains", as they are based not on spatial closeness but rather closeness in the column space of a correlation matrix. We provide examples of this method across three human proteins of varying size and function-the Nf-Kappa-Beta essential modulator, the clotting promoter Thrombin and the mismatch repair protein (dimer) complex MutS-alpha. Although the examples presented here are from all-atom molecular dynamics simulations, this visualization technique can also be used on correlations matrices built from any ensembles of conformations from experiment or computation.

MutSα's Multi-Domain Allosteric Response to Three DNA Damage Types Revealed by Machine Learning.

MutSα is a key component in the mismatch repair (MMR) pathway. This protein is responsible for initiating the signaling pathways for DNA repair or cell death. Herein we investigate this heterodimer's post-recognition, post-binding response to three types of DNA damage involving cytotoxic, anti-cancer agents-carboplatin, cisplatin, and FdU. Through a combination of supervised and unsupervised machine learning techniques along with more traditional structural and kinetic analysis applied to all-atom molecular dynamics (MD) calculations, we predict that MutSα has a distinct response to each of the three damage types. Via a binary classification tree (a supervised machine learning technique), we identify key hydrogen bond motifs unique to each type of damage and suggest residues for experimental mutation studies. Through a combination of a recently developed clustering (unsupervised learning) algorithm, RMSF calculations, PCA, and correlated motions we predict that each type of damage causes MutSα to explore a specific region of conformation space. Detailed analysis suggests a short range effect for carboplatin-primarily altering the structures and kinetics of residues within 10 angstroms of the damaged DNA-and distinct longer-range effects for cisplatin and FdU. In our simulations, we also observe that a key phenylalanine residue-known to stack with a mismatched or unmatched bases in MMR-stacks with the base complementary to the damaged base in 88.61% of MD frames containing carboplatinated DNA. Similarly, this Phe71 stacks with the base complementary to damage in 91.73% of frames with cisplatinated DNA. This residue, however, stacks with the damaged base itself in 62.18% of trajectory frames with FdU-substituted DNA and has no stacking interaction at all in 30.72% of these frames. Each drug investigated here induces a unique perturbation in the MutSα complex, indicating the possibility of a distinct signaling event and specific repair or death pathway (or set of pathways) for a given type of damage.

One-inflation and unobserved heterogeneity in population size estimation.

We present the one-inflated zero-truncated negative binomial (OIZTNB) model, and propose its use as the truncated count distribution in Horvitz-Thompson estimation of an unknown population size. In the presence of unobserved heterogeneity, the zero-truncated negative binomial (ZTNB) model is a natural choice over the positive Poisson (PP) model; however, when one-inflation is present the ZTNB model either suffers from a boundary problem, or provides extremely biased population size estimates. Monte Carlo evidence suggests that in the presence of one-inflation, the Horvitz-Thompson estimator under the ZTNB model can converge in probability to infinity. The OIZTNB model gives markedly different population size estimates compared to some existing truncated count distributions, when applied to several capture-recapture data that exhibit both one-inflation and unobserved heterogeneity.

Binding Site Configurations Probe the Structure and Dynamics of the Zinc Finger of NEMO (NF-κB Essential Modulator).

Zinc-finger proteins are regulators of critical signaling pathways for various cellular functions, including apoptosis and oncogenesis. Here, we investigate how binding site protonation states and zinc coordination influence protein structure, dynamics, and ultimately function, as these pivotal regulatory proteins are increasingly important for protein engineering and therapeutic discovery. To better understand the thermodynamics and dynamics of the zinc finger of NEMO (NF-κB essential modulator), as well as the role of zinc, we present results of 20 µs molecular dynamics trajectories, 5 µs for each of four active site configurations. Consistent with experimental evidence, the zinc ion is essential for mechanical stabilization of the functional, folded conformation. Hydrogen bond motifs are unique for deprotonated configurations yet overlap in protonated cases. Correlated motions and principal component analysis corroborate the similarity of the protonated configurations and highlight unique relationships of the zinc-bound configuration. We hypothesize a potential mechanism for zinc binding from results of the thiol configurations. The deprotonated, zinc-bound configuration alone predominantly maintains its tertiary structure throughout all 5 µs and alludes rare conformations potentially important for (im)proper zinc-finger-related protein-protein or protein-DNA interactions.

Uncovering Large-Scale Conformational Change in Molecular Dynamics without Prior Knowledge.

As the length of molecular dynamics (MD) trajectories grows with increasing computational power, so does the importance of clustering methods for partitioning trajectories into conformational bins. Of the methods available, the vast majority require users to either have some a priori knowledge about the system to be clustered or to tune clustering parameters through trial and error. Here we present non-parametric uses of two modern clustering techniques suitable for first-pass investigation of an MD trajectory. Being non-parametric, these methods require neither prior knowledge nor parameter tuning. The first method, HDBSCAN, is fast-relative to other popular clustering methods-and is able to group unstructured or intrinsically disordered systems (such as intrinsically disordered proteins, or IDPs) into bins that represent global conformational shifts. HDBSCAN is also useful for determining the overall stability of a system-as it tends to group stable systems into one or two bins-and identifying transition events between metastable states. The second method, iMWK-Means, with explicit rescaling followed by K-Means, while slower than HDBSCAN, performs well with stable, structured systems such as folded proteins and is able to identify higher resolution details such as changes in relative position of secondary structural elements. Used in conjunction, these clustering methods allow a user to discern quickly and without prior knowledge the stability of a simulated system and identify both local and global conformational changes.

Importance of long-time simulations for rare event sampling in zinc finger proteins.

Molecular dynamics (MD) simulation methods have seen significant improvement since their inception in the late 1950s. Constraints of simulation size and duration that once impeded the field have lessened with the advent of better algorithms, faster processors, and parallel computing. With newer techniques and hardware available, MD simulations of more biologically relevant timescales can now sample a broader range of conformational and dynamical changes including rare events. One concern in the literature has been under which circumstances it is sufficient to perform many shorter timescale simulations and under which circumstances fewer longer simulations are necessary. Herein, our simulations of the zinc finger NEMO (2JVX) using multiple simulations of length 15, 30, 1000, and 3000 ns are analyzed to provide clarity on this point.