Predictive Accuracy of Stroke Risk Prediction Models Across Black and White Race, Sex, and Age Groups.

Importance: Stroke is the fifth-highest cause of death in the US and a leading cause of serious long-term disability with particularly high risk in Black individuals. Quality risk prediction algorithms, free of bias, are key for comprehensive prevention strategies. Objective: To compare the performance of stroke-specific algorithms with pooled cohort equations developed for atherosclerotic cardiovascular disease for the prediction of new-onset stroke across different subgroups (race, sex, and age) and to determine the added value of novel machine learning techniques. Design, Setting, and Participants: Retrospective cohort study on combined and harmonized data from Black and White participants of the Framingham Offspring, Atherosclerosis Risk in Communities (ARIC), Multi-Ethnic Study for Atherosclerosis (MESA), and Reasons for Geographical and Racial Differences in Stroke (REGARDS) studies (1983-2019) conducted in the US. The 62 482 participants included at baseline were at least 45 years of age and free of stroke or transient ischemic attack. Exposures: Published stroke-specific algorithms from Framingham and REGARDS (based on self-reported risk factors) as well as pooled cohort equations for atherosclerotic cardiovascular disease plus 2 newly developed machine learning algorithms. Main Outcomes and Measures: Models were designed to estimate the 10-year risk of new-onset stroke (ischemic or hemorrhagic). Discrimination concordance index (C index) and calibration ratios of expected vs observed event rates were assessed at 10 years. Analyses were conducted by race, sex, and age groups. Results: The combined study sample included 62 482 participants (median age, 61 years, 54% women, and 29% Black individuals). Discrimination C indexes were not significantly different for the 2 stroke-specific models (Framingham stroke, 0.72; 95% CI, 0.72-073; REGARDS self-report, 0.73; 95% CI, 0.72-0.74) vs the pooled cohort equations (0.72; 95% CI, 0.71-0.73): differences 0.01 or less (P values >.05) in the combined sample. Significant differences in discrimination were observed by race: the C indexes were 0.76 for all 3 models in White vs 0.69 in Black women (all P values <.001) and between 0.71 and 0.72 in White men and between 0.64 and 0.66 in Black men (all P values ≤.001). When stratified by age, model discrimination was better for younger (<60 years) vs older (≥60 years) adults for both Black and White individuals. The ratios of observed to expected 10-year stroke rates were closest to 1 for the REGARDS self-report model (1.05; 95% CI, 1.00-1.09) and indicated risk overestimation for Framingham stroke (0.86; 95% CI, 0.82-0.89) and pooled cohort equations (0.74; 95% CI, 0.71-0.77). Performance did not significantly improve when novel machine learning algorithms were applied. Conclusions and Relevance: In this analysis of Black and White individuals without stroke or transient ischemic attack among 4 US cohorts, existing stroke-specific risk prediction models and novel machine learning techniques did not significantly improve discriminative accuracy for new-onset stroke compared with the pooled cohort equations, and the REGARDS self-report model had the best calibration. All algorithms exhibited worse discrimination in Black individuals than in White individuals, indicating the need to expand the pool of risk factors and improve modeling techniques to address observed racial disparities and improve model performance.

A novel algorithm-driven hybrid simulation learning method to improve acquisition of endotracheal intubation skills: a randomized controlled study.

BACKGROUND: Simulation-based training is a clinical skill learning method that can replicate real-life situations in an interactive manner. In our study, we compared a novel hybrid learning method with conventional simulation learning in the teaching of endotracheal intubation. METHODS: One hundred medical students and residents were randomly divided into two groups and were taught endotracheal intubation. The first group of subjects (control group) studied in the conventional way via lectures and classic simulation-based training sessions. The second group (experimental group) used the hybrid learning method where the teaching process consisted of distance learning and small group peer-to-peer simulation training sessions with remote supervision by the instructors. After the teaching process, endotracheal intubation (ETI) procedures were performed on real patients under the supervision of an anesthesiologist in an operating theater. Each step of the procedure was evaluated by a standardized assessment form (checklist) for both groups. RESULTS: Thirty-four subjects constituted the control group and 43 were in the experimental group. The hybrid group (88%) showed significantly better ETI performance in the operating theater compared with the control group (52%). Further, all hybrid group subjects (100%) followed the correct sequence of actions, while in the control group only 32% followed proper sequencing. CONCLUSIONS: We conclude that our novel algorithm-driven hybrid simulation learning method improves acquisition of endotracheal intubation with a high degree of acceptability and satisfaction by the learners' as compared with classic simulation-based training.

Stromal computational signatures predict upgrade to invasive carcinoma in mass-forming DCIS: A brief report of 44 cases.

Mass-forming ductal carcinoma in situ (DCIS) detected on core needle biopsy (CNB) is often a radiology-pathology discordance and thought to represent missed invasive carcinoma. This brief report applied supervised machine learning (ML) for image segmentation to investigate a series of 44 mass-forming DCIS cases, with the primary focus being stromal computational signatures. The area under the curve (AUC) for receiver operator curves (ROC) in relation to upgrade to invasive carcinoma from DCIS were as follows: high myxoid stromal ratio (MSR): 0.923, P = <0.001; low collagenous stromal percentage (CSP): 0.875, P = <0.001; and low proportionated stromal area (PSA): 0.682, P = 0.039. The use of ML in mass-forming DCIS could predict upgraded to invasive carcinoma with high sensitivity and specificity. The findings from this brief report are clinically useful and should be further validated by future studies.

Small but visible: Predicting rare bryophyte distribution and richness patterns using remote sensing-based ensembles of small models.

In Canadian boreal forests, bryophytes represent an essential component of biodiversity and play a significant role in ecosystem functioning. Despite their ecological importance and sensitivity to disturbances, bryophytes are overlooked in conservation strategies due to knowledge gaps on their distribution, which is known as the Wallacean shortfall. Rare species deserve priority attention in conservation as they are at a high risk of extinction. This study aims to elaborate predictive models of rare bryophyte species in Canadian boreal forests using remote sensing-derived predictors in an Ensemble of Small Models (ESMs) framework. We hypothesize that high ESMs-based prediction accuracy can be achieved for rare bryophyte species despite their low number of occurrences. We also assess if there is a spatial correspondence between rare and overall bryophyte richness patterns. The study area is located in western Quebec and covers 72,292 km2. We selected 52 bryophyte species with <30 occurrences from a presence-only database (214 species, 389 plots in total). ESMs were built from Random Forest and Maxent techniques using remote sensing-derived predictors related to topography and vegetation. Lee's L statistic was used to assess and map the spatial relationship between rare and overall bryophyte richness patterns. ESMs yielded poor to excellent prediction accuracy (AUC > 0.5) for 73% of the modeled species, with AUC values > 0.8 for 19 species, which confirmed our hypothesis. In fact, ESMs provided better predictions for the rarest bryophytes. Likewise, our study revealed a spatial concordance between rare and overall bryophyte richness patterns in different regions of the study area, which have important implications for conservation planning. This study demonstrates the potential of remote sensing for assessing and making predictions on inconspicuous and rare species across the landscape and lays the basis for the eventual inclusion of bryophytes into sustainable development planning.

An artificial intelligence model to predict hepatocellular carcinoma risk in Korean and Caucasian patients with chronic hepatitis B.

BACKGROUND & AIMS: Several models have recently been developed to predict risk of hepatocellular carcinoma (HCC) in patients with chronic hepatitis B (CHB). Our aims were to develop and validate an artificial intelligence-assisted prediction model of HCC risk. METHODS: Using a gradient-boosting machine (GBM) algorithm, a model was developed using 6,051 patients with CHB who received entecavir or tenofovir therapy from 4 hospitals in Korea. Two external validation cohorts were independently established: Korean (5,817 patients from 14 Korean centers) and Caucasian (1,640 from 11 Western centers) PAGE-B cohorts. The primary outcome was HCC development. RESULTS: In the derivation cohort and the 2 validation cohorts, cirrhosis was present in 26.9%-50.2% of patients at baseline. A model using 10 parameters at baseline was derived and showed good predictive performance (c-index 0.79). This model showed significantly better discrimination than previous models (PAGE-B, modified PAGE-B, REACH-B, and CU-HCC) in both the Korean (c-index 0.79 vs. 0.64-0.74; all p <0.001) and Caucasian validation cohorts (c-index 0.81 vs. 0.57-0.79; all p <0.05 except modified PAGE-B, p = 0.42). A calibration plot showed a satisfactory calibration function. When the patients were grouped into 4 risk groups, the minimal-risk group (11.2% of the Korean cohort and 8.8% of the Caucasian cohort) had a less than 0.5% risk of HCC during 8 years of follow-up. CONCLUSIONS: This GBM-based model provides the best predictive power for HCC risk in Korean and Caucasian patients with CHB treated with entecavir or tenofovir. LAY SUMMARY: Risk scores have been developed to predict the risk of hepatocellular carcinoma (HCC) in patients with chronic hepatitis B. We developed and validated a new risk prediction model using machine learning algorithms in 13,508 antiviral-treated patients with chronic hepatitis B. Our new model, based on 10 common baseline characteristics, demonstrated superior performance in risk stratification compared with previous risk scores. This model also identified a group of patients at minimal risk of developing HCC, who could be indicated for less intensive HCC surveillance.

Evaluation of the Cost-effectiveness of Drug Treatment for Alzheimer Disease in a Simulation Model That Includes Caregiver and Societal Factors.

Importance: The possibility of widespread use of a novel effective therapy for Alzheimer disease (AD) will present important clinical, policy, and financial challenges. Objective: To describe how including different patient, caregiver, and societal treatment-related factors affects estimates of the cost-effectiveness of a hypothetical disease-modifying AD treatment. Design, Setting, and Participants: In this economic evaluation, the Alzheimer Disease Archimedes Condition Event Simulator was used to simulate the prognosis of a hypothetical cohort of patients selected from the Alzheimer Disease Neuroimaging Initiative database who received the diagnosis of mild cognitive impairment (MCI). Scenario analyses that varied costs and quality of life inputs relevant to patients and caregivers were conducted. The analysis was designed and conducted from June 15, 2019, to September 30, 2020. Exposures: A hypothetical drug that would delay progression to dementia in individuals with MCI compared with usual care. Main Outcomes and Measures: Incremental cost-effectiveness ratio (ICER), measured by cost per quality-adjusted life-year (QALY) gained. Results: The model included a simulated cohort of patients who scored between 24 and 30 on the Mini-Mental State Examination and had a global Clinical Dementia Rating scale of 0.5, with a required memory box score of 0.5 or higher, at baseline. Using a health care sector perspective, which included only individual patient health care costs, the ICER for the hypothetical treatment was $192 000 per QALY gained. The result decreased to $183 000 per QALY gained in a traditional societal perspective analysis with the inclusion of patient non-health care costs. The inclusion of estimated caregiver health care costs produced almost no change in the ICER, but the inclusion of QALYs gained by caregivers led to a substantial reduction in the ICER for the hypothetical treatment, to $107 000 per QALY gained in the health sector perspective. In the societal perspective scenario, with the broadest inclusion of patient and caregiver factors, the ICER decreased to $74 000 per added QALY. Conclusions and Relevance: The findings of this economic evaluation suggest that policy makers should be aware that efforts to estimate and include the effects of AD treatments outside those on patients themselves can affect the results of the cost-effectiveness analyses that often underpin assessments of the value of new treatments. Further research and debate on including these factors in assessments that will inform discussions on fair pricing for new treatments are needed.

Nursing students' experiences of virtual simulation when using a video conferencing system - a mixed methods study.

OBJECTIVES: There is limited knowledge about students' experiences with virtual simulation when using a video conferencing system. Therefore, the aim of this study was to explore how second-year undergraduate nursing students experienced learning through virtual simulations during the COVID-19 pandemic. METHODS: The study had an exploratory design with both quantitative and qualitative approaches. In total, 69 nursing students participated in two sessions of virtual simulation during spring 2020, and 33 students answered online questionnaires at session 1. To further explore students' experiences, one focus group interview and one individual interview were conducted using a video conferencing system after session 2. In addition, system information on use during both sessions was collected. RESULTS: Changes in the students' ratings of their experiences of virtual simulation with the Body Interact™ system were statistically significant. The virtual simulation helped them to bridge gaps in both the teaching and learning processes. Four important aspects of learning were identified: 1) learning by self-training, 2) learning from the software (Body Interact™), 3) learning from peers, and 4) learning from faculty. CONCLUSIONS: We conclude that virtual simulation through a video conferencing system can be useful for student learning and feedback from both peers and faculty is important.

Constructing robust and efficient experimental designs in groundwater modeling using a Galerkin method, proper orthogonal decomposition, and metaheuristic algorithms.

Estimating parameters accurately in groundwater models for aquifers is challenging because the models are non-explicit solutions of complex partial differential equations. Modern research methods, such as Monte Carlo methods and metaheuristic algorithms, for searching an efficient design to estimate model parameters require hundreds, if not thousands of model calls, making the computational cost prohibitive. One method to circumvent the problem and gain valuable insight on the behavior of groundwater is to first apply a Galerkin method and convert the system of partial differential equations governing the flow to a discrete problem and then use a Proper Orthogonal Decomposition to project the high-dimensional model space of the original groundwater model to create a reduced groundwater model with much lower dimensions. The reduced model can be solved several orders of magnitude faster than the full model and able to provide an accurate estimate of the full model. The task is still challenging because the optimization problem is non-convex, non-differentiable and there are continuous variables and integer-valued variables to optimize. Following convention, heuristic algorithms and a combination is used search to find efficient designs for the reduced groundwater model using various optimality criteria. The main goals are to introduce new design criteria and the concept of design efficiency for experimental design research in hydrology. The two criteria have good utility but interestingly, do not seem to have been implemented in hydrology. In addition, design efficiency is introduced. Design efficiency is a method to assess how robust a design is under a change of criteria. The latter is an important issue because the design criterion may be subjectively selected and it is well known that an optimal design can perform poorly under another criterion. It is thus desirable that the implemented design has relatively high efficiencies under a few criteria. As applications, two heuristic algorithms are used to find optimal designs for a small synthetic aquifer design problem and a design problem for a large-scale groundwater model and assess their robustness properties to other optimality criteria. The results show the proof of concept is workable for finding a more informed and efficient model-based design for a water resource study.

Targeting Melanoma-Initiating Cells by Caffeine: In Silico and In Vitro Approaches.

The beneficial effects of coffee on human diseases are well documented, but the molecular mechanisms of its bioactive compounds on cancer are not completely elucidated. This is likely due to the large heterogeneity of coffee preparations and different coffee-based beverages, but also to the choice of experimental models where proliferation, differentiation and immune responses are differently affected. The aim of the present study was to investigate the effects of one of the most interesting bioactive compounds in coffee, i.e., caffeine, using a cellular model of melanoma at a defined differentiation level. A preliminary in silico analysis carried out on public gene-expression databases identified genes potentially involved in caffeine's effects and suggested some specific molecular targets, including tyrosinase. Proliferation was investigated in vitro on human melanoma initiating cells (MICs) and cytokine expression was measured in conditioned media. Tyrosinase was revealed as a key player in caffeine's mechanisms of action, suggesting a crucial role in immunomodulation through the reduction in IL-1ß, IP-10, MIP-1α, MIP-1ß and RANTES secretion onto MICs conditioned media. The potent antiproliferative effects of caffeine on MICs are likely to occur by promoting melanin production and reducing inflammatory signals' secretion. These data suggest tyrosinase as a key player mediating the effects of caffeine on melanoma.

Prediction of n-octanol/water partition coefficients and acidity constants (pKa) in the SAMPL7 blind challenge with the IEFPCM-MST model.

Within the scope of SAMPL7 challenge for predicting physical properties, the Integral Equation Formalism of the Miertus-Scrocco-Tomasi (IEFPCM/MST) continuum solvation model has been used for the blind prediction of n-octanol/water partition coefficients and acidity constants of a set of 22 and 20 sulfonamide-containing compounds, respectively. The log P and pKa were computed using the B3LPYP/6-31G(d) parametrized version of the IEFPCM/MST model. The performance of our method for partition coefficients yielded a root-mean square error of 1.03 (log P units), placing this method among the most accurate theoretical approaches in the comparison with both globally (rank 8th) and physical (rank 2nd) methods. On the other hand, the deviation between predicted and experimental pKa values was 1.32 log units, obtaining the second best-ranked submission. Though this highlights the reliability of the IEFPCM/MST model for predicting the partitioning and the acid dissociation constant of drug-like compounds compound, the results are discussed to identify potential weaknesses and improve the performance of the method.

Critical fluctuations in epidemic models explain COVID-19 post-lockdown dynamics.

As the COVID-19 pandemic progressed, research on mathematical modeling became imperative and very influential to understand the epidemiological dynamics of disease spreading. The momentary reproduction ratio r(t) of an epidemic is used as a public health guiding tool to evaluate the course of the epidemic, with the evolution of r(t) being the reasoning behind tightening and relaxing control measures over time. Here we investigate critical fluctuations around the epidemiological threshold, resembling new waves, even when the community disease transmission rate [Formula: see text] is not significantly changing. Without loss of generality, we use simple models that can be treated analytically and results are applied to more complex models describing COVID-19 epidemics. Our analysis shows that, rather than the supercritical regime (infectivity larger than a critical value, [Formula: see text]) leading to new exponential growth of infection, the subcritical regime (infectivity smaller than a critical value, [Formula: see text]) with small import is able to explain the dynamic behaviour of COVID-19 spreading after a lockdown lifting, with [Formula: see text] hovering around its threshold value.

Drug repurposing: Iron in the fire for older drugs.

Repositioning or "repurposing" of existing therapies for indications of alternative disease is an attractive approach that can generate lower costs and require a shorter approval time than developing a de novo drug. The development of experimental drugs is time-consuming, expensive, and limited to a fairly small number of targets. The incorporation of separate and complementary data should be used, as each type of data set exposes a specific feature of organism knowledge Drug repurposing opportunities are often focused on sporadic findings or on time-consuming pre-clinical drug tests which are often not guided by hypothesis. In comparison, repurposing in-silico drugs is a new, hypothesis-driven method that takes advantage of big-data use. Nonetheless, the widespread use of omics technology, enhanced data storage, data sense, machine learning algorithms, and computational modeling all give unparalleled knowledge of the methods of action of biological processes and drugs, providing wide availability, for both disease-related data and drug-related data. This review has taken an in-depth look at the current state, possibilities, and limitations of further progress in the field of drug repositioning.

In Silico Modeling of the Mitochondrial Pumping Complexes with Markov State Models.

The mechanism of proton pumping by the mitochondrial electron transport chain complexes is still enigmatic after decades of research. Recently, there has been interest in in silico Markov state models to model the mitochondrial pumping complexes at the microscopic level, and this chapter describes the methods of constructing and simulating such models.

Evaluation of log P, pKa, and log D predictions from the SAMPL7 blind challenge.

The Statistical Assessment of Modeling of Proteins and Ligands (SAMPL) challenges focuses the computational modeling community on areas in need of improvement for rational drug design. The SAMPL7 physical property challenge dealt with prediction of octanol-water partition coefficients and pKa for 22 compounds. The dataset was composed of a series of N-acylsulfonamides and related bioisosteres. 17 research groups participated in the log P challenge, submitting 33 blind submissions total. For the pKa challenge, 7 different groups participated, submitting 9 blind submissions in total. Overall, the accuracy of octanol-water log P predictions in the SAMPL7 challenge was lower than octanol-water log P predictions in SAMPL6, likely due to a more diverse dataset. Compared to the SAMPL6 pKa challenge, accuracy remains unchanged in SAMPL7. Interestingly, here, though macroscopic pKa values were often predicted with reasonable accuracy, there was dramatically more disagreement among participants as to which microscopic transitions produced these values (with methods often disagreeing even as to the sign of the free energy change associated with certain transitions), indicating far more work needs to be done on pKa prediction methods.

Using machine learning techniques to characterize sleep-deprived driving behavior.

OBJECTIVE: Sleep deprivation is known to affect driving behavior and may lead to serious car accidents similar to the effects from e.g., alcohol. In a previous study, we have demonstrated that the use of machine learning techniques allows adequate characterization of abnormal driving behavior after alprazolam and/or alcohol intake. In the present study, we extend this approach to sleep deprivation and test the model for characterization of new interventions. We aimed to classify abnormal driving behavior after sleep deprivation, and, by using a machine learning model, we tested if this model could also pick up abnormal driving behavior resulting from other interventions. METHODS: Data were collected during a previous study, in which 24 subjects were tested after being sleep-deprived and after a well-rested night. Features were calculated from several driving parameters, such as the lateral position, speed of the car, and steering speed. In the present study, we used a gradient boosting model to classify sleep deprivation. The model was validated using a 5-fold cross validation technique. Next, probability scores were used to identify the overlap of driving behavior after sleep deprivation and driving behavior affected by other interventions. In the current study alprazolam, alcohol, and placebo are used to test/validate the approach. RESULTS: The sleep deprivation model detected abnormal driving behavior in the simulator with an accuracy of 77 ± 9%. Abnormal driving behavior after alprazolam, and to a lesser extent also after alcohol intake, showed remarkably similar characteristics to sleep deprivation. The average probability score for alprazolam and alcohol measurements was 0.79, for alcohol 0.63, and for placebo only 0.27 and 0.30, matching the expected relative drowsiness. CONCLUSION: We developed a model detecting abnormal driving induced by sleep deprivation. The model shows the similarities in driving characteristics between sleep deprivation and other interventions, i.e., alcohol and alprazolam. Consequently, our model for sleep deprivation may serve as a next reference point for a driving test battery of newly developed drugs.

Comparison of infection control strategies to reduce COVID-19 outbreaks in homeless shelters in the United States: a simulation study.

BACKGROUND: COVID-19 outbreaks have occurred in homeless shelters across the US, highlighting an urgent need to identify the most effective infection control strategy to prevent future outbreaks. METHODS: We developed a microsimulation model of SARS-CoV-2 transmission in a homeless shelter and calibrated it to data from cross-sectional polymerase chain reaction (PCR) surveys conducted during COVID-19 outbreaks in five homeless shelters in three US cities from March 28 to April 10, 2020. We estimated the probability of averting a COVID-19 outbreak when an exposed individual is introduced into a representative homeless shelter of 250 residents and 50 staff over 30 days under different infection control strategies, including daily symptom-based screening, twice-weekly PCR testing, and universal mask wearing. RESULTS: The proportion of PCR-positive residents and staff at the shelters with observed outbreaks ranged from 2.6 to 51.6%, which translated to the basic reproduction number (R0) estimates of 2.9-6.2. With moderate community incidence (~ 30 confirmed cases/1,000,000 people/day), the estimated probabilities of averting an outbreak in a low-risk (R0 = 1.5), moderate-risk (R0 = 2.9), and high-risk (R0 = 6.2) shelter were respectively 0.35, 0.13, and 0.04 for daily symptom-based screening; 0.53, 0.20, and 0.09 for twice-weekly PCR testing; 0.62, 0.27, and 0.08 for universal masking; and 0.74, 0.42, and 0.19 for these strategies in combination. The probability of averting an outbreak diminished with higher transmissibility (R0) within the simulated shelter and increasing incidence in the local community. CONCLUSIONS: In high-risk homeless shelter environments and locations with high community incidence of COVID-19, even intensive infection control strategies (incorporating daily symptom screening, frequent PCR testing, and universal mask wearing) are unlikely to prevent outbreaks, suggesting a need for non-congregate housing arrangements for people experiencing homelessness. In lower-risk environments, combined interventions should be employed to reduce outbreak risk.

Early stopping in clinical PET studies: How to reduce expense and exposure.

Clinical positron emission tomography (PET) research is costly and entails exposing participants to radioactivity. Researchers should therefore aim to include just the number of subjects needed to fulfill the purpose of the study. In this tutorial we show how to apply sequential Bayes Factor testing in order to stop the recruitment of subjects in a clinical PET study as soon as enough data have been collected to make a conclusion. By using simulations, we demonstrate that it is possible to stop a study early, while keeping the number of erroneous conclusions low. We then apply sequential Bayes Factor testing to a real PET data set and show that it is possible to obtain support in favor of an effect while simultaneously reducing the sample size with 30%. Using this procedure allows researchers to reduce expense and radioactivity exposure for a range of effect sizes relevant for PET research.

Smooth ROC curve estimation via Bernstein polynomials.

The receiver operating characteristic (ROC) curve is commonly used to evaluate the accuracy of a diagnostic test for classifying observations into two groups. We propose two novel tuning parameters for estimating the ROC curve via Bernstein polynomial smoothing of the empirical ROC curve. The new estimator is very easy to implement with the naturally selected tuning parameter, as illustrated by analyzing both real and simulated data sets. Empirical performance is investigated through extensive simulation studies with a variety of scenarios where the two groups are both from a single family of distributions (symmetric or right skewed) or one from a symmetric and the other from a right skewed distribution. The new estimator is uniformly more efficient than the empirical ROC estimator, and very competitive to eleven other existing smooth ROC estimators in terms of mean integrated square errors.