Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms.

Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.

Validation of a precision tremor measurement system for multiple sclerosis.

BACKGROUND: Tremor is a debilitating symptom of Multiple Sclerosis (MS). Little is known about its pathophysiology and treatments are limited. Clinical trials investigating new interventions often rely on subjective clinical rating scales to provide supporting evidence of efficacy. NEW METHOD: We present a novel instrument (TREMBAL) which uses electromagnetic motion capture technology to quantify MS tremor. We aim to validate TREMBAL by comparison to clinical ratings using regression modelling with 310 samples of tremor captured from 13 MS participants who performed five different hand exercises during several follow-up visits. Minimum detectable change (MDC) and test-retest reliability were calculated and comparisons were made between MS tremor and data from 12 healthy volunteers. RESULTS: Velocity of the index finger was most congruent with clinical observation. Regression modelling combining different features, sensor configurations, and labelling exercises did not improve results. TREMBAL MDC was 84% of its initial measurement compared to 91% for the clinical rating. Intra-class correlations for test-retest reliability were 0.781 for TREMBAL and 0.703 for clinical ratings. Tremor was lower (p = 0.002) in healthy subjects. COMPARISON WITH EXISTING METHODS: Subjective scales have low sensitivity, suffer from ceiling effects, and mitigation against inter-rater variability is challenging. Inertial sensors are ubiquitous, however, their output is nonlinearly related to tremor frequency, compensation is required for gravitational artefacts, and their raw data cannot be intuitively comprehended. CONCLUSIONS: TREMBAL, compared with clinical ratings, gave measures in agreement with clinical observation, had marginally lower MDC, and similar test-retest reliability.

A Robust Low-Cost EEG Motor Imagery-Based Brain-Computer Interface.

Motor imagery (MI) based Brain-Computer Interfaces (BCIs) are a viable option for giving locked-in syndrome patients independence and communicability. BCIs comprising expensive medical-grade EEG systems evaluated in carefully-controlled, artificial environments are impractical for take-home use. Previous studies evaluated low-cost systems; however, performance was suboptimal or inconclusive. Here we evaluated a low-cost EEG system, OpenBCI, in a natural environment and leveraged neurofeedback, deep learning, and wider temporal windows to improve performance. $\mu-$rhythm data collected over the sensorimotor cortex from healthy participants performing relaxation and right-handed MI tasks were used to train a multi-layer perceptron binary classifier using deep learning. We showed that our method outperforms previous OpenBCI MI-based BCIs, thereby extending the BCI capabilities of this low-cost device.

Neurophysiological analysis of the clinical pull test.

Postural reflexes are impaired in conditions such as Parkinson's disease, leading to difficulty walking and falls. In clinical practice, postural responses are assessed using the "pull test," where an examiner tugs the prewarned standing patient backward at the shoulders and grades the response. However, validity of the pull test is debated, with issues including scaling and variability in administration and interpretation. It is unclear whether to assess the first trial or only subsequent repeated trials. The ecological relevance of a forewarned backward challenge is also debated. We therefore developed an instrumented version of the pull test to characterize responses and clarify how the test should be performed and interpreted. In 33 healthy participants, "pulls" were manually administered and pull force measured. Trunk and step responses were assessed with motion tracking. We probed for the StartReact phenomenon (where preprepared responses are released early by a startling stimulus) by delivering concurrent normal or "startling" auditory stimuli. We found that the first pull triggers a different response, including a larger step size suggesting more destabilization. This is consistent with "first trial effects," reported by platform translation studies, where movement execution appears confounded by startle reflex-like activity. Thus, first pull test trials have clinical relevance and should not be discarded as practice. Supportive of ecological relevance, responses to repeated pulls exhibited StartReact, as previously reported with a variety of other postural challenges, including those delivered with unexpected timing and direction. Examiner pull force significantly affected the postural response, particularly the size of stepping. NEW & NOTEWORTHY We characterized postural responses elicited by the clinical "pull test" using instrumentation. The first pull triggers a different response, including a larger step size suggesting more destabilization. Thus, first trials likely have important clinical and ecological relevance and should not be discarded as practice. Responses to repeated pulls can be accelerated with a startling stimulus, as reported with a variety of other challenges. Examiner pull force was a significant factor influencing the postural response.

Clinical validation of a precision electromagnetic tremor measurement system in participants receiving deep brain stimulation for essential tremor.

Tremor is characterized commonly through subjective clinical rating scales. Accelerometer-based techniques for objective tremor measurement have been developed in the past, yet these measures are usually presented as an unintuitive dimensionless index without measurement units. Here we have developed a tool (TREMBAL) to provide quantifiable and objective measures of tremor severity using electromagnetic motion tracking. We aimed to compare TREMBAL's objective measures with clinical tremor ratings and determine the test-retest reliability of our technique. Eight participants with ET receiving deep brain stimulation (DBS) therapy were consented. Tremor was simultaneously recorded using TREMBAL and video during DBS adjustment. After each adjustment, participants performed a hands-outstretched task (for postural tremor) and a finger-nose task (for kinetic tremor). Video recordings were de-identified, randomized, and shown to a panel of movement disorder specialists to obtain their ratings. Regression analysis and Pearson's correlations were used to determine agreement between datasets. Subsets of the trial were repeated to assess test-retest reliability. Tremor amplitude and velocity measures were in close agreement with mean clinical ratings (r > 0.90) for both postural and kinetic tremors. Test-retest reliability for both translational and rotational components of tremor showed intra-class correlations >0.80. TREMBAL assessments showed that tremor gradually improved with increasing DBS therapy-this was also supported by clinical observation. TREMBAL measurements are a sensitive, objective and reliable assessment of tremor severity. This tool may have application in clinical trials and in aiding automated optimization of deep brain stimulation.

A simple and inexpensive test-rig for evaluating the performance of motion sensors used in movement disorders research.

Since the advent of electromyogram recording, precise measures of tremor and gait have been used to study movement disorders such as Parkinson's disease. Now, a wide range of accelerometers and other motion-tracking technologies exist to better inform researchers and clinicians, yet such systems are rarely tested for accuracy or suitability before use. Our inexpensive test-rig can produce sinusoidal displacements using a simple cantilever system driven by a subwoofer. Controlled sinusoids were generated using computer software, and the displacement amplitudes of the test-rig were verified with fiducial marker tracking. To illustrate the use of the test-rig, we evaluated an accelerometer and an electromagnetic motion tracker. Accelerometry recordings were accurate to within ±0.09 g of actual peak-to-peak amplitude with a frequency response close to unity gain between 1 and 20 Hz. The electromagnetic sensor underestimated peak displacement by 2.68 mm, which was largely due to a diminishing gain with increasing frequency. Both sensors had low distortion. Overall sensitivity was limited by noise for the accelerometer and quantisation resolution for the electromagnetic sensor. Our simple and low-cost test-rig can be used to bench-test sensors used in movement disorders research. It was able to produce reliable sinusoidal displacements and worked across the 1- to 20-Hz frequency range.