Video-Based Pairwise Comparison: Enabling the Development of Automated Rating of Motor Dysfunction in Multiple Sclerosis.

OBJECTIVES: To examine the feasibility, reliability, granularity, and convergent validity of a video-based pairwise comparison technique that uses algorithmic support to enable automated rating of motor dysfunction in patients with multiple sclerosis (MS). DESIGN: Feasibility and larger cross-sectional cohort study. SETTING: The outpatient clinic of 2 specialist university medical centers. PARTICIPANTS: Selected sample from a cohort of patients with MS participating in the Assess MS study (N=42). Videos were randomly drawn from each strata of the ataxia severity-degrees as defined in the Expanded Disability Status Scale (EDSS). In Basel: 19 videos of 17 patients (mean age, 43.4±11.6y; 10 women). In Amsterdam: 50 videos of 25 patients (mean age, 50.0±10.0y; 15 women). INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: In each center, neurologists (n=13; n=10) viewed pairs of videos of patients performing standardized movements (eg, finger-to-nose test) to assess relative performance. A comparative assessment score was calculated for each video using the TrueSkill algorithm and analyzed for intrarater (test-retest; ratio of agreement) and interrater reliability (intraclass correlation coefficient [ICC] for absolute agreement) and convergent validity (Spearman ρ). Granularity was estimated from the average difference in comparative assessment scores at which 80% of neurologists considered performance to be different. RESULTS: Intrarater reliability was excellent (median ratio of agreement≥0.87). The comparative assessment scores calculated from individual neurologists demonstrated good-excellent ICCs for interrater reliability (0.89; 0.71). The comparative assessment scores correlated (very) highly with their Neurostatus-EDSS equivalent (ρ=0.78, P<.001; ρ=0.91, P<.05), suggesting a more fine-grained rating. CONCLUSIONS: Video-based pairwise comparison of motor dysfunction allows for reliable and fine-grained capturing of clinical judgment about neurologic performance, which can contribute to the development of a consistent quantified metric of motor ability in MS.

Tasks of activities of daily living (ADL) are more valuable than the classical neurological examination to assess upper extremity function and mobility in multiple sclerosis.

BACKGROUND: Accurate clinical assessment in multiple sclerosis (MS) is challenging. The Assess MS system is being developed to automatically quantify motor dysfunction in MS, including upper extremity function (UEF) and mobility. OBJECTIVE: To determine to what extent combinations of standardized movements included in the Assess MS system explain accepted measures of UEF and mobility. METHODS: MS patients were recruited at four European MS centres. Eight movements were selected, including tasks of activities of daily living (ADL) and classical neurological tests. Movements were recorded on video and rated by experienced neurologists (n = 5). Subsequently, multivariate linear regression models were performed to explain the variance of the Nine-Hole Peg Test (9HPT), Arm Function in Multiple Sclerosis Questionnaire (AMSQ) and Timed-25 Foot Walk test (T25WT). RESULTS: In total, 257 patients were included. The movements explained 62.9% to 80.1% of the variance of the 9HPT models, 43.3% and 44.3% of the AMSQ models and 70.8% of the T25WT. In all models, tasks of ADL contributed most to the variance. CONCLUSION: Combinations of movements are valuable to assess UEF and mobility. Incorporating ADL tasks into daily clinical practice and clinical trials may be more valuable than the classical neurological examination of UEF and mobility.

Impact of 3 Tesla MRI on interobserver agreement in clinically isolated syndrome: A MAGNIMS multicentre study.

BACKGROUND: Compared to 1.5 T, 3 T magnetic resonance imaging (MRI) increases signal-to-noise ratio leading to improved image quality. However, its clinical relevance in clinically isolated syndrome suggestive of multiple sclerosis remains uncertain. OBJECTIVES: The purpose of this study was to investigate how 3 T MRI affects the agreement between raters on lesion detection and diagnosis. METHODS: We selected 30 patients and 10 healthy controls from our ongoing prospective multicentre cohort. All subjects received baseline 1.5 and 3 T brain and spinal cord MRI. Patients also received follow-up brain MRI at 3-6 months. Four experienced neuroradiologists and four less-experienced raters scored the number of lesions per anatomical region and determined dissemination in space and time (McDonald 2010). RESULTS: In controls, the mean number of lesions per rater was 0.16 at 1.5 T and 0.38 at 3 T ( p = 0.005). For patients, this was 4.18 and 4.40, respectively ( p = 0.657). Inter-rater agreement on involvement per anatomical region and dissemination in space and time was moderate to good for both field strengths. 3 T slightly improved agreement between experienced raters, but slightly decreased agreement between less-experienced raters. CONCLUSION: Overall, the interobserver agreement was moderate to good. 3 T appears to improve the reading for experienced readers, underlining the benefit of additional training.

Regression-Based Norms for the Symbol Digit Modalities Test in the Dutch Population: Improving Detection of Cognitive Impairment in Multiple Sclerosis?

BACKGROUND/AIMS: Appropriate and timely screening instruments that sensitively capture the cognitive functioning of multiple sclerosis (MS) patients are the need of the hour. We evaluated newly derived regression-based norms for the Symbol Digit Modalities Test (SDMT) in a Dutch-speaking sample, as an indicator of the cognitive state of MS patients. METHODS: Regression-based norms for the SDMT were created from a healthy control sample (n = 96) and used to convert MS patients' (n = 157) raw scores to demographically adjusted Z-scores, correcting for the effects of age, age2, gender, and education. Conventional and regression-based norms were compared on their impairment-classification rates and related to other neuropsychological measures. RESULTS: The regression analyses revealed that age was the only significantly influencing demographic in our healthy sample. Regression-based norms for the SDMT more readily detected impairment in MS patients than conventional normalization methods (32 patients instead of 15). Patients changing from an SDMT-preserved to -impaired status (n = 17) were also impaired on other cognitive domains (p < 0.05), except for visuospatial memory (p = 0.34). CONCLUSIONS: Regression-based norms for the SDMT more readily detect abnormal performance in MS patients than conventional norms, identifying those patients at highest risk for cognitive impairment, which was supported by a worse performance on other neuropsychological measures.

Comparing long-term results of PASAT and SDMT scores in relation to neuropsychological testing in multiple sclerosis.

BACKGROUND AND OBJECTIVES: The Symbol Digit Modalities Test (SDMT) shows advantages over the Paced Auditory Serial Addition Test (PASAT) as a cognitive test in patients with multiple sclerosis (MS). To determine which of these tests is most valid and reliable over time as an indicator of the cognitive state of MS patients, long-term test results of both tests were compared in relation to scores of the Brief Repeatable Battery of Neuropsychological tests (BRBN). METHODS: For 485 MS patients visiting the VU University Medical Center Amsterdam for different research projects, a total number of 1078 visits with BRBN (including PASAT and SDMT) was planned. Observed and model-based correlations were used to calculate the construct validity of the SDMT and PASAT 3 seconds test (PASAT3) by comparing correlations with the BRBN-sumscore. The test-retest reliability of each test was also computed. RESULTS: For the construct validity, higher correlations were found between SDMT and BRBN compared to PASAT3 and BRBN, especially for the model-based correlations at baseline. The reliability of the measurements was good for all instruments, with the highest coefficients for the SDMT. CONCLUSION: As a single assessment tool for cognition in MS, the SDMT is more valid and reliable compared to PASAT3.

Improving Detection of Change in Motor Functioning in Multiple Sclerosis Using Video-Assisted Composite Measures.

BACKGROUND: Assessing motor functioning is important to monitor the disease course of multiple sclerosis (MS). Video-assisted rating of classic neurologic tests and activities of daily living may improve the detection of changes in motor functioning. We investigated the value of using video-assisted composite measures for the detection of changes in mobility and upper extremity function (UEF). METHODS: Forty-three patients with MS were recorded performing motor function tests before and during treatment with fampridine. Patients were classified as improved or not improved on mobility composite (MOB-COM) and UEF composite (UEF-COM) measures based on neurologists' ratings of the tests. The proportional agreements between the composite measures and the conventional measures-the Timed 25-Foot Walk test (T25FW) and the Nine-Hole Peg Test (NHPT)-were determined and compared with patient-perceived improvement, which was determined using patient-reported ratings of changes in mobility and UEF. RESULTS: Agreement between MOB-COM and T25FW was 79.5%, and agreement between UEF-COM and NHPT was 82.1%. Twenty-six of 39 patients (66.7%) reported mobility improvement; 6 of these reports were confirmed by both T25FW and MOB-COM, 4 were confirmed by T25FW only, and 2 were confirmed by MOB-COM only. For UEF, 13 of 39 patients (33.3%) reported improvement; 3 of these were confirmed by the NHPT and 3 were confirmed by the UEF-COM. CONCLUSIONS: Compared with the conventional NHPT measure, the video-assisted composite measure of UEF detected additional patient-perceived improvement. This was less evident for mobility measures. Video-assisted composite measures may enhance the detection of treatment effects in MS clinical practice and trials.

Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references.

BACKGROUND: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are needed for objective evaluation and training of new methods. OBJECTIVES: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated segmentation approach (FASTSURF). METHODS: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. RESULTS: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. CONCLUSIONS: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload.

Manual and automated tissue segmentation confirm the impact of thalamus atrophy on cognition in multiple sclerosis: A multicenter study.

BACKGROUND AND RATIONALE: Thalamus atrophy has been linked to cognitive decline in multiple sclerosis (MS) using various segmentation methods. We investigated the consistency of the association between thalamus volume and cognition in MS for two common automated segmentation approaches, as well as fully manual outlining. METHODS: Standardized neuropsychological assessment and 3-Tesla 3D-T1-weighted brain MRI were collected (multi-center) from 57 MS patients and 17 healthy controls. Thalamus segmentations were generated manually and using five automated methods. Agreement between the algorithms and manual outlines was assessed with Bland-Altman plots; linear regression assessed the presence of proportional bias. The effect of segmentation method on the separation of cognitively impaired (CI) and preserved (CP) patients was investigated through Generalized Estimating Equations; associations with cognitive measures were investigated using linear mixed models, for each method and vendor. RESULTS: In smaller thalami, automated methods systematically overestimated volumes compared to manual segmentations [ρ=(-0.42)-(-0.76); p-values < 0.001). All methods significantly distinguished CI from CP MS patients, except manual outlines of the left thalamus (p = 0.23). Poorer global neuropsychological test performance was significantly associated with smaller thalamus volumes bilaterally using all methods. Vendor significantly affected the findings. CONCLUSION: Automated and manual thalamus segmentation consistently demonstrated an association between thalamus atrophy and cognitive impairment in MS. However, a proportional bias in smaller thalami and choice of MRI acquisition system might impact the effect size of these findings.

Setwise comparison: efficient fine-grained rating of movement videos using algorithmic support - a proof of concept study.

Purpose: Clinical ordinal rating scales of movements, e.g., the Expanded Disability Status Scale, have poor intra- and interrater reliability, are insensitive to subtle differences and result in coarse-grained ratings compared to relative comparative rating methods. We therefore established video-based setwise comparison as a fine-grained, reliable and efficient rating method of motor dysfunction using algorithmic support.Materials and methods: Eight neurologists rated a set of 40 multiple sclerosis patient videos of the Finger-to-Nose-Test using both the newly developed setwise comparison and the established pairwise comparison techniques, which result in a continuous rating scale. Reliability was assessed by the intra-class correlation coefficient. Construct validity was estimated as Pearson's correlation between the continuous scale and severity ratings according to the Neurostatus scale for upper-extremity tremor/dysmetria and the Nine-hole-peg-test. Comparing the time needed for ratings assessed efficiency.Results: Intra-class correlation coefficient was 0.83 for setwise and 0.7 for pairwise comparison. Correlation to the tremor/dysmetria score of the Neurostatus was 0.86 for both rating procedures and correlation to the Nine-hole-peg-test was 0.64 (setwise) and 0.66 (pairwise). The time needed to rate 40 videos was 22.9 ± 6.9 minutes (setwise) and 77.8 ± 14.5 minutes (pairwise).Conclusions: Setwise comparison is an efficient, valid and reliable method for fine-grained rating of motor dysfunction that can be applied to larger datasets. It is substantially more efficient than pairwise comparison.Implications for rehabilitationDisability rating is crucial in clinical neurorehabilitation and in clinical trials.Humans are naturally inconsistent in rating items on ordinal scales leading to poor intra- and interrater reliability, insensitivity to subtle differences and coarse-grained ratings.Video-based setwise comparison is a new rating method enabling fine-grained, reliable and efficient ratings of motor dysfunction using algorithmic support.

Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort.

BACKGROUND: Deep grey matter (DGM) atrophy in multiple sclerosis (MS) and its relation to cognitive and clinical decline requires accurate measurements. MS pathology may deteriorate the performance of automated segmentation methods. Accuracy of DGM segmentation methods is compared between MS and controls, and the relation of performance with lesions and atrophy is studied. METHODS: On images of 21 MS subjects and 11 controls, three raters manually outlined caudate nucleus, putamen and thalamus; outlines were combined by majority voting. FSL-FIRST, FreeSurfer, Geodesic Information Flow and volBrain were evaluated. Performance was evaluated volumetrically (intra-class correlation coefficient (ICC)) and spatially (Dice similarity coefficient (DSC)). Spearman's correlations of DSC with global and local lesion volume, structure of interest volume (ROIV), and normalized brain volume (NBV) were assessed. RESULTS: ICC with manual volumes was mostly good and spatial agreement was high. MS exhibited significantly lower DSC than controls for thalamus and putamen. For some combinations of structure and method, DSC correlated negatively with lesion volume or positively with NBV or ROIV. Lesion-filling did not substantially change segmentations. CONCLUSIONS: Automated methods have impaired performance in patients. Performance generally deteriorated with higher lesion volume and lower NBV and ROIV, suggesting that these may contribute to the impaired performance.

Reference videos reduce variability of motor dysfunction assessments in multiple sclerosis.

Motor dysfunction, particularly ataxia, is one of the predominant clinical manifestations in patients with multiple sclerosis (MS). Assessment of motor dysfunction suffers from a high variability. We investigated whether the clinical rating of ataxia can be improved through the use of reference videos, covering the spectrum of severity degrees as defined in the Neurostatus-Expanded Disability Status Scale. Twenty-five neurologists participated. The variability of their assessments was significantly lower when reference videos were used (SD = 0.12; range = 0.40 vs SD = 0.26; range = 0.88 without reference videos; p = 0.013). Reference videos reduced the variability of clinical assessments and may be useful tools to improve the precision and consistency in the clinical assessment of motor functions in MS.

The role of the cerebellum in multiple sclerosis-150 years after Charcot.

Despite its functional importance and well known clinical impact in Multiple Sclerosis (MS), the cerebellum has only received significant attention over the past few years. It is now established that the cerebellum plays a key role not only in various sensory-motor networks, but also in cognitive-behavioural processes, domains primarily affected in patients with MS. Evidence from histopathological and magnetic resonance imaging (MRI) studies on cerebellar involvement in MS is increasingly available, however linking these pathological findings with clinical dysfunction remains challenging. There are promising advances in technology that are likely to improve the detection of pathological changes within the cerebellum, which may elucidate how pathology relates to disability.

Three-Tesla MRI does not improve the diagnosis of multiple sclerosis: A multicenter study.

OBJECTIVE: In the work-up of patients presenting with a clinically isolated syndrome (CIS), 3T MRI might offer a higher lesion detection than 1.5T, but it remains unclear whether this affects the fulfilment of the diagnostic criteria for multiple sclerosis (MS). METHODS: We recruited 66 patients with CIS within 6 months from symptom onset and 26 healthy controls in 6 MS centers. All participants underwent 1.5T and 3T brain and spinal cord MRI at baseline according to local optimized protocols and the MAGNIMS guidelines. Patients who had not converted to MS during follow-up received repeat brain MRI at 3-6 months and 12-15 months. The number of lesions per anatomical region was scored by 3 raters in consensus. Criteria for dissemination in space (DIS) and dissemination in time (DIT) were determined according to the 2017 revisions of the McDonald criteria. RESULTS: Three-Tesla MRI detected 15% more T2 brain lesions compared to 1.5T (p < 0.001), which was driven by an increase in baseline detection of periventricular (12%, p = 0.015), (juxta)cortical (21%, p = 0.005), and deep white matter lesions (21%, p < 0.001). The detection rate of spinal cord lesions and gadolinium-enhancing lesions did not differ between field strengths. Three-Tesla MRI did not lead to a higher number of patients fulfilling the criteria for DIS or DIT, or subsequent diagnosis of MS, at any of the 3 time points. CONCLUSION: Scanning at 3T does not influence the diagnosis of MS according to McDonald diagnostic criteria.

Usability and Acceptability of ASSESS MS: Assessment of Motor Dysfunction in Multiple Sclerosis Using Depth-Sensing Computer Vision.

BACKGROUND: Sensor-based recordings of human movements are becoming increasingly important for the assessment of motor symptoms in neurological disorders beyond rehabilitative purposes. ASSESS MS is a movement recording and analysis system being developed to automate the classification of motor dysfunction in patients with multiple sclerosis (MS) using depth-sensing computer vision. It aims to provide a more consistent and finer-grained measurement of motor dysfunction than currently possible. OBJECTIVE: To test the usability and acceptability of ASSESS MS with health professionals and patients with MS. METHODS: A prospective, mixed-methods study was carried out at 3 centers. After a 1-hour training session, a convenience sample of 12 health professionals (6 neurologists and 6 nurses) used ASSESS MS to capture recordings of standardized movements performed by 51 volunteer patients. Metrics for effectiveness, efficiency, and acceptability were defined and used to analyze data captured by ASSESS MS, video recordings of each examination, feedback questionnaires, and follow-up interviews. RESULTS: All health professionals were able to complete recordings using ASSESS MS, achieving high levels of standardization on 3 of 4 metrics (movement performance, lateral positioning, and clear camera view but not distance positioning). Results were unaffected by patients' level of physical or cognitive disability. ASSESS MS was perceived as easy to use by both patients and health professionals with high scores on the Likert-scale questions and positive interview commentary. ASSESS MS was highly acceptable to patients on all dimensions considered, including attitudes to future use, interaction (with health professionals), and overall perceptions of ASSESS MS. Health professionals also accepted ASSESS MS, but with greater ambivalence arising from the need to alter patient interaction styles. There was little variation in results across participating centers, and no differences between neurologists and nurses. CONCLUSIONS: In typical clinical settings, ASSESS MS is usable and acceptable to both patients and health professionals, generating data of a quality suitable for clinical analysis. An iterative design process appears to have been successful in accounting for factors that permit ASSESS MS to be used by a range of health professionals in new settings with minimal training. The study shows the potential of shifting ubiquitous sensing technologies from research into the clinic through a design approach that gives appropriate attention to the clinic environment.

Quantifying progression of multiple sclerosis via classification of depth videos.

This paper presents new learning-based techniques for measuring disease progression in Multiple Sclerosis (MS) patients. Our system aims to augment conventional neurological examinations by adding quantitative evidence of disease progression. An off-the-shelf depth camera is used to image the patient at the examination, during which he/she is asked to perform carefully selected movements. Our algorithms then automatically analyze the videos, assessing the quality of each movement and classifying them as healthy or non-healthy. Our contribution is three-fold: We i) introduce ensembles of randomized SVM classifiers and compare them with decision forests on the task of depth video classification; ii) demonstrate automatic selection of discriminative landmarks in the depth videos, showing their clinical relevance; iii) validate our classification algorithms quantitatively on a new dataset of 1041 videos of both MS patients and healthy volunteers. We achieve average Dice scores well in excess of the 80% mark, confirming the validity of our approach in practical applications. Our results suggest that this technique could be fruitful for depth-camera supported clinical assessments for a range of conditions.