A framework for rigorous evaluation of human performance in human and machine learning comparison studies.

Rigorous comparisons of human and machine learning algorithm performance on the same task help to support accurate claims about algorithm success rates and advances understanding of their performance relative to that of human performers. In turn, these comparisons are critical for supporting advances in artificial intelligence. However, the machine learning community has lacked a standardized, consensus framework for performing the evaluations of human performance necessary for comparison. We demonstrate common pitfalls in a designing the human performance evaluation and propose a framework for the evaluation of human performance, illustrating guiding principles for a successful comparison. These principles are first, to design the human evaluation with an understanding of the differences between human and algorithm cognition; second, to match trials between human participants and the algorithm evaluation, and third, to employ best practices for psychology research studies, such as the collection and analysis of supplementary and subjective data and adhering to ethical review protocols. We demonstrate our framework's utility for designing a study to evaluate human performance on a one-shot learning task. Adoption of this common framework may provide a standard approach to evaluate algorithm performance and aid in the reproducibility of comparisons between human and machine learning algorithm performance.

Association of Disease Severity and Socioeconomic Status in Black and White Americans With Multiple Sclerosis.

OBJECTIVE: To compare clinical and imaging features of multiple sclerosis (MS) severity between Black Americans (BAs) and White Americans (WAs) and to evaluate the role of socioeconomic status. METHODS: We compared BA and WA participants in the Multiple Sclerosis Partners Advancing Technology Health Solutions (MS PATHS) cohort with respect to MS characteristics, including self-reported disability, objective neurologic function assessments, and quantitative brain MRI measurements, after covariate adjustment (including education level, employment, or insurance as socioeconomic indicators). In a subgroup, we evaluated within-race, neighborhood-level indicators of socioeconomic status (SES) using 9-digit zip codes. RESULTS: Of 1,214 BAs and 7,530 WAs with MS, BAs were younger, had lower education level, and were more likely to have Medicaid insurance or to be disabled or unemployed than WAs. BAs had worse self-reported disability (1.47-fold greater odds of severe vs mild disability, 95% confidence interval [CI] 1.18, 1.86) and worse performances on tests of cognitive processing speed (-5.06 fewer correct, 95% CI -5.72, -4.41), walking (0.66 seconds slower, 95% CI 0.36, 0.96), and manual dexterity (2.11 seconds slower, 95% CI 1.69, 2.54). BAs had more brain MRI lesions and lower overall and gray matter brain volumes, including reduced thalamic (-0.77 mL, 95% CI -0.91, -0.64), cortical (-30.63 mL, 95% CI -35.93, -25.33), and deep (-1.58 mL, 95% CI -1.92, -1.23) gray matter volumes. While lower SES correlated with worse neuroperformance scores in WAs, this association was less clear in BAs. CONCLUSION: We observed a greater burden of disease in BAs with MS relative to WAs with MS, despite adjustment for SES indicators. Beyond SES, future longitudinal studies should also consider roles of other societal constructs (e.g., systemic racism). Such studies will be important for identifying prognostic factors; developing optimal treatment strategies among BAs with MS is warranted.