REFORMS: Consensus-based Recommendations for Machine-learning-based Science.

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

Development and testing of a computerized pain body map in patients with advanced cancer.

CONTEXT: Pain localization is an important part of pain assessment. Development of pain tools for self-report should include expert and patient input, and patient testing in large samples. OBJECTIVES: To develop a computerized pain body map (CPBM) for use in patients with advanced cancer. METHODS: Three studies were conducted: 1) an international expert survey and a pilot study guiding the contents and layout of the CPBM, 2) clinical testing in an international symptom assessment study in eight countries and 17 centers (N = 533), and 3) comparing patient pain markings on computer and paper body maps (N = 92). RESULTS: Study 1: 22 pain experts and 28 patients participated. A CPBM with anterior and posterior whole body views was developed for marking pain locations, supplemented by pain intensity ratings for each location. Study 2: 533 patients (286 male, 247 female, mean age 62 years) participated; 80% received pain medication and 81% had metastatic disease. Eighty-five percent completed CPBM as intended. Mean ± SD number of marked pain locations was 1.8 ± 1.2. Aberrant markings (15%) were mostly related to software problems. No differences were found regarding age, gender, cognitive/physical performance, or previous computer experience. Study 3: 70% of the patients had identical markings on the computer and paper maps. Only four patients had completely different markings on the two maps. CONCLUSION: This first version of CPBM was well accepted by patients with advanced cancer. However, several areas for improvement were revealed, providing a basis for the development of the next version, which is subject to further international testing.

Computer-based symptom assessment is feasible in patients with advanced cancer: results from an international multicenter study, the EPCRC-CSA.

CONTEXT: Symptom assessment by computers is only effective if it provides valid results and is perceived as useful for clinical use by the end users: patients and health care providers. OBJECTIVES: To identify factors associated with discontinuation, time expenditure, and patient preferences of the computerized symptom assessment used in an international multicenter data collection project: the European Palliative Care Research Collaborative-Computerized Symptom Assessment. METHODS: Cancer patients with incurable metastatic or locally advanced disease were recruited from 17 centers in eight countries, providing 1017 records for analyses. Observer-based registrations and patient-reported measures on pain, depression, and physical function were entered on touch screen laptop computers. RESULTS: The entire assessment was completed by 94.9% (n = 965), with median age 63 years (range 18-91 years) and median Karnofsky Performance Status (KPS) score of 70 (range 20-100). Predictive factors for noncompletion were higher age, lower KPS, and more pain (P ≤ 0.012). Time expenditure among completers increased with higher age, male gender, Norwegian nationality, number of comorbidities, and lower physical functioning (P ≤ 0.007) but was inversely related to pain levels and tiredness (P ≤ 0.03). Need for assistance was predicted by higher age, nationality other than Norwegian, lower KPS, and lower educational level (P < 0.001). More than 50% of patients preferred computerized assessment to a paper and pencil version. CONCLUSION: The high completion rate shows that symptom assessment by computers is feasible in patients with advanced cancer. However, reduced performance status reduces compliance and increases the need for assistance. Future work should aim at identifying the minimum set of valid screening questions and refine the software to optimize symptom assessment and reduce respondent burden in frail patients.