Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis.

BACKGROUND: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality. OBJECTIVE: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data. METHODS: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips. RESULTS: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts' opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance. CONCLUSIONS: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.

Pericardial tamponade: a new perspective on echocardiographic features and application of a prediction score.

Few clinical decision rules have been used to guide clinical management and predict outcomes in patients with pericardial tamponade. The objectives of this study are to identify the echocardiographic features associated with adverse outcomes in patients with pericardial effusions requiring pericardiocentesis and to apply a previously described four-point clinical and echocardiographic score to predict clinical outcomes over 24-hr, 30-day, and 1-year intervals. We performed a retrospective cohort review of patients who had transthoracic echocardiogram (TTE) performed and underwent pericardiocentesis within 48 h of emergency department presentation at two large tertiary care institutions. We constructed different stepwise logistic regression models and examined the associations of TTE characteristics and clinical features with ICU admission, hospital length of stay (h-LOS), and survival. The data set was then employed against a previously proposed scoring system to predict factors associated with clinical outcomes over 24 hr, 30 days, and 1 year. Two hundred thirty-nine patients were included in the final analysis. Echocardiographic characteristics of patients with pericardial tamponade who underwent pericardiocentesis are as follows: 69.1% right ventricular (RV) diastolic collapse, 62.3% exaggerated mitral valve (MV) inflow velocities, 56.4% inferior vena cava (IVC) plethora, and 53.4% right atrial (RA) systolic collapse. Increase in systolic blood pressure and increased variation in MV inflow velocity were associated with reduced ICU admission [OR: 0.94 (CI 0.90, 0.99), 0.28 (CI 0.09, 0.89), respectively]. In addition, a history of malignancy increased the length of hospital stay by about 3.89 days (CI 1.43-6.35, p < 0.01) and prior pericardiocentesis history was associated with 4.82-day increase in hospital stay (CI 1.19-8.45, p = 0.01). In utilizing the previously published prediction score, we found no statistically significant correlation in predicting survival. RV diastolic collapse and exaggerated MV inflow velocity were the most common echocardiographic findings in patients requiring pericardiocentesis. Contrary to prior studies, exaggerated MV inflow velocity was associated with reduced ICU admission. In addition, a previously described prediction score did not correlate with decreased survival in this cohort.