Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis.

BACKGROUND: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality. OBJECTIVE: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data. METHODS: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips. RESULTS: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts' opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance. CONCLUSIONS: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.

Severe Impairment of Microcirculatory Perfused Vessel Density Is Associated With Postoperative Lactate and Acute Organ Injury After Cardiac Surgery.

OBJECTIVE: Resuscitation after cardiac surgery needs to address multiple pathophysiological processes that are associated with significant morbidity and mortality. Functional microcirculatory derangements despite normal systemic hemodynamics have been previously described but must be tied to clinical outcomes. The authors hypothesized that microcirculatory dysfunction after cardiac surgery would include impaired capillary blood flow and impaired diffusive capacity and that subjects with the lowest quartile of perfused vessel density would have an increased postoperative lactate level and acute organ injury scores. DESIGN: Prospective, observational study. SETTING: A single, tertiary university cardiovascular surgical intensive care unit. PARTICIPANTS: 25 adults undergoing elective cardiac surgery requiring cardiopulmonary bypass. INTERVENTION: Sublingual microcirculation was imaged using incident dark field microscopy before and 2 to 4 hours after surgery in the intensive care unit. MEASUREMENTS AND MAIN RESULTS: Compared with baseline measurements, postoperative vessel-by-vessel microvascular flow index (2.9 [2.8-2.9] v 2.5 [2.4-2.7], p < 0.0001) and perfused vessel density were significantly impaired (20.7 [19.3-22.9] v 16.3 [12.8-17.9], p < 0.0001). The lowest quartile of perfused vessel density (<12.8 mm/mm2) was associated with a significantly increased postoperative lactate level (6.0 ± 2.9 v 1.8 ± 1.2, p < 0.05), peak lactate level (7.6 ± 2.8 v 2.8 ± 1.5, p = 0.03), and sequential organ failure assessment (SOFA) score at 24 and 48 hours. CONCLUSION: In patients undergoing cardiac surgery, there was a significant decrease in postoperative microcirculatory convective blood flow and diffusive capacity during early postoperative resuscitation. Severely impaired perfused vessel density, represented by the lowest quartile of distribution, is significantly related to hyperlactatemia and early organ injury.

Beyond Mean Arterial Pressure and Lactate: Perfusion End Points for Managing the Shocked Patient.

Patients in shock present frequently to the emergency department. The emergency physician must be skilled in the resuscitation of both differentiated and undifferentiated shock. Early, aggressive resuscitation of patients in shock is essential, using macrocirculatory, microcirculatory, and clinical end points to guide interventions. Therapy should focus on the restoration of oxygen delivery to match tissue demand. This article reviews the evidence supporting common end points of resuscitation for common etiologies of shock and limitations to their use.