Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis.

BACKGROUND: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality. OBJECTIVE: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data. METHODS: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips. RESULTS: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts' opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance. CONCLUSIONS: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.

Gender effects in crowdfunded business loan campaigns.

Crowdfunding is a growing source of finance for entrepreneurs. In this paper, we investigate the existence of a gender effect in the time needed to obtain a business loan through crowdfunding. Using data from three Dutch crowdfunding platforms, survival analysis of the time to completion for 934 business loan campaigns shows that female entrepreneurs have a 20% shorter campaign completion time compared to male entrepreneurs, whereas couples do not differ from males. This effect persists across the different platforms. Subsequent analysis shows that female entrepreneurs do not have the disadvantage they face in traditional lending channels when requesting funds through crowdfunding, and that herding behavior by investors benefits female entrepreneurs most.

tracerDB: a crowdsourced fluorescent tracer database for target engagement analysis.

Investigating ligand-protein complexes is essential in the areas of chemical biology and drug discovery. However, detailed information on key reagents such as fluorescent tracers and associated data for the development of widely used bioluminescence resonance energy transfer (BRET) assays including NanoBRET, time-resolved Förster resonance energy transfer (TR-FRET) and fluorescence polarization (FP) assays are not easily accessible to the research community. We created tracerDB, a curated database of validated tracers. This resource provides an open access knowledge base and a unified system for tracer and assay validation. The database is freely available at https://www.tracerdb.org/ .

Rationale and Study Checklist for Ethical Rejection of Participants on Crowdsourcing Research Platforms.

Online participant recruitment ("crowdsourcing") platforms are increasingly being used for research studies. While such platforms can rapidly provide access to large samples, there are concomitant concerns around data quality. Researchers have studied and demonstrated means to reduce the prevalence of low-quality data from crowdsourcing platforms, but approaches to doing so often involve rejecting work and/or denying payment to participants, which can pose ethical dilemmas. We write this essay as an associate professor and two institutional review board (IRB) directors to provide a perspective on the competing interests of participants/workers and researchers and to propose a checklist of steps that we believe may support workers' agency on the platform and lessen instances of unfair consequences to them while enabling researchers to definitively reject lower-quality work that might otherwise reduce the likelihood of their studies producing true results. We encourage further, explicit discussion of these issues among academics and among IRBs.

Unmet financial needs among patients crowdfunding to support gynecologic cancer care.

BACKGROUND: Patients may use crowdfunding to solicit donations, typically from multiple small donors using internet-based means, to offset the financial toxicity of cancer care. OBJECTIVE: To describe crowdfunding campaigns by gynecologic cancer patients and to compare campaign characteristics and needs expressed between patients with cervical, uterine, and ovarian cancer. STUDY DESIGN: We queried the public crowdfunding forum GoFundMe.com for "cervical cancer," "uterine cancer," and "ovarian cancer." The first 200 consecutive posts for each cancer type fundraising within the United States were analyzed. Data on campaign goals and needs expressed were manually extracted. Descriptive statistics and bivariate analyses were performed. RESULTS: Among the 600 fundraising pages, the median campaign goal was $10,000 [IQR $5000-$23,000]. Campaigns raised a median of 28.6% of their goal with only 8.7% of campaigns reaching their goal after a median of 54 days online. On average, ovarian cancer campaigns had higher monetary goals, more donors, and larger donation amounts than cervical cancer campaigns and raised more money than both cervical and uterine cancer campaigns. Campaigns were fundraising to support medical costs (80-85%) followed by lost wages (36-56%) or living expenses (27-41%). Cervical cancer campaigns reported need for non-medical costs more frequently than uterine or ovarian cancer campaigns. States without Medicaid expansions (31% of the national population) were over-represented among cervical cancer and uterine cancer, but not ovarian cancer campaigns. CONCLUSIONS: Crowdfunding pages reveal patients fundraising for out-of-pocket costs in the thousands of dollars and a wide range of unmet financial needs based on cancer type.

Charitable crowdfunding donation-intention estimation depending on emotional project images using fNIRS-based functional connectivity.

Charitable fundraising increasingly relies on online crowdfunding platforms. Project images of charitable crowdfunding use emotional appeals to promote helping behavior. Negative emotions are commonly used to motivate helping behavior because the image of a happy child may not motivate donors to donate as willingly. However, some research has found that happy images can be more beneficial. These contradictory results suggest that the emotional valence of project imagery and how fundraisers frame project images effectively remain debatable. Thus, we compared and analyzed brain activation differences in the prefrontal cortex governing human emotions depending on donation decisions using functional near-infrared spectroscopy, a neuroimaging device. We advance existing theory on charitable behavior by demonstrating that little correlation exists in donation intentions and brain activity between negative and positive project images, which is consistent with survey results on donation intentions by victim image. We also discovered quantitative brain hemodynamic signal variations between donors and nondonors, which can predict and detect donor mental brain functioning using functional connectivity, that is, the statistical dependence between the time series of electrophysiological activity and oxygenated hemodynamic levels in the prefrontal cortex. These findings are critical in developing future marketing strategies for online charitable crowdfunding platforms, especially project images.

Protocol for the crowdsourced image-based morbidity hotspot surveillance for neglected tropical diseases (CIMS-NTDs).

INTRODUCTION: Efficient NTDs elimination strategies require effective surveillance and targeted interventions. Traditional methods are costly and time-consuming, often failing to cover entire populations in case of movement restrictions. To address these challenges, a morbidity image-based surveillance system is being developed. This innovative approach which leverages the smartphone technology aims at simultaneous surveillance of multiple NTDs, enhancing cost-efficiency, reliability, and community involvement, particularly in areas with movement constraints. Moreover, it holds promise for post-elimination surveillance. METHODOLOGY: The pilot of this method will be conducted across three states in southern Nigeria. It will target people affected by Neglected Tropical Diseases and members of their communities. The new surveillance method will be introduced to target communities in the selected states through community stakeholder's advocacy meetings and awareness campaigns. The pilot which is set to span eighteen months, entails sensitizing NTDs-affected individuals and community members using signposts, posters, and handbills, to capture photos of NTDs manifestations upon notice using smartphones. These images, along with pertinent demographic information, will be transmitted to a dedicated server through WhatsApp or Telegram accounts. The received images will be reviewed and organized at backend and then forwarded to a panel of experts for identification and annotation to specific NTDs. Data generated, along with geocoordinate information, will be used to create NTDs morbidity hotspot maps using ArcGIS. Accompanying metadata will be used to generate geographic and demographic distributions of various NTDs identified. To protect privacy, people will be encouraged to send manifestation photos of the affected body part only without any identifiable features. EVALUATION PROTOCOL: NTDs prevalence data obtained using conventional surveillance methods from both the pilot and selected control states during the pilot period will be compared with data from the CIMS-NTDs method to determine its effectiveness. EXPECTED RESULTS AND CONCLUSION: It is expected that an effective, privacy-conscious, population inclusive new method for NTDs surveillance, with the potential to yield real-time data for the identification of morbidity hotspots and distribution patterns of NTDs will be established. The results will provide insights into the effectiveness of the new surveillance method in comparison to traditional approaches, potentially advancing NTDs elimination strategies.

Boosting wisdom of the crowd for medical image annotation using training performance and task features.

A crucial bottleneck in medical artificial intelligence (AI) is high-quality labeled medical datasets. In this paper, we test a large variety of wisdom of the crowd algorithms to label medical images that were initially classified by individuals recruited through an app-based platform. Individuals classified skin lesions from the International Skin Lesion Challenge 2018 into 7 different categories. There was a large dispersion in the geographical location, experience, training, and performance of the recruited individuals. We tested several wisdom of the crowd algorithms of varying complexity from a simple unweighted average to more complex Bayesian models that account for individual patterns of errors. Using a switchboard analysis, we observe that the best-performing algorithms rely on selecting top performers, weighting decisions by training accuracy, and take into account the task environment. These algorithms far exceed expert performance. We conclude by discussing the implications of these approaches for the development of medical AI.

Enabling Health Information Recommendation Using Crowdsourced Refinement in Web-Based Health Information Applications: User-Centered Design Approach and EndoZone Informatics Case Study.

BACKGROUND: In the digital age, search engines and social media platforms are primary sources for health information, yet their commercial interests-focused algorithms often prioritize irrelevant content. Web-based health applications by reputable sources offer a solution to circumvent these biased algorithms. Despite this advantage, there remains a significant gap in research on the effective integration of content-ranking algorithms within these specialized health applications to ensure the delivery of personalized and relevant health information. OBJECTIVE: This study introduces a generic methodology designed to facilitate the development and implementation of health information recommendation features within web-based health applications. METHODS: We detail our proposed methodology, covering conceptual foundation and practical considerations through the stages of design, development, operation, review, and optimization in the software development life cycle. Using a case study, we demonstrate the practical application of the proposed methodology through the implementation of recommendation functionalities in the EndoZone platform, a platform dedicated to providing targeted health information on endometriosis. RESULTS: Application of the proposed methodology in the EndoZone platform led to the creation of a tailored health information recommendation system known as EndoZone Informatics. Feedback from EndoZone stakeholders as well as insights from the implementation process validate the methodology's utility in enabling advanced recommendation features in health information applications. Preliminary assessments indicate that the system successfully delivers personalized content, adeptly incorporates user feedback, and exhibits considerable flexibility in adjusting its recommendation logic. While certain project-specific design flaws were not caught in the initial stages, these issues were subsequently identified and rectified in the review and optimization stages. CONCLUSIONS: We propose a generic methodology to guide the design and implementation of health information recommendation functionality within web-based health information applications. By harnessing user characteristics and feedback for content ranking, this methodology enables the creation of personalized recommendations that align with individual user needs within trusted health applications. The successful application of our methodology in the development of EndoZone Informatics marks a significant progress toward personalized health information delivery at scale, tailored to the specific needs of users.

Repeat it without me: Crowdsourcing the T1 mapping common ground via the ISMRM reproducibility challenge.

PURPOSE: T1 mapping is a widely used quantitative MRI technique, but its tissue-specific values remain inconsistent across protocols, sites, and vendors. The ISMRM Reproducible Research and Quantitative MR study groups jointly launched a challenge to assess the reproducibility of a well-established inversion-recovery T1 mapping technique, using acquisition details from a seminal T1 mapping paper on a standardized phantom and in human brains. METHODS: The challenge used the acquisition protocol from Barral et al. (2010). Researchers collected T1 mapping data on the ISMRM/NIST phantom and/or in human brains. Data submission, pipeline development, and analysis were conducted using open-source platforms. Intersubmission and intrasubmission comparisons were performed. RESULTS: Eighteen submissions (39 phantom and 56 human datasets) on scanners by three MRI vendors were collected at 3 T (except one, at 0.35 T). The mean coefficient of variation was 6.1% for intersubmission phantom measurements, and 2.9% for intrasubmission measurements. For humans, the intersubmission/intrasubmission coefficient of variation was 5.9/3.2% in the genu and 16/6.9% in the cortex. An interactive dashboard for data visualization was also developed: https://rrsg2020.dashboards.neurolibre.org. CONCLUSION: The T1 intersubmission variability was twice as high as the intrasubmission variability in both phantoms and human brains, indicating that the acquisition details in the original paper were insufficient to reproduce a quantitative MRI protocol. This study reports the inherent uncertainty in T1 measures across independent research groups, bringing us one step closer to a practical clinical baseline of T1 variations in vivo.

Quantifying the impact of misinformation and vaccine-skeptical content on Facebook.

Low uptake of the COVID-19 vaccine in the US has been widely attributed to social media misinformation. To evaluate this claim, we introduce a framework combining lab experiments (total N = 18,725), crowdsourcing, and machine learning to estimate the causal effect of 13,206 vaccine-related URLs on the vaccination intentions of US Facebook users (N ≈ 233 million). We estimate that the impact of unflagged content that nonetheless encouraged vaccine skepticism was 46-fold greater than that of misinformation flagged by fact-checkers. Although misinformation reduced predicted vaccination intentions significantly more than unflagged vaccine content when viewed, Facebook users' exposure to flagged content was limited. In contrast, unflagged stories highlighting rare deaths after vaccination were among Facebook's most-viewed stories. Our work emphasizes the need to scrutinize factually accurate but potentially misleading content in addition to outright falsehoods.

A Perspective on Crowdsourcing and Human-in-the-Loop Workflows in Precision Health.

Modern machine learning approaches have led to performant diagnostic models for a variety of health conditions. Several machine learning approaches, such as decision trees and deep neural networks, can, in principle, approximate any function. However, this power can be considered to be both a gift and a curse, as the propensity toward overfitting is magnified when the input data are heterogeneous and high dimensional and the output class is highly nonlinear. This issue can especially plague diagnostic systems that predict behavioral and psychiatric conditions that are diagnosed with subjective criteria. An emerging solution to this issue is crowdsourcing, where crowd workers are paid to annotate complex behavioral features in return for monetary compensation or a gamified experience. These labels can then be used to derive a diagnosis, either directly or by using the labels as inputs to a diagnostic machine learning model. This viewpoint describes existing work in this emerging field and discusses ongoing challenges and opportunities with crowd-powered diagnostic systems, a nascent field of study. With the correct considerations, the addition of crowdsourcing to human-in-the-loop machine learning workflows for the prediction of complex and nuanced health conditions can accelerate screening, diagnostics, and ultimately access to care.

The Wallet Biopsy: Medical Crowdfunding for Heart Transplantation.

Financial considerations continue to impact access to heart transplantation. Transplant recipients face various costs, including, but not limited to, the index hospitalization, immunosuppressive medications, and lodging and travel to appointments. In this study, we sought to describe the state of crowdfunding for individuals being evaluated for heart transplantation. Using the search term heart transplant, 1000 GoFundMe campaigns were reviewed. After exclusions, 634 (63.4%) campaigns were included. Most campaigns were in support of white individuals (57.8%), males (63.1%) and adults (76.7%). Approximately 15% of campaigns had not raised any funds. The remaining campaigns fundraised a median of $53.24 dollars per day. Of the patients, 44% were admitted at the time of the fundraising. Within the campaigns in the United States, the greatest proportions were in the Southeast United States in non-Medicaid expansion states. These findings highlight the significant financial toxicities associated with heart transplantation and the need for advocacy at the governmental and payer levels to improve equitable access and coverage for all.

The Impact of Health Literacy on Trust in Online Plastic Surgery Content: A Nationwide Analysis.

INTRODUCTION: On most online platforms, just about anyone can disseminate plastic surgery (PS) content regardless of their educational or professional background. This study examines the general public's perceptions of the accuracy of online PS content and the factors that contribute to the discernment of credible information. METHODS: The Amazon Mechanical Turk crowdsourcing platform was used to survey adults in the United States. The survey assessed respondent demographics, health literacy (HL), and perceptions of online PS content accuracy. T-tests, Chi-square tests, and post hoc analyses with Bonferroni corrections assessed differences between HL groups. Multivariate linear regressions assessed associations between sociodemographic variables and perceptions of online content. RESULTS: In total, 428 (92.0%) of 465 complete responses were analyzed. The median age of respondents was 32 y (interquartile range: 29-40). Online sources were predominantly perceived to have a high degree of accuracy, with mean scores of various platforms ranging from 3.8 to 4.5 (1 = not accurate at all; 5 = extremely accurate). The low HL group perceived social media sites and review sites to be more accurate than the high HL respondents, particularly for Reddit (P = 0.004), Pinterest (P = 0.040), and Snapchat (P = 0.002). CONCLUSIONS: There is a concerning relationship between low HL and the perceptions of the accuracy of online PS sources. This study underscores the need for education campaigns, the development of trustworthy online resources, and initiatives to improve HL. By fostering a more informed public, individuals seeking PS can make better informed decisions.

Crowdsourcing for mining new fungal sources for addressing the need for novel antibiotics against multidrug resistant pathogens.

There are a limited number of new antibiotics to manage the health crisis caused by the evolution and spread of antimicrobial resistant (AMR) bacteria including multidrug resistant (MDR), extensively drug-resistant (XDR) and pan-drug-resistant (PDR) ones. Bioprospecting fungi of less studied and extreme environments using new and less used older approaches could reveal novel antibiotics to manage MDR pathogens. Furthermore, I posit a crowdsourcing model which could substantially increase the chances of discovering novel antibiotics as well as new chemotypes for other therapeutic areas and considerably reduce the cost and time of this exercise.

Crowdsourcing to increase hepatitis B and C testing and reduce hepatitis stigma among medical students in Bangladesh.

This study addresses the pervasive challenges of low hepatitis B (HBV) and hepatitis C (HCV) testing rates coupled with the stigma associated with these diseases in low- and middle-income countries (LMICs) with a special focus on Bangladesh. This study aims to introduce an innovative crowdsourcing intervention that involves medical students, a crucial cohort with the potential to shape healthcare attitudes. Through a structured crowdsourcing approach, the study designs and implements a digital intervention to counter stigma and promote testing among medical students in Dhaka, Bangladesh. Participants submitted brief videos or texts aiming to encourage hepatitis testing and reduce stigma. The call, advertised through meetings, emails, and social media, welcomed entries in English or Bengali over 3 weeks. A panel of six judges evaluated each entry based on clarity, impact potential, innovation, feasibility, and sustainability, awarding prizes to students behind the highest-rated submissions. Seventeen videos and four text messages received an average score of 5.5 among 440 surveyed medical students, predominantly 22 years old (16%) and in their fourth year (21%). After viewing, 360 students underwent screening, identifying two previously undiagnosed HBV cases referred for care; no HCV infections were found. Notably, 41% expressed concerns about individuals with HBV working in hospitals or having a doctor living with HBV. In conclusion, this pilot showcases the power of medical students in spearheading campaigns to counter hepatitis stigma and encourage testing. By utilizing crowdsourcing, the study introduces an innovative approach to a persistent issue in LMICs specially in Bangladesh, offering a model that could potentially be adapted by other regions grappling with similar challenges.

Burden of atopic dermatitis in Europe: A population-centred approach leveraging web search data in 21 European countries.

BACKGROUND: The complexity, high prevalence, and substantial personal and socioeconomic burden collectively render atopic dermatitis (AD) a major public health concern. Using crowdsourced Internet data has the potential to provide unique insights into this concern, as demonstrated by several previous studies. However, a comprehensive comparison across European countries remains lacking. OBJECTIVES: The study aimed to investigate AD-related web searches across Europe to assess spatiotemporal variations and associations between disease-related and external factors. METHODS: AD-related web search data were extracted for 21 European countries between February 2019 and January 2023. Descriptive analysis and autocorrelation functions were performed to examine spatiotemporal patterns. Correlations (r) were used to evaluate the associations between web searches and disease-related, socioeconomic and meteorological data. RESULTS: Over 241 million AD-related web searches were identified, with search volume varying substantially among European countries (p < 0.001) and correlating with AD prevalence and disease burden (both r = 0.51, p = 0.019). Search volume increased between 2019 and 2023 in all countries and seasonally peaked in January and March. Negative correlations with median population age (r = -0.46, p = 0.039), number of general practitioners (r = -0.29, p = 0.226) and specialists (r = -0.27, p = 0.270) were observed. Moderate to strong correlations were found between search volume and cold, humid and windy weather with fewer sunshine hours, while higher online interest typically occurred 1-3 months after such weather conditions. CONCLUSION: The study highlights the great potential of online crowdsourced data analysis, for example, to investigate the impact of climate change or to identify unmet needs at a population level. Furthermore, the growing online interest in AD and the corresponding seasonal peaks emphasize the necessity of adapting treatment plans, intensifying public health campaigns, and disseminating reliable online information by governments and healthcare providers, especially during these periods.

Developing Crowdsourced Clinical Registry Studies.

ABSTRACT: Crowdsourced registries have been used to quickly gather information, especially during emerging public health concerns. Registries that began during the COVID-19 pandemic were used to rapidly answer key questions on coinfections, experimental treatments, and morbidity and mortality outcomes. Registries are also used more frequently to support clinical trials and track long-term outcomes in patient populations. This article reviews registry methodology, including the collection of data from crowdsourcing and real-world sources, that can be applied to nurse researcher and clinical research nurse skill sets. The authors illustrate a recently reported crowdsourced COVID-19 and cryptococcal disease registry that followed project management strategies and the Agency for Healthcare Research and Quality registry guidelines for planning, execution, and analysis of registries and other research methods.