Expanding the Immunology Toolbox: Embracing Public-Data Reuse and Crowdsourcing.

New technologies have been propelling dramatic increases in the volume and diversity of large-scale public data, which can potentially be reused to answer questions beyond those originally envisioned. However, this often requires computational and statistical skills beyond the reach of most bench scientists. The development of educational and accessible computational tools is thus critical, as are crowdsourcing efforts that utilize the community's expertise to curate public data for hypothesis generation and testing. Here we review the history of public-data reuse and argue for greater incorporation of computational and statistical sciences into the biomedical education curriculum and the development of biologist-friendly crowdsourcing tools. Finally, we provide a resource list for the reuse of public data and highlight an illustrative crowdsourcing exercise to explore public gene-expression data of human autoimmune diseases and corresponding mouse models. Through education, tool development, and community engagement, immunologists will be poised to transform public data into biological insights.

Crowdsourced RNA design discovers diverse, reversible, efficient, self-contained molecular switches.

Internet-based scientific communities promise a means to apply distributed, diverse human intelligence toward previously intractable scientific problems. However, current implementations have not allowed communities to propose experiments to test all emerging hypotheses at scale or to modify hypotheses in response to experiments. We report high-throughput methods for molecular characterization of nucleic acids that enable the large-scale video game­based crowdsourcing of RNA sensor design, followed by high-throughput functional characterization. Iterative design testing of thousands of crowdsourced RNA sensor designs produced near­thermodynamically optimal and reversible RNA switches that act as self-contained molecular sensors and couple five distinct small molecule inputs to three distinct protein binding and fluorogenic outputs. This work suggests a paradigm for widely distributed experimental bioscience.

The Wallet Biopsy: Medical Crowdfunding for Heart Transplantation.

Financial considerations continue to impact access to heart transplantation. Transplant recipients face various costs, including, but not limited to, the index hospitalization, immunosuppressive medications, and lodging and travel to appointments. In this study, we sought to describe the state of crowdfunding for individuals being evaluated for heart transplantation. Using the search term heart transplant, 1000 GoFundMe campaigns were reviewed. After exclusions, 634 (63.4%) campaigns were included. Most campaigns were in support of white individuals (57.8%), males (63.1%) and adults (76.7%). Approximately 15% of campaigns had not raised any funds. The remaining campaigns fundraised a median of $53.24 dollars per day. Of the patients, 44% were admitted at the time of the fundraising. Within the campaigns in the United States, the greatest proportions were in the Southeast United States in non-Medicaid expansion states. These findings highlight the significant financial toxicities associated with heart transplantation and the need for advocacy at the governmental and payer levels to improve equitable access and coverage for all.

A crowdsourcing open platform for literature curation in UniProt.

The UniProt knowledgebase is a public database for protein sequence and function, covering the tree of life and over 220 million protein entries. Now, the whole community can use a new crowdsourcing annotation system to help scale up UniProt curation and receive proper attribution for their biocuration work.

Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis.

BACKGROUND: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality. OBJECTIVE: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data. METHODS: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips. RESULTS: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts' opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance. CONCLUSIONS: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.

Enhancing Community Participation in Dengue Control Through Digital Crowdsourcing: An Analysis of a World Mosquito Program Digital Open Call in Sri Lanka.

BACKGROUND: Two crowdsourcing open calls were created to enhance community engagement in dengue control in Sri Lanka. We analyzed the process and outcomes of these digital crowdsourcing open calls. METHODS: We used standard World Health Organization methods to organize the open calls, which used exclusively digital methods because of coronavirus disease 2019 (COVID-19). We collected and analyzed sociodemographic information and digital engagement metrics from each submission. Submissions in the form of textual data describing community-led strategies for mosquito release were coded using grounded theory. RESULTS: The open calls received 73 submissions. Most people who submitted ideas spoke English, lived in Sri Lanka, and were 18 to 34 years old. The total Facebook reach was initially limited (16 161 impressions), prompting expansion to a global campaign, which reached 346 810 impressions over 14 days. Diverse strategies for the distribution of Wolbachia-infected mosquito boxes were identified, including leveraging traditional festivals, schools, and community networks. Fifteen submissions (21%) suggested the use of digital tools for monitoring and evaluation, sharing instructions, or creating networks. Thirteen submissions (18%) focused on social and economic incentives to prompt community engagement and catalyze community-led distribution. CONCLUSIONS: Our project demonstrates that digital crowdsourcing open calls are an effective way to solicit creative and innovative ideas in a resource-limited setting.

Struggling, helping and adapting: Crowdfunding motivations and outcomes during the early US COVID-19 pandemic.

During the early months of COVID-19, many people in the US turned to charitable crowdfunding to seek and provide assistance. Little is known about the needs, hopes or experiences that motivated US pandemic crowdfunding and how these were correlated with campaign success. This study uses a mixed-methods data analysis of a randomised cluster sample of 919 US GoFundMe campaigns during the first 7 months of the pandemic. Overall, most campaigns performed poorly, and 38% got no donations at all. The largest proportion of campaigns aimed to address individual, acute financial struggles, often arising from considerable challenges accessing or qualifying for government assistance. These campaigns, as well as those involving campaigners and beneficiaries of colour, tended to be least successful. Qualitative thematic analysis revealed three key crowdfunding motivations that reflect individualistic, agentive responses to the pandemic: struggling, helping and adapting. These motivations reveal a shift away from social suffering and collective mobilisation and towards largely individualised efforts of survival as digital crowdfunding becomes a key domain of crisis response. Crowdfunding platforms are playing an increasingly important role in mediating and influencing individual and collective responses to crisis, which has important political ramifications for how societies perceive and address health emergencies.

Exploring Novel Innovation Strategies to Close a Technology Gap in Neurosurgery: HORAO Crowdsourcing Campaign.

BACKGROUND: Scientific research is typically performed by expert individuals or groups who investigate potential solutions in a sequential manner. Given the current worldwide exponential increase in technical innovations, potential solutions for any new problem might already exist, even though they were developed to solve a different problem. Therefore, in crowdsourcing ideation, a research question is explained to a much larger group of individuals beyond the specialist community to obtain a multitude of diverse, outside-the-box solutions. These are then assessed in parallel by a group of experts for their capacity to solve the new problem. The 2 key problems in brain tumor surgery are the difficulty of discerning the exact border between a tumor and the surrounding brain, and the difficulty of identifying the function of a specific area of the brain. Both problems could be solved by a method that visualizes the highly organized fiber tracts within the brain; the absence of fibers would reveal the tumor, whereas the spatial orientation of the tracts would reveal the area's function. To raise awareness about our challenge of developing a means of intraoperative, real-time, noninvasive identification of fiber tracts and tumor borders to improve neurosurgical oncology, we turned to the crowd with a crowdsourcing ideation challenge. OBJECTIVE: Our objective was to evaluate the feasibility of a crowdsourcing ideation campaign for finding novel solutions to challenges in neuroscience. The purpose of this paper is to introduce our chosen crowdsourcing method and discuss it in the context of the current literature. METHODS: We ran a prize-based crowdsourcing ideation competition called HORAO on the commercial platform HeroX. Prize money previously collected through a crowdfunding campaign was offered as an incentive. Using a multistage approach, an expert jury first selected promising technical solutions based on broad, predefined criteria, coached the respective solvers in the second stage, and finally selected the winners in a conference setting. We performed a postchallenge web-based survey among the solvers crowd to find out about their backgrounds and demographics. RESULTS: Our web-based campaign reached more than 20,000 people (views). We received 45 proposals from 32 individuals and 7 teams, working in 26 countries on 4 continents. The postchallenge survey revealed that most of the submissions came from single solvers or teams working in engineering or the natural sciences, with additional submissions from other nonmedical fields. We engaged in further exchanges with 3 out of the 5 finalists and finally initiated a successful scientific collaboration with the winner of the challenge. CONCLUSIONS: This open innovation competition is the first of its kind in medical technology research. A prize-based crowdsourcing ideation campaign is a promising strategy for raising awareness about a specific problem, finding innovative solutions, and establishing new scientific collaborations beyond strictly disciplinary domains.

Assessing Interventions on Crowdsourcing Platforms to Nudge Patients for Engagement Behaviors in Primary Care Settings: Randomized Controlled Trial.

BACKGROUND: Engaging patients in health behaviors is critical for better outcomes, yet many patient partnership behaviors are not widely adopted. Behavioral economics-based interventions offer potential solutions, but it is challenging to assess the time and cost needed for different options. Crowdsourcing platforms can efficiently and rapidly assess the efficacy of such interventions, but it is unclear if web-based participants respond to simulated incentives in the same way as they would to actual incentives. OBJECTIVE: The goals of this study were (1) to assess the feasibility of using crowdsourced surveys to evaluate behavioral economics interventions for patient partnerships by examining whether web-based participants responded to simulated incentives in the same way they would have responded to actual incentives, and (2) to assess the impact of 2 behavioral economics-based intervention designs, psychological rewards and loss of framing, on simulated medication reconciliation behaviors in a simulated primary care setting. METHODS: We conducted a randomized controlled trial using a between-subject design on a crowdsourcing platform (Amazon Mechanical Turk) to evaluate the effectiveness of behavioral interventions designed to improve medication adherence in primary care visits. The study included a control group that represented the participants' baseline behavior and 3 simulated interventions, namely monetary compensation, a status effect as a psychological reward, and a loss frame as a modification of the status effect. Participants' willingness to bring medicines to a primary care visit was measured on a 5-point Likert scale. A reverse-coding question was included to ensure response intentionality. RESULTS: A total of 569 study participants were recruited. There were 132 in the baseline group, 187 in the monetary compensation group, 149 in the psychological reward group, and 101 in the loss frame group. All 3 nudge interventions increased participants' willingness to bring medicines significantly when compared to the baseline scenario. The monetary compensation intervention caused an increase of 17.51% (P<.001), psychological rewards on status increased willingness by 11.85% (P<.001), and a loss frame on psychological rewards increased willingness by 24.35% (P<.001). Responses to the reverse-coding question were consistent with the willingness questions. CONCLUSIONS: In primary care, bringing medications to office visits is a frequently advocated patient partnership behavior that is nonetheless not widely adopted. Crowdsourcing platforms such as Amazon Mechanical Turk support efforts to efficiently and rapidly reach large groups of individuals to assess the efficacy of behavioral interventions. We found that crowdsourced survey-based experiments with simulated incentives can produce valid simulated behavioral responses. The use of psychological status design, particularly with a loss framing approach, can effectively enhance patient engagement in primary care. These results support the use of crowdsourcing platforms to augment and complement traditional approaches to learning about behavioral economics for patient engagement.

The Success of Cancer Crowdfunding Campaigns: Project and Text Analysis.

BACKGROUND: Recent studies have analyzed the factors that contribute to variations in the success of crowdfunding campaigns for a specific cancer type; however, little is known about the influential factors among crowdfunding campaigns for multiple cancers. OBJECTIVE: The purpose of this study was to examine the relationship between project features and the success of cancer crowdfunding campaigns and to determine whether text features affect campaign success for various cancers. METHODS: Using cancer-related crowdfunding projects on the GoFundMe website, we transformed textual descriptions from the campaigns into structured data using natural language processing techniques. Next, we used penalized logistic regression and correlation analyses to examine the influence of project and text features on fundraising project outcomes. Finally, we examined the influence of campaign description sentiment on crowdfunding success using Linguistic Inquiry and Word Count software. RESULTS: Campaigns were significantly more likely to be successful if they featured a lower target amount (Goal amount, ß=-1.949, z score=-82.767, P<.001) for fundraising, a higher number of previous donations, agency (vs individual) organizers, project pages containing updates, and project pages containing comments from readers. The results revealed an inverted U-shaped relationship between the length of the text and the amount of funds raised. In addition, more spelling mistakes negatively affected the funds raised (Number of spelling errors, ß=-1.068, z score=-38.79, P<.001). CONCLUSIONS: Difficult-to-treat cancers and high-mortality cancers tend to trigger empathy from potential donors, which increases the funds raised. Gender differences were observed in the effects of emotional words in the text on the amount of funds raised. For cancers that typically occur in women, links between emotional words used and the amount of funds raised were weaker than for cancers typically occurring among men.

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study.

BACKGROUND: As trachoma is eliminated, skilled field graders become less adept at correctly identifying active disease (trachomatous inflammation-follicular [TF]). Deciding if trachoma has been eliminated from a district or if treatment strategies need to be continued or reinstated is of critical public health importance. Telemedicine solutions require both connectivity, which can be poor in the resource-limited regions of the world in which trachoma occurs, and accurate grading of the images. OBJECTIVE: Our purpose was to develop and validate a cloud-based "virtual reading center" (VRC) model using crowdsourcing for image interpretation. METHODS: The Amazon Mechanical Turk (AMT) platform was used to recruit lay graders to interpret 2299 gradable images from a prior field trial of a smartphone-based camera system. Each image received 7 grades for US $0.05 per grade in this VRC. The resultant data set was divided into training and test sets to internally validate the VRC. In the training set, crowdsourcing scores were summed, and the optimal raw score cutoff was chosen to optimize kappa agreement and the resulting prevalence of TF. The best method was then applied to the test set, and the sensitivity, specificity, kappa, and TF prevalence were calculated. RESULTS: In this trial, over 16,000 grades were rendered in just over 60 minutes for US $1098 including AMT fees. After choosing an AMT raw score cut point to optimize kappa near the World Health Organization (WHO)-endorsed level of 0.7 (with a simulated 40% prevalence TF), crowdsourcing was 95% sensitive and 87% specific for TF in the training set with a kappa of 0.797. All 196 crowdsourced-positive images received a skilled overread to mimic a tiered reading center and specificity improved to 99%, while sensitivity remained above 78%. Kappa for the entire sample improved from 0.162 to 0.685 with overreads, and the skilled grader burden was reduced by over 80%. This tiered VRC model was then applied to the test set and produced a sensitivity of 99% and a specificity of 76% with a kappa of 0.775 in the entire set. The prevalence estimated by the VRC was 2.70% (95% CI 1.84%-3.80%) compared to the ground truth prevalence of 2.87% (95% CI 1.98%-4.01%). CONCLUSIONS: A VRC model using crowdsourcing as a first pass with skilled grading of positive images was able to identify TF rapidly and accurately in a low prevalence setting. The findings from this study support further validation of a VRC and crowdsourcing for image grading and estimation of trachoma prevalence from field-acquired images, although further prospective field testing is required to determine if diagnostic characteristics are acceptable in real-world surveys with a low prevalence of the disease.

Digital, Crowdsourced, Multilevel Intervention to Promote HIV Testing Among Men Who Have Sex With Men: Cluster Randomized Controlled Trial.

BACKGROUND: Despite great efforts in HIV prevention worldwide, HIV testing uptake among men who have sex with men (MSM) remains suboptimal. The effectiveness of digital, crowdsourced, multilevel interventions in improving HIV testing is still unclear. OBJECTIVE: The aim of this study was to evaluate the effect of a digital, crowdsourced, multilevel intervention in improving HIV testing uptake among MSM in China. METHODS: We conducted a 2-arm cluster randomized controlled trial among MSM in 11 cities in Shandong province, China, from August 2019 to April 2020. Participants were men who were HIV seronegative or had unknown serum status, had anal sex with a man in the past 12 months, and had not been tested for HIV in the past 3 months. Participants were recruited through a gay dating app and community-based organizations from preselected cities; these cities were matched into 5 blocks (2 clusters per block) and further randomly assigned (1:1) to receive a digital, crowdsourced, multilevel intervention (intervention arm) or routine intervention (control arm). The digital multilevel intervention was developed through crowdsourced open calls tailored for MSM, consisting of digital intervention images and videos, the strategy of providing HIV self-testing services through digital tools, and peer-moderated discussion within WeChat groups. The primary outcome was self-reported HIV testing uptake in the previous 3 months. An intention-to-treat approach was used to examine the cluster-level effect of the intervention in the 12-month study period using generalized linear mixed models and the individual-level effect using linear mixed models. RESULTS: A total of 935 MSM were enrolled (404 intervention participants and 531 controls); 751 participants (80.3%) completed at least one follow-up survey. Most participants were younger than 30 years (n=601, 64.3%), single (n=681, 72.8%), had a college degree or higher (n=629, 67.3%), and had an HIV testing history (n=785, 84%). Overall, the proportion of testing for HIV in the past 3 months at the 3-, 6-, 9-, and 12-month follow-ups was higher in the intervention arm (139/279, 49.8%; 148/266, 55.6%; 189/263, 71.9%; and 171/266, 64.3%, respectively) than the control arm (183/418, 43.8%; 178/408, 43.6%; 206/403, 51.1%; and 182/397, 48.4%, respectively), with statistically significant differences at the 6-, 9-, and 12-month follow-ups. At the cluster level, the proportion of participants who had tested for HIV increased 11.62% (95% CI 0.74%-22.5%; P=.04) with the intervention. At the individual level, participants in the intervention arm had 69% higher odds for testing for HIV in the past 3 months compared with control participants, but the result was not statistically significant (risk ratio 1.69, 95% CI 0.87-3.27; P=.11). CONCLUSIONS: The intervention effectively improved HIV testing uptake among Chinese MSM. Our findings highlight that digital, crowdsourced, multilevel interventions should be made more widely available for HIV prevention and other public health issues. TRIAL REGISTRATION: Chinese Clinical Trial Registry ChiCTR1900024350; http://www.chictr.org.cn/showproj.aspx?proj=36718. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1186/s13063-020-04860-8.

Thematic and Demographic Analysis of Crowdsourcing Practices for Plastic Surgery.

BACKGROUND: In this study, we investigate the characterization of medical crowdsourcing on GoFundMe for plastic surgery procedures, with overall funds raised being the primary end point. HYPOTHESIS: Certain demographic factors such as sex and race mentioned in campaign narratives are associated with the effectiveness of medical crowdfunding campaigns. METHODS: Search terms were used to aggregate fundraising campaigns for plastic surgery medical procedures on GoFundMe. These studies were then stratified by demographics based on campaign text or author consensus, and were further subdivided into categories based on procedure type. RESULTS: Men were found to have higher median shares than women-raising an average of $609 more than female counterparts ( P < 0.05). Fundraising for themes such as lack of insurance, travel costs, lifesaving treatment, and end-of-life expenses were more successful than the theme of psychosocial effects of disease or social impairment. In addition, those that included a smiling picture of the recipient and those created by a friend/relative raised more funds. Although no significant difference was found in fundraising between demographics based on race, a majority (72.8%) of campaigners were White. Across ~2000 plastic surgery campaigns, a total of $10,186,687 were raised from these data. CONCLUSIONS: We identified both modifiable and nonmodifiable factors that influence success. These successful campaigns can serve as a learning opportunity for many who have been marginalized by the medical and pharmaceutical industry, and they demonstrate a promising area for demographic studies.

Evaluating Collaborative and Autonomous Agents in Data-Stream-Supported Coordination of Mobile Crowdsourcing.

Mobile crowdsourcing refers to systems where the completion of tasks necessarily requires physical movement of crowdworkers in an on-demand workforce. Evidence suggests that in such systems, tasks often get assigned to crowdworkers who struggle to complete those tasks successfully, resulting in high failure rates and low service quality. A promising solution to ensure higher quality of service is to continuously adapt the assignment and respond to failure-causing events by transferring tasks to better-suited workers who use different routes or vehicles. However, implementing task transfers in mobile crowdsourcing is difficult because workers are autonomous and may reject transfer requests. Moreover, task outcomes are uncertain and need to be predicted. In this paper, we propose different mechanisms to achieve outcome prediction and task coordination in mobile crowdsourcing. First, we analyze different data stream learning approaches for the prediction of task outcomes. Second, based on the suggested prediction model, we propose and evaluate two different approaches for task coordination with different degrees of autonomy: an opportunistic approach for crowdshipping with collaborative, but non-autonomous workers, and a market-based model with autonomous workers for crowdsensing.

A Model for Cognitive Personalization of Microtask Design.

The study of data quality in crowdsourcing campaigns is currently a prominent research topic, given the diverse range of participants involved. A potential solution to enhancing data quality processes in crowdsourcing is cognitive personalization, which involves appropriately adapting or assigning tasks based on a crowd worker's cognitive profile. There are two common methods for assessing a crowd worker's cognitive profile: administering online cognitive tests, and inferring behavior from task fingerprinting based on user interaction log events. This article presents the findings of a study that investigated the complementarity of both approaches in a microtask scenario, focusing on personalizing task design. The study involved 134 unique crowd workers recruited from a crowdsourcing marketplace. The main objective was to examine how the administration of cognitive ability tests can be used to allocate crowd workers to microtasks with varying levels of difficulty, including the development of a deep learning model. Another goal was to investigate if task fingerprinting can be used to allocate crowd workers to different microtasks in a personalized manner. The results indicated that both objectives were accomplished, validating the usage of cognitive tests and task fingerprinting as effective mechanisms for microtask personalization, including the development of a deep learning model with 95% accuracy in predicting the accuracy of the microtasks. While we achieved an accuracy of 95%, it is important to note that the small dataset size may have limited the model's performance.

Frustration and ennui among Amazon MTurk workers.

Academics are increasingly turning to crowdsourcing platforms to recruit research participants. Their endeavors have benefited from a proliferation of studies attesting to the quality of crowdsourced data or offering guidance on managing specific challenges associated with doing crowdsourced research. Thus far, however, relatively little is known about what it is like to be a participant in crowdsourced research. Our analysis of almost 1400 free-text responses provides insight into the frustrations encountered by workers on one widely used crowdsourcing site: Amazon's MTurk. Some of these frustrations stem from inherent limitations of the MTurk platform and cannot easily be addressed by researchers. Many others, however, concern factors that are directly controllable by researchers and that may also be relevant for researchers using other crowdsourcing platforms such as Prolific or CrowdFlower. Based on participants' accounts of their experiences as crowdsource workers, we offer recommendations researchers might consider as they seek to design online studies that demonstrate consideration for respondents and respect for their time, effort, and dignity.

Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk.

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.

Plant health emergencies demand open science: Tackling a cereal killer on the run.

Outbreaks of emerging plant diseases and insect pests are increasing at an alarming rate threatening the food security needs of a booming world population. The role of plant pathologists in addressing these threats to plant health is critical. Here, we share our personal experience with the appearance in Bangladesh of a destructive new fungal disease called wheat blast and stress the importance of open-science platforms and crowdsourced community responses in tackling emerging plant diseases. Benefits of the open-science approach include recruitment of multidisciplinary experts, application of cutting-edge methods, and timely replication of data analyses to increase the robustness of the findings. Based on our experiences, we provide some general recommendations and practical guidance for responding to emerging plant diseases.

Validation and tuning of in situ transcriptomics image processing workflows with crowdsourced annotations.

Recent advancements in in situ methods, such as multiplexed in situ RNA hybridization and in situ RNA sequencing, have deepened our understanding of the way biological processes are spatially organized in tissues. Automated image processing and spot-calling algorithms for analyzing in situ transcriptomics images have many parameters which need to be tuned for optimal detection. Having ground truth datasets (images where there is very high confidence on the accuracy of the detected spots) is essential for evaluating these algorithms and tuning their parameters. We present a first-in-kind open-source toolkit and framework for in situ transcriptomics image analysis that incorporates crowdsourced annotations, alongside expert annotations, as a source of ground truth for the analysis of in situ transcriptomics images. The kit includes tools for preparing images for crowdsourcing annotation to optimize crowdsourced workers' ability to annotate these images reliably, performing quality control (QC) on worker annotations, extracting candidate parameters for spot-calling algorithms from sample images, tuning parameters for spot-calling algorithms, and evaluating spot-calling algorithms and worker performance. These tools are wrapped in a modular pipeline with a flexible structure that allows users to take advantage of crowdsourced annotations from any source of their choice. We tested the pipeline using real and synthetic in situ transcriptomics images and annotations from the Amazon Mechanical Turk system obtained via Quanti.us. Using real images from in situ experiments and simulated images produced by one of the tools in the kit, we studied worker sensitivity to spot characteristics and established rules for annotation QC. We explored and demonstrated the use of ground truth generated in this way for validating spot-calling algorithms and tuning their parameters, and confirmed that consensus crowdsourced annotations are a viable substitute for expert-generated ground truth for these purposes.

Crowdsourcing biocuration: The Community Assessment of Community Annotation with Ontologies (CACAO).

Experimental data about gene functions curated from the primary literature have enormous value for research scientists in understanding biology. Using the Gene Ontology (GO), manual curation by experts has provided an important resource for studying gene function, especially within model organisms. Unprecedented expansion of the scientific literature and validation of the predicted proteins have increased both data value and the challenges of keeping pace. Capturing literature-based functional annotations is limited by the ability of biocurators to handle the massive and rapidly growing scientific literature. Within the community-oriented wiki framework for GO annotation called the Gene Ontology Normal Usage Tracking System (GONUTS), we describe an approach to expand biocuration through crowdsourcing with undergraduates. This multiplies the number of high-quality annotations in international databases, enriches our coverage of the literature on normal gene function, and pushes the field in new directions. From an intercollegiate competition judged by experienced biocurators, Community Assessment of Community Annotation with Ontologies (CACAO), we have contributed nearly 5,000 literature-based annotations. Many of those annotations are to organisms not currently well-represented within GO. Over a 10-year history, our community contributors have spurred changes to the ontology not traditionally covered by professional biocurators. The CACAO principle of relying on community members to participate in and shape the future of biocuration in GO is a powerful and scalable model used to promote the scientific enterprise. It also provides undergraduate students with a unique and enriching introduction to critical reading of primary literature and acquisition of marketable skills.