Identification of Social Engagement Indicators Associated With Autism Spectrum Disorder Using a Game-Based Mobile App: Comparative Study of Gaze Fixation and Visual Scanning Methods.

BACKGROUND: Autism spectrum disorder (ASD) is a widespread neurodevelopmental condition with a range of potential causes and symptoms. Standard diagnostic mechanisms for ASD, which involve lengthy parent questionnaires and clinical observation, often result in long waiting times for results. Recent advances in computer vision and mobile technology hold potential for speeding up the diagnostic process by enabling computational analysis of behavioral and social impairments from home videos. Such techniques can improve objectivity and contribute quantitatively to the diagnostic process. OBJECTIVE: In this work, we evaluate whether home videos collected from a game-based mobile app can be used to provide diagnostic insights into ASD. To the best of our knowledge, this is the first study attempting to identify potential social indicators of ASD from mobile phone videos without the use of eye-tracking hardware, manual annotations, and structured scenarios or clinical environments. METHODS: Here, we used a mobile health app to collect over 11 hours of video footage depicting 95 children engaged in gameplay in a natural home environment. We used automated data set annotations to analyze two social indicators that have previously been shown to differ between children with ASD and their neurotypical (NT) peers: (1) gaze fixation patterns, which represent regions of an individual's visual focus and (2) visual scanning methods, which refer to the ways in which individuals scan their surrounding environment. We compared the gaze fixation and visual scanning methods used by children during a 90-second gameplay video to identify statistically significant differences between the 2 cohorts; we then trained a long short-term memory (LSTM) neural network to determine if gaze indicators could be predictive of ASD. RESULTS: Our results show that gaze fixation patterns differ between the 2 cohorts; specifically, we could identify 1 statistically significant region of fixation (P<.001). In addition, we also demonstrate that there are unique visual scanning patterns that exist for individuals with ASD when compared to NT children (P<.001). A deep learning model trained on coarse gaze fixation annotations demonstrates mild predictive power in identifying ASD. CONCLUSIONS: Ultimately, our study demonstrates that heterogeneous video data sets collected from mobile devices hold potential for quantifying visual patterns and providing insights into ASD. We show the importance of automated labeling techniques in generating large-scale data sets while simultaneously preserving the privacy of participants, and we demonstrate that specific social engagement indicators associated with ASD can be identified and characterized using such data.

Crowd annotations can approximate clinical autism impressions from short home videos with privacy protections.

Artificial Intelligence (A.I.) solutions are increasingly considered for telemedicine. For these methods to serve children and their families in home settings, it is crucial to ensure the privacy of the child and parent or caregiver. To address this challenge, we explore the potential for global image transformations to provide privacy while preserving the quality of behavioral annotations. Crowd workers have previously been shown to reliably annotate behavioral features in unstructured home videos, allowing machine learning classifiers to detect autism using the annotations as input. We evaluate this method with videos altered via pixelation, dense optical flow, and Gaussian blurring. On a balanced test set of 30 videos of children with autism and 30 neurotypical controls, we find that the visual privacy alterations do not drastically alter any individual behavioral annotation at the item level. The AUROC on the evaluation set was 90.0% ±7.5% for unaltered videos, 85.0% ±9.0% for pixelation, 85.0% ±9.0% for optical flow, and 83.3% ±9.3% for blurring, demonstrating that an aggregation of small changes across behavioral questions can collectively result in increased misdiagnosis rates. We also compare crowd answers against clinicians who provided the same annotations for the same videos as crowd workers, and we find that clinicians have higher sensitivity in their recognition of autism-related symptoms. We also find that there is a linear correlation (r = 0.75, p < 0.0001) between the mean Clinical Global Impression (CGI) score provided by professional clinicians and the corresponding score emitted by a previously validated autism classifier with crowd inputs, indicating that the classifier's output probability is a reliable estimate of the clinical impression of autism. A significant correlation is maintained with privacy alterations, indicating that crowd annotations can approximate clinician-provided autism impression from home videos in a privacy-preserved manner.

Machine learning models using mobile game play accurately classify children with autism.

Digitally-delivered healthcare is well suited to address current inequities in the delivery of care due to barriers of access to healthcare facilities. As the COVID-19 pandemic phases out, we have a unique opportunity to capitalize on the current familiarity with telemedicine approaches and continue to advocate for mainstream adoption of remote care delivery. In this paper, we specifically focus on the ability of GuessWhat? a smartphone-based charades-style gamified therapeutic intervention for autism spectrum disorder (ASD) to generate a signal that distinguishes children with ASD from neurotypical (NT) children. We demonstrate the feasibility of using "in-the-wild", naturalistic gameplay data to distinguish between ASD and NT by children by training a random forest classifier to discern the two classes (AU-ROC = 0.745, recall = 0.769). This performance demonstrates the potential for GuessWhat? to facilitate screening for ASD in historically difficult-to-reach communities. To further examine this potential, future work should expand the size of the training sample and interrogate differences in predictive ability by demographic.

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study.

BACKGROUND: Automated emotion classification could aid those who struggle to recognize emotions, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult emotion and therefore underperform when applied to child faces. OBJECTIVE: We designed a strategy to gamify the collection and labeling of child emotion-enriched images to boost the performance of automatic child emotion recognition models to a level closer to what will be needed for digital health care approaches. METHODS: We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort, called HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotion-centric database (>30 times larger than existing public pediatric emotion data sets), we trained a convolutional neural network (CNN) computer vision classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions evoked by children. RESULTS: The classifier achieved a 66.9% balanced accuracy and 67.4% F1-score on the entirety of the Child Affective Facial Expression (CAFE) as well as a 79.1% balanced accuracy and 78% F1-score on CAFE Subset A, a subset containing at least 60% human agreement on emotions labels. This performance is at least 10% higher than all previously developed classifiers evaluated against CAFE, the best of which reached a 56% balanced accuracy even when combining "anger" and "disgust" into a single class. CONCLUSIONS: This work validates that mobile games designed for pediatric therapies can generate high volumes of domain-relevant data sets to train state-of-the-art classifiers to perform tasks helpful to precision health efforts.

A Mobile Game Platform for Improving Social Communication in Children with Autism: A Feasibility Study.

BACKGROUND: Many children with autism cannot receive timely in-person diagnosis and therapy, especially in situations where access is limited by geography, socioeconomics, or global health concerns such as the current COVD-19 pandemic. Mobile solutions that work outside of traditional clinical environments can safeguard against gaps in access to quality care. OBJECTIVE: The aim of the study is to examine the engagement level and therapeutic feasibility of a mobile game platform for children with autism. METHODS: We designed a mobile application, GuessWhat, which, in its current form, delivers game-based therapy to children aged 3 to 12 in home settings through a smartphone. The phone, held by a caregiver on their forehead, displays one of a range of appropriate and therapeutically relevant prompts (e.g., a surprised face) that the child must recognize and mimic sufficiently to allow the caregiver to guess what is being imitated and proceed to the next prompt. Each game runs for 90 seconds to create a robust social exchange between the child and the caregiver. RESULTS: We examined the therapeutic feasibility of GuessWhat in 72 children (75% male, average age 8 years 2 months) with autism who were asked to play the game for three 90-second sessions per day, 3 days per week, for a total of 4 weeks. The group showed significant improvements in Social Responsiveness Score-2 (SRS-2) total (3.97, p <0.001) and Vineland Adaptive Behavior Scales-II (VABS-II) socialization standard (5.27, p = 0.002) scores. CONCLUSION: The results support that the GuessWhat mobile game is a viable approach for efficacious treatment of autism and further support the possibility that the game can be used in natural settings to increase access to treatment when barriers to care exist.

Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder.

Crowd-powered telemedicine has the potential to revolutionize healthcare, especially during times that require remote access to care. However, sharing private health data with strangers from around the world is not compatible with data privacy standards, requiring a stringent filtration process to recruit reliable and trustworthy workers who can go through the proper training and security steps. The key challenge, then, is to identify capable, trustworthy, and reliable workers through high-fidelity evaluation tasks without exposing any sensitive patient data during the evaluation process. We contribute a set of experimentally validated metrics for assessing the trustworthiness and reliability of crowd workers tasked with providing behavioral feature tags to unstructured videos of children with autism and matched neurotypical controls. The workers are blinded to diagnosis and blinded to the goal of using the features to diagnose autism. These behavioral labels are fed as input to a previously validated binary logistic regression classifier for detecting autism cases using categorical feature vectors. While the metrics do not incorporate any ground truth labels of child diagnosis, linear regression using the 3 correlative metrics as input can predict the mean probability of the correct class of each worker with a mean average error of 7.51% for performance on the same set of videos and 10.93% for performance on a distinct balanced video set with different children. These results indicate that crowd workers can be recruited for performance based largely on behavioral metrics on a crowdsourced task, enabling an affordable way to filter crowd workforces into a trustworthy and reliable diagnostic workforce.

Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels.

Background/Introduction: Emotion detection classifiers traditionally predict discrete emotions. However, emotion expressions are often subjective, thus requiring a method to handle compound and ambiguous labels. We explore the feasibility of using crowdsourcing to acquire reliable soft-target labels and evaluate an emotion detection classifier trained with these labels. We hypothesize that training with labels that are representative of the diversity of human interpretation of an image will result in predictions that are similarly representative on a disjoint test set. We also hypothesize that crowdsourcing can generate distributions which mirror those generated in a lab setting. Methods: We center our study on the Child Affective Facial Expression (CAFE) dataset, a gold standard collection of images depicting pediatric facial expressions along with 100 human labels per image. To test the feasibility of crowdsourcing to generate these labels, we used Microworkers to acquire labels for 207 CAFE images. We evaluate both unfiltered workers as well as workers selected through a short crowd filtration process. We then train two versions of a ResNet-152 neural network on soft-target CAFE labels using the original 100 annotations provided with the dataset: (1) a classifier trained with traditional one-hot encoded labels, and (2) a classifier trained with vector labels representing the distribution of CAFE annotator responses. We compare the resulting softmax output distributions of the two classifiers with a 2-sample independent t-test of L1 distances between the classifier's output probability distribution and the distribution of human labels. Results: While agreement with CAFE is weak for unfiltered crowd workers, the filtered crowd agree with the CAFE labels 100% of the time for happy, neutral, sad and "fear + surprise", and 88.8% for "anger + disgust". While the F1-score for a one-hot encoded classifier is much higher (94.33% vs. 78.68%) with respect to the ground truth CAFE labels, the output probability vector of the crowd-trained classifier more closely resembles the distribution of human labels (t=3.2827, p=0.0014). Conclusions: For many applications of affective computing, reporting an emotion probability distribution that accounts for the subjectivity of human interpretation can be more useful than an absolute label. Crowdsourcing, including a sufficient filtering mechanism for selecting reliable crowd workers, is a feasible solution for acquiring soft-target labels.

Crowdsourced privacy-preserved feature tagging of short home videos for machine learning ASD detection.

Standard medical diagnosis of mental health conditions requires licensed experts who are increasingly outnumbered by those at risk, limiting reach. We test the hypothesis that a trustworthy crowd of non-experts can efficiently annotate behavioral features needed for accurate machine learning detection of the common childhood developmental disorder Autism Spectrum Disorder (ASD) for children under 8 years old. We implement a novel process for identifying and certifying a trustworthy distributed workforce for video feature extraction, selecting a workforce of 102 workers from a pool of 1,107. Two previously validated ASD logistic regression classifiers, evaluated against parent-reported diagnoses, were used to assess the accuracy of the trusted crowd's ratings of unstructured home videos. A representative balanced sample (N = 50 videos) of videos were evaluated with and without face box and pitch shift privacy alterations, with AUROC and AUPRC scores > 0.98. With both privacy-preserving modifications, sensitivity is preserved (96.0%) while maintaining specificity (80.0%) and accuracy (88.0%) at levels comparable to prior classification methods without alterations. We find that machine learning classification from features extracted by a certified nonexpert crowd achieves high performance for ASD detection from natural home videos of the child at risk and maintains high sensitivity when privacy-preserving mechanisms are applied. These results suggest that privacy-safeguarded crowdsourced analysis of short home videos can help enable rapid and mobile machine-learning detection of developmental delays in children.

Feature replacement methods enable reliable home video analysis for machine learning detection of autism.

Autism Spectrum Disorder is a neuropsychiatric condition affecting 53 million children worldwide and for which early diagnosis is critical to the outcome of behavior therapies. Machine learning applied to features manually extracted from readily accessible videos (e.g., from smartphones) has the potential to scale this diagnostic process. However, nearly unavoidable variability in video quality can lead to missing features that degrade algorithm performance. To manage this uncertainty, we evaluated the impact of missing values and feature imputation methods on two previously published autism detection classifiers, trained on standard-of-care instrument scoresheets and tested on ratings of 140 children videos from YouTube. We compare the baseline method of listwise deletion to classic univariate and multivariate techniques. We also introduce a feature replacement method that, based on a score, selects a feature from an expanded dataset to fill-in the missing value. The replacement feature selected can be identical for all records (general) or automatically adjusted to the record considered (dynamic). Our results show that general and dynamic feature replacement methods achieve a higher performance than classic univariate and multivariate methods, supporting the hypothesis that algorithmic management can maintain the fidelity of video-based diagnostics in the face of missing values and variable video quality.

Making sense of what exactly public health does: a typology of public health interventions.

SETTING: Montréal. INTERVENTION: The lack of common knowledge about what public health does is a hindrance to its recognition and capacity to act. Montréal's regional public health department set an explicit goal to clarify and better communicate its specific contributions when it developed its 2016-2021 action plan. This article briefly describes the efforts made to classify public health practice, introduces a typology of public health interventions and discusses its application and benefits. OUTCOMES: The typology that was developed defines 29 types of interventions grouped into four categories: direct action targeting the population; advocacy (persuading partners to take action); support (helping partners take action); collaboration (taking action with partners). The analysis of Montreal's most recent action plan, completely drafted in terms of the typology, provides an insightful characterization of public health practice. Globally, four out of five interventions target partners (indirect), with more than half falling within the support category. Other indirect interventions are divided almost equally between advocacy and collaboration. Following a rigorous planning process and enforcing the use of the typology also had a significant structuring effect on the organization and its teams and enabled greater synergy with partners from other sectors. IMPLICATIONS: Very few people are familiar with everything public health does, sometimes not even the responsible political decision-makers. This situation poses a threat to the survival of its prevention mission. The typology of public health interventions is an innovative tool that can be used to better inform the public and decision-makers.

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition.

Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers-defined as vetted members of popular crowdsourcing platforms-to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine.

Influence of a Pay-for-Performance Program on Glycemic Control in Patients Living with Diabetes by Family Physicians in a Canadian Province.

OBJECTIVES: We evaluated the influence of the introduction of a pay-for-performance program implemented in 2010 for family physicians on the glycemic control of patients with diabetes. METHODS: Administrative data for all 583 eligible family physicians and 83,580 adult patients with diabetes in New Brunswick over 10 years were used. We compared the probability of receiving at least 2 tests for glycated hemoglobin (A1C) levels and achieving glycemic control before (2005-2009) and after (2010-2014) the implementation of the program and between patients divided based on whether a physician claimed the incentive or did not. RESULTS: Patients living with diabetes showed greater odds of receiving at least 2 A1C tests per year if the detection of their diabetes occurred after (vs. before) the implementation of the program (OR, 99% CI=1.23, 1.18 to 1.28), if a physician claimed the incentive (vs. not claiming it) for their care (1.92, 1.87 to 1.96) in the given year, and if they were followed by a physician who ever (vs. never) claimed the incentive (1.24, 1.15 to 1.34). In a cohort-based analysis, patients for whom an incentive was claimed (vs. not claimed) had greater odds of receiving at least 2 A1C tests per year before implementation of the incentive, and these odds increased by 56% (1.49 to 1.62) following its implementation. However, there was no difference in A1C values among the various comparison groups. CONCLUSIONS: Introduction of the incentive was associated with greater odds of having a minimum of 2 A1C tests per year, which may suggest that it led physicians to provide better follow-up care for patients with diabetes. However, the incentive program has not been associated with differences in glycemic control.

Factors that could explain the increasing prevalence of type 2 diabetes among adults in a Canadian province: a critical review and analysis.

BACKGROUND: The prevalence of diabetes has increased since the last decade in New Brunswick. Identifying factors contributing to the increase in diabetes prevalence will help inform an action plan to manage the condition. The objective was to describe factors that could explain the increasing prevalence of type 2 diabetes in New Brunswick since 2001. METHODS: A critical literature review was conducted to identify factors potentially responsible for an increase in prevalence of diabetes. Data from various sources were obtained to draw a repeated cross-sectional (2001-2014) description of these factors concurrently with changes in the prevalence of type 2 diabetes in New Brunswick. Linear regressions, Poisson regressions and Cochran Armitage analysis were used to describe relationships between these factors and time. RESULTS: Factors identified in the review were summarized in five categories: individual-level risk factors, environmental risk factors, evolution of the disease, detection effect and global changes. The prevalence of type 2 diabetes has increased by 120% between 2001 and 2014. The prevalence of obesity, hypertension, prediabetes, alcohol consumption, immigration and urbanization increased during the study period and the consumption of fruits and vegetables decreased which could represent potential factors of the increasing prevalence of type 2 diabetes. Physical activity, smoking, socioeconomic status and education did not present trends that could explain the increasing prevalence of type 2 diabetes. During the study period, the mortality rate and the conversion rate from prediabetes to diabetes decreased and the incidence rate increased. Suggestion of a detection effect was also present as the number of people tested increased while the HbA1c and the age at detection decreased. Period and birth cohort effect were also noted through a rise in the prevalence of type 2 diabetes across all age groups, but greater increases were observed among the younger cohorts. CONCLUSIONS: This study presents a comprehensive overview of factors potentially responsible for population level changes in prevalence of type 2 diabetes. Recent increases in type 2 diabetes in New Brunswick may be attributable to a combination of some individual-level and environmental risk factors, the detection effect, the evolution of the disease and global changes.

The time course of the contingent spatial blink.

Attentional capture is the unintentional deployment of attention to a task-irrelevant but attentionally salient object. The contingent involuntary orienting hypothesis states that it occurs only if a distractor's property matches current top-down attentional control settings (Folk, Remington, & Johnston, 1992). Folk, Leber, and Egeth (2002) found that monitoring a central RSVP stream for a coloured target led to spatial attentional capture by a peripheral distractor that matched the target colour. Using a similar paradigm, we explored the time course of this spatial blink. Implications of this study for current accounts of the attentional capture phenomenon are discussed.

How do selected arrows guide visuospatial attention? Dissociating symbolic value and spatial proximity.

Previous research on the control of visuospatial attention showed that overlearned symbols like arrows have the potential to induce involuntary shifts of attention. Following work on the role of attentional control settings and of the content of working memory in the involuntary deployment of visuospatial attention, Pratt and Hommel (2003) found that this unintentional orienting by an arrow depended on its top-down selection, contingent on the attentional control settings, that is to say, the target selection cue. However, in this study, each arrow was closer to the location it indicated than to any other location, raising the issue of attention being drawn to the arrow location, facilitating processing at adjacent locations, rather than pushed to the symbolically cued location. In the present study, we dissociated symbolic cueing and spatial proximity cueing by the selected arrow. The results support the proximity cueing hypothesis.

Contingent capture of visual-spatial attention depends on capacity-limited central mechanisms: evidence from human electrophysiology and the psychological refractory period.

It has recently been demonstrated that a lateralized distractor that matches the individual's top-down control settings elicits an N2pc wave, an electrophysiological index of the focus of visual-spatial attention, indicating that contingent capture has a visual-spatial locus. Here, we investigated whether contingent capture required capacity-limited central resources by incorporating a contingent capture task as the second task of a psychological refractory period (PRP) dual-task paradigm. The N2pc was used to monitor where observers were attending while they performed concurrent central processing known to cause the PRP effect. The N2pc elicited by the lateralized distractor that matched the top-down control settings was attenuated in high concurrent central load conditions, indicating that although involuntary, the deployment of visual-spatial attention occurring during contingent capture depends on capacity-limited central resources.

Tracking the location of visuospatial attention in a contingent capture paradigm.

Currently, there is considerable controversy regarding the degree to which top-down control can affect attentional capture by salient events. According to the contingent capture hypothesis, attentional capture by a salient stimulus is contingent on a match between the properties of the stimulus and top-down attentional control settings. In contrast, bottom-up saliency accounts argue that the initial capture of attention is determined solely by the relative salience of the stimulus, and the effect of top-down attentional control is limited to effects on the duration of attentional engagement on the capturing stimulus. In the present study, we tested these competing accounts by utilizing the N2pc event-related potential component to track the locus of attention during an attentional capture task. The results were completely consistent with the contingent capture hypothesis: An N2pc wave was elicited only by distractors that possessed the target-defining attribute. In a second experiment, we expanded upon this finding by exploring the effect of target-distractor similarity on the duration that attention dwells at the distractor location. In this experiment, only distractors possessing the target-defining attribute (color) captured visuospatial attention to their location and the N2pc increased in duration and in magnitude when the capture distractor also shared a second target attribute (category membership). Finally, in three additional control experiments, we replicated the finding of an N2pc generated by distractors, only if they shared the target-defining attribute. Thus, our results demonstrate that attentional control settings influence both which stimuli attract attention and to what extent they are processed.