The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse.

Even when guided by strong theories and sound methods, researchers must often choose a singular course of action from multiple viable alternatives. Regardless of the choice, it, along with all other choices made during the research process, individually and collectively affects study results, often in unpredictable ways. The inability to disentangle how much of an observed effect is attributable to the phenomenon of interest, and how much is attributable to what have come to be known as researcher degrees of freedom (RDF), slows theoretical progress and stymies practical implementation. However, if one could examine the results from a particular set of RDF (known as a universe) against a systematically and comprehensively determined background of alternative viable universes (known as a multiverse), then the effects of RDF can be directly examined to provide greater context and clarity to future researchers, and greater confidence in the recommendations to practitioners. This tutorial demonstrates a means to map result variability directly and efficiently, and empirically investigate RDF impact on conclusions via multiverse analysis. Using the R package multiverse, we outline best practices in planning, executing and interpreting of multiverse analyses.

Evaluating the Use of Uncertainty Visualisations for Imputations of Data Missing At Random in Scatterplots.

Most real-world datasets contain missing values yet most exploratory data analysis (EDA) systems only support visualising data points with complete cases. This omission may potentially lead the user to biased analyses and insights. Imputation techniques can help estimate the value of a missing data point, but introduces additional uncertainty. In this work, we investigate the effects of visualising imputed values in charts using different ways of representing data imputations and imputation uncertainty-no imputation, mean, 95% confidence intervals, probability density plots, gradient intervals, and hypothetical outcome plots. We focus on scatterplots, which is a commonly used chart type, and conduct a crowdsourced study with 202 participants. We measure users' bias and precision in performing two tasks-estimating average and detecting trend-and their self-reported confidence in performing these tasks. Our results suggest that, when estimating averages, uncertainty representations may reduce bias but at the cost of decreasing precision. When estimating trend, only hypothetical outcome plots may lead to a small probability of reducing bias while increasing precision. Participants in every uncertainty representation were less certain about their response when compared to the baseline. The findings point towards potential trade-offs in using uncertainty encodings for datasets with a large number of missing values. This paper and the associated analysis materials are available at: https://osf.io/q4y5r/.

An Automated Approach to Reasoning About Task-Oriented Insights in Responsive Visualization.

Authors often transform a large screen visualization for smaller displays through rescaling, aggregation and other techniques when creating visualizations for both desktop and mobile devices (i.e., responsive visualization). However, transformations can alter relationships or patterns implied by the large screen view, requiring authors to reason carefully about what information to preserve while adjusting their design for the smaller display. We propose an automated approach to approximating the loss of support for task-oriented visualization insights (identification, comparison, and trend) in responsive transformation of a source visualization. We operationalize identification, comparison, and trend loss as objective functions calculated by comparing properties of the rendered source visualization to each realized target (small screen) visualization. To evaluate the utility of our approach, we train machine learning models on human ranked small screen alternative visualizations across a set of source visualizations. We find that our approach achieves an accuracy of 84% (random forest model) in ranking visualizations. We demonstrate this approach in a prototype responsive visualization recommender that enumerates responsive transformations using Answer Set Programming and evaluates the preservation of task-oriented insights using our loss measures. We discuss implications of our approach for the development of automated and semi-automated responsive visualization recommendation.