lab.js: A free, open, online study builder.

Web-based data collection is increasingly popular in both experimental and survey-based research because it is flexible, efficient, and location-independent. While dedicated software for laboratory-based experimentation and online surveys is commonplace, researchers looking to implement experiments in the browser have, heretofore, often had to manually construct their studies' content and logic using code. We introduce lab.js, a free, open-source experiment builder that makes it easy to build studies for both online and in-laboratory data collection. Through its visual interface, stimuli can be designed and combined into a study without programming, though studies' appearance and behavior can be fully customized using HTML, CSS, and JavaScript code if required. Presentation and response times are kept and measured with high accuracy and precision heretofore unmatched in browser-based studies. Experiments constructed with lab.js can be run directly on a local computer and published online with ease, with direct deployment to cloud hosting, export to web servers, and integration with popular data collection platforms. Studies can also be shared in an editable format, archived, re-used and adapted, enabling effortless, transparent replications, and thus facilitating open, cumulative science. The software is provided free of charge under an open-source license; further information, code, and extensive documentation are available from https://lab.js.org/ .

Estimating Reliability of Within-Person Couplings in a Multilevel Framework.

Within-person couplings play a prominent role in psychological research and previous studies have shown that interindividual differences in within-person couplings predict future behavior. For example, stress reactivity-operationalized as the within-person coupling of stress and positive or negative affect-is an important predictor of various (mental) health outcomes and has often been assumed to be a more or less stable personality trait. However, issues of reliability of these couplings have been largely neglected so far. In this work, we present an estimate for the reliability of within-person couplings that can be easily obtained using the user-modifiable R code accompanying this work. Results of a simulation study show that this index performs well even in the context of unbalanced data due to missing values. We demonstrate the application of this index in a measurement burst study targeting the reliability and test-retest correlation of stress reactivity estimates operationalized as within-person couplings in a daily diary design. Reliability and test-retest correlations of stress reactivity estimates were rather low, challenging the implicit assumption of stress reactivity as a stable person-level variable. We highlight key factors that researchers planning studies targeting interindividual differences in within-person couplings should consider to maximize reliability.

Amortized Bayesian Model Comparison With Evidential Deep Learning.

Comparing competing mathematical models of complex processes is a shared goal among many branches of science. The Bayesian probabilistic framework offers a principled way to perform model comparison and extract useful metrics for guiding decisions. However, many interesting models are intractable with standard Bayesian methods, as they lack a closed-form likelihood function or the likelihood is computationally too expensive to evaluate. In this work, we propose a novel method for performing Bayesian model comparison using specialized deep learning architectures. Our method is purely simulation-based and circumvents the step of explicitly fitting all alternative models under consideration to each observed dataset. Moreover, it requires no hand-crafted summary statistics of the data and is designed to amortize the cost of simulation over multiple models, datasets, and dataset sizes. This makes the method especially effective in scenarios where model fit needs to be assessed for a large number of datasets, so that case-based inference is practically infeasible. Finally, we propose a novel way to measure epistemic uncertainty in model comparison problems. We demonstrate the utility of our method on toy examples and simulated data from nontrivial models from cognitive science and single-cell neuroscience. We show that our method achieves excellent results in terms of accuracy, calibration, and efficiency across the examples considered in this work. We argue that our framework can enhance and enrich model-based analysis and inference in many fields dealing with computational models of natural processes. We further argue that the proposed measure of epistemic uncertainty provides a unique proxy to quantify absolute evidence even in a framework which assumes that the true data-generating model is within a finite set of candidate models.

BayesFlow: Learning Complex Stochastic Models With Invertible Neural Networks.

Estimating the parameters of mathematical models is a common problem in almost all branches of science. However, this problem can prove notably difficult when processes and model descriptions become increasingly complex and an explicit likelihood function is not available. With this work, we propose a novel method for globally amortized Bayesian inference based on invertible neural networks that we call BayesFlow. The method uses simulations to learn a global estimator for the probabilistic mapping from observed data to underlying model parameters. A neural network pretrained in this way can then, without additional training or optimization, infer full posteriors on arbitrarily many real data sets involving the same model family. In addition, our method incorporates a summary network trained to embed the observed data into maximally informative summary statistics. Learning summary statistics from data makes the method applicable to modeling scenarios where standard inference techniques with handcrafted summary statistics fail. We demonstrate the utility of BayesFlow on challenging intractable models from population dynamics, epidemiology, cognitive science, and ecology. We argue that BayesFlow provides a general framework for building amortized Bayesian parameter estimation machines for any forward model from which data can be simulated.

On the difficulty to think in ratios: a methodological bias in Stevens' magnitude estimation procedure.

In the field of new psychophysics, the magnitude estimation procedure is one of the most frequently used methods. It requires participants to assess the intensity of a stimulus in relation to a reference. In three studies, we examined whether difficulties of thinking in ratios influence participants' intensity perceptions. In Study 1, a standard magnitude estimation procedure was compared to an adapted procedure in which the numerical response dimension was reversed so that smaller (larger) numbers indicated brighter (darker) stimuli. In Study 2, participants first had to indicate whether a stimulus was brighter or darker compared to the reference, and only afterwards they estimated the magnitude of this difference, always using ratings above the reference to indicate their perception. In Study 3, we applied the same procedure as in Study 2 to a different physical dimension (red saturation). Results from Study 1 (N = 20) showed that participants in the reversal condition used more (less) extreme ratings for brighter (darker) stimuli compared to the standard condition. Data from the unidirectional method applied in Study 2 (N = 34) suggested a linear psychophysical function for brightness perception. Similar results were found for red saturation in Study 3 (N = 36) with a less curved power function describing the association between objective red saturation and perceived redness perception. We conclude that the typical power functions that emerge when using a standard magnitude estimation procedure might be biased due to difficulties experienced by participants to think in ratios.

Towards end-to-end likelihood-free inference with convolutional neural networks.

Complex simulator-based models with non-standard sampling distributions require sophisticated design choices for reliable approximate parameter inference. We introduce a fast, end-to-end approach for approximate Bayesian computation (ABC) based on fully convolutional neural networks. The method enables users of ABC to derive simultaneously the posterior mean and variance of multidimensional posterior distributions directly from raw simulated data. Once trained on simulated data, the convolutional neural network is able to map real data samples of variable size to the first two posterior moments of the relevant parameter's distributions. Thus, in contrast to other machine learning approaches to ABC, our approach allows us to generate reusable models that can be applied by different researchers employing the same model. We verify the utility of our method on two common statistical models (i.e., a multivariate normal distribution and a multiple regression scenario), for which the posterior parameter distributions can be derived analytically. We then apply our method to recover the parameters of the leaky competing accumulator (LCA) model and we reference our results to the current state-of-the-art technique, which is the probability density estimation (PDA). Results show that our method exhibits a lower approximation error compared with other machine learning approaches to ABC. It also performs similarly to PDA in recovering the parameters of the LCA model.