A framework for quantifying individual and collective common sense.

The notion of common sense is invoked so frequently in contexts as diverse as everyday conversation, political debates, and evaluations of artificial intelligence that its meaning might be surmised to be unproblematic. Surprisingly, however, neither the intrinsic properties of common sense knowledge (what makes a claim commonsensical) nor the degree to which it is shared by people (its "commonness") have been characterized empirically. In this paper, we introduce an analytical framework for quantifying both these elements of common sense. First, we define the commonsensicality of individual claims and people in terms of the latter's propensity to agree on the former and their awareness of one another's agreement. Second, we formalize the commonness of common sense as a clique detection problem on a bipartite belief graph of people and claims, defining [Formula: see text] common sense as the fraction [Formula: see text] of claims shared by a fraction [Formula: see text] of people. Evaluating our framework on a dataset of [Formula: see text] raters evaluating [Formula: see text] diverse claims, we find that commonsensicality aligns most closely with plainly worded, fact-like statements about everyday physical reality. Psychometric attributes such as social perceptiveness influence individual common sense, but surprisingly demographic factors such as age or gender do not. Finally, we find that collective common sense is rare: At most, a small fraction [Formula: see text] of people agree on more than a small fraction [Formula: see text] of claims. Together, these results undercut universalistic beliefs about common sense and raise questions about its variability that are relevant both to human and artificial intelligence.

Replies to commentaries on beyond playing 20 questions with nature.

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as "playing twenty questions with nature," theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm-and with far greater efficiency.

Empirica: a virtual lab for high-throughput macro-level experiments.

Virtual labs allow researchers to design high-throughput and macro-level experiments that are not feasible in traditional in-person physical lab settings. Despite the increasing popularity of online research, researchers still face many technical and logistical barriers when designing and deploying virtual lab experiments. While several platforms exist to facilitate the development of virtual lab experiments, they typically present researchers with a stark trade-off between usability and functionality. We introduce Empirica: a modular virtual lab that offers a solution to the usability-functionality trade-off by employing a "flexible defaults" design strategy. This strategy enables us to maintain complete "build anything" flexibility while offering a development platform that is accessible to novice programmers. Empirica's architecture is designed to allow for parameterizable experimental designs, reusable protocols, and rapid development. These features will increase the accessibility of virtual lab experiments, remove barriers to innovation in experiment design, and enable rapid progress in the understanding of human behavior.

Inducing Vascular Grammars for Anomaly Classification in Brain Angiograms.

As machine learning is used to make strides in medical diagnostics, few methods provide heuristics from which human doctors can learn directly. This work introduces a method for leveraging human observable structures, such as macroscale vascular formations, for producing assessments of medical conditions with relatively few training cases, and uncovering patterns that are potential diagnostic aids. The approach draws on shape grammars, a rule-based technique, pioneered in design and architecture, and accelerated through a recursive subgraph mining algorithm. The distribution of rule instances in the data from which they are induced is then used as an intermediary representation enabling common classification and anomaly detection approaches to identify indicative rules with relatively small data sets. The method is applied to seven-tesla time-of-flight angiography MRI (n = 54) of human brain vasculature. The data were segmented and induced to generate representative grammar rules. Ensembles of rules were isolated to implicate vascular conditions reliably. This application demonstrates the power of automated structured intermediary representations for assessing nuanced biological form relationships, and the strength of shape grammars, in particular for identifying indicative patterns in complex vascular networks.