Elicitation and aggregation of multimodal estimates improve wisdom of crowd effects on ordering tasks.

We present a wisdom of crowds study where participants are asked to order a small set of images based on the number of dots they contain and then to guess the respective number of dots in each image. We test two input elicitation interfaces-one elicits the two modalities of estimates jointly and the other independently. We show that the latter interface yields higher quality estimates, even though the multimodal estimates tend to be more self-contradictory. The inputs are aggregated via optimization and voting-rule based methods to estimate the true ordering of a larger universal set of images. We demonstrate that the quality of collective estimates from the simpler yet more computationally-efficient voting methods is comparable to that achieved by the more complex optimization model. Lastly, we find that using multiple modalities of estimates from one group yields better collective estimates compared to mixing numerical estimates from one group with the ordinal estimates from a different group.

Measuring trust: a text analysis approach to compare, contrast, and select trust questionnaires.

Introduction: Trust has emerged as a prevalent construct to describe relationships between people and between people and technology in myriad domains. Across disciplines, researchers have relied on many different questionnaires to measure trust. The degree to which these questionnaires differ has not been systematically explored. In this paper, we use a word-embedding text analysis technique to identify the differences and common themes across the most used trust questionnaires and provide guidelines for questionnaire selection. Methods: A review was conducted to identify the existing trust questionnaires. In total, we included 46 trust questionnaires from three main domains (i.e., Automation, Humans, and E-commerce) with a total of 626 items measuring different trust layers (i.e., Dispositional, Learned, and Situational). Next, we encoded the words within each questionnaire using GloVe word embeddings and computed the embedding for each questionnaire item, and for each questionnaire. We reduced the dimensionality of the resulting dataset using UMAP to visualize these embeddings in scatterplots and implemented the visualization in a web app for interactive exploration of the questionnaires (https://areen.shinyapps.io/Trust_explorer/). Results: At the word level, the semantic space serves to produce a lexicon of trust-related words. At the item and questionnaire level, the analysis provided recommendation on questionnaire selection based on the dispersion of questionnaires' items and at the domain and layer composition of each questionnaire. Along with the web app, the results help explore the semantic space of trust questionnaires and guide the questionnaire selection process. Discussion: The results provide a novel means to compare and select trust questionnaires and to glean insights about trust from spoken dialog or written comments.

Evaluating the Impact of Uncertainty Visualization on Model Reliance.

Machine learning models have gained traction as decision support tools for tasks that require processing copious amounts of data. However, to achieve the primary benefits of automating this part of decision-making, people must be able to trust the machine learning model's outputs. In order to enhance people's trust and promote appropriate reliance on the model, visualization techniques such as interactive model steering, performance analysis, model comparison, and uncertainty visualization have been proposed. In this study, we tested the effects of two uncertainty visualization techniques in a college admissions forecasting task, under two task difficulty levels, using Amazon's Mechanical Turk platform. Results show that (1) people's reliance on the model depends on the task difficulty and level of machine uncertainty and (2) ordinal forms of expressing model uncertainty are more likely to calibrate model usage behavior. These outcomes emphasize that reliance on decision support tools can depend on the cognitive accessibility of the visualization technique and perceptions of model performance and task difficulty.

Trusting Automation: Designing for Responsivity and Resilience.

OBJECTIVE: This paper reviews recent articles related to human trust in automation to guide research and design for increasingly capable automation in complex work environments. BACKGROUND: Two recent trends-the development of increasingly capable automation and the flattening of organizational hierarchies-suggest a reframing of trust in automation is needed. METHOD: Many publications related to human trust and human-automation interaction were integrated in this narrative literature review. RESULTS: Much research has focused on calibrating human trust to promote appropriate reliance on automation. This approach neglects relational aspects of increasingly capable automation and system-level outcomes, such as cooperation and resilience. To address these limitations, we adopt a relational framing of trust based on the decision situation, semiotics, interaction sequence, and strategy. This relational framework stresses that the goal is not to maximize trust, or to even calibrate trust, but to support a process of trusting through automation responsivity. CONCLUSION: This framing clarifies why future work on trust in automation should consider not just individual characteristics and how automation influences people, but also how people can influence automation and how interdependent interactions affect trusting automation. In these new technological and organizational contexts that shift human operators to co-operators of automation, automation responsivity and the ability to resolve conflicting goals may be more relevant than reliability and reliance for advancing system design. APPLICATION: A conceptual model comprising four concepts-situation, semiotics, strategy, and sequence-can guide future trust research and design for automation responsivity and more resilient human-automation systems.

A review of mathematical models of human trust in automation.

Understanding how people trust autonomous systems is crucial to achieving better performance and safety in human-autonomy teaming. Trust in automation is a rich and complex process that has given rise to numerous measures and approaches aimed at comprehending and examining it. Although researchers have been developing models for understanding the dynamics of trust in automation for several decades, these models are primarily conceptual and often involve components that are difficult to measure. Mathematical models have emerged as powerful tools for gaining insightful knowledge about the dynamic processes of trust in automation. This paper provides an overview of various mathematical modeling approaches, their limitations, feasibility, and generalizability for trust dynamics in human-automation interaction contexts. Furthermore, this study proposes a novel and dynamic approach to model trust in automation, emphasizing the importance of incorporating different timescales into measurable components. Due to the complex nature of trust in automation, it is also suggested to combine machine learning and dynamic modeling approaches, as well as incorporating physiological data.

Bridging psychology and engineering to make technology work for people.

Engineering grand challenges and big ideas not only demand innovative engineering solutions, but also typically involve and affect human thought, behavior, and quality of life. To solve these types of complex problems, multidisciplinary teams must bring together experts in engineering and psychological science, yet fusing these distinct areas can be difficult. This article describes how Human Systems Engineering (HSE) researchers have confronted such challenges at the interface of humans and technological systems. Two narrative cases are reported-computer game-based cognitive assessments and medical device reprocessing-and lessons learned are shared. The article then discusses 2 strategies currently being explored to enact such lessons and enhance these kinds of multidisciplinary engineering teams: a "top-down" administrative approach that supports team formation and productivity through a university research center, and a "bottom-up" engineering education approach that prepares students to work at the intersection of psychology and engineering. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Cooperation in Human-Agent Systems to Support Resilience: A Microworld Experiment.

OBJECTIVE: This study uses a dyadic approach to understand human-agent cooperation and system resilience. BACKGROUND: Increasingly capable technology fundamentally changes human-machine relationships. Rather than reliance on or compliance with more or less reliable automation, we investigate interaction strategies with more or less cooperative agents. METHOD: A joint-task microworld scenario was developed to explore the effects of agent cooperation on participant cooperation and system resilience. To assess the effects of agent cooperation on participant cooperation, 36 people coordinated with a more or less cooperative agent by requesting resources and responding to requests for resources in a dynamic task environment. Another 36 people were recruited to assess effects following a perturbation in their own hospital. RESULTS: Experiment 1 shows people reciprocated the cooperative behaviors of the agents; a low-cooperation agent led to less effective interactions and less resource sharing, whereas a high-cooperation agent led to more effective interactions and greater resource sharing. Experiment 2 shows that an initial fast-tempo perturbation undermined proactive cooperation-people tended to not request resources. However, the initial fast tempo had little effect on reactive cooperation-people tended to accept resource requests according to cooperation level. CONCLUSION: This study complements the supervisory control perspective of human-automation interaction by considering interdependence and cooperation rather than the more common focus on reliability and reliance. APPLICATION: The cooperativeness of automated agents can influence the cooperativeness of human agents. Design and evaluation for resilience in teams involving increasingly autonomous agents should consider the cooperative behaviors of these agents.

Quantitative ethnographic study of physician workflow and interactions with electronic health record systems.

This study explores the relationship between primary care physicians' interactions with health information technology and primary care workflow. Clinical encounters were recorded with high-resolution video cameras to capture physicians' workflow and interaction with two objects of interest, the electronic health record (EHR) system, and their patient. To analyze the data, a coding scheme was developed based on a validated list of primary care tasks to define the presence or absence of a task, the time spent on each task, and the sequence of tasks. Results revealed divergent workflows and significant differences between physicians' EHR use surrounding common workflow tasks: gathering information, documenting information, and recommend/discuss treatment options. These differences suggest impacts of EHR use on primary care workflow, and capture types of workflows that can be used to inform future studies with larger sample sizes for more effective designs of EHR systems in primary care clinics. Future research on this topic and design strategies for effective health information technology in primary care are discussed.

Organizational and technological correlates of nurses' trust in a smart intravenous pump.

The aim of this study was to understand technology and system characteristics that contribute to nurses' ratings of trust in a smart intravenous pump. Nurses' trust in new technologies can influence how technologies are used. Trust in technology is defined as a person's belief that a technology will not fail them. Potential outcomes of trust in technology are appropriate trust, overtrust, distrust, and mistrust. Trust in technology is also related to several use-specific outcomes, including appropriate use and inappropriate use such as overreliance, disuse or rejection, or misuse. Understanding trust in relation to outcomes can contribute to designs that facilitate appropriate trust in new technologies. A survey was completed by 391 nurses a year after the implementation of a new smart intravenous pump. The survey assessed trust in the intravenous pump and other elements of the sociotechnical system, individual characteristics, technology characteristics, and organizational characteristics. Results show that perceptions of usefulness, safety, ease of use, and usability are related to ratings of trust in smart intravenous pumps. Other work systemfactors such as perception of work environment, age, experience, quality of work, and perception of work performance are also related to ratings of trust. Nurses' trust in smart intravenous pumps is influenced by both characteristics of the technology and the sociotechnical system. Findings from this research have implications for the design of future smart intravenous pumps and health systems. Recommendations for appropriately trustworthy smart intravenous pumps are discussed. Findings also have implications for how trust in health technologies can be measured and conceptualized in complex sociotechnical systems.

Identifying design considerations for a shared decision aid for use at the point of outpatient clinical care: An ethnographic study at an inner city clinic.

BACKGROUND AND OBJECTIVE: Computerized decision aids could facilitate shared decision-making at the point of outpatient clinical care. The objective of this study was to investigate whether a computerized shared decision aid would be feasible to implement in an inner-city clinic by evaluating the current practices in shared decision-making, clinicians' use of computers, patient and clinicians' attitudes and beliefs toward computerized decision aids, and the influence of time on shared decision-making. METHODS: Qualitative data analysis of observations and semi-structured interviews with patients and clinicians at an inner-city outpatient clinic. FINDINGS: The findings provided an exploratory look at the prevalence of shared decision-making and attitudes about health information technology and decision aids. A prominent barrier to clinicians engaging in shared decision-making was a lack of perceived patient understanding of medical information. Some patients preferred their clinicians make recommendations for them rather than engage in formal shared decision-making. Health information technology was an integral part of the clinic visit and welcomed by most clinicians and patients. Some patients expressed the desire to engage with health information technology such as viewing their medical information on the computer screen with their clinicians. All participants were receptive to the idea of a decision aid integrated within the clinic visit although some clinicians were concerned about the accuracy of prognostic estimates for complex medical problems. IMPLICATIONS: We identified several important considerations for the design and implementation of a computerized decision aid including opportunities to: bridge clinician-patient communication about medical information while taking into account individual patients' decision-making preferences, complement expert clinician judgment with prognostic estimates, take advantage of patient waiting times, and make tasks involved during the clinic visit more efficient. These findings should be incorporated into the design and implementation of a computerized shared decision aid at an inner-city hospital.