Impact of a mobile-based (mHealth) tool to support community health nurses in early identification of depression and suicide risk in Pacific Island Countries.

OBJECTIVE: To convert screening tools for depression and suicide risk into algorithmic decision support on smartphones for use by community health nurses (CHNs), and to evaluate the efficiency, effectiveness, and usability of the mHealth tool in providing mental health (MH) care. METHOD: Two scenarios of depression and suicide risk were developed and presented to 48 nurses using paper-based and mobile-based guidelines under laboratory (nonclinical) conditions. Participants read through the case scenarios to provide summaries, diagnoses, and management recommendations. Audiotapes were transcribed and analyzed for accuracy in scoring guidelines, therapy decisions, and time for tasks completion. The validated System Usability Scale (SUS) was used to measure mobile app usability. RESULTS: Using mHealth-based guidelines, nurses took significantly less time to complete their tasks, and generated no errors of addition, as compared to paper-based guidelines. Although coding errors were noted when using the mHealth app, it did not influence treatment recommendations. The system usability scores for both guidelines were over 84%. CONCLUSIONS: Usable mHealth technology can support task-sharing for CHNs in Fiji, for the efficient and accurate screening of patients for depression and suicide risks in a nonclinical setting. Studies on clinical implementation of the mHealth tool are needed and planned.

EHR audit logs: A new goldmine for health services research?

A byproduct of the transition to electronic health records (EHRs) is the associated observational data that capture EHR users' granular interactions with the medical record. Often referred to as audit log data or event log data, these datasets capture and timestamp user activity while they are logged in to the EHR. These data - alone and in combination with other datasets - offer a new source of insights, which cannot be gleaned from claims data or clinical data, to support health services research and those studying healthcare processes and outcomes. In this commentary, we seek to promote broader awareness of EHR audit log data and to stimulate their use in many contexts. We do so by describing EHR audit log data and offering a framework for their potential uses in quality domains (as defined by the National Academy of Medicine). The framework is illustrated with select examples in the safety and efficiency domains, along with their accompanying methodologies, which serve as a proof of concept. This article also discusses insights and challenges from working with EHR audit log data. Ensuring that researchers are aware of such data, and the new opportunities they offer, is one way to assure that our healthcare system benefits from the digital revolution.

Overlaying multiple sources of data to identify bottlenecks in clinical workflow.

The pursuit of increased efficiency and quality of clinical care based on the analysis of workflow has seen the introduction of several modern technologies into medical environments. Electronic health records (EHRs) remain central to analysis of workflow, owing to their wide-ranging impact on clinical processes. The two most common interventions to facilitate EHR-related workflow analysis are automated location tracking using sensor-based technologies and EHR usage data logs. However, to maximize the potential of these technologies, and especially to facilitate workflow redesign, it is necessary to overlay these quantitative findings on the contextual data from qualitative methods such as ethnography. Such a complementary approach promises to yield more precise measures of clinical workflow that provide insights into how redesign could address inefficiencies. In this paper, we categorize clinical workflow in the Emergency Department (ED) into three types (perceived, real and ideal) to create a structured approach to workflow redesign using the available data. We use diverse data sources: sensor-based location tracking through Radio-Frequency Identification (RFID), summary EHR usage data logs, and data from physician interviews augmented by direct observations (through clinician shadowing). Our goal is to discover inefficiencies and bottlenecks that can be addressed to achieve a more ideal workflow state relative to its real and perceived state. We thereby seek to demonstrate a novel data-driven approach toward iterative workflow redesign that generalizes for use in a variety of settings. We also propose types of targeted support or adjustments to offset some of the inefficiencies we noted.

A visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS).

BACKGROUND: Vast volumes of data, coded through hierarchical terminologies (e.g., International Classification of Diseases, Tenth Revision-Clinical Modification [ICD10-CM], Medical Subject Headings [MeSH]), are generated routinely in electronic health record systems and medical literature databases. Although graphic representations can help to augment human understanding of such data sets, a graph with hundreds or thousands of nodes challenges human comprehension. To improve comprehension, new tools are needed to extract the overviews of such data sets. We aim to develop a visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS) as an online, and publicly accessible tool. The ultimate goals are to filter, summarize the health data sets, extract insights, compare and highlight the differences between various health data sets by using VIADS. The results generated from VIADS can be utilized as data-driven evidence to facilitate clinicians, clinical researchers, and health care administrators to make more informed clinical, research, and administrative decisions. We utilized the following tools and the development environments to develop VIADS: Django, Python, JavaScript, Vis.js, Graph.js, JQuery, Plotly, Chart.js, Unittest, R, and MySQL. RESULTS: VIADS was developed successfully and the beta version is accessible publicly. In this paper, we introduce the architecture design, development, and functionalities of VIADS. VIADS includes six modules: user account management module, data sets validation module, data analytic module, data visualization module, terminology module, dashboard. Currently, VIADS supports health data sets coded by ICD-9, ICD-10, and MeSH. We also present the visualization improvement provided by VIADS in regard to interactive features (e.g., zoom in and out, customization of graph layout, expanded information of nodes, 3D plots) and efficient screen space usage. CONCLUSIONS: VIADS meets the design objectives and can be used to filter, summarize, compare, highlight and visualize large health data sets that coded by hierarchical terminologies, such as ICD-9, ICD-10 and MeSH. Our further usability and utility studies will provide more details about how the end users are using VIADS to facilitate their clinical, research or health administrative decision making.

A method for the analysis and visualization of clinical workflow in dynamic environments.

The analysis of clinical workflow offers many challenges, especially in settings characterized by rapid dynamic change. Typically, some combination of approaches drawn from ethnography and grounded theory-based qualitative methods are used to develop relevant metrics. Medical institutions have recently attempted to introduce technological interventions to develop quantifiable quality metrics to supplement existing purely qualitative analyses. These interventions range from automated location tracking to repositories of clinical data (e.g., electronics health record (EHR) data, medical equipment logs). Our goal in this paper is to present a cohesive framework that combines a set of analytic techniques that can potentially complement traditional human observations to derive a deeper understanding of clinical workflow and thereby to enhance the quality, safety, and efficiency of care offered in that environment. We present a series of theoretically-guided techniques to perform analysis and visualization of data developed using location tracking, with illustrations using the Emergency Department (ED) as an example. Our framework is divided into three modules: (i) transformation, (ii) analysis, and (iii) visualization. We describe the methods used in each of these modules, and provide a series of visualizations developed using location-tracking data collected at the Mayo Clinic ED (Phoenix, AZ). Our innovative analytics go beyond qualitative study, and includes user data collected from a relatively modern but increasingly ubiquitous technique of location tracking, with the goal of creating quantitative workflow metrics. Although we believe that the methods we have developed will generalize well to other settings, additional work will be required to demonstrate their broad utility beyond our single study environment.

Contextual Computing: A Bluetooth based approach for tracking healthcare providers in the emergency room.

Hospital Emergency Departments (EDs) frequently experience crowding. One of the factors that contributes to this crowding is the "door to doctor time", which is the time from a patient's registration to when the patient is first seen by a physician. This is also one of the Meaningful Use (MU) performance measures that emergency departments report to the Center for Medicare and Medicaid Services (CMS). Current documentation methods for this measure are inaccurate due to the imprecision in manual data collection. We describe a method for automatically (in real time) and more accurately documenting the door to physician time. Using sensor-based technology, the distance between the physician and the computer is calculated by using the single board computers installed in patient rooms that log each time a Bluetooth signal is seen from a device that the physicians carry. This distance is compared automatically with the accepted room radius to determine if the physicians are present in the room at the time logged to provide greater precision. The logged times, accurate to the second, were compared with physicians' handwritten times, showing automatic recordings to be more precise. This real time automatic method will free the physician from extra cognitive load of manually recording data. This method for evaluation of performance is generic and can be used in any other setting outside the ED, and for purposes other than measuring physician time.

Methodological framework for evaluating clinical processes: A cognitive informatics perspective.

We propose a methodological framework for evaluating clinical cognitive activities in complex real-world environments that provides a guiding framework for characterizing the patterns of activities. This approach, which we refer to as a process-based approach, is particularly relevant to cognitive informatics (CI) research-an interdisciplinary domain utilizing cognitive approaches in the study of computing systems and applications-as it provides new ways for understanding human information processing, interactions, and behaviors. Using this approach involves the identification of a process of interest (e.g., a clinical workflow), and the contributing sequences of activities in that process (e.g., medication ordering). A variety of analytical approaches can then be used to characterize the inherent dependencies and relations within the contributing activities within the considered process. Using examples drawn from our own research and the extant research literature, we describe the theoretical foundations of the process-based approach, relevant practical and pragmatic considerations for using such an approach, and a generic framework for applying this approach for evaluation studies in clinical settings. We also discuss the potential for this approach in future evaluations of interactive clinical systems, given the need for new approaches for evaluation, and significant opportunities for automated, unobtrusive data collection.

Affordable, web-based surgical skill training and evaluation tool.

Advances in the medical field have increased the need to incorporate modern techniques into surgical resident training and surgical skills learning. To facilitate this integration, one approach that has gained credibility is the incorporation of simulator based training to supplement traditional training programs. However, existing implementations of these training methods still require the constant presence of a competent surgeon to assess the surgical dexterity of the trainee, which limits the evaluation methods and relies on subjective evaluation. This research proposes an efficient, effective, and economic video-based skill assessment technique for minimally invasive surgery (MIS). It analyzes a surgeon's hand and surgical tool movements and detects features like smoothness, efficiency, and preciseness. The system is capable of providing both real time on-screen feedback and a performance score at the end of the surgery. Finally, we present a web-based tool where surgeons can securely upload MIS training videos and receive evaluation scores and an analysis of trainees' performance trends over time.

Characterizing the structure and content of nurse handoffs: A Sequential Conversational Analysis approach.

Effective communication during nurse handoffs is instrumental in ensuring safe and quality patient care. Much of the prior research on nurse handoffs has utilized retrospective methods such as interviews, surveys and questionnaires. While extremely useful, an in-depth understanding of the structure and content of conversations, and the inherent relationships within the content is paramount to designing effective nurse handoff interventions. In this paper, we present a methodological framework-Sequential Conversational Analysis (SCA)-a mixed-method approach that integrates qualitative conversational analysis with quantitative sequential pattern analysis. We describe the SCA approach and provide a detailed example as a proof of concept of its use for the analysis of nurse handoff communication in a medical intensive care unit. This novel approach allows us to characterize the conversational structure, clinical content, disruptions in the conversation, and the inherently phasic nature of nurse handoff communication. The characterization of communication patterns highlights the relationships underlying the verbal content of nurse handoffs with specific emphasis on: the interactive nature of conversation, relevance of role-based (incoming, outgoing) communication requirements, clinical content focus on critical patient-related events, and discussion of pending patient management tasks. We also discuss the applicability of the SCA approach as a method for providing in-depth understanding of the dynamics of communication in other settings and domains.

Cognitive informatics in biomedicine and healthcare.

Cognitive Informatics (CI) is a burgeoning interdisciplinary domain comprising of the cognitive and information sciences that focuses on human information processing, mechanisms and processes within the context of computing and computer applications. Based on a review of articles published in the Journal of Biomedical Informatics (JBI) between January 2001 and March 2014, we identified 57 articles that focused on topics related to cognitive informatics. We found that while the acceptance of CI into the mainstream informatics research literature is relatively recent, its impact has been significant - from characterizing the limits of clinician problem-solving and reasoning behavior, to describing coordination and communication patterns of distributed clinical teams, to developing sustainable and cognitively-plausible interventions for supporting clinician activities. Additionally, we found that most research contributions fell under the topics of decision-making, usability and distributed team activities with a focus on studying behavioral and cognitive aspects of clinical personnel, as they performed their activities or interacted with health information systems. We summarize our findings within the context of the current areas of CI research, future research directions and current and future challenges for CI researchers.

Error recovery by dialysis technicians.

Experts are believed to make fewer errors than novices. Researchers in other domains have shown that experts not only make less errors, they also detect and recover from these errors better than non-experts. To investigate this phenomenon among dialysis technicians working in hemodialysis, we evaluated the ability of dialysis technicians to detect and recover from healthcare errors. Two clinical cases with embedded errors were created by an expert nephrology nurse. Twenty-four dialysis technician subjects read the cases aloud and then answered a set of related questions. Subjects' error detection and recovery responses were scored against the clinical cases. We found that there was no significant difference between the ability of expert and non-expert dialysis technicians to detect errors. However, expert dialysis technicians recovered from significantly more healthcare errors than less experienced non-expert dialysis technicians. This has implications for training dialysis technicians in better error detection and recovery strategies.

Evaluation of a Patient Decision Aid for Refractive Eye Surgery.

BACKGROUND: We developed a prototype patient decision aid, EyeChoose, to assist college-aged students in selecting a refractive surgery. EyeChoose can educate patients on refractive errors and surgeries, generate evidence-based recommendations based on a user's medical history and personal preferences, and refer patients to local refractive surgeons. OBJECTIVES: We conducted an evaluative study on EyeChoose to assess the alignment of surgical modality recommendations with a user's medical history and personal preferences, and to examine the tool's usefulness and usability. METHODS: We designed a mixed methods study on EyeChoose through simulations of test cases to provide a quantitative measure of the customized recommendations, an online survey to evaluate the usefulness and usability, and a focus group interview to obtain an in-depth understanding of user experience and feedback. RESULTS: We used stratified random sampling to generate 245 test cases. Simulated execution indicated EyeChoose's recommendations aligned with the reference standard in 243 (99%). A survey of 55 participants with 16 questions on usefulness, usability, and general impression showed that 14 questions recorded more than 80% positive responses. A follow-up focus group with 10 participants confirmed EyeChoose's useful features of patient education, decision assistance, surgeon referral, as well as good usability with multimedia resources, visual comparison among the surgical modalities, and the overall aesthetically pleasing design. Potential areas for improvement included offering nuances in soliciting user preferences, providing additional details on pricing, effectiveness, and reversibility of surgeries, expanding the function of surgeon referral, and fixing specific usability issues. CONCLUSION: The initial evaluation of EyeChoose suggests that it could provide effective patient education, generate appropriate recommendations, connect to local refractive surgeons, and demonstrate good system usability in a test environment. Future research is required to enhance the system functions, fully implement and evaluate the tool in naturalistic settings, and examine the findings' generalizability to other populations.

Data-Driven Hypothesis Generation in Clinical Research: What We Learned from a Human Subject Study?

Hypothesis generation is an early and critical step in any hypothesis-driven clinical research project. Because it is not yet a well-understood cognitive process, the need to improve the process goes unrecognized. Without an impactful hypothesis, the significance of any research project can be questionable, regardless of the rigor or diligence applied in other steps of the study, e.g., study design, data collection, and result analysis. In this perspective article, the authors provide a literature review on the following topics first: scientific thinking, reasoning, medical reasoning, literature-based discovery, and a field study to explore scientific thinking and discovery. Over the years, scientific thinking has shown excellent progress in cognitive science and its applied areas: education, medicine, and biomedical research. However, a review of the literature reveals the lack of original studies on hypothesis generation in clinical research. The authors then summarize their first human participant study exploring data-driven hypothesis generation by clinical researchers in a simulated setting. The results indicate that a secondary data analytical tool, VIADS-a visual interactive analytic tool for filtering, summarizing, and visualizing large health data sets coded with hierarchical terminologies, can shorten the time participants need, on average, to generate a hypothesis and also requires fewer cognitive events to generate each hypothesis. As a counterpoint, this exploration also indicates that the quality ratings of the hypotheses thus generated carry significantly lower ratings for feasibility when applying VIADS. Despite its small scale, the study confirmed the feasibility of conducting a human participant study directly to explore the hypothesis generation process in clinical research. This study provides supporting evidence to conduct a larger-scale study with a specifically designed tool to facilitate the hypothesis-generation process among inexperienced clinical researchers. A larger study could provide generalizable evidence, which in turn can potentially improve clinical research productivity and overall clinical research enterprise.

Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools.

Objectives: To compare how clinical researchers generate data-driven hypotheses with a visual interactive analytic tool (VIADS, a visual interactive analysis tool for filtering and summarizing large datasets coded with hierarchical terminologies) or other tools. Methods: We recruited clinical researchers and separated them into "experienced" and "inexperienced" groups. Participants were randomly assigned to a VIADS or control group within the groups. Each participant conducted a remote 2-hour study session for hypothesis generation with the same study facilitator on the same datasets by following a think-aloud protocol. Screen activities and audio were recorded, transcribed, coded, and analyzed. Hypotheses were evaluated by seven experts on their validity, significance, and feasibility. We conducted multilevel random effect modeling for statistical tests. Results: Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid. The VIADS and control groups generated a similar number of hypotheses. The VIADS group took a significantly shorter time to generate one hypothesis (e.g., among inexperienced clinical researchers, 258 s versus 379 s, p = 0.046, power = 0.437, ICC = 0.15). The VIADS group received significantly lower ratings than the control group on feasibility and the combination rating of validity, significance, and feasibility. Conclusion: The role of VIADS in hypothesis generation seems inconclusive. The VIADS group took a significantly shorter time to generate each hypothesis. However, the combined validity, significance, and feasibility ratings of their hypotheses were significantly lower. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development.

The quality of data-driven hypotheses generated by inexperienced clinical researchers: A case study.

Objectives: We invited inexperienced clinical researchers to analyze coded health datasets and develop hypotheses. We recorded and analyzed their hypothesis generation process. All the hypotheses generated in the process were rated by the same group of seven experts by using the same metrics. This case study examines the higher quality (i.e., higher ratings) and lower quality of hypotheses and participants who generated them. We characterized the contextual factors associated with the quality of hypotheses. Methods: All participants (i.e., clinical researchers) completed a 2-hour study session to analyze data and generate scientific hypotheses using the think-aloud method. Participants' screen activity and audio were recorded and transcribed. These transcriptions were used to measure the time used to generate each hypothesis and to code cognitive events (i.e., cognitive activities used when generating hypotheses, for example, "Seeking for Connection" describes an attempt to draw connections between data points). The hypothesis ratings by the expert panel were used as the quality of the hypotheses during the analysis. We analyzed the factors associated with (1) the five highest and (2) five lowest rated hypotheses and (3) the participants who generated them, including the number of hypotheses per participant, the validity of those hypotheses, the number of cognitive events used for each hypothesis, as well as the participant's research experience and basic demographics. Results: Participants who generated the five highest-rated hypotheses used similar lengths of time (difference 3:03), whereas those who generated the five lowest-rated hypotheses used more varying lengths of time (difference 7:13). Participants who generated the five highest-rated hypotheses also utilized slightly fewer cognitive events on average compared to the five lowest-rated hypotheses (4 per hypothesis vs. 4.8 per hypothesis). When we examine the participants (who generated the five highest and five lowest hypotheses) and their total hypotheses generated during the 2-hour study sessions, the participants with the five highest-rated hypotheses again had a shorter range of time per hypothesis on average (0:03:34 vs. 0:07:17). They (with the five highest ratings) used fewer cognitive events per hypothesis (3.498 vs. 4.626). They (with the five highest ratings) also had a higher percentage of valid rate (75.51% vs. 63.63%) and generally had more experience with clinical research. Conclusion: The quality of the hypotheses was shown to be associated with the time taken to generate them, where too long or too short time to generate hypotheses appears to be negatively associated with the hypotheses' quality ratings. Also, having more experience seems to positively correlate with higher ratings of hypotheses and higher valid rates. Validity is a quality dimension used by the expert panel during rating. However, we acknowledge that our results are anecdotal. The effect may not be simply linear, and future research is necessary. These results underscore the multi-factor nature of hypothesis generation.

How team clinical reasoning occurs on ward rounds: Implications for learning.

BACKGROUND: Ward rounds offer a rich environment for learning about team clinical reasoning. We aimed to assess how team clinical reasoning occurs on ward rounds to inform efforts to enhance the teaching of clinical reasoning. METHODS: We performed focused ethnography of ward rounds over a 6-week period, during which we observed five different teams. Each day team comprised one senior physician, one senior resident, one junior resident, two interns and one medical student. Twelve 'night-float' residents who discussed new patients with the day team were also included. Field notes were analysed using content analysis. FINDINGS: We analysed 41 new patient presentations and discussions on 23 different ward rounds. The median duration of case presentations and discussions was 13.0 minutes (IQR, 10.0-18.0 minutes). More time was devoted to information sharing (median 5.5 minutes; IQR, 4.0-7.0 minutes) than any other activity, followed by discussion of management plans (median 4.0 minutes; IQR, 3.0-7.8 minutes). Nineteen (46%) cases did not include discussion of a differential diagnosis for the chief concern. We identified two themes relevant to learning: (1) linear versus iterative approaches to team-based diagnosis and (2) the influence of hierarchy on participation in clinical reasoning discussions. CONCLUSION: The ward teams we observed spent far less time discussing differential diagnoses compared with information sharing. Junior learners such as medical students and interns contributed less frequently to team clinical reasoning discussions. In order to maximise student learning, strategies to engage junior learners in team clinical reasoning discussions on ward rounds may be needed.

Development, validation, and usage of metrics to evaluate the quality of clinical research hypotheses.

Objectives: Metrics and instruments can provide guidance for clinical researchers to assess their potential research projects at an early stage before significant investment. Furthermore, metrics can also provide structured criteria for peer reviewers to assess others' clinical research manuscripts or grant proposals. This study aimed to develop, test, validate, and use evaluation metrics and instruments to accurately, consistently, and conveniently assess the quality of scientific hypotheses for clinical research projects. Materials and Methods: Metrics development went through iterative stages, including literature review, metrics and instrument development, internal and external testing and validation, and continuous revisions in each stage based on feedback. Furthermore, two experiments were conducted to determine brief and comprehensive versions of the instrument. Results: The brief version of the instrument contained three dimensions: validity, significance, and feasibility. The comprehensive version of metrics included novelty, clinical relevance, potential benefits and risks, ethicality, testability, clarity, interestingness, and the three dimensions of the brief version. Each evaluation dimension included 2 to 5 subitems to evaluate the specific aspects of each dimension. For example, validity included clinical validity and scientific validity. The brief and comprehensive versions of the instruments included 12 and 39 subitems, respectively. Each subitem used a 5-point Likert scale. Conclusion: The validated brief and comprehensive versions of metrics can provide standardized, consistent, and generic measurements for clinical research hypotheses, allow clinical researchers to prioritize their research ideas systematically, objectively, and consistently, and can be used as a tool for quality assessment during the peer review process.

A Visual Analytic Tool (VIADS) to Assist the Hypothesis Generation Process in Clinical Research: Mixed Methods Usability Study.

BACKGROUND: Visualization can be a powerful tool to comprehend data sets, especially when they can be represented via hierarchical structures. Enhanced comprehension can facilitate the development of scientific hypotheses. However, the inclusion of excessive data can make visualizations overwhelming. OBJECTIVE: We developed a visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS). In this study, we evaluated the usability of VIADS for visualizing data sets of patient diagnoses and procedures coded in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). METHODS: We used mixed methods in the study. A group of 12 clinical researchers participated in the generation of data-driven hypotheses using the same data sets and time frame (a 1-hour training session and a 2-hour study session) utilizing VIADS via the think-aloud protocol. The audio and screen activities were recorded remotely. A modified version of the System Usability Scale (SUS) survey and a brief survey with open-ended questions were administered after the study to assess the usability of VIADS and verify their intense usage experience with VIADS. RESULTS: The range of SUS scores was 37.5 to 87.5. The mean SUS score for VIADS was 71.88 (out of a possible 100, SD 14.62), and the median SUS was 75. The participants unanimously agreed that VIADS offers new perspectives on data sets (12/12, 100%), while 75% (8/12) agreed that VIADS facilitates understanding, presentation, and interpretation of underlying data sets. The comments on the utility of VIADS were positive and aligned well with the design objectives of VIADS. The answers to the open-ended questions in the modified SUS provided specific suggestions regarding potential improvements for VIADS, and the identified problems with usability were used to update the tool. CONCLUSIONS: This usability study demonstrates that VIADS is a usable tool for analyzing secondary data sets with good average usability, good SUS score, and favorable utility. Currently, VIADS accepts data sets with hierarchical codes and their corresponding frequencies. Consequently, only specific types of use cases are supported by the analytical results. Participants agreed, however, that VIADS provides new perspectives on data sets and is relatively easy to use. The VIADS functionalities most appreciated by participants were the ability to filter, summarize, compare, and visualize data. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/39414.

How do clinical researchers generate data-driven scientific hypotheses? Cognitive events using think-aloud protocol.

Objectives: This study aims to identify the cognitive events related to information use (e.g., "Analyze data", "Seek connection") during hypothesis generation among clinical researchers. Specifically, we describe hypothesis generation using cognitive event counts and compare them between groups. Methods: The participants used the same datasets, followed the same scripts, used VIADS (a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies) or other analytical tools (as control) to analyze the datasets, and came up with hypotheses while following the think-aloud protocol. Their screen activities and audio were recorded and then transcribed and coded for cognitive events. Results: The VIADS group exhibited the lowest mean number of cognitive events per hypothesis and the smallest standard deviation. The experienced clinical researchers had approximately 10% more valid hypotheses than the inexperienced group. The VIADS users among the inexperienced clinical researchers exhibit a similar trend as the experienced clinical researchers in terms of the number of cognitive events and their respective percentages out of all the cognitive events. The highest percentages of cognitive events in hypothesis generation were "Using analysis results" (30%) and "Seeking connections" (23%). Conclusion: VIADS helped inexperienced clinical researchers use fewer cognitive events to generate hypotheses than the control group. This suggests that VIADS may guide participants to be more structured during hypothesis generation compared with the control group. The results provide evidence to explain the shorter average time needed by the VIADS group in generating each hypothesis.