More, better feedback please: are learning analytics dashboards (LAD) the solution to a wicked problem?

There is a long-standing lack of learner satisfaction with quality and quantity of feedback in health professions education (HPE) and training. To address this, university and training programmes are increasingly using technological advancements and data analytic tools to provide feedback. One such educational technology is the Learning Analytic Dashboard (LAD), which holds the promise of a comprehensive view of student performance via partial or fully automated feedback delivered to learners in real time. The possibility of displaying performance data visually, on a single platform, so users can access and process feedback efficiently and constantly, and use this to improve their performance, is very attractive to users, educators and institutions. However, the mainstream literature tends to take an atheoretical and instrumentalist view of LADs, a view that uncritically celebrates the promise of LAD's capacity to provide a 'technical fix' to the 'wicked problem' of feedback in health professions education. This paper seeks to recast the discussion of LADs as something other than a benign material technology using the lenses of Miller and Rose's technologies of government and Barry's theory of Technological Societies, where such technical devices are also inherently agentic and political. An examination of the purpose, design and deployment of LADs from these theoretical perspectives can reveal how these educational devices shape and govern the HPE learner body in different ways, which in turn, may produce a myriad of unintended- and ironic- effects on the feedback process. In this Reflections article we wish to encourage health professions education scholars to examine the practices and consequences thereof of the ever-expanding use of LADs more deeply and with a sense of urgency.

Early development of local data dashboards to depict the substance use care cascade for youth involved in the legal system: qualitative findings from end users.

INTRODUCTION: Rates of substance use are high among youth involved in the legal system (YILS); however, YILS are less likely to initiate and complete substance use treatment compared to their non legally-involved peers. There are multiple steps involved in connecting youth to needed services, from screening and referral within the juvenile legal system to treatment initiation and completion within the behavioral health system. Understanding potential gaps in the care continuum requires data and decision-making from these two systems. The current study reports on the development of data dashboards that integrate these systems' data to help guide decisions to improve substance use screening and treatment for YILS, focusing on end-user feedback regarding dashboard utility. METHODS: Three focus groups were conducted with n = 21 end-users from juvenile legal systems and community mental health centers in front-line positions and in decision-making roles across 8 counties to gather feedback on an early version of the data dashboards; dashboards were then modified based on feedback. RESULTS: Qualitative analysis revealed topics related to (1) important aesthetic features of the dashboard, (2) user features such as filtering options and benchmarking to compare local data with other counties, and (3) the centrality of consistent terminology for data dashboard elements. Results also revealed the use of dashboards to facilitate collaboration between legal and behavioral health systems. CONCLUSIONS: Feedback from end-users highlight important design elements and dashboard utility as well as the challenges of working with cross-system and cross-jurisdiction data.

A Collection of Components to Design Clinical Dashboards Incorporating Patient-Reported Outcome Measures: Qualitative Study.

BACKGROUND: A clinical dashboard is a data-driven clinical decision support tool visualizing multiple key performance indicators in a single report while minimizing time and effort for data gathering. Studies have shown that including patient-reported outcome measures (PROMs) in clinical dashboards supports the clinician's understanding of how treatments impact patients' health status, helps identify changes in health-related quality of life at an early stage, and strengthens patient-physician communication. OBJECTIVE: This study aims to determine design components for clinical dashboards incorporating PROMs to inform software producers and users (ie, physicians). METHODS: We conducted interviews with software producers and users to test preselected design components. Furthermore, the interviews allowed us to derive additional components that are not outlined in existing literature. Finally, we used inductive and deductive coding to derive a guide on which design components need to be considered when building a clinical dashboard incorporating PROMs. RESULTS: A total of 25 design components were identified, of which 16 were already surfaced during the literature search. Furthermore, 9 additional components were derived inductively during our interviews. The design components are clustered in a generic dashboard, PROM-related, adjacent information, and requirements for adoption components. Both software producers and users agreed on the primary purpose of a clinical dashboard incorporating PROMs to enhance patient communication in outpatient settings. Dashboard benefits include enhanced data visualization and improved workflow efficiency, while interoperability and data collection were named as adoption challenges. Consistency in dashboard design components is preferred across different episodes of care, with adaptations only for disease-specific PROMs. CONCLUSIONS: Clinical dashboards have the potential to facilitate informed treatment decisions if certain design components are followed. This study establishes a comprehensive framework of design components to guide the development of effective clinical dashboards incorporating PROMs in health care practice.

Public Health Dashboards in Overdose Prevention: The Rhode Island Approach to Public Health Data Literacy, Partnerships, and Action.

As the field of public health rises to the demands of real-time surveillance and rapid data-sharing needs in a postpandemic world, it is time to examine our approaches to the dissemination and accessibility of such data. Distinct challenges exist when working to develop a shared public health language and narratives based on data. It requires that we assess our understanding of public health data literacy, revisit our approach to communication and engagement, and continuously evaluate our impact and relevance. Key stakeholders and cocreators are critical to this process and include people with lived experience, community organizations, governmental partners, and research institutions. In this viewpoint paper, we offer an instructive approach to the tools we used, assessed, and adapted across 3 unique overdose data dashboard projects in Rhode Island, United States. We are calling this model the "Rhode Island Approach to Public Health Data Literacy, Partnerships, and Action." This approach reflects the iterative lessons learned about the improvement of data dashboards through collaboration and strong partnerships across community members, state agencies, and an academic research team. We will highlight key tools and approaches that are accessible and engaging and allow developers and stakeholders to self-assess their goals for their data dashboards and evaluate engagement with these tools by their desired audiences and users.

Trainee-supervisor collaboration, progress-visualisation, and coaching: a survey on challenges in assessment of ICU trainees.

BACKGROUND: Assessing trainees is crucial for development of their competence, yet it remains a challenging endeavour. Identifying contributing and influencing factors affecting this process is imperative for improvement. METHODS: We surveyed residents, fellows, and intensivists working in an intensive care unit (ICU) at a large non-university hospital in Switzerland to investigate the challenges in assessing ICU trainees. Thematic analysis revealed three major themes. RESULTS: Among 45 physicians, 37(82%) responded. The first theme identified is trainee-intensivist collaboration discontinuity. The limited duration of trainees' ICU rotations, large team size operating in a discordant three-shift system, and busy and unpredictable day-planning hinder sustained collaboration. Potential solutions include a concise pre-collaboration briefing, shared bedside care, and post-collaboration debriefing involving formative assessment and reflection on collaboration. The second theme is the lack of trainees' progress visualisation, which is caused by unsatisfactory familiarisation with the trainees' development. The lack of an overview of a trainee's previous achievements, activities, strengths, weaknesses, and goals may result in inappropriate assessments. Participants suggested implementing digital assessment tools, a competence committee, and dashboards to facilitate progress visualisation. The third theme we identified is insufficient coaching and feedback. Factors like personality traits, hierarchy, and competing interests can impede coaching, while high-quality feedback is essential for correct assessment. Skilled coaches can define short-term goals and may optimise trainee assessment by seeking feedback from multiple supervisors and assisting in both formative and summative assessment. Based on these three themes and the suggested solutions, we developed the acronym "ICU-STAR" representing a potentially powerful framework to enhance short-term trainee-supervisor collaboration in the workplace and to co-scaffold the principles of adequate assessment. CONCLUSIONS: According to ICU physicians, trainee-supervisor collaboration discontinuity, the lack of visualisation of trainee's development, and insufficient coaching and feedback skills of supervisors are the major factors hampering trainees' assessment in the workplace. Based on suggestions by the survey participants, we propose the acronym "ICU-STAR" as a framework including briefing, shared bedside care, and debriefing of the trainee-supervisor collaboration at the workplace as its core components. With the attending intensivists acting as coaches, progress visualisation can be enhanced by actively collecting more data points. TRIAL REGISTRATION: N/A.

Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Informatics progress of the Global Burden of Animal Diseases programme towards data for One Health.

The Global Burden of Animal Diseases (GBADs) programme will provide data-driven evidence that policy-makers can use to evaluate options, inform decisions, and measure the success of animal health and welfare interventions. The GBADs' Informatics team is developing a transparent process for identifying, analysing, visualising and sharing data to calculate livestock disease burdens and drive models and dashboards. These data can be combined with data on other global burdens (human health, crop loss, foodborne diseases) to provide a comprehensive range of information on One Health, required to address such issues as antimicrobial resistance and climate change. The programme began by gathering open data from international organisations (which are undergoing their own digital transformations). Efforts to achieve an accurate estimate of livestock numbers revealed problems in finding, accessing and reconciling data from different sources over time. Ontologies and graph databases are being developed to bridge data silos and improve the findability and interoperability of data. Dashboards, data stories, a documentation website and a Data Governance Handbook explain GBADs data, now available through an application programming interface. Sharing data quality assessments builds trust in such data, encouraging their application to livestock and One Health issues. Animal welfare data present a particular challenge, as much of this information is held privately and discussions continue regarding which data are the most relevant. Accurate livestock numbers are an essential input for calculating biomass, which subsequently feeds into calculations of antimicrobial use and climate change. The GBADs data are also essential to at least eight of the United Nations Sustainable Development Goals.

Le programme " Impact mondial des maladies animales " (GBADs) a pour but de réunir des éléments probants axés sur des données, qui soient exploitables par les décideurs politiques pour évaluer les solutions envisagées, fonder leurs décisions et mesurer le succès des interventions dans les domaines de la santé et du bien-être des animaux. L'équipe informatique du GBADs a conçu un processus transparent pour l'identification, l'analyse, la visualisation et le partage des données, grâce auquel il sera possible d'estimer l'impact des maladies du bétail et de réaliser des modèles et des tableaux de bord sur le sujet. Les données ainsi réunies peuvent être combinées avec celles couvrant d'autres problématiques ayant un impact mondial (santé humaine, pertes de récoltes, maladies d'origine alimentaire) afin de fournir l'éventail complet d'informations Une seule santé requis pour faire face à des enjeux tels que la résistance aux agents antimicrobiens ou le changement climatique. La première phase du programme a consisté à recueillir des données ouvertes auprès de diverses organisations internationales (qui procèdent également à leur propre transformation numérique). Les efforts déployés pour parvenir à une estimation précise des effectifs des cheptels ont mis en lumière les difficultés à trouver les données détenues par différentes sources, à y accéder et à les recouper au fil du temps. Des ontologies et des bases de données graphiques sont en cours d'élaboration pour résoudre le problème des silos de données et pour améliorer la facilité de recherche et l'interopérabilité des données. Les données du GBADs sont désormais expliquées sous forme de tableaux de bord, de récits construits à partir des données, ainsi que dans un site web documentaire et un Manuel de gouvernance des données, tous disponibles via une interface de programmation d'applications. Le partage des évaluations de la qualité des données renforce la confiance dans ces dernières et encourage à les appliquer pour traiter les problématiques affectant l'élevage ou relevant de l'approche Une seule santé. Les données relatives au bien-être animal présentent une difficulté particulière : elles sont, pour l'essentiel, détenues à titre privé et la question de savoir quelles sont les données les plus pertinentes est toujours en discussion. Les effectifs des cheptels doivent avoir été déterminés de manière précise afin de calculer la biomasse animale, élément qui entre par la suite dans le calcul des quantités d'agents antimicrobiens utilisés et des indicateurs du changement climatique. Les données du programme GBADs sont également essentielles au regard d'au moins huit des objectifs de développement durable des Nations Unies.

El programa sobre el Impacto Global de las Enfermedades Animales (GBADs) proporcionará información contrastada y basada en el uso de datos de la que luego puedan servirse los planificadores de políticas para valorar distintas opciones, decidir con conocimiento de causa y medir la eficacia de una u otra intervención en materia de sanidad y bienestar animales. El equipo informático encargado del GBADs está preparando un proceso transparente destinado a seleccionar, analizar, visualizar y poner en común datos que ayuden a calcular la carga de enfermedades del ganado y a guiar la elaboración de modelos y paneles de control. Estos datos pueden ser combinados con datos referidos a otros grandes problemas planetarios (salud humana, pérdida de cultivos, enfermedades de transmisión alimentaria) para obtener el repertorio completo de información en clave de Una sola salud que se necesita para abordar problemáticas como la resistencia a los antimicrobianos o el cambio climático. El programa empezó por reunir datos abiertos procedentes de organizaciones internacionales (inmersas, por otra parte, en su propio proceso de transformación digital). La labor emprendida para estimar con exactitud las cifras de ejemplares del mundo pecuario reveló ciertos problemas a la hora de encontrar, obtener y conciliar datos de distintas fuentes a lo largo del tiempo. Ahora se están elaborando ontologías y bases de datos gráficos para crear conexiones entre los "silos de datos" y lograr que los datos sean a la vez más compatibles entre sí y más fáciles de localizar. Paneles de control, interpretaciones narrativas de los datos ("data stories"), un sitio web de documentación y un manual de gestión de datos ayudan a explicar y aprehender los datos del GBADs, accesibles ahora por medio de una interfaz de programación de aplicaciones. El hecho de poner en común las evaluaciones de la calidad de los datos genera mayor confianza en esta información, promoviendo con ello su aplicación en temas de ganadería y de Una sola salud. Los datos de bienestar animal plantean una particular dificultad, pues gran parte de esta información está en manos privadas y todavía no está claro cuáles son los datos de mayor interés. Disponer de cifras exactas sobre el número de cabezas de ganado es fundamental para efectuar los cálculos de biomasa que después se utilizan para hacer otros cómputos referidos al uso de antimicrobianos y al cambio climático. Los datos del GBADs son asimismo esenciales para al menos ocho de los Objetivos de Desarrollo Sostenible de las Naciones Unidas.

Toward comprehensive value assessment for Alzheimer's disease innovations.

INTRODUCTION: Assessing medical technologies for Alzheimer's disease (AD) creates challenges for current methods of value assessment. New value assessment approaches for AD are also needed. METHODS: We adapted concepts from health economics to help guide decision makers to more informed decisions about AD therapies and diagnostics. RESULTS: We propose a value framework based on five categories: perspective, value elements, analysis, reporting, and decision making. AD value assessments should include the perspective of the patient-caregiver dyad. We propose a broader array of value elements than currently used. Analytics and decision methods can synthesize evidence for all elements of value. Decisions should use a "deliberative appraisal" approach informed by the composite evidence and be transparently reported. DISCUSSION: Using the proposed framework, the value of forthcoming innovations for AD may be more thoroughly assessed for and by all stakeholders. It can guide decision makers to carefully consider all relevant elements of value contributing to more holistic and transparent decision making. RESEARCH HIGHLIGHTS: Alzheimer's disease challenges common methods of evaluating medical technology. Using current methods, new AD innovations might not be appropriately valued. Poor value assessments will adversely affect patient access to AD innovations. A full AD value framework expands perspective, elements, analysis, decision-making, reporting.

Protocol for developing a dashboard for interactive cohort analysis of oral health-related data.

INTRODUCTION: A working knowledge of data analytics is becoming increasingly important in the digital health era. Interactive dashboards are a useful, accessible format for presenting and disseminating health-related information to a wide audience. However, many oral health researchers receive minimal data visualisation and programming skills. OBJECTIVES: The objective of this protocols paper is to demonstrate the development of an analytical, interactive dashboard, using oral health-related data from multiple national cohort surveys. METHODS: The flexdashboard package was used within the R Studio framework to create the structure-elements of the dashboard and interactivity was added with the Shiny package. Data sources derived from the national longitudinal study of children in Ireland and the national children's food survey. Variables for input were selected based on their known associations with oral health. The data were aggregated using tidyverse packages such as dplyr and summarised using ggplot2 and kableExtra with specific functions created to generate bar-plots and tables. RESULTS: The dashboard layout is structured by the YAML (YAML Ain't Markup Language) metadata in the R Markdown document and the syntax from Flexdashboard. Survey type, wave of survey and variable selector were set as filter options. Shiny's render functions were used to change input to automatically render code and update output. The deployed dashboard is openly accessible at https://dduh.shinyapps.io/dduh/ . Examples of how to interact with the dashboard for selected oral health variables are illustrated. CONCLUSION: Visualisation of national child cohort data in an interactive dashboard allows viewers to dynamically explore oral health data without requiring multiple plots and tables and sharing of extensive documentation. Dashboard development requires minimal non-standard R coding and can be quickly created with open-source software.

Big Data Health Care Innovations: Performance Dashboarding as a Process of Collective Sensemaking.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/16779.

Toward Better Data Dashboards for US Drug Value Assessments.

OBJECTIVES: To explore the use of data dashboards to convey information about a drug's value, and reduce the need to collapse dimensions of value to a single measure. METHODS: Review of the literature on US Drug Value Assessment Frameworks, and discussion of the value of data dashboards to improve the manner in which information on value is displayed. RESULTS: The incremental cost per quality-adjusted life-year ratio is a useful starting point for conversation about a drug's value, but it cannot reflect all of the elements of value about which different audiences care deeply. Data dashboards for drug value assessments can draw from other contexts. Decision makers should be presented with well-designed value dashboards containing various metrics, including conventional cost per quality-adjusted life-year ratios as well as measures of a drug's impact on clinical and patient-centric outcomes, and on budgetary and distributional consequences, to convey a drug's value along different dimensions. CONCLUSIONS: The advent of US drug value frameworks in health care has forced a concomitant effort to develop appropriate information displays. Researchers should formally test different formats and elements.

How Do We Use Dashboards to Enhance Quality in Cardiac Anesthesia?

The use of clinical dashboards has expanded significantly in healthcare in recent years in a variety of settings. The ability to analyze data related to quality metrics in one screen is highly desirable for cardiac anesthesiologists, as they have considerable influence on important clinical outcomes. Building a robust quality program within cardiac anesthesia relies on consistent access and review of quality outcome measures, process measures, and operational measures through a clinical dashboard. Signals and trends in these measures may be compared to other cardiac surgical programs to analyze gaps and areas for quality improvement efforts. In this article, the authors describe how they designed a clinical cardiac anesthesia dashboard for quality efforts at their institution.

Analysis of a Web-Based Dashboard to Support the Use of National Audit Data in Quality Improvement: Realist Evaluation.

BACKGROUND: Dashboards can support data-driven quality improvements in health care. They visualize data in ways intended to ease cognitive load and support data comprehension, but how they are best integrated into working practices needs further investigation. OBJECTIVE: This paper reports the findings of a realist evaluation of a web-based quality dashboard (QualDash) developed to support the use of national audit data in quality improvement. METHODS: QualDash was co-designed with data users and installed in 8 clinical services (3 pediatric intensive care units and 5 cardiology services) across 5 health care organizations (sites A-E) in England between July and December 2019. Champions were identified to support adoption. Data to evaluate QualDash were collected between July 2019 and August 2021 and consisted of 148.5 hours of observations including hospital wards and clinical governance meetings, log files that captured the extent of use of QualDash over 12 months, and a questionnaire designed to assess the dashboard's perceived usefulness and ease of use. Guided by the principles of realist evaluation, data were analyzed to understand how, why, and in what circumstances QualDash supported the use of national audit data in quality improvement. RESULTS: The observations revealed that variation across sites in the amount and type of resources available to support data use, alongside staff interactions with QualDash, shaped its use and impact. Sites resourced with skilled audit support staff and established reporting systems (sites A and C) continued to use existing processes to report data. A number of constraints influenced use of QualDash in these sites including that some dashboard metrics were not configured in line with user expectations and staff were not fully aware how QualDash could be used to facilitate their work. In less well-resourced services, QualDash automated parts of their reporting process, streamlining the work of audit support staff (site B), and, in some cases, highlighted issues with data completeness that the service worked to address (site E). Questionnaire responses received from 23 participants indicated that QualDash was perceived as useful and easy to use despite its variable use in practice. CONCLUSIONS: Web-based dashboards have the potential to support data-driven improvement, providing access to visualizations that can help users address key questions about care quality. Findings from this study point to ways in which dashboard design might be improved to optimize use and impact in different contexts; this includes using data meaningful to stakeholders in the co-design process and actively engaging staff knowledgeable about current data use and routines in the scrutiny of the dashboard metrics and functions. In addition, consideration should be given to the processes of data collection and upload that underpin the quality of the data visualized and consequently its potential to stimulate quality improvement. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1136/bmjopen-2019-033208.

COVID-19 Knowledge Resource Categorization and Tracking: Conceptual Framework Study.

BACKGROUND: Since the declaration of COVID-19 as a global pandemic by the World Health Organization, the disease has gained momentum with every passing day. Various private and government sectors of different countries allocated funding for research in multiple capacities. A significant portion of efforts has been devoted to information technology and service infrastructure development, including research on developing intelligent models and techniques for alerts, monitoring, early diagnosis, prevention, and other relevant services. As a result, many information resources have been created globally and are available for use. However, a defined structure to organize these resources into categories based on the nature and origin of the data is lacking. OBJECTIVE: This study aims to organize COVID-19 information resources into a well-defined structure to facilitate the easy identification of a resource, tracking information workflows, and to provide a guide for a contextual dashboard design and development. METHODS: A sequence of action research was performed that involved a review of COVID-19 efforts and initiatives on a global scale during the year 2020. Data were collected according to the defined structure of primary, secondary, and tertiary categories. Various techniques for descriptive statistical analysis were employed to gain insights into the data to help develop a conceptual framework to organize resources and track interactions between different resources. RESULTS: Investigating diverse information at the primary, secondary, and tertiary levels enabled us to develop a conceptual framework for COVID-19-related efforts and initiatives. The framework of resource categorization provides a gateway to access global initiatives with enriched metadata, and assists users in tracking the workflow of tertiary, secondary, and primary resources with relationships between various fragments of information. The results demonstrated mapping initiatives at the tertiary level to secondary level and then to the primary level to reach firsthand data, research, and trials. CONCLUSIONS: Adopting the proposed three-level structure allows for a consistent organization and management of existing COVID-19 knowledge resources and provides a roadmap for classifying future resources. This study is one of the earliest studies to introduce an infrastructure for locating and placing the right information at the right place. By implementing the proposed framework according to the stated guidelines, this study allows for the development of applications such as interactive dashboards to facilitate the contextual identification and tracking of interdependent COVID-19 knowledge resources.

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study.

BACKGROUND: Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration. OBJECTIVE: We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time. METHODS: We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point. RESULTS: Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards. CONCLUSIONS: Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

Predict or describe? How learning analytics dashboard design influences motivation and statistics anxiety in an online statistics course.

Based on the achievement goal theory, this experimental study explored the influence of predictive and descriptive learning analytics dashboards on graduate students' motivation and statistics anxiety in an online graduate-level statistics course. Participants were randomly assigned into one of three groups: (a) predictive dashboard, (b) descriptive dashboard, or (c) control (i.e., no dashboard). Measures of motivation and statistical anxiety were collected in the beginning and the end of the semester via the Motivated Strategies for Learning Questionnaire and Statistical Anxiety Rating Scale. Individual semi-structured interviews were used to understand learners' perceptions of the course and whether the use of the dashboards influenced the meaning of their learning experiences. Results indicate that, compared to the control group, the predictive dashboard significantly reduced learners' interpretation anxiety and had an effect on intrinsic goal orientation that depended on learners' lower or higher initial levels of intrinsic goal orientation. In comparison to the control group, both predictive and descriptive dashboards reduced worth of anxiety (negative attitudes towards statistics) for learners who started the course with higher levels of worth anxiety. Thematic analysis revealed that learners who adopted a more performance-avoidance goal orientation approach demonstrated higher levels of anxiety regardless of the dashboard used. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s11423-021-09998-z.

Empowering Implementation Teams with a Learning Health System Approach: Leveraging Data to Improve Quality of Care for Transient Ischemic Attack.

BACKGROUND: Questions persist about how learning healthcare systems should integrate audit and feedback (A&F) into quality improvement (QI) projects to support clinical teams' use of performance data to improve care quality. OBJECTIVE: To identify how a virtual "Hub" dashboard that provided performance data for patients with transient ischemic attack (TIA), a resource library, and a forum for sharing QI plans and tools supported QI activities among newly formed multidisciplinary clinical teams at six Department of Veterans Affairs (VA) medical centers. DESIGN: An observational, qualitative evaluation of how team members used a web-based Hub. PARTICIPANTS: External facilitators and multidisciplinary team members at VA facilities engaged in QI to improve the quality of TIA care. APPROACH: Qualitative implementation process and summative evaluation of observational Hub data (interviews with Hub users, structured field notes) to identify emergent, contextual themes and patterns of Hub usage. KEY RESULTS: The Hub supported newly formed multidisciplinary teams in implementing QI plans in three main ways: as an information interface for integrated monitoring of TIA performance; as a repository used by local teams and facility champions; and as a tool for team activation. The Hub enabled access to data that were previously inaccessible and unavailable and integrated that data with benchmark and scientific evidence to serve as a common data infrastructure. Led by champions, each implementation team used the Hub differently: local adoption of the staff and patient education materials; benchmarking facility performance against national rates and peer facilities; and positive reinforcement for QI plan development and monitoring. External facilitators used the Hub to help teams leverage data to target areas of improvement and disseminate local adaptations to promote resource sharing across teams. CONCLUSIONS: As a dynamic platform for A&F operating within learning health systems, hubs represent a promising strategy to support local implementation of QI programs by newly formed, multidisciplinary teams.

Case-mix adjusted postanaesthesia care unit length of stay and business intelligence dashboards for feedback to anaesthetists.

BACKGROUND: Despite advances in business intelligence software and evidence that feedback to doctors can improve outcomes, objective feedback regarding patient outcomes for individual anaesthetists is hampered by lack of useful benchmarks. We aimed to address this issue by producing case-mix and risk-adjusted postanaesthesia care unit (PACU) length of stay (LOS) benchmarks for integration into modern reporting tools. METHODS: We extended existing hospital information systems to calculate predicted PACU LOS using a neural network trained on patient age, surgery duration, sex, operating specialty, urgency, weekday, and insurance status (n=100 511). We then calculated the difference between observed mean and predicted PACU LOS for individual doctors, and compared the results with and without case-mix adjustment. We report practical implications of using visual analytics dashboards displaying the difference between observed and predicted PACU LOS to provide feedback to anaesthetic doctors. RESULTS: The neural network accounted for over half of observed variation in individual doctors' mean PACU LOS (mean predicted and mean actual LOS Spearman's r2=0.57). Account for case-mix reduced apparent spread, with 80% of individual doctors falling in a band of 4.3 min after case-mix adjusting, compared with a range of 24 min without adjustment. Case-mix adjusting also identified different individual doctors as outliers (Weighted Cohen's kappa [κ]=0.27). Finally, we demonstrated that we were able to integrate the adjusted metrics into routine reporting tools. CONCLUSION: With caution, case-mix adjustment of anaesthetic outcome measures such as PACU LOS potentially provides a useful continuous quality improvement tool. Unadjusted outcome measures are imprecise at best and misleading at worst.

A Dynamic Dashboarding Application for Fleet Monitoring Using Semantic Web of Things Technologies.

In industry, dashboards are often used to monitor fleets of assets, such as trains, machines or buildings. In such industrial fleets, the vast amount of sensors evolves continuously, new sensor data exchange protocols and data formats are introduced, new visualization types may need to be introduced and existing dashboard visualizations may need to be updated in terms of displayed sensors. These requirements motivate the development of dynamic dashboarding applications. These, as opposed to fixed-structure dashboard applications, allow users to create visualizations at will and do not have hard-coded sensor bindings. The state-of-the-art in dynamic dashboarding does not cope well with the frequent additions and removals of sensors that must be monitored-these changes must still be configured in the implementation or at runtime by a user. Also, the user is presented with an overload of sensors, aggregations and visualizations to select from, which may sometimes even lead to the creation of dashboard widgets that do not make sense. In this paper, we present a dynamic dashboard that overcomes these problems. Sensors, visualizations and aggregations can be discovered automatically, since they are provided as RESTful Web Things on a Web Thing Model compliant gateway. The gateway also provides semantic annotations of the Web Things, describing what their abilities are. A semantic reasoner can derive visualization suggestions, given the Thing annotations, logic rules and a custom dashboard ontology. The resulting dashboarding application automatically presents the available sensors, visualizations and aggregations that can be used, without requiring sensor configuration, and assists the user in building dashboards that make sense. This way, the user can concentrate on interpreting the sensor data and detecting and solving operational problems early.

High Density Real-Time Air Quality Derived Services from IoT Networks.

In recent years, there is an increasing attention on air quality derived services for the final users. A dense grid of measures is needed to implement services such as conditional routing, alerting on data values for personal usage, data heatmaps for Dashboards in control room for the operators, and for web and mobile applications for the city users. Therefore, the challenge consists of providing high density data and services starting from scattered data and regardless of the number of sensors and their position to a large number of users. To this aim, this paper is focused on providing an integrated solution addressing at the same time multiple aspects: To create and optimize algorithms for data interpolation (creating regular data from scattered), making it possible to cope with the scalability and providing support for on demand services to provide air quality data in any point of the city with dense data. To this end, the accuracy of different interpolation algorithms has been evaluated comparing the results with respect to real values. In addition, the trends of heatmaps interpolation errors have been exploited to detected devices' dysfunctions. Such anomalies may often be useful to request a maintenance action. The solution proposed has been integrated as a Micro Services providing data analytics in a data flow real time process based on Node.JS Node-RED, called in the paper IoT Applications. The specific case presented in this paper refers to the data and the solution of Snap4City for Helsinki. Snap4City, which has been developed as a part of Select4Cities PCP of the European Commission, and it is presently used in a number of cities and areas in Europe.