Evaluation of Digitalisation in Healthcare and the Quantification of the "Unmeasurable".

Evaluating healthcare digitalisation, where technology implementation and adoption transforms existing socio-organisational processes, presents various challenges for outcome assessments. Populations are diverse, interventions are complex and evolving over time, meaningful comparisons are difficult as outcomes vary between settings, and outcomes take a long time to materialise and stabilise. Digitalisation may also have unanticipated impacts. We here discuss the limitations of evaluating the digitalisation of healthcare, and describe how qualitative and quantitative approaches can complement each other to facilitate investment and implementation decisions. In doing so, we argue how existing approaches have focused on measuring what is easily measurable and elevating poorly chosen values to inform investment decisions. Limited attention has been paid to understanding processes that are not easily measured even though these can have significant implications for contextual transferability, sustainability and scale-up of interventions. We use what is commonly known as the McNamara Fallacy to structure our discussions. We conclude with recommendations on how we envisage the development of mixed methods approaches going forward in order to address shortcomings.

Socio-Organizational Dimensions: The Key to Advancing the Shared Care Record Agenda in Health and Social Care.

Integrating health and social care delivery with the help of digital technologies is a grand challenge. We argue that previous attempts have largely failed to achieve their objectives because implementers and decision makers disregard the complex socio-organizational dimensions of change associated with initiatives. These include structural and organizational complexity inhibiting the development of shared care pathways; professional jurisdictions, interests, and expertise; and existing data and governance structures. We provide an overview of those dimensions that can inform strategic decisions going forward, thereby contributing to the chances of success of shared care initiatives.

Optimising ePrescribing in hospitals through the interoperability of systems and processes: a qualitative study in the UK, US, Norway and the Netherlands.

BACKGROUND: Investment in the implementation of hospital ePrescribing systems has been a priority in many economically-developed countries in order to modernise the delivery of healthcare. However, maximum gains in the safety, quality and efficiency of care are unlikely to be fully realised unless ePrescribing systems are further optimised in a local context. Typical barriers to optimal use are often encountered in relation to a lack of systemic capacity and preparedness to meet various levels of interoperability requirements, including at the data, systems and services levels. This lack of systemic interoperability may in turn limit the opportunities and benefits potentially arising from implementing novel digital heath systems. METHODS: We undertook n = 54 qualitative interviews with key stakeholders at nine digitally advanced hospital sites across the UK, US, Norway and the Netherlands. We included hospitals featuring 'standalone, best of breed' systems, which were interfaced locally, and multi-component and integrated electronic health record enterprise systems. We analysed the data inductively, looking at strategies and constraints for ePrescribing interoperability within and beyond hospital systems. RESULTS: Our thematic analysis identified 4 main drivers for increasing ePrescribing systems interoperability: (1) improving patient safety (2) improving integration & continuity of care (3) optimising care pathways and providing tailored decision support to meet local and contextualised care priorities and (4) to enable full patient care services interoperability in a variety of settings and contexts. These 4 interoperability dimensions were not always pursued equally at each implementation site, and these were often dependent on the specific national, policy, organisational or technical contexts of the ePrescribing implementations. Safety and efficiency objectives drove optimisation targeted at infrastructure and governance at all levels. Constraints to interoperability came from factors such as legacy systems, but barriers to interoperability of processes came from system capability, hospital policy and staff culture. CONCLUSIONS: Achieving interoperability is key in making ePrescribing systems both safe and useable. Data resources exist at macro, meso and micro levels, as do the governance interventions necessary to achieve system interoperability. Strategic objectives, most notably improved safety, often motivated hospitals to push for evolution across the entire data architecture of which they formed a part. However, hospitals negotiated this terrain with varying degrees of centralised coordination. Hospitals were heavily reliant on staff buy-in to ensure that systems interoperability was built upon to achieve effective data sharing and use. Positive outcomes were founded on a culture of agreement about the usefulness of access by stakeholders, including prescribers, policymakers, vendors and lab technicians, which was reflected in an alignment of governance goals with system design.

Understanding Public Perceptions of COVID-19 Contact Tracing Apps: Artificial Intelligence-Enabled Social Media Analysis.

BACKGROUND: The emergence of SARS-CoV-2 in late 2019 and its subsequent spread worldwide continues to be a global health crisis. Many governments consider contact tracing of citizens through apps installed on mobile phones as a key mechanism to contain the spread of SARS-CoV-2. OBJECTIVE: In this study, we sought to explore the suitability of artificial intelligence (AI)-enabled social media analyses using Facebook and Twitter to understand public perceptions of COVID-19 contact tracing apps in the United Kingdom. METHODS: We extracted and analyzed over 10,000 relevant social media posts across an 8-month period, from March 1 to October 31, 2020. We used an initial filter with COVID-19-related keywords, which were predefined as part of an open Twitter-based COVID-19 dataset. We then applied a second filter using contract tracing app-related keywords and a geographical filter. We developed and utilized a hybrid, rule-based ensemble model, combining state-of-the-art lexicon rule-based and deep learning-based approaches. RESULTS: Overall, we observed 76% positive and 12% negative sentiments, with the majority of negative sentiments reported in the North of England. These sentiments varied over time, likely influenced by ongoing public debates around implementing app-based contact tracing by using a centralized model where data would be shared with the health service, compared with decentralized contact-tracing technology. CONCLUSIONS: Variations in sentiments corroborate with ongoing debates surrounding the information governance of health-related information. AI-enabled social media analysis of public attitudes in health care can help facilitate the implementation of effective public health campaigns.

Interorganizational Knowledge Sharing to Establish Digital Health Learning Ecosystems: Qualitative Evaluation of a National Digital Health Transformation Program in England.

BACKGROUND: The English Global Digital Exemplar (GDE) program is one of the first concerted efforts to create a digital health learning ecosystem across a national health service. OBJECTIVE: This study aims to explore mechanisms that support or inhibit the exchange of interorganizational digital transformation knowledge. METHODS: We conducted a formative qualitative evaluation of the GDE program. We used semistructured interviews with clinical, technical, and managerial staff; national program managers and network leaders; nonparticipant observations of knowledge transfer activities through attending meetings, workshops, and conferences; and documentary analysis of policy documents. The data were thematically analyzed by drawing on a theory-informed sociotechnical coding framework. We used a mixture of deductive and inductive methods, supported by NVivo software, to facilitate coding. RESULTS: We conducted 341 one-on-one and 116 group interviews, observed 86 meetings, and analyzed 245 documents from 36 participating provider organizations. We also conducted 51 high-level interviews with policy makers and vendors; performed 77 observations of national meetings, workshops, and conferences; and analyzed 80 national documents. Formal processes put in place by the GDE program to initiate and reinforce knowledge transfer and learning have accelerated the growth of informal knowledge networking and helped establish the foundations of a learning ecosystem. However, formal networks were most effective when supported by informal networking. The benefits of networking were enhanced (and costs reduced) by geographical proximity, shared culture and context, common technological functionality, regional and strategic alignments, and professional agendas. CONCLUSIONS: Knowledge exchange is most effective when sustained through informal networking driven by the mutual benefits of sharing knowledge and convergence between group members in their organizational and technological setting and goals. Policy interventions need to enhance incentives and reduce barriers to sharing across the ecosystem, be flexible in tailoring formal interventions to emerging needs, and promote informal knowledge sharing.

Theoretical and methodological considerations in evaluating large-scale health information technology change programmes.

BACKGROUND: Attempts to achieve digital transformation across the health service have stimulated increasingly large-scale and more complex change programmes. These encompass a growing range of functions in multiple locations across the system and may take place over extended timeframes. This calls for new approaches to evaluate these programmes. MAIN BODY: Drawing on over a decade of conducting formative and summative evaluations of health information technologies, we here build on previous work detailing evaluation challenges and ways to tackle these. Important considerations include changing organisational, economic, political, vendor and markets necessitating tracing of evolving networks, relationships, and processes; exploring mechanisms of spread; and studying selected settings in depth to understand local tensions and priorities. CONCLUSIONS: Decision-makers need to recognise that formative evaluations, if built on solid theoretical and methodological foundations, can help to mitigate risks and help to ensure that programmes have maximum chances of success.

Can Disinfection Robots Reduce the Risk of Transmission of SARS-CoV-2 in Health Care and Educational Settings?

We explore the opportunities and challenges surrounding the use of disinfection robots to reduce the risk of SARS-CoV-2 transmission in health care and educational settings. Although there is some potential for deploying robots to help with manual cleaning, the evidence base is mixed, and we highlight that there needs to be work to establish and enhance the effectiveness of these robots in inactivating the virus.

Can Robots Improve Testing Capacity for SARS-CoV-2?

There is currently increasing interest internationally in deploying robotic applications for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) testing, as these can help to reduce the risk of transmission of the virus to health care staff and patients. We provide an overview of key recent developments in this area. We argue that, although there is some potential for deploying robots to help with SARS-CoV-2 testing, the potential of patient-facing applications is likely to be limited. This is due to the high costs associated with patient-facing functionality, and risks of potentially adverse impacts on health care staff work practices and patient interactions. In contrast, back-end laboratory-based robots dealing with sample extraction and amplification, that effectively integrate with established processes, software, and interfaces to process samples, are much more likely to result in safety and efficiency gains. Consideration should therefore be given to deploying these at scale.

Developing and Applying a Formative Evaluation Framework for Health Information Technology Implementations: Qualitative Investigation.

BACKGROUND: There is currently a lack of comprehensive, intuitive, and usable formative evaluation frameworks for health information technology (HIT) implementations. We therefore sought to develop and apply such a framework. This study describes the Technology, People, Organizations, and Macroenvironmental factors (TPOM) framework we developed. OBJECTIVE: The aim was to develop and apply a formative evaluation framework for HIT implementations, highlighting interrelationships between identified dimensions and offering guidance for implementers. METHODS: We drew on an initial prototype framework developed as part of a literature review exploring factors for the effective implementation of HIT. In addition, we used qualitative data from three national formative evaluations of different HIT interventions (electronic health record, electronic prescribing, and clinical decision support functionality). The combined data set comprised 19 case studies of primarily hospital settings, and included 703 semistructured interviews, 663 hours of observations, and 864 documents gathered from a range of care settings across National Health Service (NHS) England and NHS Scotland. Data analysis took place over a period of 10 years and was guided by a framework informed by the existing evidence base. RESULTS: TPOM dimensions are intimately related and each include a number of subthemes that evaluators need to consider. Although technological functionalities are crucial in getting an initiative off the ground, system design needs to be cognizant of the accompanying social and organizational transformations required to ensure that technologies deliver the desired value for a variety of stakeholders. Wider structural changes, characterized by shifting policy landscapes and markets, influence technologies and the ways they are used by organizations and staff. CONCLUSIONS: The TPOM framework supports formative evaluations of HIT implementation and digitally enabled transformation efforts. There is now a need for prospective application of the TPOM framework to determine its value.

Accelerating Innovation in Health Care: Insights From a Qualitative Inquiry Into United Kingdom and United States Innovation Centers.

BACKGROUND: Digital health innovations are being prioritized on international policy agendas in the hope that they will help to address the existing health system challenges. OBJECTIVE: The aim of this study was to explore the setup, design, facilities, and strategic priorities of leading United Kingdom and United States health care innovation centers to identify transferable lessons for accelerating their creation and maximizing their impact. METHODS: We conducted qualitative case studies consisting of semistructured, audio-recorded interviews with decision makers and center staff in 6 innovation centers. We also conducted nonparticipant observations of meetings and center tours, where we took field notes. Qualitative data were analyzed initially within and then across cases facilitated by QSR International's NVivo software. RESULTS: The centers had different institutional arrangements, including university-associated institutes or innovation laboratories, business accelerators or incubators, and academic health science partnership models. We conducted interviews with 34 individuals, 1 group interview with 3 participants, and observations of 4 meetings. Although the centers differed significantly in relation to their mission, structure, and governance, we observed key common characteristics. These included high-level leadership support and incentives to engage in innovation activities, a clear mission to address identified gaps within their respective organizational and health system settings, physical spaces that facilitated networking through open-door policies, flat managerial structures characterized by new organizational roles for which boundary spanning was key, and a wider innovation ecosystem that was strategically and proactively engaged with the center facilitating external partnerships. CONCLUSIONS: Although innovation in health care settings is unpredictable, we offer insights that may help those establishing innovation centers. The key in this respect is the ability to support different kinds of innovations at different stages through adequate support structures, including the development of new career pathways.

Opportunities and Challenges Surrounding the Use of Data From Wearable Sensor Devices in Health Care: Qualitative Interview Study.

BACKGROUND: Wearable sensors connected via networked devices have the potential to generate data that may help to automate processes of care, engage patients, and increase health care efficiency. The evidence of effectiveness of such technologies is, however, nascent and little is known about unintended consequences. OBJECTIVE: Our objective was to explore the opportunities and challenges surrounding the use of data from wearable sensor devices in health care. METHODS: We conducted a qualitative, theoretically informed, interview-based study to purposefully sample international experts in health care, technology, business, innovation, and social sciences, drawing on sociotechnical systems theory. We used in-depth interviews to capture perspectives on development, design, and use of data from wearable sensor devices in health care, and employed thematic analysis of interview transcripts with NVivo to facilitate coding. RESULTS: We interviewed 16 experts. Although the use of data from wearable sensor devices in health and care has significant potential in improving patient engagement, there are a number of issues that stakeholders need to negotiate to realize these benefits. These issues include the current gap between data created and meaningful interpretation in health and care contexts, integration of data into health care professional decision making, negotiation of blurring lines between consumer and medical care, and pervasive monitoring of health across previously disconnected contexts. CONCLUSIONS: Stakeholders need to actively negotiate existing challenges to realize the integration of data from wearable sensor devices into electronic health records. Viewing wearables as active parts of a connected digital health and care infrastructure, in which various business, personal, professional, and health system interests align, may help to achieve this.

Correction: Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study.

[This corrects the article DOI: .].

Technological Capabilities to Assess Digital Excellence in Hospitals in High Performing Health Care Systems: International eDelphi Exercise.

BACKGROUND: Hospitals worldwide are developing ambitious digital transformation programs as part of broader efforts to create digitally advanced health care systems. However, there is as yet no consensus on how best to characterize and assess digital excellence in hospitals. OBJECTIVE: Our aim was to develop an international agreement on a defined set of technological capabilities to assess digital excellence in hospitals. METHODS: We conducted a two-stage international modified electronic Delphi (eDelphi) consensus-building exercise, which included a qualitative analysis of free-text responses. In total, 31 international health informatics experts participated, representing clinical, academic, public, and vendor organizations. RESULTS: We identified 35 technological capabilities that indicate digital excellence in hospitals. These are divided into two categories: (a) capabilities within a hospital (n=20) and (b) capabilities enabling communication with other parts of the health and social care system, and with patients and carers (n=15). The analysis of free-text responses pointed to the importance of nontechnological aspects of digitally enabled change, including social and organizational factors. Examples included an institutional culture characterized by a willingness to transform established ways of working and openness to risk-taking. The availability of a range of skills within digitization teams, including technological, project management and business expertise, and availability of resources to support hospital staff, were also highlighted. CONCLUSIONS: We have identified a set of criteria for assessing digital excellence in hospitals. Our findings highlight the need to broaden the focus from technical functionalities to wider digital transformation capabilities.

Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study.

BACKGROUND: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE: The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS: Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide." RESULTS: The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle." CONCLUSIONS: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Health Care Robotics: Qualitative Exploration of Key Challenges and Future Directions.

BACKGROUND: The emergence of robotics is transforming industries around the world. Robot technologies are evolving exponentially, particularly as they converge with other functionalities such as artificial intelligence to learn from their environment, from each other, and from humans. OBJECTIVE: The goal of the research was to understand the emerging role of robotics in health care and identify existing and likely future challenges to maximize the benefits associated with robotics and related convergent technologies. METHODS: We conducted qualitative semistructured one-to-one interviews exploring the role of robotic applications in health care contexts. Using purposive sampling, we identified a diverse range of stakeholders involved in conceiving, procuring, developing, and using robotics in a range of national and international health care settings. Interviews were digitally recorded, transcribed verbatim, and analyzed thematically, supported by NVivo 10 (QSR International) software. Theoretically, this work was informed by the sociotechnical perspective, where social and technical systems are understood as being interdependent. RESULTS: We conducted 21 interviews and these accounts suggested that there are significant opportunities for improving the safety, quality, and efficiency of health care through robotics, but our analysis identified 4 major barriers that need to be effectively negotiated to realize these: (1) no clear pull from professionals and patients, (2) appearance of robots and associated expectations and concerns, (3) disruption of the way work is organized and distributed, and (4) new ethical and legal challenges requiring flexible liability and ethical frameworks. CONCLUSIONS: Sociotechnical challenges associated with the effective integration of robotic applications in health care settings are likely to be significant, particularly for patient-facing functions. These need to be identified and addressed for effective innovation and adoption.

Does sharing the electronic health record in the consultation enhance patient involvement? A mixed-methods study using multichannel video recording and in-depth interviews in primary care.

BACKGROUND: Sharing the electronic health-care record (EHR) during consultations has the potential to facilitate patient involvement in their health care, but research about this practice is limited. METHODS: We used multichannel video recordings to identify examples and examine the practice of screen-sharing within 114 primary care consultations. A subset of 16 consultations was viewed by the general practitioner and/or patient in 26 reflexive interviews. Screen-sharing emerged as a significant theme and was explored further in seven additional patient interviews. Final analysis involved refining themes from interviews and observation of videos to understand how screen-sharing occurred, and its significance to patients and professionals. RESULTS: Eighteen (16%) of 114 videoed consultations involved instances of screen-sharing. Screen-sharing occurred in six of the subset of 16 consultations with interviews and was a significant theme in 19 of 26 interviews. The screen was shared in three ways: 'convincing' the patient of a diagnosis or treatment; 'translating' between medical and lay understandings of disease/medication; and by patients 'verifying' the accuracy of the EHR. However, patients and most GPs perceived the screen as the doctor's domain, not to be routinely viewed by the patient. CONCLUSIONS: Screen-sharing can facilitate patient involvement in the consultation, depending on the way in which sharing comes about, but the perception that the record belongs to the doctor is a barrier. To exploit the potential of sharing the screen to promote patient involvement, there is a need to reconceptualise and redesign the EHR.

Taxonomy of delays in the implementation of hospital computerized physician order entry and clinical decision support systems for prescribing: a longitudinal qualitative study.

BACKGROUND: Implementation delays are common in health information technology (HIT) projects. In this paper, we sought to explore the reasons for delays in implementing major hospital-based HIT, through studying computerized physician order entry (CPOE) and clinical decision support (CDS) systems for prescribing and to develop a provisional taxonomy of causes of implementation delays. METHODS: We undertook a series of longitudinal, qualitative case studies to investigate the implementation and adoption of CPOE and CDS systems for prescribing in hospitals in the U.K. We used a combination of semi-structured interviews from six case study sites and two whole day expert roundtable discussions to collect data. Interviews were carried out with users, implementers and suppliers of CPOE/CDS systems. We used thematic analysis to examine the results, drawing on perspectives surrounding the biography of artefacts. RESULTS: We identified 15 major factors contributing to delays in implementation of CPOE and CDS systems. These were then categorized in a two-by-two delay classification matrix: one axis distinguishing tactical versus unintended causes of delay, and the second axis illustrating internal i.e., (the adopting hospital) versus external (i.e., suppliers, other hospitals, policymakers) related causes. CONCLUSIONS: Our taxonomy of delays in HIT implementation should enable system developers, implementers and policymakers to better plan and manage future implementations. More detailed planning at the outset, considering long-term strategies, sustained user engagement, and phased implementation approaches appeared to reduce the risks of delays. It should however be noted that whilst some delays are likely to be preventable, other delays cannot be easily avoided and taking steps to minimize these may negatively affect the longer-term use of the system.

Concerted adoption as an emerging strategy for digital transformation of healthcare-lessons from Australia, Canada, and England.

OBJECTIVES: With an increasing focus on the digitalization of health and care settings, there is significant scope to learn from international approaches to promote concerted adoption of electronic health records. MATERIALS AND METHODS: We review three large-scale initiatives from Australia, Canada, and England, and extract common lessons for future health and social care transformation strategy. RESULTS: We discuss how, despite differences in contexts, concerted adoption enables sharing of experience and learning to streamline the digital transformation of health and care. DISCUSSION AND CONCLUSION: Concerted adoption can be accelerated through building communities of expertise and partnerships promoting knowledge transfer and circulation of expertise; commonalities in geographical and cultural contexts; and commonalities in technological systems.

Global research priorities to better understand the burden of iatrogenic harm in primary care: an international Delphi exercise.

Using a modified Delphi exercise, Aziz Sheikh and colleagues identify research priorities for patient safety research in primary care contexts. Please see later in the article for the Editors' Summary.