A pediatric virtual care evaluation framework and its evolution using consensus methods.

BACKGROUND: The use of virtual care has increased dramatically in response to the COVID-19 pandemic, yet evidence is lacking regarding the impact of virtual care on patient outcomes, particularly in pediatrics. A standardized evaluation approach is required to support the integration of virtual care into pediatric health care delivery programs. The objective of this work was to develop a comprehensive and structured framework for pediatric virtual care evaluation. This framework is intended to engage and guide care providers, health centres, and stakeholders towards the development of a standardized approach to the evaluation of pediatric virtual care. METHODS: We brought together a diverse multidisciplinary team, including pediatric clinicians, researchers, digital health leads and analysts, program leaders, a human factors engineer, a family advisor and our manager of health equity and diversity. The team reviewed the literature, including published evaluation frameworks, and used a consensus-based method to develop a virtual care evaluation framework applicable to a broad spectrum of pediatric virtual care programs. We used an iterative process to develop framework components, including domains and sub-domains, examples of evaluation questions, measures, and data sources. Team members met repeatedly over seven months to generate and provide feedback on all components of the framework, making revision as needed until consensus was reached. The framework was then applied to an existing virtual care program. RESULTS: The resulting framework includes four domains (health outcomes, health delivery, individual experience, and program implementation) and 19 sub-domains designed to support the development and evaluation of pediatric virtual care programs. We also developed guidance on how to use the framework and illustrate its utility by applying it to an existing pediatric virtual care program. CONCLUSIONS: This virtual care evaluation framework expands on previously developed frameworks by providing additional detail and a structure that supports practical application. It can be used to evaluate a wide range of pediatric virtual care programs in a standardized manner. Use of this comprehensive yet easy to use evaluation framework will inform appropriate implementation and integration of virtual care into routine practice and support its sustainability and continuous improvement.

Evaluation Strategies for Understanding Experiences With Virtual Care in Canada: Mixed Methods Study.

BACKGROUND: Virtual care was rapidly integrated into pediatric health services during the COVID-19 pandemic. While virtual care offers many benefits, it is necessary to better understand the experiences of those who receive, deliver, and coordinate virtual care in order to support sustainable, high-quality, and patient-centered health care. To date, methods implemented to evaluate users' experiences of virtual care have been highly variable, making comparison and data synthesis difficult. OBJECTIVE: This study aims to describe evaluation strategies currently used to understand personal experiences with pediatric virtual care in Canada. METHODS: In this mixed methods environmental scan, we first distributed a web-based questionnaire to clinical, research, and operational leaders delivering and evaluating pediatric virtual care in Canada. The questionnaire collected information about how experiences with virtual care have been or are currently being evaluated and whether these evaluations included the perspectives of children or youth, families, providers, or support staff. Second, respondents were asked to share the questions they used in their evaluations, and a content analysis was performed to identify common question categories. Third, we conducted semistructured interviews to further explore our respondents' evaluation experiences across 4 domains-evaluation approaches, distribution methods, response rates, and lessons learned-and interest in a core set of questions for future evaluations. RESULTS: There were 72 respondents to the web-based questionnaire; among those who had conducted an evaluation, we identified 15 unique evaluations, and 14 of those provided a copy of the tools used to evaluate virtual care. These evaluations measured the virtual care experiences of parents or caregivers (n=15, 100%), children or youth (n=11, 73%), health care providers (n=11, 73%), and support staff (n=4, 27%). The most common data collection method used was electronic questionnaires distributed by email. Two respondents used validated tools; the remainder modified existing tools or developed new tools. Content analysis of the 14 submitted questionnaires revealed that the most common questions were about overall participant satisfaction, the comparison of virtual care to in-person care, and whether participants would choose virtual care options in the future. Interview findings indicate respondents frequently relied on methods used by peers and that a standardized, core set of questions to evaluate experiences with virtual care would be helpful to improve evaluation practices and support pediatric health care delivery. CONCLUSIONS: At our institution and elsewhere in Canada, experiences with pediatric virtual care have been evaluated using a variety of methods. A more consistent evaluation approach using standardized tools may enable more regular comparisons of experiences with virtual care and the synthesis of findings across health care settings. In turn, this may better inform our approach to virtual care, improve its integration into health systems, and facilitate sustainable, high-quality, patient-centered care.

Virtual family-centered rounds: a quality improvement initiative to adapt inpatient care during COVID-19 using a human-centred participatory design approach.

BACKGROUND: Family-centered rounds (FCR) are fundamental to pediatric inpatient care. During the COVID-19 pandemic, we aimed to design and implement a virtual family-centered rounds (vFCR) process that allowed continuation of inpatient rounds while following physical distancing guidelines and preserving personal protective equipment (PPE). METHODS: A multidisciplinary team developed the vFCR process using a participatory design approach. From April through July 2020, quality improvement methods were used to iteratively evaluate and improve the process. Outcome measures included satisfaction, perceived effectiveness, and perceived usefulness of vFCR. Data were collected via questionnaire distributed to patients, families, staff and medical staff, and analyzed using descriptive statistics and content analysis. Virtual auditors monitored time per patient round and transition time between patients as balancing measures. RESULTS: Seventy-four percent (51/69) of health care providers surveyed and 79% (26/33) of patients and families were satisfied or very satisfied with vFCR. Eighty eight percent (61/69) of health care providers and 88% (29/33) of patients and families felt vFCR were useful. Audits revealed an average vFCR duration of 8.4 min (SD = 3.9) for a single patient round and transition time between patients averaged 2.9 min (SD = 2.6). CONCLUSION: Virtual family-centered rounds are an acceptable alternative to in-person FCR in a pandemic scenario, yielding high levels of stakeholder satisfaction and support. We believe vFCR are a useful method to support inpatient rounds, physical distancing, and preservation of PPE that may also be valuable beyond the pandemic. A rigorous process evaluation of vFCR is underway.