Digital Transformation of Face-To-Face Focus Group Methodology: Engaging a Globally Dispersed Audience to Manage Institutional Change at the World Health Organization.

Focus group discussions (FGDs) are widely used to obtain qualitative data from purposely selected groups of people. This paper describes how the Learning and Capacity Development (LCD) unit of the World Health Organization (WHO) Health Emergencies Programme (WHE) digitalized FGDs to engage with WHO staff from around the world, to listen, share, and collect their feedback in the development of a WHO learning framework. The impact of the COVID-19 pandemic and the introduction of local lockdowns and travel restrictions resulted in the wide use of digital platforms, such as Zoom, for employee communications and collaboration capable of reaching employees wherever they are working. The LCD/WHE team drew upon the experience of WHO colleagues from human resources, country, and regional offices to set up and hold FGDs in 6 languages with participants from all WHO regions. Building on the findings of a 2019 WHO staff survey, which was part of a comprehensive, organization-wide career development initiative, the digitalized FGDs allowed for the exchange of substantive feedback, novel ideas, and alignment, connecting across different geographies, disciplines, and levels of seniority. As a result, FGDs can be successfully conducted online, but it is essential to remove barriers to participation by adopting a multilingual and flexible approach in multinational and international organizations such as the WHO.

Global Reach of an Online COVID-19 Course in Multiple Languages on OpenWHO in the First Quarter of 2020: Analysis of Platform Use Data.

BACKGROUND: At the onset of the coronavirus outbreak, the World Health Organization's (WHO) Health Emergencies Learning and Capacity Development Unit, together with the WHO's health technical lead on coronaviruses, developed a massive open online course within 3 weeks as part of the global response to the emergency. The introductory coronavirus disease (COVID­19) course was launched on January 26, 2020, on the health emergencies learning platform OpenWHO.org. OBJECTIVE: The aim of this paper is to investigate the geographic reach of different language courses accessed by a worldwide audience seeking information on COVID-19. Users' professional identities and backgrounds were explored to inform course owners on the use case. The course was developed and delivered via the open-access learning platform OpenWHO.org. The self-paced resources are available in a total of 13 languages and were produced between January 26 and March 25, 2020. METHODS: Data were collected from the online courses' statistical data and metrics reporting system on the OpenWHO platform. User patterns and locations were analyzed based on Google Analytics and the platform's own statistics capabilities, and data sets were overlaid. This analysis was conducted based on user location, with the data disaggregated according to the six WHO regions, the top 10 countries, and the proportion of use for each language version. Data included affiliation, gender, age, and other parameters for 32.43% (52,214/161,007) of the users who indicated their background. RESULTS: As of March 25, 2020, the introductory COVID-19 course totaled 232,890 enrollments across all languages. The Spanish language course was comprised of more than half (n=118,754, 50.99%) of all course enrollments, and the English language course was comprised of 38.21% (n=88,988) of enrollments. The WHO's Region of the Americas accounted for most of the course enrollments, with more than 72.47% (138,503/191,130) enrollment across all languages. Other regions were more evenly distributed with less than 10% enrollment for each. A total of 32.43% (52,214/161,007) of users specified a professional affiliation by choosing from the 12 most common backgrounds in the OpenWHO user profiles. Before the COVID-19 pandemic, users were spread over the 11 distinct affiliations, with a small fraction of users identifying themselves as "Other." With the COVID-19 introductory course, the largest number of users selected "Other" (16,527/52,214, 31.65%), suggesting a large number of users who were not health professionals or academics. The top 10 countries with the most users across all languages were Argentina, Chile, Colombia, Ecuador, India, Mexico, Peru, Spain, the United Kingdom, and the United States. CONCLUSIONS: The online course has addressed a worldwide learning need by providing WHO's technical guidance packaged in simple formats for access and use. The learning material development was expedited to meet the onset of the epidemic. Initial data suggest that the various language versions of the course, in particular Spanish, have reached new user groups, fulfilling the platform's aim of providing learning everywhere to anyone that is interested. User surveys will be carried out to measure the real impact.

Learning interventions and training methods in health emergencies: A scoping review.

BACKGROUND: Keeping the health workforce and the public informed about the latest evolving health information during a health emergency is critical to preventing, detecting and responding to infectious disease outbreaks or other health emergencies. Having a well-informed, ready, willing, and skilled workforce and an informed public can help save lives, reduce diseases and suffering, and minimize socio-economic loss in affected communities and countries. Providing "just in time" support and opportunities for learning in health emergencies is much needed for capacity building. In this paper, 'learning intervention' refers to the provision of ad-hoc, focused, or personalized training sessions with the goal of preparing the health workers for emergencies or filling specific knowledge or skill gaps. We refer to 'training methods' as instructional design strategies used to teach someone the necessary knowledge and skills to perform a task. METHODS: We conducted a scoping review to map and better understand what learning interventions and training methods have been used in different types of health emergencies and by whom. Studies were identified using six databases (Pubmed/Medline, Embase, Hinari, WorldCat, CABI and Web of Science) and by consulting with experts. Characteristics of studies were mapped and displayed and major topic areas were identified. RESULTS: Of the 319 records that were included, contexts most frequently covered were COVID-19, disasters in general, Ebola and wars. Four prominent topic areas were identified: 1) Knowledge acquisition, 2) Emergency plans, 3) Impact of the learning intervention, and 4) Training methods. Much of the evidence was based on observational methods with few trials, which likely reflects the unique context of each health emergency. Evolution of methods was apparent, particularly in virtual learning. Learning during health emergencies appeared to improve knowledge, general management of the situation, quality of life of both trainers and affected population, satisfaction and clinical outcomes. CONCLUSION: This is the first scoping review to map the evidence, which serves as a first step in developing urgently needed global guidance to further improve the quality and reach of learning interventions and training methods in this context.

Health Care Workers' Motivations for Enrolling in Massive Open Online Courses During a Public Health Emergency: Descriptive Analysis.

Background: Massive open online courses (MOOCs) are increasingly used to educate health care workers during public health emergencies. In early 2020, the World Health Organization (WHO) developed a series of MOOCs for COVID-19, introducing the disease and strategies to control its outbreak, with 6 courses specifically targeting health care workers as learners. In 2020, Stanford University also launched a MOOC designed to deliver accurate and timely education on COVID-19, equipping health care workers across the globe to provide health care safely and effectively to patients with the novel infectious disease. Although the use of MOOCs for just-in-time training has expanded during the pandemic, evidence is limited regarding the factors motivating health care workers to enroll in and complete courses, particularly in low-income countries (LICs) and lower-middle-income countries (LMICs). Objective: This study seeks to gain insights on the characteristics and motivations of learners turning to MOOCs for just-in-time training, to provide evidence that can better inform MOOC design to meet the needs of health care workers. We examine data from learners in 1 Stanford University and 6 WHO COVID-19 courses to identify (1) the characteristics of health care workers completing the courses and (2) the factors motivating them to enroll. Methods: We analyze (1) course registration data of the 49,098 health care workers who completed the 7 focal courses and (2) survey responses from 6272 course completers. The survey asked respondents to rank their motivations for enrollment and share feedback about their learning experience. We use descriptive statistics to compare responses by health care profession and by World Bank country income classification. Results: Health care workers completed the focal courses from all regions of the world, with nearly one-third (14,159/49,098, 28.84%) practicing in LICs and LMICs. Survey data revealed a diverse range of professional roles among the learners, including physicians (2171/6272, 34.61%); nurses (1599/6272, 25.49%); and other health care professionals such as allied health professionals, community health workers, paramedics, and pharmacists (2502/6272, 39.89%). Across all health care professions, the primary motivation to enroll was for personal learning to improve clinical practice. Continuing education credit was also an important motivator, particularly for nonphysicians and learners in LICs and LMICs. Course cost (3423/6272, 54.58%) and certification (4238/6272, 67.57%) were also important to a majority of learners. Conclusions: Our results demonstrate that a diverse range of health care professionals accessed MOOCs for just-in-time training during a public health emergency. Although all health care workers were motivated to improve their clinical practice, different factors were influential across professions and locations. These factors should be considered in MOOC design to meet the needs of health care workers, particularly those in lower-resource settings where alternative avenues for training may be limited.

Online learning for WHO priority diseases with pandemic potential: evidence from existing courses and preparing for Disease X.

BACKGROUND: OpenWHO is the open-access learning platform of the World Health Organization (WHO) that provides online learning for health emergencies with essential health knowledge for emergencies. There is emphasis for courses on severe emerging diseases with epidemic and pandemic potential to help frontline health workers prevent, control and respond to infectious diseases. This research addresses the question of how the existing OpenWHO online courses on infectious disease were used in the countries of disease occurrence and how to prepare for disease X, a novel or unknown pathogen with pandemic potential. METHODS: OpenWHO collects self-declared demographic data from learners among which there is data on geographical location of learners. Data in infectious disease courses use on OpenWHO was collected and examined and additionally information languages used in the outbreak locations was collected. RESULTS: For most diseases in focus the online learning materials were used in countries with burden of disease. This suggests the learning material production needs to be targeted for outbreak and epidemic events. CONCLUSIONS: Findings inform the use of learning materials in disease outbreaks. Further, this use case data confirms learning providers need to add offerings in languages spoken in outbreak impacted areas.

Using Digital Tools to Train Health Emergencies Personnel in Fragile Contexts.

The Leadership in Emergencies learning programme, launched in 2019, was designed to strengthen the competencies of World Health Organization (WHO) and Member State staff in teamwork, decision-making and communication, key skills required to lead effectively in emergencies. While the programme was initially used to train 43 staff in a workshop setting, the COVID-19 pandemic required a new remote approach. An online learning environment was developed using a variety of digital tools including WHO's open learning platform, OpenWHO.org. The strategic use of these technologies enabled WHO to dramatically expand access to the programme for personnel responding to health emergencies in fragile contexts and increase participation among key groups that were previously underserved.

Scale Up Multilingualism in Health Emergency Learning: Developing an Automated Transcription and Translation Tool.

World Health Organization's (WHO) emergency learning platform OpenWHO provided by Hasso Plattner Institut (HPI) delivered online learning in real-time and in multiple languages during the COVID-19 pandemic. The challenge was to move from manual transcription and translation to automated to increase the speed and quantity of materials and languages available. TransPipe tool was introduced to facilitate this task. We describe the TransPipe development, analyze its functioning and report key results achieved. TransPipe successfully connects existing services and provides a suitable workflow to create and maintain video subtitles in different languages. By the end of 2022, the tool transcribed nearly 4,700 minutes of video content and translated 1,050,700 characters of video subtitles. Automated transcription and translation have enormous potential as a public health learning tool, allowing the near-simultaneous availability of video subtitles on OpenWHO in many languages, thus improving the usability of the learning materials in multiple languages for wider audiences.

Online capacity building for the health workforce: the case of the Integrated Disease Surveillance and Response for the African region.

The World Health Organisation (WHO) Regional Office for Africa (AFRO) has developed a comprehensive capacity devel-opment programme to support the successful implementation of the Integrated Disease Surveillance and Response 3rd edition Technical Guidelines (IDSR). As part of the learning program, a series of asyn-chronous online courses are offered on OpenWHO in English, French and Portuguese. This paper describes the use of five IDSR online courses and reports on feedback received from learners on Course 1 in the English series. An online learner survey was developed, and a descriptive analysis was conducted. This paper also reports on use related empirical metadata from the OpenWHO platform. Overall, learners (97%-n/N) of Course 1 IDSR English series indicated a positive perception toward their online learning experience because of the quality of course content, its organization, ease of use and relevance to their workplace needs. In addition, 88% (n/N) of learners reported that they had used their acquired knowledge at least sometimes and 54.4% (n/N) had shared their learning with others. Lastly, the quiz analyses showed an average of right answers of 78.97% for quiz 1 and 69.94% for quiz 2. Online learning is an essential component of a blended capacity development programme and provides cost effective, equitable and impactful learning. Learners who have a learning goal and find their needs met in courses tend to show more satisfaction and motivation to share their learning.

Insights on Public Health Professionals Non-technical Skills in an Emergency Response (Multi-Team System) Environment.

This paper provides practitioner and academic insights into the importance of examining non-technical skills in a multiteam system emergency response. The case of public health professionals is highlighted, illustrated with unique qualitative field data which focused upon the use of non-technical skills at a meso level of analysis. Results reflected the importance of context upon the multiteam system and highlighted seven non-technical skills used by public health professionals to support an effective response. Recommendations for future research and implications for practice are noted for this hard to access professional group, located within emerging advances in the scientific inquiry of complex and increasingly evident, multi-team systems.

Superusers of Self-Paced Online Learning on OpenWHO.

INTRODUCTION: OpenWHO provides open-access, online, free and real-time learning responses to health emergencies, which includes capacitating healthcare providers, first liners, medical students and even the general public. During the pandemic and to date, an additional 40 courses for COVID-19 response have led to a massive increase in the number of learners and a change in user's trends. This paper presents initial findings on enrollment trends, use and completion rates of health emergency courses offered on OpenWHO. METHODS: The enrolment data statistics were drawn from OpenWHO's built-in reporting system, which tracks learners' enrolments, completion rates, demographics and other key course-related data, This information was collected from the beginning of the OpenWHO launch in 2017 up until October 2021. RESULTS: Average course completion rate on OpenWHO including all courses and languages was equal to 45.9%. Nearly half (46.4%) of all OpenWHO learners have enrolled in at least 2 courses and 71 000 superusers have completed at least 10 courses on the platform. CONCLUSION: WHO's learning platform during the pandemic registered record high completion rates and repeat learners enrollment. This highlights the massive impact of the OpenWHO online learning platform for health emergencies and the tangible knowledge transfer and access to health literacy.

Multilingual Approach to COVID-19 Online Learning Response on OpenWHO.org.

In pursuit of equitable access to emergency-related knowledge, the World Health Organization (WHO) translates COVID-19 and other infectious disease courses into multiple languages on its open-access online learning platform OpenWHO.org. Languages spoken by vulnerable or underserved populations in low- and middle-income countries and in outbreak-prone and affected areas are prioritized. Accessing learning in preferred languages enhances uptake and comprehension. In this study, we assess and compare the initial enrollment levels and global reach of these multilingual courses. On average, OpenWHO's 38 COVID-19 courses have each been translated into 4.8 languages. The platform hosts courses in 55 different languages with 10.4 million words translated. The findings identify which available languages were most utilized for COVID-19 learning to inform course production and outreach strategies. Languages were used differently across geographic regions, calling for localized learning offerings. A streamlined multilingual publishing scheme, ensuring quick and effective delivery of diverse languages, is critical to achieving greater equity of access to knowledge.

Could Linguistic Complexity Be Automatically Evaluated? A Multilingual Study on WHO's Emergency Learning Platform.

The ability of assessing any type of linguistic complexity of any given contents could potentially improve knowledge reproduction, especially tacit knowledge which can be expensive during a pandemic. In this paper, we develop a simple and crosslinguistic model of complexity which considers formal accounts on the study of linguistic systems, but can be easily implemented by non-linguists' groups, e.g., communication experts and policymakers. To test our model, we conduct a study on a corpus extracted from the World Health Organization (WHO)'s emergency learning platform in 6 languages. Data extracted from open-access encyclopaedic entries act as control groups. The results show that the measurements adopted signal a trend for a minimization of complexity and can be exploited as features for (automatic) text classification.

Digitalizing WHO's Health Emergency Leadership Training During the Pandemic.

The WHO Health Emergencies Programme combined online tools with adult learning techniques to develop an innovative online leadership course. A combination of self-paced learning and online classes were used to deliver skills-based leadership training tailored to public health staff working in emergency response roles. Although using an online approach was considered a temporary solution to counter travel restrictions during the pandemic, the advantages have challenged preconceptions that effective learning, networking and peer exchange for leadership can only be achieved through face-to-face learning.

Evaluating Complexity of Digital Learning in a Multilingual Context: A Cross-Linguistic Study on WHO's Emergency Learning Platform.

Reproduction of knowledge, especially tacit knowledge can be expensive during a pandemic. One of the most common causes is the reduced information accessibility during the translation process. Having the ability to assess the linguistic complexity of any given contents could potentially improve knowledge reproduction. Authors conduct two cross-linguistic studies on the World Health Organization (WHO)'s emergency learning platform to assess the linguistic complexity of two online courses in 10 languages. Morpho-syntactically annotated treebanks, unannotated materials from Wikipedia and language-specific corpora are set as control groups. Preliminary findings reveal a clear reduced complexity of learning contents in the most candidate languages while retaining the maximum amount of information. Creating a baseline study on low-resourced languages on the learning genre could be potentially useful for measuring impact of normative products at country and local level.

Delivering WHO's Life-Saving Information in Real-Time During a Pandemic Through an Online Learning Platform: Evidence from Global Use.

Accessibility, multilingualism and real-time knowledge transfer are the tenets of the OpenWHO learning platform. A descriptive analysis was conducted using anonymised statistical datasets retrieved from the OpenWHO reporting system to understand global use trends. When examining OpenWHO use in the areas with the highest burden of COVID-19 cases, a strong correlation was identified between the burden of COVID-19 and OpenWHO user activity. Further, health professionals are the top users of COVID-19 materials. The massive demand for OpenWHO courses confirms the need for real-time learning materials in accessible, multilingual formats during a pandemic. Learning is a global public good and is an essential component of the response to health emergencies.

Changes in Users Trends Before and During the COVID-19 Pandemic on WHO's Online Learning Platform.

OpenWHO provides open access, online, free and real time learning responses to health emergencies. Before the pandemic, courses on 18 diseases were provided. The increase to 38 courses in response to COVID-19 have led to a massive increase in the number of new learners. As a result, the COVID-19 pandemic affected learners' trends. This paper presents initial findings of changes perceived in the use and user groups' attendance to the World Health Organization's (WHO) health emergency learning platform OpenWHO. Enrolment statistics were based on data collected in December 2019 and March 2021. A descriptive analysis was conducted to explore changes in the usage pattern of the platform. Several user characteristics shifted between before and during the pandemic. More women, younger and older learners joined the learning during the pandemic. Public health education leaned toward a more equitable reach including previously underrepresented groups.

Learning From a Massive Open Online COVID-19 Vaccination Training Experience: Survey Study.

BACKGROUND: To prepare key stakeholders for the global COVID-19 vaccination rollout, the World Health Organization and partners developed online vaccination training packages. The online course was launched in December 2020 on the OpenWHO learning platform. This paper presents the findings of an evaluation of this course. OBJECTIVE: The aim of this evaluation was to provide insights into user experiences and challenges, measure the impact of the course in terms of knowledge gained, and anticipate potential interest in future online vaccination courses. METHODS: The primary source of data was the anonymized information on course participants, enrollment, completion, and scores from the OpenWHO platform's statistical data and metric reporting system. Data from the OpenWHO platform were analyzed from the opening of the courses in mid-December 2020 to mid-April 2021. In addition, a learner feedback survey was sent by email to all course participants to complete within a 3-week period (March 19 to April 9, 2021). The survey was designed to determine the perceived strengths and weaknesses of the training packages and to understand barriers to access. RESULTS: During the study period, 53,593 learners enrolled in the course. Of them, 30,034 (56.0%) completed the course, which is substantially higher than the industry benchmark of 5%-10% for a massive open online course (MOOC). Overall, learners averaged 76.5% on the prequiz compared to 85% on the postquiz, resulting in an increase in average score of 9%. A total of 2019 learners from the course participated in the survey. Nearly 98% (n=1647 fully agree, n=308 somewhat agree; N=1986 survey respondents excluding missing values) of respondents fully or somewhat agreed that they had more confidence in their ability to support COVID-19 vaccination following completion of this course. CONCLUSIONS: The online vaccine training was well received by the target audience, with a measurable impact on knowledge gained. The key benefits of online training were the convenience, self-paced nature, access to downloadable material, and ability to replay material, as well as an increased ability to concentrate. Online training was identified as a timely, cost-effective way of delivering essential training to a large number of people to prepare for the COVID-19 vaccination rollout.

One Year of Pandemic Learning Response: Benefits of Massive Online Delivery of the World Health Organization's Technical Guidance.

The World Health Organization (WHO) launched the first web-based learning course on COVID-19 on January 26, 2020, four days before the director general of the WHO declared a public health emergency of international concern. The WHO is expanding access to web-based learning for COVID-19 through its open-learning platform for health emergencies, OpenWHO. Throughout the pandemic, OpenWHO has continued to publish learning offerings based on the WHO's emerging evidence-based knowledge for managing the COVID-19 pandemic. This study presents the various findings derived from the analysis of the performance of the OpenWHO platform during the pandemic, along with the core benefits of massive web-based learning formats.

An Analysis of the Growth in Uptake of OpenWHO's Online Learning Resources for COVID-19.

From 26 January - 21 April 2020, 9 online courses for COVID-19 were published on OpenWHO.org. The courses are available in 18 different languages, totalling 53 learning resources and more than 1.5 million course enrolments. This paper analyses key trends in the growth of enrolments in these resources. The number of enrolments increased significantly between 26 January and 21 April, with distinct spikes in growth preceded by important global milestones in the timeline of the outbreak. The surge in users demonstrates that the platform is serving as a source of digitized learning for COVID-19, helping meet the broad demand for outbreak-related information.

Identifying and Assessing Competencies for Staff Working in Public Health Emergencies.

Competency-based learning involves identifying the knowledge, skills and attributes required for carrying out workplace roles. In 2018, the World Health Organization's (WHO) Health Emergencies Programme (WHE) introduced a competency framework with competency-based assessment (CBA) methodology. The CBA focuses on behavioural indicators that participants demonstrate and training faculty observe during training events. This article introduces the CBA of WHE. It is essential to use the framework to design learning programmes and provide the basis to appraise and manage WHE personnel and team performance.