Lessons learnt from the applying the Centers for Disease Control and Prevention (CDC) evaluation framework to the measles incident management system response, USA, 2020-2021.

The functionality and performance of public health programmes at all levels of government play a critical role in preventing, detecting, mitigating and responding to public health threats, including infectious disease outbreaks. Multiple and concurrent outbreaks in recent years, such as COVID-19, Ebola and Zika, have highlighted the importance of documenting lessons learnt from public health responses of national and global agencies. In February 2020, the US Centers for Disease Control and Prevention (CDC) Center for Global Health (CGH) activated the Measles Incident Management System (MIMS) to accelerate the ability to detect, mitigate and respond to measles outbreaks globally and advance progress towards regional measles elimination goals. The activation was triggered by a global resurgence in reported measles cases during 2018-2019 and supported emergency response activities conducted by partner organisations and countries. MIMS leadership decided early in the response to form an evaluation team to design and implement an evaluation approach for producing real-time data to document progress of response activities and inform timely decision-making. In this manuscript, we describe how establishing an evaluation unit within MIMS, and engaging MIMS leadership and subject matter experts in the evaluation activities, was critical to monitor progress and document lessons learnt to inform decision making. We also explain the CDC's Framework for Evaluation in Public Health Practice applied to evaluate the dynamic events throughout the MIMS response. Evaluators supporting emergency response should use a flexible framework that can be adaptable in dynamic contexts and document response activities in real-time.

CDC's COVID-19 International Vaccine Implementation and Evaluation Program and Lessons from Earlier Vaccine Introductions.

The US Centers for Disease Control and Prevention (CDC) supports international partners in introducing vaccines, including those against SARS-CoV-2 virus. CDC contributes to the development of global technical tools, guidance, and policy for COVID-19 vaccination and has established its COVID-19 International Vaccine Implementation and Evaluation (CIVIE) program. CIVIE supports ministries of health and their partner organizations in developing or strengthening their national capacities for the planning, implementation, and evaluation of COVID-19 vaccination programs. CIVIE's 7 priority areas for country-specific technical assistance are vaccine policy development, program planning, vaccine confidence and demand, data management and use, workforce development, vaccine safety, and evaluation. We discuss CDC's work on global COVID-19 vaccine implementation, including priorities, challenges, opportunities, and applicable lessons learned from prior experiences with Ebola, influenza, and meningococcal serogroup A conjugate vaccine introductions.

An assessment of the contribution of National Stop Transmission of Polio Program to Nigeria's Immunization Program.

Introduction: In July 2012, the National Stop Transmission of Polio (NSTOP) program was established to support the Government of Nigeria in interrupting transmission of poliovirus and strengthen routine immunization (RI). NSTOP has approximately 300 staff members with the majority based at the Local Government Area (LGA) level in northern Nigeria. Methods: An internal assessment of NSTOP was conducted from November 2015 to February 2016 to document the program´s contribution to Nigeria´s immunization program and plan future NSTOP engagement. A mixed methods design was used, with data gathered from health facility, LGA, state, and national levels, through structured surveys, interviews, focus group discussions, and review of program records. Survey and expenditure data were summarized by frequency and trends over time, while interview and focus group data were analyzed qualitatively for key themes. Results: The majority of the 111 non-NSTOP LGA respondents reported that NSTOP officers supported polio campaigns (100%) and supervised RI sessions (99.1%). Out of 181 respondents at health facility level, the majority reported that NSTOP trainings improved their knowledge (83.3%) and skills (76.2%) on RI, and NSTOP officers regularly supervised their RI sessions (96.7%). Most respondents reported that there would be a negative impact on immunization activities if NSTOP officers were withdrawn. Conclusion: Future implementation of NSTOP should be realigned to (a) give highest priority to mentoring LGA staff to build institutional capacity, (b) ensure increased capacity translates to improved provision of RI services, and (c) improve routine review of program monitoring data to assess progress in both polio and RI programs.

Use of a district health information system 2 routine immunization dashboard for immunization program monitoring and decision making, Kano State, Nigeria.

Introduction: a district health information system 2 tool with a customized routine immunization (RI) module and indicator dashboard was introduced in Kano State, Nigeria, in November 2014 to improve data management and analysis of RI services. We assessed the use of the module for program monitoring and decision-making, as well as the enabling factors and barriers to data collection and use. Methods: a mixed-methods approach was used to assess user experience with the RI data module and dashboard, including 1) a semi-structured survey questionnaire administered at 60 health facilities administering vaccinations and 2) focus group discussions and 16 in-depth interviews conducted with immunization program staff members at the local government area (LGA) and state levels. Results: in health facilities, a RI monitoring chart was used to review progress toward meeting vaccination coverage targets. At the LGA, staff members used RI dashboard data to prioritize health facilities for additional support. At the State level, immunization program staff members use RI data to make policy decisions. They viewed the provision of real-time data through the RI dashboard as a "game changer". Use of immunization data is facilitated through review meetings and supportive supervision visits. Barriers to data use among LGA staff members included inadequate understanding of the data collection tools and computer illiteracy. Conclusion: the routine immunization data dashboard facilitated access to and use of data for decision-making at the LGA, State and national levels, however, use at the health facility level remains limited. Ongoing data review meetings and training on computer skills and data collection tools are recommended.

Implementing the routine immunisation data module and dashboard of DHIS2 in Nigeria, 2014-2019.

In 2010, Nigeria adopted the use of web-based software District Health Information System, V.2 (DHIS2) as the platform for the National Health Management Information System. The platform supports real-time data reporting and promotes government ownership and accountability. To strengthen its routine immunisation (RI) component, the US Centers for Disease Control and Prevention (CDC) through its implementing partner, the African Field Epidemiology Network-National Stop Transmission of Polio, in collaboration with the Government of Nigeria, developed the RI module and dashboard and piloted it in Kano state in 2014. The module was scaled up nationally over the next 4 years with funding from the Bill & Melinda Gates Foundation and CDC. One implementation officer was deployed per state for 2 years to support operations. Over 60 000 RI healthcare workers were trained on data collection, entry and interpretation and each local immunisation officer in the 774 local government areas (LGAs) received a laptop and stock of RI paper data tools. Templates for national-level and state-level RI bulletins and LGA quarterly performance tools were developed to promote real-time data use for feedback and decision making, and enhance the performance of RI services. By December 2017, the DHIS2 RI module had been rolled out in all 36 states and the Federal Capital Territory, and all states now report their RI data through the RI Module. All states identified at least one government DHIS2 focal person for oversight of the system's reporting and management operations. Government officials routinely collect RI data and use them to improve RI vaccination coverage. This article describes the implementation process-including planning and implementation activities, achievements, lessons learnt, challenges and innovative solutions-and reports the achievements in improving timeliness and completeness rates.

Strengths, pitfalls, and lessons learned in implementing electronic collection of childhood vaccination data in Zambia: The SmartCare experience.

BACKGROUND: Despite widespread interest in computerized vaccination information systems, evaluation of the data quality in these systems and their acceptability to frontline healthcare workers in low and middle-income countries aren't well addressed in the literature. OBJECTIVES: Evaluation of vaccination data quality and facility-level staff perspectives on the strengths and challenges of a vaccination data module in a widely used electronic health record (EHR) system in Zambia. METHODS: After a desk review of data from two provinces, a cross-sectional mixed methods study was designed, including quantitative analysis of data quality and qualitative analysis of the module's acceptability to facility staff, using the Information System Success model as the framework for evaluation of system quality, service quality, and information quality. Data were collected from 10 purposively sampled health facilities. RESULTS: There was low current use of the vaccination module by facilities in the study area (2%). Daily power outages presented a practical challenge. Staff who had used previous EHRs had concerns about sustainability. SYSTEM QUALITY: While the module was user-friendly, there were concerns about EHR compatibility with vaccination workflow and outreach settings, where vaccines are commonly administered to older children. SERVICE QUALITY: The module was viewed as dependable; perceptions were influenced by computer literacy. INFORMATION QUALITY: The database contained incomplete and incongruous data. Staff perceived data as accurate but incomplete; easy access to data was a strength. CONCLUSIONS: Potential benefits of the vaccination module were frequently unrealized due to infrastructure, workflow, and data flow challenges that resulted in low module use and poor information quality. Elements to optimize vaccination information system implementation could include robust engagement of facility-level staff in system design, system suitability to the vaccination setting and workflow, and comprehensive planning for data flow, sustainability, data monitoring and feedback. Adaptability to the outreach setting might be increasingly important as vaccination schedules extend past infancy.