Can we design the next generation of digital health communication programs by leveraging the power of artificial intelligence to segment target audiences, bolster impact and deliver differentiated services? A machine learning analysis of survey data from rural India.

OBJECTIVES: Direct to beneficiary (D2B) mobile health communication programmes have been used to provide reproductive, maternal, neonatal and child health information to women and their families in a number of countries globally. Programmes to date have provided the same content, at the same frequency, using the same channel to large beneficiary populations. This manuscript presents a proof of concept approach that uses machine learning to segment populations of women with access to phones and their husbands into distinct clusters to support differential digital programme design and delivery. SETTING: Data used in this study were drawn from cross-sectional survey conducted in four districts of Madhya Pradesh, India. PARTICIPANTS: Study participant included pregnant women with access to a phone (n=5095) and their husbands (n=3842) RESULTS: We used an iterative process involving K-Means clustering and Lasso regression to segment couples into three distinct clusters. Cluster 1 (n=1408) tended to be poorer, less educated men and women, with low levels of digital access and skills. Cluster 2 (n=666) had a mid-level of digital access and skills among men but not women. Cluster 3 (n=1410) had high digital access and skill among men and moderate access and skills among women. Exposure to the D2B programme 'Kilkari' showed the greatest difference in Cluster 2, including an 8% difference in use of reversible modern contraceptives, 7% in child immunisation at 10 weeks, 3% in child immunisation at 9 months and 4% in the timeliness of immunisation at 10 weeks and 9 months. CONCLUSIONS: Findings suggest that segmenting populations into distinct clusters for differentiated programme design and delivery may serve to improve reach and impact. TRIAL REGISTRATION NUMBER: NCT03576157.

Optimising the reach of mobile health messaging programmes: an analysis of system generated data for the Kilkari programme across 13 states in India.

Kilkari is an outbound service that makes weekly, stage-based, prerecorded calls about reproductive, maternal, neonatal and child health directly to families' mobile phones, starting from the second trimester of pregnancy and until the child is 1 year old. Since its initiation in 2012-2013, Kilkari has scaled to 13 states across India. In this analysis article, we explored the subscriber's journey from entry to programme to engagement with calls. Data sources included call data records and household survey data from the 2015 National Family Health Survey. In 2018, of the 13.6 million records received by MOTECH, the technology platform that powers Kilkari, 9.5 million (~70%) were rejected and 4.1 million new subscribers were created. Overall, 21% of pregnant women across 13 states were covered by the programme in 2018, with West Bengal and Himachal Pradesh reaching a coverage of over 50%. Among new subscriptions in 2018, 63% were subscribed during pregnancy and 37% after childbirth. Of these, over 80% were ever reached by Kilkari calls and 39% retained in the programme. The main causes for deactivation of subscribers from the system were low listenership and calls going unanswered for six continuous weeks. Globally, Kilkari is the largest maternal mobile messaging programme of its kind in terms of number of subscribers but the coverage among pregnant women remains low. While call reach appears to be on the higher side, subscriber retention is low; this highlights broader challenges with providing mobile health services at scale across India.

The impact of a direct to beneficiary mobile communication program on reproductive and child health outcomes: a randomised controlled trial in India.

BACKGROUND: Direct-to-beneficiary communication mobile programmes are among the few examples of digital health programmes to have scaled widely in low-resource settings. Yet, evidence on their impact at scale is limited. This study aims to assess whether exposure to mobile health information calls during pregnancy and postpartum improved infant feeding and family planning practices. METHODS: We conducted an individually randomised controlled trial in four districts of Madhya Pradesh, India. Study participants included Hindi speaking women 4-7 months pregnant (n=5095) with access to a mobile phone and their husbands (n=3842). Women were randomised to either an intervention group where they received up to 72 Kilkari messages or a control group where they received none. Intention-to-treat (ITT) and instrumental variable (IV) analyses are presented. RESULTS: An average of 65% of the 2695 women randomised to receive Kilkari listened to ≥50% of the cumulative content of calls answered. Kilkari was not observed to have a significant impact on the primary outcome of exclusive breast feeding (ITT, relative risk (RR): 1.04, 95% CI 0.88 to 1.23, p=0.64; IV, RR: 1.10, 95% CI 0.67 to 1.81, p=0.71). Across study arms, Kilkari was associated with a 3.7% higher use of modern reversible contraceptives (RR: 1.12, 95% CI 1.03 to 1.21, p=0.007), and a 2.0% lower proportion of men or women sterilised since the birth of the child (RR: 0.85, 95% CI 0.74 to 0.97, p=0.016). Higher reversible method use was driven by increases in condom use and greatest among those women exposed to Kilkari with any male child (9.9% increase), in the poorest socioeconomic strata (15.8% increase), and in disadvantaged castes (12.0% increase). Immunisation at 10 weeks was higher among the children of Kilkari listeners (2.8% higher; RR: 1.03, 95% CI 1.00 to 1.06, p=0.048). Significant differences were not observed for other maternal, newborn and child health outcomes assessed. CONCLUSION: Study findings provide evidence to date on the effectiveness of the largest mobile health messaging programme in the world. TRIAL REGISTRATION NUMBER: Trial registration clinicaltrials.gov; ID 90075552, NCT03576157.

Examining the reach and exposure of a mobile phone-based training programme for frontline health workers (ASHAs) in 13 states across India.

Mobile phones are increasingly used to facilitate in-service training for frontline health workers (FLHWs). Mobile learning (mLearning) programmes have the potential to provide FLHWs with high quality, inexpensive, standardised learning at scale, and at the time and location of their choosing. However, further research is needed into FLHW engagement with mLearning content at scale, a factor which could influence knowledge and service delivery. Mobile Academy is an interactive voice response training course for FLHWs in India, which aims to improve interpersonal communication skills and refresh knowledge of preventative reproductive, maternal, neonatal and child health. FLHWs dial in to an audio course consisting of 11 chapters, each with a 4-question true/false quiz, resulting in a cumulative pass/fail score. In this paper, we analyse call data records from the national version of Mobile Academy to explore coverage, user engagement and completion. Over 158 596 Accredited Social Health Activists (ASHAs) initiated the national version, while 111 994 initiated the course on state-based platforms. Together, this represents 41% of the estimated total number of ASHAs registered in the government database across 13 states. Of those who initiated the national version, 81% completed it; and of those, over 99% passed. The initiation and completion rates varied by state, with Rajasthan having the highest initiation rate. Many ASHAs made multiple calls in the afternoons and evenings but called in for longer durations earlier in the day. Findings from this analysis provide important insights into the differential reach and uptake of the programme across states.

Assessing exposure to Kilkari: a big data analysis of a large maternal mobile messaging service across 13 states in India.

The Kilkari programme is being implemented by the Government of India in 13 states. Designed by BBC Media Action and scaled in collaboration with the Ministry of Health and Family Welfare from January 2016, Kilkari had provided mobile health information to over 10 million subscribers by the time BBC Media Action transitioned the service to the government in April 2019. Despite the reach of Kilkari in terms of the absolute number of subscribers, no longitudinal analysis of subscriber exposure to health information content over time has been conducted, which may underpin effectiveness and changes in health outcomes. In this analysis, we draw from call data records to explore exposure to the Kilkari programme in India for the 2018 cohort of subscribers. We start by assessing the timing of the first successful call answered by subscribers on entry to the programme during pregnancy or postpartum, and then assess call volume, delivery, answering and listening rates over time. Findings suggest that over half of subscribers answer their first call after childbirth, with the remaining starting in the pregnancy period. The system handles upwards of 1.2 million calls per day on average. On average, 50% of calls are picked up on the first call attempt, 76% by the third and 99.5% by the ninth call attempt. Among calls picked up, over 48% were listened to for at least 50% of the total content duration and 43% were listened to for at least 75%. This is the first analysis of its kind of a maternal mobile messaging programme at scale in India. Study analyses suggest that multiple call attempts may be required to reach subscribers. However, once answered, subscribers tend to listen the majority of the call-a figure consistent across states, over time, and by health content area.

SMS feedback system as a quality assurance mechanism: experience from a household survey in rural India.

The increasing use of digital health solutions to support data capture both as part of routine delivery of health services and through special surveys presents unique opportunities to enhance quality assurance measures. This study aims to demonstrate the feasibility and acceptability of using back-end data analytics and machine learning to identify impediments in data quality and feedback issues requiring follow-up to field teams using automated short messaging service (SMS) text messages. Data were collected as part of a postpartum women's survey (n=5095) in four districts of Madhya Pradesh, India, from October 2019 to February 2020. SMSs on common errors found in the data were sent to supervisors and coordinators. Before/after differences in time to correction of errors were examined, and qualitative interviews conducted with supervisors, coordinators, and enumerators. Study activities resulted in declines in the average number of errors per week after the implementation of automated feedback loops. Supervisors and coordinators found the direct format, complete information, and automated nature of feedback convenient to work with and valued the more rapid notification of errors. However, coordinators and supervisors reported preferring group WhatsApp messages as compared with individual SMSs to each supervisor/coordinator. In contrast, enumerators preferred the SMS system over in-person group meetings where data quality impediments were discussed. This study demonstrates that automated SMS feedback loops can be used to enhance survey data quality at minimal cost. Testing is needed among data capture applications in use by frontline health workers in India and elsewhere globally.

Does exposure to health information through mobile phones increase immunisation knowledge, completeness and timeliness in rural India?

INTRODUCTION: Immunisation plays a vital role in reducing child mortality and morbidity against preventable diseases. As part of a randomised controlled trial in rural Madhya Pradesh, India to assess the impact of Kilkari, a maternal messaging programme, we explored determinants of parental immunisation knowledge and immunisation practice (completeness and timeliness) for children 0-12 months of age from four districts in Madhya Pradesh. METHODS: Data were drawn from a cross-sectional survey of women (n=4423) with access to a mobile phone and their spouses (n=3781). Parental knowledge about immunisation and their child's receipt of vaccines, including timeliness and completeness, was assessed using self-reports and vaccination cards. Ordered logistic regressions were used to analyse the factors associated with parental immunisation knowledge. A Heckman two-stage probit model was used to analyse completeness and timeliness of immunisation after correcting for selection bias from being able to produce the immunisation card. RESULTS: One-third (33%) of women and men knew the timing for the start of vaccinations, diseases linked to immunisations and the benefits of Vitamin-A. Less than half of children had received the basic package of 8 vaccines (47%) and the comprehensive package of 19 vaccines (44%). Wealth was the most significant determinant of men's knowledge and of the child receiving complete and timely immunisation for both basic and comprehensive packages. Exposure to Kilkari content on immunisation was significantly associated with an increase in men's knowledge (but not women's) about child immunisation (OR: 1.23, 95% CI 1.02 to1.48) and an increase in the timeliness of the child receiving vaccination at birth (Probit coefficient: 0.08, 95% CI 0.08 to 0.24). CONCLUSION: Gaps in complete and timely immunisation for infants persist in rural India. Mobile messaging programmes, supported by mass media messages, may provide one important source for bolstering awareness, uptake and timeliness of immunisation services. TRIAL REGISTRATION NUMBER: NCT03576157.

Can health information through mobile phones close the divide in health behaviours among the marginalised? An equity analysis of Kilkari in Madhya Pradesh, India.

Kilkari is one of the largest maternal mobile messaging programmes in the world. It makes weekly prerecorded calls to new and expectant mothers and their families from the fourth month of pregnancy until 1-year post partum. The programme delivers reproductive, maternal, neonatal and child health information directly to subscribers' phones. However, little is known about the reach of Kilkari among different subgroups in the population, or the differentiated benefits of the programme among these subgroups. In this analysis, we assess differentials in eligibility, enrolment, reach, exposure and impact across well-known proxies of socioeconomic position-that is, education, caste and wealth. Data are drawn from a randomised controlled trial (RCT) in Madhya Pradesh, India, including call data records from Kilkari subscribers in the RCT intervention arm, and the National Family Health Survey-4, 2015. The analysis identifies that disparities in household phone ownership and women's access to phones create inequities in the population eligible to receive Kilkari, and that among enrolled Kilkari subscribers, marginalised caste groups and those without education are under-represented. An analysis of who is left behind by such interventions and how to reach those groups through alternative communication channels and platforms should be undertaken at the intervention design phase to set reasonable expectations of impact. Results suggest that exposure to Kilkari has improved levels of some health behaviours across marginalised groups but has not completely closed pre-existing gaps in indicators such as wealth and education.

Does having a mobile phone matter? Linking phone access among women to health in India: An exploratory analysis of the National Family Health Survey.

BACKGROUND: The disruptive potential of mobile phones in catalyzing development is increasingly being recognized. However, numerous gaps remain in access to phones and their influence on health care utilization. In this cross-sectional study from India, we assess the gaps in women's access to phones, their influencing factors, and their influence on health care utilization. METHODS: Data drawn from the 2015 National Family Health Survey (NFHS) in India included a national sample of 45,231 women with data on phone access. Survey design weighted estimates of household phone ownership and women's access among different population sub-groups are presented. Multilevel logistic models explored the association of phone access with a wide range of maternal and child health indicators. Blinder-Oaxaca (BO) decomposition is used to decompose the gaps between women with and without phone access in health care utilization into components explained by background characteristics influencing phone access (endowments) and unexplained components (coefficients), potentially attributable to phone access itself. FINDINGS: Phone ownership at the household level was 92·8% (95% CI: 92·6-93·0%), with rural ownership at 91·1% (90·8-91·4%) and urban at 97.1% (96·7-97·3%). Women's access to phones was 47·8% (46·7-48·8%); 41·6% in rural areas (40·5-42·6%) and 62·7% (60·4-64·8%) in urban. Phone access in urban areas was positively associated with skilled birth attendance, postnatal care and use of modern contraceptives and negatively associated with early antenatal care. Phone access was not associated with improvements in utilization indicators in rural settings. Phone access (coefficient components) explained large gaps in the use of modern contraceptives, moderate gaps in postnatal care and early antenatal care, and smaller differences in the use of skilled birth attendance and immunization. For full antenatal car, phone access was associated with reducing gaps in utilization. INTERPRETATION: Women of reproductive age have significantly lower phone access use than the households they belong to and marginalized women have the least phone access. Existing phone access for rural women did not improve their health care utilization but was associated with greater utilization for urban women. Without addressing these biases, digital health programs may be at risk of worsening existing health inequities.

Does women's mobile phone ownership matter for health? Evidence from 15 countries.

Mobile phones have the potential to increase access to health information, improve patient-provider communication, and influence the content and quality of health services received. Evidence on the gender gap in ownership of mobile phones is limited, and efforts to link phone ownership among women to care-seeking and practices for reproductive maternal newborn and child health (RMNCH) have yet to be made. This analysis aims to assess household and women's access to phones and its effects on RMNCH health outcomes in 15 countries for which Demographic and Health Surveys data on phone ownership are available. Multilevel logistic regression models were used to explore factors associated with women's phone ownership, along with the association of phone ownership to a wide range of RMNCH indicators. Study findings suggest that (1) gender gaps in mobile phone ownership vary, but they can be substantial, with less than half of women owning mobile phones in several countries; (2) the gender gap in phone ownership is larger for rural and poorer women; (3) women's phone ownership is generally associated with better RMNCH indicators; (4) among women phone owners, utilisation of RMNCH care-seeking and practices differs based on their income status; and (5) more could be done to unleash the potential of mobile phones on women's health if data gaps and varied metrics are addressed. Findings reinforce the notion that without addressing the gender gap in phone ownership, digital health programmes may be at risk of worsening existing health inequities.

Use of Big Data and Machine Learning Methods in the Monitoring and Evaluation of Digital Health Programs in India: An Exploratory Protocol.

BACKGROUND: Digital health programs, which encompass the subsectors of health information technology, mobile health, electronic health, telehealth, and telemedicine, have the potential to generate "big data." OBJECTIVE: Our aim is to evaluate two digital health programs in India-the maternal mobile messaging service (Kilkari) and the mobile training resource for frontline health workers (Mobile Academy). We illustrate possible applications of machine learning for public health practitioners that can be applied to generate evidence on program effectiveness and improve implementation. Kilkari is an outbound service that delivers weekly gestational age-appropriate audio messages about pregnancy, childbirth, and childcare directly to families on their mobile phones, starting from the second trimester of pregnancy until the child is one year old. Mobile Academy is an Interactive Voice Response audio training course for accredited social health activists (ASHAs) in India. METHODS: Study participants include pregnant and postpartum women (Kilkari) as well as frontline health workers (Mobile Academy) across 13 states in India. Data elements are drawn from system-generated databases used in the routine implementation of programs to provide users with health information. We explain the structure and elements of the extracted data and the proposed process for their linkage. We then outline the various steps to be undertaken to evaluate and select final algorithms for identifying gaps in data quality, poor user performance, predictors for call receipt, user listening levels, and linkages between early listening and continued engagement. RESULTS: The project has obtained the necessary approvals for the use of data in accordance with global standards for handling personal data. The results are expected to be published in August/September 2019. CONCLUSIONS: Rigorous evaluations of digital health programs are limited, and few have included applications of machine learning. By describing the steps to be undertaken in the application of machine learning approaches to the analysis of routine system-generated data, we aim to demystify the use of machine learning not only in evaluating digital health education programs but in improving their performance. Where articles on analysis offer an explanation of the final model selected, here we aim to emphasize the process, thereby illustrating to program implementors and evaluators with limited exposure to machine learning its relevance and potential use within the context of broader program implementation and evaluation. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/11456.