Analysing 3429 digital supervisory interactions between Community Health Workers in Uganda and Kenya: the development, testing and validation of an open access predictive machine learning web app.

BACKGROUND: Despite the growth in mobile technologies (mHealth) to support Community Health Worker (CHW) supervision, the nature of mHealth-facilitated supervision remains underexplored. One strategy to support supervision at scale could be artificial intelligence (AI) modalities, including machine learning. We developed an open access, machine learning web application (CHWsupervisor) to predictively code instant messages exchanged between CHWs based on supervisory interaction codes. We document the development and validation of the web app and report its predictive accuracy. METHODS: CHWsupervisor was developed using 2187 instant messages exchanged between CHWs and their supervisors in Uganda. The app was then validated on 1242 instant messages from a separate digital CHW supervisory network in Kenya. All messages from the training and validation data sets were manually coded by two independent human coders. The predictive performance of CHWsupervisor was determined by comparing the primary supervisory codes assigned by the web app, against those assigned by the human coders and calculating observed percentage agreement and Cohen's kappa coefficients. RESULTS: Human inter-coder reliability for the primary supervisory category of messages across the training and validation datasets was 'substantial' to 'almost perfect', as suggested by observed percentage agreements of 88-95% and Cohen's kappa values of 0.7-0.91. In comparison to the human coders, the predictive accuracy of the CHWsupervisor web app was 'moderate', suggested by observed percentage agreements of 73-78% and Cohen's kappa values of 0.51-0.56. CONCLUSIONS: Augmenting human coding is challenging because of the complexity of supervisory exchanges, which often require nuanced interpretation. A realistic understanding of the potential of machine learning approaches should be kept in mind by practitioners, as although they hold promise, supportive supervision still requires a level of human expertise. Scaling-up digital CHW supervision may therefore prove challenging. TRIAL REGISTRATION: This was not a clinical trial and was therefore not registered as such.

Participatory approaches, local stakeholders and cultural relevance facilitate an impactful community-based project in Uganda.

Sanitation is a major global challenge that is often addressed at national and international levels, while community opinions and beliefs are neglected. To promote water, sanitation and hygiene (WASH) we organized a cross-cultural knowledge exchange workshop to assess participatory methods for engaging local stakeholders. The workshop included 22 participants from all sectors of society. Practical solutions to sanitation challenges were identified and later shared with a local community. Qualitative and quantitative analyses were used to assess impact and showed participatory methods were highly valued to encourage information sharing among widely varied stakeholders, and that video was a particularly successful approach when engaging with local communities. An 8-month follow-up survey of village members revealed excellent information recall, positive behaviour changes and a desire for future visits. Our evidence suggests that community-based participation helped identify solutions to WASH issues affecting rural communities in resource-poor settings. Engaging in a multicultural knowledge-share was particularly valuable as it enabled participants to recognize they have common challenges and allowed them to share low-cost solutions from their different communities. Our use of video was widely viewed as an ideal means of circulating findings, as it communicated information to people with a wide variety of community roles and to all age groups. Its relevance was increased by adopting a culturally appropriate context by involving local communities in workshop activities. We recommend that research in low- and middle-income countries should be mindful of the environmental context in which WASH is implemented, and encourage acceptance by engaging with communities through the use of varied participatory methods.