Behavioral Analytics for Myopic Agents.

Many multi-agent systems have a single coordinator providing incentives to a large number of agents. Two challenges faced by the coordinator are a finite budget from which to allocate incentives, and an initial lack of knowledge about the utility function of the agents. Here, we present a behavioral analytics approach for solving the coordinator's problem when the agents make decisions by maximizing utility functions that depend on prior system states, inputs, and other parameters that are initially unknown. Our behavioral analytics framework involves three steps: first, we develop a model that describes the decision-making process of an agent; second, we use data to estimate the model parameters for each agent and predict their future decisions; and third, we use these predictions to optimize a set of incentives that will be provided to each agent. The framework and approaches we propose in this paper can then adapt incentives as new information is collected. Furthermore, we prove that the incentives computed by this approach are asymptotically optimal with respect to a loss function that describes the coordinator's objective. We optimize incentives with a decomposition scheme, where each sub-problem solves the coordinator's problem for a single agent, and the master problem is a pure integer program. We conclude with a simulation study to evaluate the effectiveness of our approach for designing a personalized weight loss program. The results show that our approach maintains efficacy of the program while reducing its costs by up to 60%, while adaptive heuristics provide substantially less savings.

Behavioral Modeling in Weight Loss Interventions.

Designing systems with human agents is difficult because it often requires models that characterize agents' responses to changes in the system's states and inputs. An example of this scenario occurs when designing treatments for obesity. While weight loss interventions through increasing physical activity and modifying diet have found success in reducing individuals' weight, such programs are difficult to maintain over long periods of time due to lack of patient adherence. A promising approach to increase adherence is through the personalization of treatments to each patient. In this paper, we make a contribution towards treatment personalization by developing a framework for predictive modeling using utility functions that depend upon both time-varying system states and motivational states evolving according to some modeled process corresponding to qualitative social science models of behavior change. Computing the predictive model requires solving a bilevel program, which we reformulate as a mixed-integer linear program (MILP). This reformulation provides the first (to our knowledge) formulation for Bayesian inference that uses empirical histograms as prior distributions. We study the predictive ability of our framework using a data set from a weight loss intervention, and our predictive model is validated by comparison to standard machine learning approaches. We conclude by describing how our predictive model could be used for optimization, unlike standard machine learning approaches which cannot.

Personalizing Mobile Fitness Apps using Reinforcement Learning.

Despite the vast number of mobile fitness applications (apps) and their potential advantages in promoting physical activity, many existing apps lack behavior-change features and are not able to maintain behavior change motivation. This paper describes a novel fitness app called CalFit, which implements important behavior-change features like dynamic goal setting and self-monitoring. CalFit uses a reinforcement learning algorithm to generate personalized daily step goals that are challenging but attainable. We conducted the Mobile Student Activity Reinforcement (mSTAR) study with 13 college students to evaluate the efficacy of the CalFit app. The control group (receiving goals of 10,000 steps/day) had a decrease in daily step count of 1,520 (SD ± 740) between baseline and 10-weeks, compared to an increase of 700 (SD ± 830) in the intervention group (receiving personalized step goals). The difference in daily steps between the two groups was 2,220, with a statistically significant p = 0.039.

Evaluating Machine Learning-Based Automated Personalized Daily Step Goals Delivered Through a Mobile Phone App: Randomized Controlled Trial.

BACKGROUND: Growing evidence shows that fixed, nonpersonalized daily step goals can discourage individuals, resulting in unchanged or even reduced physical activity. OBJECTIVE: The aim of this randomized controlled trial (RCT) was to evaluate the efficacy of an automated mobile phone-based personalized and adaptive goal-setting intervention using machine learning as compared with an active control with steady daily step goals of 10,000. METHODS: In this 10-week RCT, 64 participants were recruited via email announcements and were required to attend an initial in-person session. The participants were randomized into either the intervention or active control group with a one-to-one ratio after a run-in period for data collection. A study-developed mobile phone app (which delivers daily step goals using push notifications and allows real-time physical activity monitoring) was installed on each participant's mobile phone, and participants were asked to keep their phone in a pocket throughout the entire day. Through the app, the intervention group received fully automated adaptively personalized daily step goals, and the control group received constant step goals of 10,000 steps per day. Daily step count was objectively measured by the study-developed mobile phone app. RESULTS: The mean (SD) age of participants was 41.1 (11.3) years, and 83% (53/64) of participants were female. The baseline demographics between the 2 groups were similar (P>.05). Participants in the intervention group (n=34) had a decrease in mean (SD) daily step count of 390 (490) steps between run-in and 10 weeks, compared with a decrease of 1350 (420) steps among control participants (n=30; P=.03). The net difference in daily steps between the groups was 960 steps (95% CI 90-1830 steps). Both groups had a decrease in daily step count between run-in and 10 weeks because interventions were also provided during run-in and no natural baseline was collected. CONCLUSIONS: The results showed the short-term efficacy of this intervention, which should be formally evaluated in a full-scale RCT with a longer follow-up period. TRIAL REGISTRATION: ClinicalTrials.gov: NCT02886871; https://clinicaltrials.gov/ct2/show/NCT02886871 (Archived by WebCite at http://www.webcitation.org/6wM1Be1Ng).