Optimal Routing in Stochastic Networks with Reliability Guarantees.

Optimal routing in highly congested street networks where the travel times are often stochastic is a challenging problem with significant practical interest. While most approaches to this problem use minimizing the expected travel time as the sole objective, such a solution is not always desired, especially when the variance of travel time is high. In this work, we pose the problem of finding a routing policy that minimizes the expected travel time under the hard constraint of retaining a specified probability of on-time arrival. Our approach to this problem models the stochastic travel time on each segment in the road network as a discrete random variable, thus translating the model of interest into a Markov decision process. Such a setting enables us to interpret the problem as a linear program. Our work also includes a case study on the street of Manhattan, New York where we constructed the model of travel times using real-world data, and employed our approach to generate optimal routing policies.

Fuel in Markov Decision Processes (FiMDP): A Practical Approach to Consumption.

Consumption Markov Decision Processes (CMDPs) are probabilistic decision-making models of resource-constrained systems. We introduce FiMDP, a tool for controller synthesis in CMDPs with LTL objectives expressible by deterministic Büchi automata. The tool implements the recent algorithm for polynomial-time controller synthesis in CMDPs, but extends it with many additional features. On the conceptual level, the tool implements heuristics for improving the expected reachability times of accepting states, and a support for multi-agent task allocation. On the practical level, the tool offers (among other features) a new strategy simulation framework, integration with the Storm model checker, and FiMDPEnv - a new set of CMDPs that model real-world resource-constrained systems. We also present an evaluation of FiMDP on these real-world scenarios.

Safety-Guaranteed, Accelerated Learning in MDPs with Local Side Information.

In environments with uncertain dynamics, synthesis of optimal control policies mandates exploration. The applicability of classical learning algorithms to real-world problems is often limited by the number of time steps required for learning the environment model. Given some local side information about the differences in transition probabilities of the states, potentially obtained from the agent's onboard sensors, we generalize the idea of indirect sampling for accelerated learning to propose an algorithm that balances between exploration and exploitation. We formalize this idea by introducing the notion of the value of information in the context of a Markov decision process with unknown transition probabilities, as a measure of the expected improvement in the agent's current estimate of transition probabilities by taking a particular action. By exploiting available local side information and maximizing the estimated value of learned information at each time step, we accelerate the learning process and subsequent synthesis of the optimal control policy. Further, we define the notion of agent safety, a vital consideration for physical systems, in the context of our problem. Under certain assumptions, we provide guarantees on the safety of an agent exploring with our algorithm that exploits local side information. We illustrate agent safety and the improvement in learning speed using numerical experiments in the setting of a Mars rover, with data from onboard sensors acting as the local side information.

PROTRIP: Probabilistic Risk-Aware Optimal Transit Planner.

Optimal routing in urban transit networks, where variable congestion levels often lead to stochastic travel times, is usually studied with the least expected travel time (LET) as the performance criteria under the assumption of travel time independence on different road segments. However, a LET path might be subjected to high variability of travel time and therefore might not be desirable to transit users seeking a predictable arrival time. Further, there exists a spatial correlation in urban travel times due to the cascading effect of congestion across the road network. In this work, we propose a methodology and a tool that, given an origin-destination pair, a travel time budget, and a measure of the passenger's tolerance for uncertainty, provide the optimal online route choice in a transit network by balancing the objectives of maximizing on-time arrival probability and minimizing expected travel time. Our framework takes into account the correlation between travel time of different edges along a route and updates downstream distributions by taking advantage of upstream real-time information. We demonstrate the utility and performance of our algorithm with the help of realistic numerical experiments conducted on a fixed-route bus system that serves the residents of the Champaign-Urbana metropolitan area.