Hierarchical Solution of Markov Decision Processes using Macro-actions
Milos Hauskrecht, Nicolas Meuleau, Leslie Kaelbling, Thomas Dean, Craig Boutilier
We investigate the use of temporally abstract actions, or macro-actions, in the solution of Markov decision processes. Unlike current models that combine both primitive actions and macro-actions and leave the state space unchanged, we propose a hierarchical model (using an abstract MDP) that works with macro-actions only, and that significantly reduces the size of the state space. This is achieved by treating macroactions as local policies that act in certain regions of state space, and by restricting states in the abstract MDP to those at the boundaries of regions. The abstract MDP approximates the original and can be solved more efficiently. We discuss several ways in which macro-actions can be generated to ensure good solution quality. Finally, we consider ways in which macro-actions can be reused to solve multiple, related MDPs; and we show that this can justify the computational overhead of macro-action generation.
PDF Link: /papers/98/p220-hauskrecht.pdf
AUTHOR = "Milos Hauskrecht
and Nicolas Meuleau and Leslie Kaelbling and Thomas Dean and Craig Boutilier",
TITLE = "Hierarchical Solution of Markov Decision Processes using Macro-actions",
BOOKTITLE = "Proceedings of the Fourteenth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-98)",
PUBLISHER = "Morgan Kaufmann",
ADDRESS = "San Francisco, CA",
YEAR = "1998",
PAGES = "220--229"