Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight
  Campaign Design
  This paper develops a hierarchical reinforcement learning architecture for
multimission spaceflight campaign design under uncertainty, including vehicle
design, infrastructure deployment planning, and space transportation
scheduling. This problem involves a high-dimensional design space and is
challenging especially with uncertainty present. To tackle this challenge, the
developed framework has a hierarchical structure with reinforcement learning
and network-based mixed-integer linear programming (MILP), where the former
optimizes campaign-level decisions (e.g., design of the vehicle used throughout
the campaign, destination demand assigned to each mission in the campaign),
whereas the latter optimizes the detailed mission-level decisions (e.g., when
to launch what from where to where). The framework is applied to a set of human
lunar exploration campaign scenarios with uncertain in situ resource
utilization performance as a case study. The main value of this work is its
integration of the rapidly growing reinforcement learning research and the
existing MILP-based space logistics methods through a hierarchical framework to
handle the otherwise intractable complexity of space mission design under
uncertainty. This unique framework is expected to be a critical steppingstone
for the emerging research direction of artificial intelligence for space
mission design.
