A Multi-step and Resilient Predictive Q-learning Algorithm for IoT with
  Human Operators in the Loop: A Case Study in Water Supply Networks
  We consider the problem of recommending resilient and predictive actions for
an IoT network in the presence of faulty components, considering the presence
of human operators manipulating the information of the environment the agent
sees for containment purposes. The IoT network is formulated as a directed
graph with a known topology whose objective is to maintain a constant and
resilient flow between a source and a destination node. The optimal route
through this network is evaluated via a predictive and resilient Q-learning
algorithm which takes into account historical data about irregular operation,
due to faults, as well as the feedback from the human operators that are
considered to have extra information about the status of the network concerning
locations likely to be targeted by attacks. To showcase our method, we utilize
anonymized data from Arlington County, Virginia, to compute predictive and
resilient scheduling policies for a smart water supply system, while avoiding
(i) all the locations indicated to be attacked according to human operators
(ii) as many as possible neighborhoods detected to have leaks or other faults.
This method incorporates both the adaptability of the human and the computation
capability of the machine to achieve optimal implementation containment and
recovery actions in water distribution.
