{'fraction_non_alphanumeric': 0.03578982291709866, 'fraction_numerical': 0.009496951851698254, 'mean_word_length': 4.935023381737632, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Crossmodal Attentive Skill Learner

This paper presents the Crossmodal Attentive Skill Learner (CASL), integrated with the recently-introduced Asynchronous Advantage Option-Critic (A2OC) architecture [Harb et al., 2017] to enable hierarchical reinforcement learning across multiple sensory inputs. We provide concrete examples where the approach not only improves performance in a single task, but accelerates transfer to new tasks. We demonstrate the attention mechanism anticipates and identifies useful latent features, while filtering irrelevant sensor modalities during execution. We modify the Arcade Learning Environment [Bellemare et al., 2013] to support audio queries, and conduct evaluations of crossmodal learning in the Atari 2600 game Amidar. Finally, building on the recent work of Babaeizadeh et al. [2017], we open-source a fast hybrid CPU-GPU implementation of CASL.


Introduction
Intelligent agents should be capable of disambiguating local sensory streams to realize long-term goals. In recent years, the combined progress of computational capabilities and algorithmic innovations has afforded reinforcement learning (RL) [Sutton and Barto, 1998] approaches the ability to achieve this desiderata in impressive domains, exceeding expert-level human performance in tasks such as Atari and Go [Mnih et al., 2015. Nonetheless, many of these algorithms thrive primarily in well-defined mission scenarios learned in isolation from one another; such monolithic approaches are not sufficiently scalable for missions where goals may be less clearly defined, and sensory inputs found salient in one domain may be less relevant in another.
How should agents learn effectively in domains of high dimensionality, where tasks are durative, agents receive sparse feedback, and sensors compete for limited computational resources? One promising avenue is hierarchical reinforcement learning (HRL), focusing on problem decomposition for learning transferable skills. Temporal abstraction enables exploitation of domain regularities to provide the agent hierarchical guidance in the form of options or sub-goals [Sutton et al., 1999, Kulkarni et al., 2016. Options help agents improve learning by mitigating scalability issues in long-duration missions, by reducing the effective number of decision epochs. In the parallel field of supervised learning, temporal dependencies have been captured proficiently using attention mechanisms applied to encoder-decoder based sequence-to-sequence models [Bahdanau et al., 2014, Luong et al., 2015. Attention empowers the learner to focus on the most pertinent stimuli and capture longer-term correlations in its encoded state, for instance to conduct neural machine translation or video captioning [Yeung et al., 2015, Yang et al., 2016. Recent works also show benefits of spatio-temporal attention in RL [Mnih et al., 2014, Sorokin et al., 2015.
One can interpret the above approaches as conducting dimensionality reduction, where the target dimension is time. In view of this insight, this paper proposes an RL paradigm exploiting hierarchies in the dimensions of time and sensor modalities. Our aim is to learn rich skills that attend to and exploit pertinent crossmodal (multi-sensor) signals at the appropriate moments. The introduced crossmodal skill learning approach largely benefits an agent learning in a high-dimensional domain (e.g., a robot equipped with many sensors). Instead of the expensive operation of processing and/or storing data from all sensors, we demonstrate that our approach enables such an agent to focus on important sensors; this, in turn, leads to more efficient use of the agent's limited computational and storage resources (e.g., its finite-sized memory).
In this paper, we focus on combining two sensor modalities: audio and video. While these modalities have been previously used for supervised learning [Ngiam et al., 2011], to our knowledge they have yet to be exploited for crossmodal skill learning. We provide concrete examples where the proposed HRL approach not only improves performance in a single task, but accelerates transfer to new tasks. We demonstrate the attention mechanism anticipates and identifies useful latent features, while filtering irrelevant sensor modalities during execution. We also show preliminary results in the Arcade Learning Environment [Bellemare et al., 2013], which we modified to support audio queries. In addition, we provide insight into how our model functions internally by analyzing the interactions of attention and memory. Building on the recent work of Babaeizadeh et al. [2017], we open-source a fast hybrid CPU-GPU implementation of our framework. Finally, note that despite this paper's focus on audio-video sensors, the framework presented is general and readily applicable to additional sensory inputs.

Background
POMDPs This work considers an agent operating in a partially-observable stochastic environment, modeled as a POMDP S, A, O, T , O, R, γ [Kaelbling et al., 1998]. S, A, and O are, respectively, the state, action, and observation spaces. At timestep t, the agent executes action a ∈ A in state s ∈ S, transitions to state s ∼ T (s, a, s ), receives observation o ∼ O(o, s , a), and reward r t = R(s, a) ∈ R. The value of state s under policy π : Dist(S) → A is the expected return V π (s) = E[ H t =t γ t −t r t ], given horizon H and discount factor γ ∈ [0, 1). The objective is to learn an optimal policy π * , which maximizes the value.
Options The framework of options provides an RL agent the ability to plan using temporallyextended actions [Sutton et al., 1999]. Option ω ∈ Ω is defined by initiation set I ⊆ S, intra-option policy π ω : S → Dist(A), and termination condition β ω : S → [0, 1]. Initially, a policy over options π Ω chooses an option among those that satisfy the initiation set. The selected option executes its intra-option policy until termination, upon which a new option is chosen. This process iterates until the goal state is reached. Recently, the Asynchronous Advantage Actor-Critic framework (A3C) [Mnih et al., 2016] has been applied to POMDP learning in a computationally-efficient manner by combining parallel actor-learners and Long Short-Term Memory (LSTM) cells [Hochreiter and Schmidhuber, 1997]. Asynchronous Advantage Option-Critic (A2OC) extends A3C and enables learning option-value functions, intra-option policies, and termination conditions in an end-to-end fashion . The option-value function models the value of state s ∈ S in option ω ∈ Ω, Q Ω (s, ω) = a π ω (a|s) r(s, a) + γ s T (s |s, a)U (s , ω) , where a ∈ A is a primitive action and U (s , ω) represents the option utility function, A2OC introduces deliberation cost, c, in the utility function to address the issue of options terminating too frequently. Intuitively, the role of c is to impose an added penalty when options terminate, leading them to terminate less frequently. The value function over options, V Ω , is defined, where π Ω is the policy over options (e.g., an epsilon-greedy policy over Q Ω ). Assuming use of a differentiable representation, option parameters are learned using gradient descent.

Approach
Our goal is to design a mechanism that enables the learner to modulate high-dimensional sensory inputs, focusing on pertinent stimuli that may lead to more efficient skill learning. This section presents motivations behind attentive skill learning, then introduces the proposed framework.

Attentive Mechanisms
Before presenting the proposed architecture, let us first motivate our interests towards attentive skill learning. One might argue that the combination of deep learning and RL already affords agents the representation learning capabilities necessary for proficient decision-making in high-dimensional domains; i.e., why the need for crossmodal attention?
Our ideas are motivated by the studies in behavioral neuroscience that suggest the interplay of attention and choice bias humans' value of information during learning, playing a key factor in solving tasks with high-dimensional information streams [Leong et al., 2017]. Works studying learning in the brain also suggest a natural pairing of attention and hierarchical learning, where domain regularities are embedded as priors into skills and combined with attention to alleviate the curse of dimensionality [Niv et al., 2015]. Works also suggest attention plays a role in the intrinsic curiosity of agents during learning, through direction of focus to regions predicted to have high reward [Mackintosh, 1975], high uncertainty [Pearce and Hall, 1980], or both [Pearce and Mackintosh, 2010].
In view of these studies, we conjecture that crossmodal attention, in combination with HRL, improves representations of relevant environmental features that lead to superior learning and decision-making. Specifically, using crossmodal attention, agents combine internal beliefs with external stimuli to more effectively exploit multiple modes of input features for learning. As we later demonstrate, our approach captures temporal crossmodal dependencies, and enables faster and more proficient learning of skills in the domains examined.

Crossmodal Attentive Skill Learner
We propose Crossmodal Attentive Skill Learner (CASL), a novel end-to-end framework for HRL. One may consider many blueprints for integration of multi-sensory attention into the options framework. Our proposed architecture is primarily motivated by the literature that taxonomizes attention into two classes: exogeneous and endogeneous. The former is an involuntary mechanism triggered automatically by the inherent saliency of the sensory inputs, whereas the latter is driven by the intrinsic and possibly long-term goals, intents, and beliefs of the agent [Carrasco, 2011]. Previous attention-based neural architectures take advantage of both classes, for instance, to solve natural language processing problems [Vinyals et al., 2015]. Our approach follows this schema.
The CASL network architecture is visualized in Fig Weight matrices W m , W h , W z and bias vectors b m , b h , b z are trainable parameters and nonlinearities are applied element-wise.
Both exogeneous attention over sensory features x t m and endogeneous attention over LSTM hidden state h t−1 are captured in (4). The sensory feature extractor used in experiments consists of 3 convolutional layers, each with 32 filters of size 3 × 3, stride 2, and ReLU activations. Attended features α t m x t m may be combined via summation or concatenation (per (6)), then fed to an LSTM cell. The LSTM output captures temporal dependencies used to estimate option values, intra-option policies, and termination conditions (Q Ω , π ω , β ω in Fig. 1, respectively), π ω (a|s) = softmax(W π,ω h t + b π,ω ), where weight matrices W Ω,ω , W π,ω , W β,ω and bias vectors b Ω,ω , b π,ω , b β,ω are trainable parameters for the current option ω, and σ(·) is the sigmoid function. Network parameters are updated using gradient descent. Entropy regularization of attention outputs α t was found to encourage exploration of crossmodal attention behaviors during training.

Evaluation
The proposed framework is evaluated on a variety of learning tasks with inherent reward sparsity and transition noise. We evaluate our approach in three domains: a door puzzle domain, a 2D-Minecraft like domain, and the Arcade Learning Environment [Bellemare et al., 2013]. These environments include challenging combinations of reward sparsity and/or complex audio-video sensory input modalities that may not always be useful to the agent. The first objective of our experiments is to analyze performance of CASL in terms of learning rate and transfer learning. The second objective is  to understand relationships between attention and memory mechanisms (as captured in the LSTM cell state). Finally, we modify the Arcade Learning Environment to support audio queries, and evaluate crossmodal learning in the Atari 2600 game Amidar.

Crossmodal Learning and Transfer
We first evaluate crossmodal attention in a sequential door puzzle game, where the agent spawns in a 2D world with two locked doors and a key at fixed positions. The key type is randomly generated, and its observable color indicates the associated door. The agent hears a fixed sound (but receives no reward) when adjacent to the key, and hears noise otherwise. The agent must find and pick up the key (which then disappears), then find and open the correct door to receive +1 reward (with discount γ = 0.99). The game terminates upon opening of either door. The agent's sensory inputs x t m are vision (grayscale image) and audio spectrogram. This task was designed in such a way that audio is not necessary to achieve the task -the agent can certainly focus on learning a policy mapping from visual features to open the correct door. However, audio provides potentially useful signals that may accelerate learning, making this a domain of interest for analyzing the interplay of attention and sensor modalities. Figure 2a shows ablative training results for several network architectures. The three LSTM-based skill learners (including CASL) converge to the optimal value. Interestingly, the network that ignores audio inputs (V-O-LSTM) converges faster than its audio-enabled counterpart (V-A-O-LSTM), indicating the latter is overwhelmed by the extra sensory modality. Introduction of crossmodal attention enables CASL to converge faster than all other networks, using roughly half the training data of the others. The feedforward networks all fail to attain optimal value, with the non-option cases (V-A-FF and V-FF) repeatedly opening one door due to lack of memory of key color. Notably, the option-based feedforward nets exploit the option index to implicitly remember the key color, leading to higher value. Interplay between explicit memory mechanisms and use of options as pseudo-memory may be an interesting line of future work.

Attention Improves Learning Rate
Attention Accelerates Transfer We also evaluate crossmodal attention for transfer learning (Fig. 2b), using the more promising option-based networks. The door puzzle domain is modified to randomize the key position, with pre-trained options from the fixed-position variant used for initialization. All networks benefit from an empirical return jumpstart of 0.2 at the beginning of training, due to skill transfer. Once again, CASL converges fastest, indicating more effective use of the available audio-video data. While the asymptotic performance of CASL is only slightly higher than the V-A-O-LSTM network, the reduction in number of samples needed to achieve a high score (e.g, after 100K episodes) makes it advantageous for domains with high sampling cost.
Attention Necessary to Learn in Some Domains Temporal behaviors of the attention mechanism are also evaluated in a 2D Minecraft-like domain, where the agent must pick an appropriate tool (pickaxe or shovel) to mine either gold or iron ore (Figs. 3a to 3c). Critically, the agent observes identical images for both ore types, but unique audio features when near the ore, making long-term audio storage necessary for selection of the correct tool. The agent receives +10 reward for correct (a) Gold ore should be mined with the pickaxe.
(b) Iron ore should be mined with the shovel.   tool selection, −10 for incorrect selection, and −1 step cost. Compared to the door puzzle game, the mining domain is posed in such a way that the interplay between audio-video features is emphasized. Specifically, an optimal policy for this task must utilize both audio and video features: visual inputs enable detection of locations of the ore, agent, tools, whereas audio is used to identify the ore type.
Visual occlusion of the ore type, interplay of audio-video features, and sparse positive rewards cause the non-attentive network to fail to learn in the mining domain, as opposed to the attentive case (Fig. 3d). Figure 4a plots a sequence of frames where the agent anticipates salient audio features as it nears the ore at t = 6, gradually increasing audio attention, then sharply reducing it to 0 after hearing the signal. Video >2500

Interactions of Attention and Memory
While the anticipatory nature of crossmodal attention in the mining domain is interesting, it also points to additional lines of investigation regarding interactions of attention and updates of the agent's internal belief (as encoded in its LSTM cell state). Specifically, one might wonder whether it is necessary for the agent to place any attention on the non-useful audio signals prior to timestep t = 6 in Fig. 4a, and also whether this behavior implies inefficient usage of its finite-size memory state.
Motivated by the above concerns, we conduct more detailed analysis of the interplay between the agent's attention and memory mechanisms as used in the CASL architecture (Fig. 1). Readers are referred to the appendix (Section 6.1) for details on how this analysis was conducted, as well as a brief overview of LSTM units. Given the sequence of audio-video inputs in Fig. 4a, we plot overall activations of the forget and input LSTM gates (averaged across all cell state elements), in Fig. 4b and Fig. 4c, respectively. Critically, these plots also indicate the relative influence of the forget and input LSTM gates' contributing variables (audio input, video input, hidden state, and bias term) to the overall activation.
Interestingly, prior to timestep t = 6, the contribution of audio to the forget gate and input gates is essentially zero, despite the positive attention on audio (in Fig. 4a). Recall a low forget gate activation corresponds to complete forgetting of the previous LSTM cell state element, whereas a high input gate activation corresponds to complete throughput of the corresponding input element. At t = 6, the forget gate activation drops, while the input gate experiences a sudden increase, indicating major overwriting of previous memory states with new information. Critically, the plots indicate that the attended audio input is the key contributing factor of both behaviors. In Fig. 4a, after the agent hears the necessary audio signal, it moves attention entirely to video; the contribution of audio to the forget and input activations also drops to zero. These behaviors indicate that the agent attends to audio in anticipation of an upcoming pertinent signal, but chooses not to embed it into memory until the appropriate moment. Attention filters irrelevant sensor modalities, given the contextual clues provided by exogeneous and endogeneous input features; it, therefore, enables the LSTM gates to focus on learning when and how to update the agent's internal state.

Early Experiments in the Arcade Learning Environment
Preliminary evaluation of crossmodal attention was conducted in the Arcade Learning Environment (ALE) [Bellemare et al., 2013]. We modified ALE to support audio queries, as it previously did not have this feature; we plan to add this code to the ALE repository.
Experiments were conducted in the Atari 2600 game Amidar (Table 1). This line of investigation considers impacts of crossmodal attention on Atari agent behavior, even without use of multiple (hierarchical) options; these results use CASL with a single option, hence tagged "non-hierarchical" in the table. Amidar was one of the games in which deep Q-networks failed to exceed human-level performance [Mnih et al., 2015]. The objective in Amidar is to collect rewards in a rectilinear maze while avoiding patrolling enemies. The agent is rewarded for painting segments of the maze, killing enemies at opportune moments, or collecting bonuses. Background audio plays throughout the game, and specific audio signals play when the agent crosses previously-unseen segment vertices. Figure 5  reveals that the agent anticipates and increases audio attention when near these critical vertices, which are especially difficult to observe when the agent sprite is overlapping them (e.g., zoom into RGB sequences of Fig. 5).
Our crossmodal attentive agent achieves a mean score of 900 in Amidar, over 30 test runs, outperforming the other non-hierarchical methods. Note agent also beats the score of the hierarchical approach of . We emphasize these are not direct comparisons due to our method leveraging additional sensory inputs, but are meant to highlight the performance benefits of crossmodal learning. We also note that the state-of-the-art hierarchical approach FeUdal [Vezhnevets et al., 2017] beats our agent's score, and may achieve even higher score with crossmodal inputs.

Contribution
This work introduced the Crossmodal Attentive Skill Learner (CASL), integrated with the recentlyintroduced Asynchronous Advantage Option-Critic (A2OC) architecture  to enable hierarchical reinforcement learning across multiple sensory inputs. We provided concrete examples where CASL not only improves performance in a single task, but accelerates transfer to new tasks. We demonstrated the learned attention mechanism anticipates and identifies useful sensory features, while filtering irrelevant sensor modalities during execution. We modified the Arcade Learning Environment [Bellemare et al., 2013] to support audio queries, and evaluations of crossmodal learning were conducted in the Atari 2600 game Amidar. Finally, building on the recent work of Babaeizadeh et al. [2017], we open-source a fast hybrid CPU-GPU implementation of CASL. This investigation indicates crossmodal skill learning as a promising avenue for future works in HRL that target domains with high-dimensional, multimodal inputs.

Analyzing Interactions of Attention and Memory
We provide a brief overview of LSTM networks to enable more rigorous discussion of attentionmemory interactions. At timestep t, LSTM cell state C t encodes the agent's memory given its previous stream of inputs. The cell state is updated as follows, where f t is the forget gate activation vector, i t is the input gate activation vector, h t−1 is the previous hidden state vector, x ⊕ is attended feature vector, and refers to the Hadamard product. Weights W f , W i , W C and biases b f , b i , b C are trainable parameters. The cell state update in (12) first forgets certain elements (f t term), and then adds contributions from new inputs (i t term). Note that a forget gate activation of 0 corresponds to complete forgetting of the previous cell state element, and that an input gate activation of 1 corresponds to complete throughput of the corresponding input element.
Our goal is to not only analyze the overall forget/input activations throughout the gameplay episode, but also to quantify the relative impact of each contributing variable (audio input, video input, hidden state, and bias term) to the overall activations. Many methods may be used for analysis of the contribution of explanatory variables in nonlinear models (i.e., (10) to (12)). We introduce a means of quantifying the correlation of each variable with respect to the corresponding activation function.
In the following, we focus on the forget gate activation, but the same analysis applies to the input gate. First, expanding the definition of forget gate activation in (10), assuming use of concatenated attention (per (6)), yields, where x a and x v are, respectively, the audio and video input features, and I is the identity matrix. Definef t m as the forget gate activation if the m-th contributing variable were removed. For example, if audio input x a were to be removed, then, Define the forget gate activation residual asf t m = |f t −f t m | (i.e., the difference in output resulting from removal of the m-th contributing variable). Then, one can define a 'pseudo correlation' of the m-th contributing variable with respect to the true activation, This provides an approximate quantification of the relative contribution of the m-th variable (audio input, video input, hidden unit, or bias) to the overall activation of the forget and input gates. Armed with this toolset, we can analyze the interplay between attention and LSTM memory.