{'fraction_non_alphanumeric': 0.03039129479006375, 'fraction_numerical': 0.010661683886568477, 'mean_word_length': 4.828635490070468, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Early Improving Recurrent Elastic Highway Network

To model time-varying nonlinear temporal dynamics in sequential data, a recurrent network capable of varying and adjusting the recurrence depth between input intervals is examined. The recurrence depth is extended by several intermediate hidden state units, and the weight parameters involved in determining these units are dynamically calculated. The motivation behind the paper lies on overcoming a deficiency in Recurrent Highway Networks and improving their performances which are currently at the forefront of RNNs: 1) Determining the appropriate number of recurrent depth in RHN for different tasks is a huge burden and just setting it to a large number is computationally wasteful with possible repercussion in terms of performance degradation and high latency. Expanding on the idea of adaptive computation time (ACT), with the use of an elastic gate in the form of a rectified exponentially decreasing function taking on as arguments as previous hidden state and input, the proposed model is able to evaluate the appropriate recurrent depth for each input. The rectified gating function enables the most significant intermediate hidden state updates to come early such that significant performance gain is achieved early. 2) Updating the weights from that of previous intermediate layer offers a richer representation than the use of shared weights across all intermediate recurrence layers. The weight update procedure is just an expansion of the idea underlying hypernetworks. To substantiate the effectiveness of the proposed network, we conducted three experiments: regression on synthetic data, human activity recognition, and language modeling on the Penn Treebank dataset. The proposed networks showed better performance than other state-of-the-art recurrent networks in all three experiments.


Introduction
Recurrent Neural Networks (RNNs) have been successfully applied to a diverse array of tasks, such as speech recognition [1], language modeling [2], machine translation [3], music generation [4], image description [5], video description [6]. In these tasks, the non-linear transition of the internal state in the RNNs represented the temporal dynamic behavior of the sequence. This paper investigates the effectiveness of elastically varying the intermediate recurrence depth of the RNN in modeling the time-varying temporal dynamics of the sequential input data. The intermediate recurrence depth at each time interval will depend on the input and hidden state at a corresponding time interval.
While deeper networks have been known to be more efficient in representing the input-output relationship compared to shallow networks, they require more data to train and are far more susceptible to the vanishing and exploding gradient problem. As a measure to safeguard against such problem when constructing a very deep network, ResNet [7] and Highway Networks [8] were proposed. Residual units in these models provide learning stability in building a very deep model. In creating temporally deep RNNs, Long Short-Term Memory (LSTM, [9]), Gated Recurrent Unit (GRU, [3]), and Recurrent Highway Network (RHN, [10]) were introduced to both find learning stability and model long-range temporal dependency.
One factor that affects the computational time is the depth (or the number of layers) of the network. Given a training dataset, to get good performance, various models with different depth are empirically compared, and this can be a very time-consuming procedure if the dataset is large.
Recently, two studies on determining appropriate depth of a network have been considered. A general concept of time-dependent adaptive computation time (ACT) is introduced into RNN in [11]. The algorithm learns the number of computational steps to take between receiving an input and emitting an output. Here, a sigmoidal halting unit determines the probability for continuing the computation. In Spatially Adaptive Computation Time [12], the number of executed layers for the regions of the image is dynamically adjusted.
To model time-varying nonlinear temporal dynamics in sequential data, a recurrent network capable of varying and adjusting the recurrence depth between input intervals is examined. The recurrence depth is extended by several intermediate hidden state units, and the weight parameters involved in determining these units are dynamically calculated. The motivation behind the paper lies on overcoming a deficiency in Recurrent Highway Networks and improving their performances which are currently at the forefront of RNNs: 1) Determining the appropriate number of recurrent depth in RHN for different tasks is a huge burden and just setting it to a large number is computationally wasteful with possible repercussion in terms of performance degradation and high latency. Expanding on the idea recently proposed in ACT [11], with the use of an elastic gate in the form of a rectified exponentially decreasing function taking on as arguments as previous hidden state and input, the proposed model is able to evaluate the appropriate recurrent depth for each input. The rectified gating function enables the most significant intermediate hidden state updates to come early such that significant performance gain is achieved early. 2) Updating the weights from that of previous intermediate layer offers a richer representation than the use of shared weights across all intermediate recurrence layers. The weight update procedure is just an expansion of the idea underlying hypernetworks [13]. This structure is referred to as Early Improving Recurrent Elastic Highway Network (EI-REHN).
This paper is organized as follows. Section 2 describes background information for the proposed method. Section 3 presents the proposed method of EI-REHN. Section 4 shows the experimental results, and finally, Section 5 concludes the paper.

Background
Consider a hidden state transition model S of a recurrent network, where x t ∈ R Dx and h t−1 ∈ R D h are the input vector at time step t and hidden state vector at time step t − 1, respectively.
The hidden state transition model of the standard RNN can be represented as follows, where W R ∈ R D h ×(D h +Dx+1) and tanh are a weight matrix including bias and element-wise hyperbolic tangent function, respectively.
On the other hand, the hidden state transition model of an LSTM is as follows, where Here, , sigm, and ⊗ are cell proposal, input gate, forget gate, output gate, cell state, weight matrix, element-wise sigmoid function, and element-wise multiplication, respectively. The hidden state transition model of RHN with a fixed recurrence depth is as follows, where Here, , are residual component, transform gate, carry gate, and weight matrix, respectively.

Early Improving Recurrent Elastic Highway Network
In this section, Early Improving Recurrent Elastic Highway Network (EI-REHN) which is a recurrent network capable of varying and adjusting the recurrence depth between input intervals is introduced. The intermediate state transition from the (r − 1)-th intermediate recurrence layer to the r-th layer is given as where g i t and s r t are a gating function for adaptive recurrence depth and residual component, respectively. The gating function is designed to exponentially decrease as the intermediate recurrence layer increases. Consequently, intermediate recurrence state transition halts when the gating function reaches zero. The exponentially decreasing gating function enables the most significant intermediate hidden state updates to come early such that significant performance gain is achieved early. In order to reduce the number of parameters, a recurrence relationship is formulated between parameters of adjacent layers such that only the significant parameters are updated. And the weight parameters for all the intermediate recurrence layers are calculated based on a hypernetwork [13] that is a sub-network to generate the weights for another network.

Adaptive recurrence depth
A gating function to adaptively determine the intermediate recurrence depth depending on x t and h t−1 is given as follows: where d r t andĝ r t are elastic gating and residual gating, respectively. The elastic gating function is obtained as follows, where β, α, α t , and W a ∈ R D h ×(D h +Dx+1) are initial gating bias, global decreasing rate, local decreasing rate, and weight matrix for residual gating function, respectively. The model parameters ofα andβ are estimated when training, while the local decreasing rate, α t , is time-dependent and activated when forward propagation. It makes the model to have adaptive recurrence depth at every time step.
At time t, adaptive recurrence depth R t is determined as the maximum depth that satisfies ||g r t || 1 > 0. This means that the recurrence layer repeats until all hidden units have zero gating activations. Based on α and β, upper-bound to R t for all the time steps is Figure 1 shows examples of elastic gating functions for a hidden unit when α r t = 0. The elastic gating function is exponentially decreasing with respect to recurrence depth. Smaller value of α and bigger value of β give longer adaptive recurrence depth.
If an element of d r=1 t is zero, the corresponding hidden unit is not updated at time step t and the hidden information from the previous time step is passed to the next time step due to the exactly zero gating activation.

Dynamic weight matrix
At each intermediate recurrence layer, the residual component is calculated as follows, where W x ∈ R D h ×Dx , W r s ∈ R D h ×D h , and b r s ∈ R D h are input-to-residual weight matrix, hiddento-residual weight matrix, and residual bias, respectively. The residual gatingĝ r t is also calculated in similar manner by replacing 'tanh' with 'sigm'. Here, W r s and b r are dependent on the recurrence layer r. In the proposed model, the recurrence depth is time-varying and can be very deep. Learning all weight matrices for all possible depth is impractical. One solution is to use a shared weight matrix. However, this constraint reduces the ability of feature representation. To solve this problem, we propose a dynamic weight matrix utilizing the concept of hypernetworks [13].
At t time step and r intermediate recurrence layer, for the calculations of the residual component and residual gating, we use a dynamic weight matrix that is different for each calculation and defined as follows.
where, ∆W r t is obtained by hypernetwork that is a simple RNN with small number of hidden units compared with the number of main hidden units. The hypernetwork takes input as the previous residual component s r−1 t and residual gatingĝ r−1 t and calculates the hidden hypernetwork state where W zh ∈ R Dz×D h , W zg ∈ R Dz×D h , W z ∈ R Dz×Dz , and b z ∈ R Dz are model parameters of the hypernetwork. From the hidden state of the hypernetwork, we calculate a diagonal ∆W r t as follows, where P ∈ R D h ×Dz is a projection matrix. With the ∆W r t , we consider gated weight update as follows.ḡ We tried various ways of updating the parameters (e.g. weight scaling of hypernetworks in [13], weight matrix decomposition proposed by Jurgen Schmidhuber in [14], etc.) and found diagonal update to be the most effective.
Finally, the procedure of the state transition with adaptive recurrence depth is described in Algorithm 1. Computation graph of the proposed network is shown in Figure 2.  (2)) end for Add sequence [x1, · · · , xT ] to X end for Output: X

Synthetic data
This section aims to show the effectiveness of the adaptive recurrence depth on a synthetic dataset by comparing with other recurrent networks. To this end, we first constructed a synthetic dataset for sequential regression task.
The considered task is defined as predicting two-dimensional real vector of next step after observing real vector sequence up to the current step. The synthetic dataset is generated as described in Algorithm 2. The inputs, N, T, R max , θ, are the number of samples, sequence length, maximum recurrence depth, and affine transform parameter, respectively. We set the inputs as (N, T, R max , θ) = (10000, 21, 10, π/6). The synthetic dataset is divided into the training set, validation set, and test set with the size of 8000, 1000, and 1000, respectively. For all the experiments in this paper, Tensorflow toolkit [15] was used. For training the network, Adam optimizer [16] was adopted with 20 mini-batch size, 100 epochs, and 0.01 learning rate. Each model was trained five times with different initial conditions.  To confirm the effectiveness of the elastic gating, we compared 1) an RHN with shared parameters for the intermediate recurrent layers (SRHN), with 2) an RHN with shared parameters and elastic gate (SREHN) for the synthetic dataset. The hidden dimension was set to 20. SREHN showed 0.54 MSE, Now, performances by varying the recurrent depth of SRHN from 1 to 6 produced 0.81, 0.63, 0.58, 0.55, 0.55, and 0.54 MSE. From the results, we can say that the elastic gate helps in finding a recurrent depth that performs comparable to the best RHN with fixed recurrent depth. And the proposed model EI-REHN which is a model that the dynamic weight matrix is added to SREHN showed 0.47 MSE. From this result, we can say that the dynamic weight matrix gives rise to performance gain from 0.54 to 0.47.
We compare the proposed model with RNN, LSTM and RHN on the synthetic dataset by varying model structure parameters such as hidden dimension D h and recurrence depth R. Table 1 shows the MSE results with the number of parameters corresponding to each model. As shown in Table 1, RNN shows the worst performance. The proposed model shows better MSE performance with a smaller size model than other models.

Human activity recognition
In this subsection, we describe sequence classification experiments by using Human Activity Recognition Using Smartphones Data Set (HAR) [17]. In the HAR dataset, 30 persons performed six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STAND-ING, LAYING) wearing a smartphone on the waist. 3-axial linear acceleration and 3-axial angular velocity at are captured by using its embedded accelerometer and gyroscope. The dataset is divided into 7352 training sequences and 2947 test sequences.
For the human activity recognition on the HAR dataset, the input sequence is modeled by RNN, LSTM, RHN, and EI-REHN. Two-layer recurrent networks are used for this experiments. Six-class softmax layer is added to the last hidden units in the top hidden sequence for classification. To train the models, cross-entropy loss and Adam optimizer with 200 mini-batch size, 100 epochs, and 0.0025 learning rate are used. Table 2 shows the sequence classification results. In sequence classification, it is important that the last hidden units in recurrent neural networks have to contain information about the whole sequence.
In the RHN case, as the recurrence depth increases, the performance decreases. We believe that the performance degradation of the RHNs from shallow recurrence depth occurs since the intermediate state transitions vanish the early information of sequence contained in the hidden units in spite of the short-cut path in the RHNs. Nonetheless, the proposed model shows better performance than other models in terms of classification accuracy.

Language modeling
In this subsection, word level language modeling on the Penn TreeBank dataset [18] is described.
The model architecture is represented in Figure 3. For vocabulary of size C, the one-hot vector is used to represent the word input at time t, w t ∈ R C and word embedding vector is obtained by Uw t , where U ∈ R H×C . The word embedding vector is passed into EI-REHN. The hidden activation of EI-REHN, h t , is multiplied by the transposed word embedding matrix, U T based on [19,20]. Finally, softmax layer is applied to calculate the next word predictionŵ t+1 .  In Table 3, we compare EI-REHN with RHN under the fixed number of hidden units. For the parameter optimization, we follow the experimental settings described in [10]. We just replace the 'RHNCell' class in the code of [10] with an 'EIREHNCell' class that is an implementation of the proposed model. The recurrence depth for RHN is varied from 2 to 5 while the maximum recurrent depth is set from 2 to 5. The hidden state size D h and the hypernetwork state size D z are set to 1200 and 600, respectively. The number of parameters of RHN increases as the recurrence depth increases, while the number of parameters of EI-REHN doesn't. The EI-REHN with the maximum recurrence depth of 4 shows the best performance in this experiments. This result supports that the elastic gating of EI-REHN makes the model to fit in early recurrence depths compared with RHN.
In Table 4, we compare our models (D h = 1000, 1200) with the basic LSTM [21], Pointer Sentinel networks [22], Ensemble of LSTMs [21] and RHN [10]. From the results, the proposed model shows comparable performance with RHN and better than the other models in terms of perplexity. The reported result of RHN was with ten recurrence depth, whereas the proposed model reached the maximum performance with four recurrence depth.

Conclusion
To model time-varying nonlinear temporal dynamics in sequential data, a recurrent network capable of varying and adjusting the recurrence depth between input intervals was examined. By incorporating into the recurrent network that combines a shortcut path with a residual path with a rectified residual gating function which is best described as a rectified exponentially decreasing function, the network is capable of having varying recurrence depth. Moreover, we propose dynamic weight matrix construction for recurrence layers. This capability extends the capacity of existing recurrent network.
To substantiate the effectiveness of the proposed network, we conducted three experiments that are a regression on the synthetic data, human activity recognition, and language modeling on the Penn Treebank dataset. The proposed networks showed better performance than other state-of-the-art recurrent networks in all three experiments.