{'fraction_non_alphanumeric': 0.03168560712086436, 'fraction_numerical': 0.009768795834966131, 'mean_word_length': 4.628071833648393, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Genesis of Basic and Multi-Layer Echo State Network Recurrent Autoencoders for Efficient Data Representations

It is a widely accepted fact that data representations intervene noticeably in machine learning tools. The more they are well defined the better the performance results are. Feature extraction-based methods such as autoencoders are conceived for finding more accurate data representations from the original ones. They efficiently perform on a specific task in terms of 1) high accuracy, 2) large short term memory and 3) low execution time. Echo State Network (ESN) is a recent specific kind of Recurrent Neural Network which presents very rich dynamics thanks to its reservoir-based hidden layer. It is widely used in dealing with complex non-linear problems and it has outperformed classical approaches in a number of tasks including regression, classification, etc. In this paper, the noticeable dynamism and the large memory provided by ESN and the strength of Autoencoders in feature extraction are gathered within an ESN Recurrent Autoencoder (ESN-RAE). In order to bring up sturdier alternative to conventional reservoir-based networks, not only single layer basic ESN is used as an autoencoder, but also Multi-Layer ESN (ML-ESN-RAE). The new features, once extracted from ESN's hidden layer, are applied to classification tasks. The classification rates rise considerably compared to those obtained when applying the original data features. An accuracy-based comparison is performed between the proposed recurrent AEs and two variants of an ELM feed-forward AEs (Basic and ML) in both of noise free and noisy environments. The empirical study reveals the main contribution of recurrent connections in improving the classification performance results.

this labor-intensive work does not guarantee performance to confront new problems. The need to automatically extract better features has led to develop corresponding machine learning techniques. They tend to provide a convenient illustration of data in order to ensure promising accuracies. Machine learning-based feature extraction is a dominant subject following the achievements of deep learning [14]. Indeed, many researches attested the insightful modeling capability of Deep Neural Networks (DNNs). They have been hugely applied in various fields related to images and signal treatments [15]. Autoencoder is a kind of DNN models. It transforms an input sequence into output while minimizing as much as possible the distortion amount [16]. It is an artificial unsupervised feed-forward neural network which provides efficient data coding. It learns certain data representation so as to render a dimensionality alteration of these data (either reduction or expansion or keeping unaltered). Lately, the concept of AE was more widely indented in learning generative data models. Many variants of AEs have been developed in the literature. For instance, denoising AE is a stochastic version of AE that performs two things. It tries to encode and preserve the inputs while undoing the impact of a corruption process (noise) applied to these input. Bengio et al. [1] gave a model of Denoising AE (DAE) in order to reconstruct a clean repaired input from a corrupted one. They provided therefore a new way to extract robust features. Su et al. [17] used a stacked denoising AE to detect frequent changes from multi-temporal images. Sun et al. [18] proposed a sparse AE-based approach for induction motor faults classification. B Vincent et al. [19] used sparse AE for human body detection. A plenty of other works are using AE for feature extraction in various domains. Overall, the majority of implemented AEs are based on a feed-forward fashion. Although they have get good classification results, they suffer from several deficiencies. The main drawback of AE in general is the fact that its training is based on gradient-based methods such as the back-propagation algorithm. In fact, this last is likely to converge to local minima. Also, the retro-propagation of the error signal throughout the whole network during several iterations may result to its explosion or vanishing. In fact, Feed-forward neural networks (FNN) [20] have been well investigated and highly used since the emergence of the famous Back-Propagation (BP) algorithm [21]. Traditional BP is a first order gradient algorithm for parameter settings. It is used in Neural Networks to optimize weights values. However, BP can result on very slow convergence as well as local minimum problem [22]. Also, the error signal, when back-propagated throughout layers and neurons may either vanish or explode. To remedy this problem, researchers have studied several methods to boost the optimality of BP-based training, such as second order optimization methods, subset selection methods or global optimization methods [23]. Although, they came up with esteemed results namely fast training speed and/or higher generalization ability, the majority of these methods are still unable to guarantee a global optimality. Extreme Learning Machine (ELM) [24] is a popular single hidden layer feed-forward network. It is renowned for the quick learning speed and the high accuracy that it reflects. It has been implemented in multiple applications [24] [25]. In ELM, hidden nodes are randomly initiated then stay unaltered without the need of iterative tuning. The unique degree of freedom is located at the level of readout weights which relate the hidden neurons and the output ones. ELM boils down to solving fastly many problems. Inspired by the efficiency of the AE in feature representations and the effectiveness of ELM in systems modeling, researchers proposed ELM based AE (ELM-AE). In this purpose, Zhu et al. [26] implemented a hierarchical ELM-AE based approach to learn representations from massive unlabeled data. Wang et al. [27] developed a convolutional AE extreme learning machine network for 3D feature learning. On the other hand, Recurrent Neural Networks RNNs [28] [29] are known by their capability to deal with complex nonlinear problems as compared with FNNs. The recurrent feedback connections create a memory in which past captured information are stored and used to model the coming ones. This fact creates dynamism within the network. Echo State Network [30] is a recent RNN based on a hidden random reservoir at the level of the hidden layer. It requires a reduced execution time to converge [31] [32]. Its training method is alike that of ELM. Some researchers have got to a conclusion that ESN is the recurrent form of ELM [33]. ESN has been successfully involved in several tasks such as prediction, classification, etc [34]. It is believed frequently that RNNs can perform only on temporal data as they are recurrent. However, there are many works in the literature that used recurrent architectures such as works in [35] and [36]. They studied an ESN-based approach to deal with classification problems. The efficiency of ESN dynamics provided by Reservoir Computing (RC) [37] [38] method has encouraged the idea of bestowing it as an AE. Typically, ESN is a three layered network. It has an input layer randomly connected to the next hidden layer which is the reservoir [39]. The reservoir neurons are linked to each other through a pre-determined sparse weight matrix. Only the synaptic weights from the reservoir to the output layer are trained using linear regression training. In this paper, we will extend the focus on Multi-Layer ESN [36]. Thus, a number of additional hidden weights will lead to a stronger synaptic connection between neurons of more than one reservoir. The main idea of this paper is to mix the makings of ESN (a recurrent network) and those of AE in an ESN-RAE framework to create efficient, effective and more accurate data representations. ESN is considered here as a new recurrent representational space of the original data patterns. ESN is studied as a basic and multi-layer network. It does not matter whether the data are temporal or not. What counts is the quality of the new features when extracted from the original ones via ESN-RAE. In the literature, there is just a few researches which used the common AE to inject new data representations in ESN for classification or recognition purposes such as the work in [40]. Practically it is the first time that ESN is used as an AE. To ensure the efficiency of the proposed RAE, this latter's behavior is studied when a noise is randomly squirted into the training and / or testing databases. The novelty of our contribution can be summarized as follows. 1) A novel non gradient-based recurrent ESN-RAE is proposed. This last avoids the problems of vanishing or exploding gradient. It provides a very simple training minimizing thereafter both the complexity and the execution time of the approach.
2) This study is extended to deep ESN with a multitude of reservoirs. Each hidden layer (reservoir) adds its own level of non-linearity that cannot be contained in a single layer. Hence, the data undergo many transformations before achieving the last one. New expressive features may appear after a series of transformations. 3) Adding a denoising aspect to the already proposed RAEs variants by injecting different gaussian noises into the studied datasets. The stability of the proposed RAE's behavior is studied. The remainder of this paper is partitioned into four sections. In section 2, a thorough study about AE is performed. In section 3, Basic and Multi-layer Echo State Network AE ESN-RAE and ML-ESN-RAE approaches are described. Details about basic and ML-ESN are given too. The already proposed frameworks are applied to a number of data sets for classification purposes throughout section 4. Several simulation results are given and a comparison with ELM feed-forward AE (Basic and ML) is performed in section 4. Two cases are studied: noise free and noisy environments. Conclusions and outlook for future work are drawn by the end of the paper.

BASICS OF AUTOENCODER
Autoencoders [41] are one of the most promising tools capable of automatically furnishing efficient features from original unlabeled data. An Autoencoder [42] is a fully connected feed-forward neural network. It is based on unsupervised learning. It is an automatic framework resulting on specific data representations. It pre-processes data to be used for classification purposes. The goal of an AE is to make input unlabeled data to be equal to the output ones. Two parts are to be distinguished in this kind of network; the encoding and the decoding phases. In the case of single-layer AE, the encoder and the decoder are gathered in a symmetrical structure (see Fig.1). The coding phase provides learned features extracted from the training data set. This extraction is performed following an unsupervised learning scheme. One-layer AE has one input, one output and one hidden layers. As shows Fig. 1, the input weight matrix linking the inputs to the hidden layer constitutes the encoder. The readout weights between the hidden neurons and the readout ones comprise the decoder. The dynamics of the AE are computed according to equations (1) and (2).
x = fe(W in u + be) where be and bd and denote the bias vectors. fe and fd represent activation functions of the encoder and the decoder, respectively. u, x and y are the input, hidden and output states. As the AEs tend to make an approximation of the identity function, the output y is similar to u. W in and W out designate the input and output weight matrices.

BASIC AND MULTI-LAYER ECHO STATE NET-WORK AUTOENCODERS
Recurrent Neural Networks (RNN) constitute an attractive kind of computational models successfully serving for nonlinear processing. To overcome RNN shortcomings especially those related to vanishing gradient, Jaeger [30] designed a specific recurrent architecture under the name of Echo State Network. This last is overall based on Reservoir Computing (RC) concept, [43]. Recently, RC has ceaselessly attracted machine learning community. Reservoir-based networks are characterized by a highly non-linear expansion provided by randomly recurrent connected neurons [44]. Basic and Multi-layer ESN variants are described in the coming subsections.

Basic Echo State Network(ESN)
The basic model of an ESN is shown in Fig. 2. Two parts are to be distinguished in such networks. The first part includes a random sparse recurrent hidden layer. It is called reservoir. It is a highly non-linear space allowing the inputs dimensions increasing. The dynamic neurons interconnected within this reservoir are activated via nonlinear function (hyperbolic tangent). The second part is a simple readout layer where the outputs are obtained by a product of the reservoir states by an output weight matrix [30] [45].

Fig. 2: Basic ESN model
Suppose K, N and L the input, hidden and output neurons, respectively. W in ∈ R N*(K+1) and W ∈ R N*N designate the inputs plus the bias. W out ∈ R L*N denotes the readout weight matrix. Only these weights are to be trained.
., yL(n)] represent, respectively, the inputs, hidden states and outputs of the network of the n th pattern . p is the number of patterns within the training dataset. The dynamics of ESN are represented by equations (3) and (4).
f and f out are the activation functions of the reservoir and the output layer, respectively. Generally, the hidden function is non-linear while the output one is linear. Once the initial parameters are set, the training can be launched according to teacher-forced data. Thereafter, the reservoir states are aggregated in a matrix H. Each column in H corresponds to the hidden activation states for one pattern. The output weights are computed according to equation (5) then injected to matrix R.
pinv(H) is the pseudo-inverse of H. yt represents the desired outputs corresponding to the teacher-forced inputs. The network outputs y are calculated based on equation (4).
During the test phase, inputs from testing dataset are squirted to the network. Once computed, the obtained outputs are compared to the desired ones. The result of this comparison indicates whether the reservoir is doing well his job or not. As the Dynamic Reservoir (DR) is the key concept of ESN, it should be well designed in order to maintain a good performance of the whole network. The number of neurons as well as the connectivity rate intervene both in conceiving the structure of the DR. In conventional ESN, there are no precise formulas in choosing these parameters. There are just some recommendations to use a big number of neurons which are sparsely connected. Enlarging the reservoir as much as it is possible may give good precision results in terms of errors but it can lead to network complexity increase. Therefore, choosing optimal parameters to design a good reservoir is very crucial in order to maximize the networks accuracy while minimizing its complexity. The accuracy of the network is computed according to equation (6).
Commonly, the setting of the DR size seems to be among the most challenging tasks. In fact, the number of hidden neurons intervenes in the choice of the number of non-zero connections within the reservoir. Furthermore, it monitors the networkj s memory capacity. Many literature findings have declared that choosing a large reservoir is very likely to offer an asset in modeling complex problems [30] as well as enhancing the inputs' non-linear expansion. In fact, having randomly connected neurons improves the projection of nonlinear expansions of input signals. Furthermore, maintaining stably rich dynamics is directly related to the sparsity degree of the hidden neurons. Sparsity rate means the number of zero weight connections whereas connectivity rate presents the number of non-zero connections. To do so, W is scaled by its spectral radius ρ which in its turn should be lower than 1 to realize the Echo State Property ESP. According to ρ, whose expression figures in equation (7), there will be either densely connected neurons with smaller synaptic reservoir weights or sparsely connected units with higher weights values [46]. Stability can be ensured in both of situations meanwhile other networkjs characteristics may be altered.

Basic Echo State Network Autoencoder (ESN-RAE)
In view of its achievements in dealing with several machine learning problems, ESN is used throughout this paper for creating new data representations. It plays the role of a recurrent AE. Inspired by the concept of AE, an ESN-based Autoencoder is proposed in this section. The ESN preserves its normal structure. The outputs are set to be equal to the inputs. Basically, ESN training is performed in two phases. The first step consists of a random feature mapping of the inputs. The second is a linear parameters solving. In fact, the hidden layer, once initialized, maps the input data into another feature space. This mapping is provided by the non-linear transformation when applying the hidden activation function (see Fig. 3). ESN is distinguished by its random feature mapping as it uses in general the tanh function. Thus, it differs from other learning machine techniques which use usually kernel functions to create data features.   Therefore the new equations of ESN-RAE are declared in the expressions (8) and (9).
The output y is equal to u as we aim to approximate the inputs. The same proceeding is to be applied to ML-ESN-RAE.

Multi-layer Echo State Network (ML-ESN)
Very early on, researchers insisted on the idea that multilayer networks may ensure better performance. Lately, this idea has virtually matured thanks to the impressive successes realized across a wide range of tasks [47] [48].
Recent studies have shown that deep architectures are capable of learning complex data distributions while achieving good generalization performance and efficient representation of patterns in challenging recognition tasks [36]. Deep architecture networks have many levels of nonlinearity, giving them the ability to compactly represent highly nonlinear complex mappings. However, they are difficult to train, since there are many hidden layers with many connections which cause gradient-based optimization with random initialization to get stuck in poor solutions [36].
The difference between ML-ESN and Basic ESN is the multitude of reservoirs within the network. In fact, ML-ESN is defined by a succession of interconnected reservoirs. The model of ML-ESN is visualized in Fig. 5. Overall, the aim of this paper is to exploit the high nonlinearity provided by the ESN js reservoir in creating another efficient representation of input patterns. Normally, the AE is used for that purpose.

Multi-layer Echo State Network Autoencoder (ML-ESN-RAE)
The architecture of ML-ESN-RAE is visualized in Fig. 6. The encoding part in ML-ESN-RAE is extended compared to that in the basic model. This time the new code is defined by the activations of the last reservoir neurons.  (14)- (16).
sume that all the reservoirs are of equal size K. x i designates the states vector of the i th reservoir, i = 1..M . W i denotes the inner recurrent weight matrix of reservoir i. W inter(i) defines the weights between neurons in reservoir i and those of the reservoir i + 1. It is of size K* * K. W in presents the input weight matrix relating the input neurons to those of the first reservoir. The readout weight matrix W out relates the last reservoir to the output layer. The dynamics of the (15) . . . reservoirs' states are defined in the equations (10)- (12). .
The data patterns, while propagating throughout the reservoirs, undergo several non-linear transformations. Indeed, each reservoir adds a non-linear transformation of these data. These transformations may extract many new expressive features that were non-visible in their original first form.
The decoding equation is given in (17).
where u(n) = (u1(n), ..., uK(n)) presents the input sequence of the pattern n of the database. The output y is equal to u as we aim to approximate the inputs.
The processing of the basic ESN is kept as it is except that we do not need ESN outputs anymore. Obtaining the reservoir states vector is the objective of this work. For each input sequence whether it was image descriptors or time series sequence, the corresponding states of the reservoir are collected in a matrix A. This last constitutes the new data presentation of the previous inputs. This new representation will be squirted into a classifier to fulfill the classification task. In this work, it is hypothesized that the optimal reservoir connectivity and size depend essentially on the problem to be solved. A number of ESNs with different random weights and architectures is generated. For all the generated (16) networks, the outputs are made to be equal to the inputs. This is to ensure the specificity of the AE. The inputs are processed to all the ESNs. For each network, a normal learning scheme is processed till deriving W out . The error metric is calculated for every network. The ESN (ML-ESN) architecture that provides the most reduced error value is picked out. Subsequently, the output weight matrix W out of the chosen network is injected to the input weight matrix to ensure the already cited condition (W in = (W out ) T ). It is the transposed matrix of W out . Therefore, the input weight matrix acquires the values of the output one. Consequently, the hidden neurons states should be recomputed because W in is already altered. In the case

EXPERIMENTAL RESULTS
Throughout this section, a thorough empirical study of ESN-RAE (basic and ML) is carried out. The proposed framework is tested on a number of time series classification problems. Experiments are performed on a number of databases extracted from the public "UCR Time Series Data Mining Archive". These benchmarks come from various background knowledge including a broad range of application fields. The evaluation metric is represented by the classification Error Rate (ER). This last is computed according to equation (18).
of ML-ESN, all the reservoirs activations are re-calculated too. In general, the training of overall existing feed-forward ML-AEs is done layer by layer. It means that the first hidden = (18) layer is trained first of all then its outputs are squirted as inputs to the second layer and so on till reaching the last hidden layer. This training subdivision is referred to the difficult training of these AEs. As the training of ESN is very simple and quick, we propose to train the layers at the same time. Fig. 7, the approach is mainly partitioned into two parts. The first part highlights the auto-encoding provided by the ESN-RAE (Basic and ML). This is the main phase in the approach during which a new data representation is derived from the application of input sequences to ESN-RAE. The reservoir states obtained after squirting every pattern from the dataset constitute the new representation of this pattern during the second phase. To boost the interest on recurrent architectures, two variants of feed-forward Autoencoders are implemented to be compared to the recurrent ones. They are based on ELM as it is a feed-forward AE non trained by gradient-based methods. Also, its resemblance to ESN training and architecture lead us to implement it and show the added value of recurrent networks versus feed-forward ones. They are as follows:

As indicates
 ELM-AE: When original data is pre-processed by the Basic Extreme Learning Machine Autoencoder then the corresponding hidden states are introduced to the SVM classifier.  ML-ELM-AE: When original data is pre-processed by the Multi-layer Extreme Learning Machine Autoencoder then the corresponding hidden states of the last layer are introduced to the SVM classifier.
All of these methods are compared to each others and to other already existent literature approaches. They are executed ten times and the classification ER corresponding to each run is stored. Then, the average of the sum of ERs is taken for evaluation.

Datasets representation
As already said, the studied recurrent RAEs are applied to a number of time series classification datasets except the Breast cancer dataset. They are briefly described below.
5.1.1. ECG 200 dataset ECG200 dataset was created by R. Olszewski [50]. It contains 200 heart beats signals are recorded. Each series or signal consists of 96 inputs. Two classes are defined here: a normal heartbeat and a Myocardial Infarction.

Breast Cancer dataset
This benchmark was generated by Dr. Wolberg [51] based on consecutive patients. Data items are extracted from digitized images. They contain illustrations of breast mass Fine Needle Aspirate (FNA). The diagnosis result will be either malignant or benign.

Coffee dataset
Two coffee bean varieties have attracted widely economic interest. They are Coffea Arabica and Coffea Canephora named also Robusta [52]. Commonly, Arabica species are more valued by the trade as they are characterized by a finer flavor.

Olive Oil dataset
Chemometrics, a task that has distinguishable applications in food safety and quality assurance, employ food spectrographs to classify food types. Each class of this data set is an extra virgin olive oil from alternative countries [52].

Earthquakes dataset
This dataset provides a prediction of whether a major earthquake is about to occur based on a set of recent readings within certain surrounding area [52].

Meat dataset
This dataset is a part of chemometrics. It is a food classification dataset. The classes are chicken, pork and turkey. Duplicate acquisitions are taken from 60 independent samples [52].
The specifications of the datasets are gathered in Table 1 where the train and test sizes designate the size of the training and testing datasets. Length denotes the input size. The number of classes for each dataset is given too. Seen that each dataset requires a specific ESN-RAE design, the configurations of the reservoirs used for every database figure in Table 2. N and β denote the reservoir's size and the inner connectivity, respectively. For the ML-ESN, the number of layers is chosen to be equal to 2. In fact, in [36] experiments were performed to set this number and they found that two reservoirs give better performance than just one or more than two. Also, in the same work [36] One other defiance is that in order to deal with real-world problems, a high-dimensional reservoir is usually chosen. Nevertheless, data cannot be usually obtained for a long time ago due to problems related to the shortage of data acquisition techniques [34]. Consequently, the datasets may not be so large. Here, an ill-posed problem may occur especially when the training samples number is less than the reservoir size. Significant further attention should be paid in this kind of situations. Achieving efficient networks' specifications is a hard labor task. Table 2 represents the final settings of the proposed recurrent AEs. Taking into consideration the already cited challenges, each dataset has its specific parameters. For some of them, a compression is made (the number of hidden neurons is less than the input one). For the others, a sparse expansion is performed.

Accuracy results
To permit a fair comparison between recurrent and feedforward AEs, the specification design conceived for ESN-RAE and ML-ESN-RAE is applied to ELM-AE and ML-ELM-AE. The performances of the proposed AEs are studied according to two scenarios: with and without a corruption process designated by a noise signal.

Case 1: Noise free results
As already said, creating new data features based on original ones permits sometimes to provide fitter data to the task at hand. Fig. 8 visualizes and highlights the difference between the original input data and the newly obtained features after being encoded by ML-ESN-RAE for some datasets. As a first remark, the initial input data sequences look alike. They have almost the same gait with some differences. Whereas, after being autoencoded by ML-ESN-RAE, the data get richer and more various forms regarding the original data forms. According to Fig. 8, the new obtained features (Fig.8 b),d),f)and h)) are restricted to the interval [-1,1]. The curves prove also that the obtained hidden activation states get some values nearby zero the fact that proves the sparsity aspect of the proposed ESN-RAE (Basic and ML). Tables 3-5 enumerate empirical accuracy-based results. In this case, the results are given in accordance to the classification error rate (ER). Each of the tables gives a comparison study based on the classification Error Rate (ER) between the proposed frameworks and other existent approaches that have already focused on this dataset. The tables highlight the efficiency of using ESN-RAE and especially the impact of adding another hidden layer within the ESN for the three datasets. This is thanks to the high non-linearity provided by ESN-RAE reservoir(s). The added value of data encoding is emphasized throughout the tables as the ER when squirting the original data directly into the classifier (SVM) is higher than that obtained after their encoding. The drop surpasses 15% (for all the datasets) for ESN-RAE and 30% for ML-ESN-RAE.  [53] 0.300 SVM [54] 0.290 LPP [55] 0.290 NCC [53] 0.230 N5S2 [56] 0.230 EDTW [55] 0.175 Jeongs Method [54] 0.160 N8S5 [56] 0.160 DSVM [55] 0.145  0.180 C-RT [51] 0.101 NB [47] 0.070 CART with feature selection [58] 0.054 C.45 [51] 0.052

0.047
Based on what reports the tables, it comes out that ESN-RAE realizes prominent classification accuracy. ML-ESN-  [52] 0.178 NN [52] 0.142 NB [52] 0.142 NNDTW [52] 0.071 Naive Bayes [59] 0.071  Table 3, ML-ESN-RAE's ER realizes a drop of more than 26% of that achieved by ESN-RAE. According to Table 4, for Breast cancer dataset, ESN-RAE (Basic and ML) shows noticeable performances as compared to other already developed methods. The new Breast Cancer data representation based on the reservoir states has improved considerably the classification error. In this case, ML-ESN-RAE achieves an ER lessening of about 27% of ESN-RAE's ER. Table 5 boosts the fact that Multi-layer RAE (ML-ESN-RAE) performs better than the single layer one and other proposed approaches in terms of accuracy. So far, providing a multitude of reservoirs comes out with an enhancement in the classification accuracy according to the three precedent tables. In order to ensure further the strength of the proposed approaches, additional tests are performed on another set of benchmarks taken from the UCR archive [38]. The testing results are gathered in Table 6. Also, a comparison with existent approaches figures within the table. From a dataset to another, the makings of the proposed ESN-RAE variants give proof of their efficiency and leverage. This is especially proved when comparing their results with those obtained when squirting the original data into the SVM classifier. This is the reason behind the insisting on the importance of all the variants of recurrent AEs conceived in this work. Overall, the tables ensure the potential of the studied ESN based AEs. Also, the use of more than one hidden reservoir influences highly the classification accuracy. In view of the results obtained so far, it appears that furnishing more than one reservoir in the hidden layer

Case 2: Gaussian noise addition
In order to study the robustness of the proposed recurrent AEs, a gaussian noise signal is squirted randomly in both of training and testing databases. Four noise levels are studied. Each noise signal is designated by a signal to noise ratio SNR. The noise level is inversely proportional to the SNR. The lower the SNR is set, the higher the noise becomes. The SNR is expressed in dB and the four levels correspond to SNR=50dB, SNR=10dB, SNR=1dB and SNR=0.5dB. Table 7 illustrates a statistical comparison study between ESN-RAE, ML-ESN-RAE, ELM-AE and ML-ELM-AE according to the noise level. The best ER for each situation is written in bold. It is a normal fact that the results go down at each time the amount of noise signal increases. Since the recurrent AEs are supposed to overtake the feed-forward ones in the presence of uncertainties, the accuracy evolution of the four AEs is tracked according to the amount of noise. To analyze further the effect of noise squirting, the evolution of accuracy results is visualized throughout the histograms in Fig. 9. According to Fig. 9, recurrent AEs surpass feed-forward ones in overall the cases. Even with noise supplying, the precedence of ESN-based AEs is preserved in front of feed-forward ELM-AE variants.
In accordance with Fig. 9 and Table 7, it is quite clear that even when the noise level rises-up, the proposed approaches perform better than ELM-based AEs. For instance, in an extremely noisy environment when the SNR=0. 5 are computed, in Table 8, for all of noise levels in order to show thoroughly the advance of our recurrent AEs. They are multiplied by 100 in order to get the results in percentage. We can say that the ML-ESN-RAE's ER is equal to P 2% of ML-ELM-AE's ER and P 1% of ESN-RAE. This last, in its turn, gives an error equal to P 3% of that reached by ELM-AE.
As shows Table 8, the difference between feed-forward and recurrent AEs (Columns of P2 and P3) seems to be remarkable as it reaches sometimes more than 40% and 30% between ML-ESN-RAE and ML-ELM-AE. Even for the single layer AEs, the recurrent one is the winner in terms of classification accuracy. Based on the obtained values of P1, using more than a single reservoir reflects a prominent improvement in terms of precision results.
Hence, seen what reports the precedent tables and figures, it seems that recurrent AEs have greatly better noise tolerance as they retained their robustness and advance with and without noisy data compared to other methods. Handling high and low levels of disturbance ensures the efficiency and effectiveness of the conceived recurrent ESN-AEs.

CONCLUSION
Throughout this paper, Echo State Network is exploited as an Autoencoder for data representations. The high nonlinearity expansion of ESN and the strength of Autoencoder in dealing with feature extraction are merged together to bear ESN-RAE. This last is trained to provide new representation of the original data. This representation is obtained from the reservoir activation states. It encompasses other details about the data more meaningful than those of the original form. The new encoded data are squirted into a classifier. Two variants of ESN-RAE are proposed in this work which are the basic ESN-RAE and the ML-ESN-RAE. Both of ESN-RAE and ML-ESN-RAE yield considerable results in terms of classification accuracy when applied to a number of datasets. They realized an outperformance compared with existent literature approaches. In fact, the successive non-linear transformations created from a hidden layer to another may extract more significant data features. Experimental results are carried out in two kinds of environments: noise free and noisy data. A comparison study is performed between the proposed methods and other already existent approaches on a number of wellknown databases. The analysis of the obtained results shows eminent potential of ESN-AE and ML-ESN-AE in terms of classification precision. Also, they reveal the importance of providing recurrent connections within the AE. This fact is proven when comparing feed-forward ELM-AE (Basic and ML) to our recurrent AEs with and without disturbance process. Indeed, the ESN-AE's variants are more powerful in terms of accuracy and stability. Other variants of ESN-RAE are under investigation to improve the results already obtained further.