{'fraction_non_alphanumeric': 0.03561764705882353, 'fraction_numerical': 0.011941176470588235, 'mean_word_length': 4.426268751994893, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Stochastic Configuration Networks Ensemble for Large-Scale Data Analytics

This paper presents a fast decorrelated neuro-ensemble with heterogeneous features for large-scale data analytics, where stochastic configuration networks (SCNs) are employed as base learner models and the well-known negative correlation learning (NCL) strategy is adopted to evaluate the output weights. By feeding a large number of samples into the SCN base models, we obtain a huge sized linear equation system which is difficult to be solved by means of computing a pseudo-inverse used in the least squares method. Based on the group of heterogeneous features, the block Jacobi and Gauss-Seidel methods are employed to iteratively evaluate the output weights, and a convergence analysis is given with a demonstration on the uniqueness of these iterative solutions. Experiments with comparisons on two large-scale datasets are carried out, and the system robustness with respect to the regularizing factor used in NCL is given. Results indicate that the proposed ensemble learning techniques have good potential for resolving large-scale data modelling problems.


Introduction
Machine learning has received considerable attention over the past years due to its significant role for data analytics [13]. Under big data setting with decentralized information structure, advanced machine learning algorithms with robust and parallel implementations are needed along with the growth of data [24,25]. Various ensemble learning frameworks, aiming to improve the generalization performance of a learning system, have been developed over the last two decades, and many interesting ideas and theoretical works, including bagging, boosting, AdaBoost and random forests can be found in [1-7, 9, 12, 15, 16, 21, 23, 26]. Generally speaking, learning-based ensembles share some common nature in system design, such as data sampling and the output integration. The basis of ensemble learning theory lies in a rational sampling implementation for building each base learner model, which may provide a sound predictability though learning a subset of the whole data set.
For neural network ensembles [5,12,16,23], the base models are trained by the error backpropagation (BP) algorithm and the regularizing factor used in the negative correlated cost function can be determined by the cross-validation method. Unfortunately, BP algorithm suffers from the sensitive setting of the learning rate, local minima and very slow convergence. Therefore, it is challenging to apply the existing ensemble methods for large-scale data sets. To overcome this problem, we employed random vector functional-link (RVFL) networks [11,19] to develop a fast decorrelated neuro-ensemble (termed DNNE) in [1]. From our experience, DNNE can perform well on smaller data sets [1,15]. However, it is quite limited for dealing with large scale data because of its high computational complexity, the scalability of numerical algorithms for the least squares solution, and hardware constraint (here mainly referring to the PC memory). Recall that physical data may come from different types of sensors, localized information source or potential features extracted from multiple runs of some certain feature selection algorithms [6,10,17,18,20,22]. Thus, for largescale data analytics, it is useful and significant to develop a generalized neuro-ensemble framework with heterogeneous features. This paper is built on our previous work reported in [1], which is a specific implementation of the well-known NCL learning scheme using RVFL networks with a default scope setting of the random weights and biases. From theoretical statements on the universal approximation property in [11] and our empirical results on RVFL networks in [14], the default scope setting (i.e., [-1, 1]) for the random weights and biases cannot ensure the modelling performance at all. Therefore, readers should be aware of this pitfall and must be careful in making use of our code 2 . Limits of DNNE mainly come from the following aspects: (i) the system inputs are centralized or combined with different types of features; and (ii) the analysed method of computing the output weights becomes infeasible for largescale data sets, which is related to the nature of the base learner model (i.e., the number of nodes at the hidden layer must be sufficiently large to achieve sound performance). To relax these constraints and emphasize on the fast building of neuro-ensembles with heterogeneous features, we generalize the classical NCL-based ensemble framework into a more general form, where a set of input features are feed into the SCN base models separately. This work also provides a feasible solution by using two iterative methods for evaluating the output weights of the SCN ensemble (SCNE). In addition, some analyses and discussions on the convergence of these iterative schemes are given through a demonstration on the correlations among the iterative solutions and the pseudo-inverse solution.
The remainder of the paper is organized as follows: Section 2 provides some technical supports, including the basics of the SCN model, a generalized version of the ensemble generalization error and the negative correlation learning scheme. Section 3 describes the proposed SCNE with heterogeneous features, details two iterative learning algorithms and discusses their convergence. Section 4 reports some experimental results on two large-scale data sets, including a robustness analysis on the system performance with respect to the regularizing factor used in NCL. Section 5 concludes this paper with some remarks on further studies.

Technical supports
This section briefly reviews the stochastic configuration networks, extends the ensemble generalization error with heterogeneous features, followed by the negative correlation learning scheme for building ensemble models.

Revisit of stochastic configuration networks
SCNs are a class of randomized learner models which are recently developed in [28]. The unique characteristics of the SCN model, different from the classical randomized learner model (i.e., RVFL networks), is the way of generating the random input weights and biases. In contrast to RVFL networks, SCNs are built incrementally according to a supervisory mechanism, which constrains the random input weights and biases to take values in a data-dependent territory, namely stochastic configuration support (SCS). This constructive approach for building SCNs guarantees the universal approximation property of the resulting SCN model for a given nonlinear map. For the sake of completeness, we revisit the main theoretical result in Theorem 1 below.
Given a target function f : R d → R m . Suppose that an SCN model has already been built with L − 1 hidden nodes, i.e., is an activation function of the l-th hidden node with random input weights w l and bias b l . Denoted the residual error by e * ..} be a set of real-valued functions, and span(Γ) denote a function space spanned by Γ; L 2 (D) denote the space of all Lebesgue measurable functions f = [f 1 , f 2 , . . . , f m ] : The inner product of θ = [θ 1 , θ 2 , . . . , θ m ] : R d → R m and f is defined as Theorem 1 (Wang and Li [28]). Suppose that span(Γ) is dense in L 2 space and for any φ ∈ Γ, 0 < φ < b φ for some b φ ∈ R + . Given 0 < r < 1 and a nonnegative real number sequence {µ L } with lim L→+∞ µ L = 0 subjected to µ L ≤ (1 − r). For L = 1, 2, . . ., denoted by If the random basis function φ L is generated to satisfy the following inequalities: and the output weights are evaluated by Given a training data set with N sample pairs {(x n , y n ), n = 1, 2, . . . , N }, where x n ∈ R d and y n ∈ R m . Let X ∈ R N ×d and Y ∈ R N ×m represent the input and output data matrix, respectively; e L−1 (X) ∈ R N ×m be the residual error matrix, where each column e L−1,q (X) = [e L−1,q (x 1 ), . . . , e L−1,q (x N )] T ∈ R N , q = 1, 2, . . . , m. Denote the output vector of the L-th hidden node φ L for the input X by Thus, the hidden layer output matrix of f L can be expressed as H L = [h 1 , h 2 , . . . , h L ]. Denoted by The SC-III algorithm reported in [28] firstly generates a large pool of T max candidate nodes, namely {φ L , . . . , φ (Tmax) L }, in varying intervals. Then, it picks up those candidate nodes whose minimal value of the set {ξ L,1 , . . . , ξ L,m } is positive. Then, the candidate node φ * L with the largest value of ξ L = m q=1 ξ L,q will be assigned as the L-th hidden node for f L . Thus, the output weight matrix of the SCN model, β = [β 1 , β 2 , . . . , β L ] T ∈ R L×m , could be computed by the standard least squares method, that is, where H † L is the Moore-Penrose generalized inverse of the matrix H L , and · F represents the Frobenius norm [8].
If there is no candidate node that satisfies these conditions, the SC algorithm will automatically increase the value of r to relax the constraints and further adjust the scoping interval to generate a new pool of candidate nodes. This process continues until the residual error decreases to a predefined tolerance τ . To speed up the procedure of building SCN models, we could add the top n B ranked (according to the values of ξ L ) candidate nodes as a batch in each incremental loop of the SC algorithm. With a proper setting on n B , the batch version of SC-III algorithm could greatly shorten the training time without weakening the prediction performance of the SCN model. Readers could find more details about the proof of the theorem and the SC algorithms in [28].

Generalization error of ensembles with heterogeneous features
For data regression, an ensemble modelf that approximates an unknown target function g could be accomplished by fitting a collection of training samples D t = {(x n , y n ), n = 1, 2, . . . , N }, where x n ∈ R d , y n ∈ R, and the sample pair (x n , y n ) is an independent and identically distributed (i.i.d) sample from an unknown joint probability distribution p(x, y). Note that the training set D t is a realization of a random sequence D that shares the same distribution p(x, y).
Suppose that the input x is concatenated by M parts: Since the estimated θ * m depends on the given D m , we write θ * m (D m ) to clarify the dependency of D m . Therefore, an output value of the f m for an input x (m) should be written as f m (x (m) , θ * m (D m )); for simplicity, denoted by f m (x (m) ; D m ). The ensemble output for an input x is defined as where M m=1 a m = 1 and 0 < a m < 1, which is the weight of the m-th base model. For the case of a m = 1/M , the base models independently and equally contribute to the ensemble output, termed as the Naive method in this paper.
Let (X 0 , Y 0 ) be a random sample, taken from D but independent of D t , where X 0 = [X (1) 0 , X ]. In the light of the bias/variance decomposition by Geman, Bienenstock and Doursat [7], the ensemble generalization error expression by Ueda and Nakano [27], and the diversity discussion by Brown and Wyatt [4] for the ensemble modelling, we extend the result on the generalization error of the ensemble estimator as follows.

Theorem 2.
Let G E (f ) denote the generalization error of an ensemblef in Eq. (10). Then, we have where V ar(X 0 ), Cov(X 0 ) and Bias(X 0 ) are the average conditional variance, conditional covariance and conditional bias of the M models with heterogeneous features for X 0 , respectively, that is Bias Cov where Therefore, managing the covariance term Cov(X 0 ) explicitly helps in controlling the divergence of the base models, leading to a better generalized ensemble. In practice, samples from a validation data set D v , which is i.i.d to D t , could represent the random sample pair (X 0 , Y 0 ) in Theorem 2, and the validation error could be regarded as a realization of the generalization error of the ensemble.

Negative correlation learning
The negative correlation learning (NCL) is a typical training scheme to build neural network ensembles [7,9]. The key idea behind this learning algorithm lies in reducing the covariance among the base models while keeping the variance and bias terms of the ensemble not to be increased. Mathematically, the cost function of the m-th base model over the training data D m in NCL is given by where and 0 < λ < 1 is the regularizing factor. Note that Hence, Eq. (15) can be rewritten as The NCL scheme aims to find all the parameters (including the weights, biases and regularizing factor) of the M models through minimizing every e m .

Stochastic configuration networks ensemble
In [1], RVFL networks are employed as the base models to build a neural network ensemble where all the base models share the same input and identical architecture. In the construction of the DNNE, we used the pseudo-inverse method to evaluate the output weights of RVFL models. However, RVFL networks cannot perform at all if the scope of the random weights and biases is not chosen properly [14,28,29]. Also, it is difficult to estimate the number of hidden node of each RVFL base model in the ensemble. Due to the merits of the SCN model for overcoming these issues associated with RVFL networks, we employ SCNs as the base learner models with heterogeneous features to build SCNE.
For large-scale data modelling, the pseudo-inverse method used in the DNNE for evaluating the output weights of the ensemble is limited. Therefore, the block Jacobi method and block Gauss-Seidel method [30], are used to calculate the output weights of SCNE model. The proposed SCNE ∈ R dm and y n ∈ R. Each set will be used to build the SCN base model in the ensemble. The partition of the full feature set could be done by a prior knowledge or various feature selection schemes.
The procedure of building the SCNE model could be divided into two steps: (i) the SCN base model generation, that is, each SCN model will be built independently according to the SC-III algorithm in [28]; and (ii) the ensemble construction by means of the NCL algorithm, where the random weights and biases of the SCN models generated in the first step are fixed.

SCN base model generation
The SCN base models can be constructed individually with L m nodes in the hidden layer on the data set D m , and the output weights β m are also calculated according the stochastic configuration algorithm (SC-III) in [28]. One of the important issues in building SCNs is associated with the overfitting problem when too many hidden nodes are added incrementally. Fortunately, this problem can be solved by monitoring the modelling performance over a validation set with a quick pruning method. Alternatively, we can apply the trial-and-error method to determine an appropriate number of the hidden nodes of the SCN base model.

SCNE construction
In the ensemble construction, the input weights and biases of the base models are fixed. The output weights β 1 , β 2 , . . . , β M of the SCNE model can be obtained by using the NCL scheme, that is, The following sections detail the pseudo-inverse method and two iterative methods for computing the output weights. To understand some properties of the solutions obtained by these three methods, a demonstration over a 2-D function approximation is given to see the time cost and the impact of the regularized model on the correlationship among the solutions.

Analytical solution
By Eq. (18), the cost function e m of the SCN model f m with the NCL penalty term could be rewritten in the following matrix form: where . (21) The H m is the output matrix at the hidden layer of the m-th base model; B is the output weighs of the whole ensemble (all the output weights of the base models); y is the target; L is the total number of the hidden nodes the SCNE model. Simple computations give that From Eq. (22), a huge sized linear equation system can be obtained, that is, where If all the H T m H m s are invertible, which means all the SCN models are built with a full column rank of H m , then H is invertible if the regularizing factor λ takes a sufficiently smaller value. In this case, B = H −1 H T y. However, we cannot ensure full rank condition for every matrix H T m H m in system implementation. In such a case, the output weights of the SCNE model can be evaluated by where the H † represents the pseudo-inverse of the matrix H [8].

Iterative solutions
Due to some constraints on either the numerical algorithms for computing the pseudo inverse of a huge sized matrix or computing device with limited RAM memory, the analytical solution described in Algorithm 1 becomes impractical and/or time-consuming for large-scale datasets. Therefore, it is necessary and important to develop iterative algorithms for problem solving. Instead of evaluating the whole output weights B at once, we update gradually the β of each SCN base model to reduce the computing time and memory cost.
The minimum of the cost function Eq. (20) with respect to the local output weights β m at k-th iteration can be obtained by solving the following equation: where X ∈ {J, G}, indexing the Jacobi and Gauss-Seidel iteration schemes, respectively [30].
Taking the initial values of the output weights as (27), we can derive the following block iterative algorithms: The pseudo codes of these two algorithms are given in Algorithms 2 and 3 below. The main difference between these two schemes occurs at line 13. In practice, we provide a tolerance τ and a maximum iteration number k max experimentally to guarantee that the algorithms will be terminated if either the SCNE error E ens < τ or the iteration time reaches the k max . Note that Eqs. (28) and (30) are rewritten in discrete summation forms in the algorithms for the convenience of programming.    [30] converge to the same solution as obtained by the pseudo-inverse method. To do so, let us rewrite the block Jacobi iterative scheme in the the following matrix form: (33) Similarly, the block Gauss-Seidel iterative scheme can be rewritten as Thus, we conclude that the two iterative schemes converge to the same solution from the pseudo-inverse method provided that (i) every H m is full column rank; (ii) the regularizing factor λ is chosen so that the spectral radius of the iteration matrix is less than 1, that is, By using Greschgorin Circle Theorem [8] for block matrices, we know that any eigenvalue z of the matrix D −1 R must be constrained by one of the following inequalities: Hence, a theoretical estimate of the regularizing factor λ should be subjected to c 2 c −1 Simple computations give that .    (28) and (30), the new initial B (0) and the iterative equations of β m can be updated as follows: To reduce the algorithm complexity, we take equal values for r 1 , r 2 , . . . , r M , namely r, in Eq. (42).

Demonstration
A synthetic dataset is used to demonstrate the training time along with the number of the hidden nodes L in constructing the SCNEs with the pseudo-inverse, block Jacobi and block Gauss-Seidel methods, respectively. The reason behind this set-up is that a normal PC can run this task as the L grows up, and the memory cost of the computer is controllable compared to the large datasets. In this section, we report two sets of results obtained from the original SCNE and the modified SCNE with a regularization factor r = 0.1. From this comparison, we can infer that a similar consequence holds for large-scale datasets.
The synthetic dataset D = {(x 1n , x 2n , y n ), n = 1, ..., 5000} contains 4,000 training samples and 1,000 test samples, all generated by a simple function y = cos(2 × x 2 )/e x1 where x 2 = sin(x 1 ), 5,5]. This dataset is easy to learn by the SCNE, so we omit the training and test results in the context in order to focus on the time comparisons. In this demonstration, all the SCN base models use the same training datasets. By setting the M = 10 (10 base models), λ = 0.1, τ = 10 −6 , k max = 5, and gradually increasing the hidden node of each base model L m from 10 to 500, the computing time spent on SCNEs construction by using the three algorithms is presented in Fig. 2a (original) and Fig. 2c (regularized), respectively.
It is clear to see that the computing time of the pseudo-inverse method grows exponentially, while the computing time of the iterative methods grow linearly. The iterative methods may cost more time when the hidden node is small, but when L becomes larger (L > 2000 in this demonstration), the pseudo-inverse approach is clearly inefficient and infeasible for large-scale datasets.
With regard to the convergence of the output weights B from different algorithms, Fig. 2b shows the correlations between these three algorithms. Here, A1, A2 and A3 represent the pseudo-inverse, the block Jacobi and the block Gauss-Seidel methods, respectively. The results show that with these settings, the correlations of the A1-A2 and A1-A3 decrease along with the increasing L, meanwhile, the correlation coefficient of A2-A3 is consistent, implying that the iterative methods converge into the same solution. This phenomenon is caused by the use of the pseudo-inverse in evaluating the output weights. With the increasing L, the rank of H drops sharply, leading to an uncorrelated result to the iterative methods. When all the pseudo-inverse calculations of the algorithms are replaced by a regularized form with r = 0.1, the correlations of B between all these algorithms are almost one, as shown in Fig. 2d. Note that these correlations are not strictly one for the smaller L, this is because the H m is of full column rank in this case.

Performance evaluation
This section reports the experimental results of the SCNE on two large-scale datasets. We first introduce the datasets and explain the heterogeneous feature generation process. Then we present all the details of the experimental settings and results. The comparisons between the SCNE and the DNNE are carried out with remarks, and a robustness analysis of the SCNE is discussed at the end.

Large-scale datasets
The SCNE are used to explore two large-scale datasets, which are (i) Buzz Prediction on Twitter (Twitter) 3 4 . Table 1 shows the samples, features, values and training/testing split. These datasets have been analysed previously with some results.

and (ii) Year Prediction MSD (Year)
The Twitter dataset is used for the annotation prediction of the mean number of active discussion (NAD). This target is a positive integer that describes the popularity of the sample's topic. Each sample covers weeks of observation for a specific topic and is described by 77 features. All the observations are independent and identically distributed.
Year dataset is used for the prediction of the release year of a song from audio features. Songs are mostly western, commercial tracks ranging from 1922 to 2011, with a peak in the year 2000s. Among the 90 features, there are 12 belongs to timbre averages and 78 belongs to timbre covariance. The first value is the year (target), ranging from 1922 to 2011. All the features extracted from the timbre features from The-Echo-Nest-API 5 . The average and covariance are taken over all segments, each segment being described by a 12-dimensional timbre vector.

Heterogeneous feature generation
There are various methods to generate the heterogeneous feature sets for the base models. In our experiments, the 77 features of Twitter dataset are partitioned into 11 groups based on the introduction of the dataset. Each group has its specific meaning. The details, such as group names, feature indexes and explanations are listed in Table 2. According to the data description of the Year dataset,  the total 90 features could be partitioned into two groups, timbre averages group (12 features) and timbre covariance (78 features) group. We equally split the timbre covariance feature group into 6 small groups (TC-a to TC-f). Thus each small timbre covariance feature group contains 13 features, which is close to the timbre average feature group. The details could be found in Table 3.

Experimental setup
As a common knowledge in machine learning, data preprocessing plays a crucial role before modelling. Normalization and standardization are two widely used methods for rescaling data. The 0-1 normalization could scale all numeric variables into the range [0, 1], one possible formula is The z-score standardization transforms the data to have zero mean and unit variance by the formula x new = (x −x)/σ, which indicates how many standard deviations an element is from the mean. However, if there were outliers in the dataset, normalization will certainly scale the "normal" data to a very small interval. Our experiments aim to demonstrate the capability of the SCNE for large-scale datasets, here we choose the 0-1 normalization method for data preprocessing and assume there is no outlier in the dataset. To show the data distribution, we randomly select a small batch of samples from each dataset and present them in Figs. 3 and 4, respectively.    All the experiments are designed, repeated and followed the same procedure. Fig. 5 presents the general experimental diagram. The arrows indicate the direction of the data feeds. The experiments are designed in two stages, training and testing, indicated by the dash-lined box in the diagram. In the training stage, the training dataset is used to build the SCNE, and the validation data is used to adjust and refine the hyper-parameters of the ensemble, such as {S}, M , λ, L max and k max (In our experiments, the heterogeneous feature set {S} and the base model number M is predefined). When all these parameters are properly estimated, the training and validation data will be used together as one combined set to retrain the SCNE again; then this final ensemble will be tested on the testing dataset. The testing results could be seen as a good indicator for the generalization ability of the SCNE. Unlike some benchmark dataset, there is no ready-made training or testing sets of Twitter data, which leaves us various choices to partition the data. For Twitter data, we apply 70% of the total samples for training, 15% for validation and the rest 15% for testing, denoted as "70-15-15". The Year dataset has been specified with the testing data (almost 10%), so we randomly split the rest 90% samples into two parts 70% for training and 20% for validation, denoted as "70-20-10".

Results and discussion
The following section presents all the training, validation, testing results and parameter estimations of SCNE on Twitter and Year datasets. All the experimental conditions and details are stated and specified as follows for the convenience to repeat the experiments. The comparative ensemble is the DNNE with the same structure of the SCNE.
For large datasets, such as the Twitter and Year in this paper, the general training-validation procedure for estimating hyper-parameters or preventing model overfitting usually costs a large amount of time. According to the SC-III algorithm, each base model needs to compute the results of all its training inputs, and then evaluate all the candidate nodes to add one or a few hidden nodes in one SC-search loop. This would take more time to estimate the proper hidden node number L m of each SCN models of the ensemble. To accelerate this procedure, a down-sampling approach is applied, that is, instead of using all the samples, we randomly select two sub-datasets to estimate the parameters from the training and validation datasets, respectively, and repeat this procedure for several times to obtain statistic results. In this paper, we choose 70,000 samples from the training set and 30,000 from the validation set with heterogeneous features as the training-validation datasets for the estimation of the hidden node number of SCNs in the ensemble. Then build each SCN model with increasing hidden nodes to find the hidden node number with respect to the minimal validation error. The search range for the L of each SCN model is [1,120]. For fair comparisons, the DNNE shares the same architecture as the SCNE, that is, each RVFL base model has the same number of hidden nodes as the corresponding SCN base model. According to [14], the range of the random weights and biases of the RVFL networks, [−α, +α], needs to be estimated to improve the training performance. In our experiments, we compare the RVFL models with varying α from 0.5 to 1.4. The results are listed in Tables 6 and 7. The training data and validation data are sampled as the same as those for the SCN model. Each result is the average of 10 independent repeats with the same α. For each RVFL base model, the α * with the minimal validation error is chosen for the RVFL base model used to construct the final DNNE.  The following results show the advantages of the SCNE on Twitter and Year datasets. When the hidden node numbers of the SCNE and random weights ranges of the DNNE are estimated according to the previous experiments, we retrain the SCNE and DNNE with the full training datasets with the regularizing factor λ = 0.10 and the iteration number k max = 10. Tables 8 and 9 present the final training and testing results (i.e., the root mean square error, RMSE) of SCNE and DNNE with respect to 4 construction algorithms. All the results are the averages of 10 independent repeats. It is clear to see that the SCNE has lower training and testing RMSE than the DNNE on both datasets. The pseudo-inverse method and iterative schemes got much lower errors than the naive algorithm. Despite the training time cost, these results clearly indicate that the proposed SCNE works better than the DNNE.

Remark 3.
In SCNE construction, it should be emphasized that the pseudo-inverse method costs much more computing time and memory than the iterative methods with a large L. Ideally, applying the iterative methods, block Jacobi and block Gauss-Seidel methods, could lower the complexity to approximate O( M m=1 L 3 m ). In the Twitter experiments, the peak of the memory cost of the pseudoinverse method is about 7.9GB, which is the limit of our computer (8GB RAM) and it tends to take more. However, for iterative methods, the peaks are at almost the same level as the naive method, never beyond 5.4GB, which means on the same machine, the iterative methods have more potentials than the pseudo-inverse method for large-scale data modelling.

Robustness analysis
The regularizing factor λ = 0.10 is used for the construction of SCNE in previous experiments. To investigate the reliability of the proposed SCNE system, an analysis on the robustness of the modelling performance with respect to some key learning parameters should be carried out. In this work, we focus on the robustness analysis for the regularizing factor λ used in the NCL. The hidden nodes and data partitions for SCN base models are the same as the previous experiments. Only the value of the λ is changed here. Fig. 7 depicts the distribution of 10 testing results of the SCNE in box-plots with respect to λ = [0.08, 0.09, 0.10, 0.11, 0.12]. On each box, the central red mark indicates the median, and the bottom and top edges of the box indicate the first and third percentiles (Q1 and Q3), respectively. The top and bottom bars indicate minimum and maximum within the inner fence [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR] where IQR = Q3 − Q1. The values beyond the inner fence are suspected outliers plotted individually using the red '+' symbol. For both datasets, the results show that the λ affects the SCNE testing performance slightly but not significantly within a certain range.

Conclusion
Analysing more data quickly with higher accuracy turns to be significant nowadays because of a vast number of real-world applications from various domains. Traditional machine learning techniques, such as neural networks with optimization-based learning algorithms, support vector machines and decision trees, are hardly applied for large-scale datasets. Ensemble learning with its theoretical framework helps in improving the generalization performance of the base learner models, but it still has some limitations on the efficiency and scalability for dealing with large-scale data modelling problems.
This paper contributes to the development of randomized neuro-ensemble with heterogeneous features, where the stochastic configuration networks are employed as base learners and the well-known negative correlation learning strategy is adopted to evaluate the output weights of the SCNE model. To overcome the challenge in computing a pseudo-inverse of a huge sized linear equation system used in the least squares method, we suggest to utilize the block Jacobi and block Gauss-Seidel iterative schemes for problem solving. Some analyses and discussions on these solutions for evaluating the output weights are given by a demonstration. Simulation results clearly indicate that it is necessary to apply the ridge regression method for building the SCN base models, so that the resulting SCNE models with the iterative schemes can be consistent with the one built by using the non-iterative method in terms of the correlationship of the output weights.
The reported results in the demonstration implies that the statement on the speediness of the pseudoinverse-based solution for building randomized learner models (either for single or ensemble models) is valid only for smaller datasets. Indeed, its computational complexity and time cost are very high, even infeasible, for large-scale datasets. Experimental results with comparisons over two large-scale benchmark datasets show that the proposed SCNE always outperforms the DNNE. Robustness analysis on the modelling performance with respect to the regularizing factor used in NCL reveals that our proposed ensemble system performs robustly with good potential for large-scale data analytics. Further researches on improved feature grouping methodology, robust large-scale data regression [29] and enhancement of the generalization performance of the ensemble system are being expected.