{'fraction_non_alphanumeric': 0.03506986212639956, 'fraction_numerical': 0.01744239844545202, 'mean_word_length': 4.605549792531121, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep learning-based multi-output quantile forecasting of PV generation

This paper develops probabilistic PV forecasters by taking advantage of recent breakthroughs in deep learning. It tailored forecasting tool, named encoder-decoder, is implemented to compute intraday multi-output PV quantiles forecasts to efficiently capture the time correlation. The models are trained using quantile regression, a non-parametric approach that assumes no prior knowledge of the probabilistic forecasting distribution. The case study is composed of PV production monitored on-site at the University of Li\`ege (ULi\`ege), Belgium. The weather forecasts from the regional climate model provided by the Laboratory of Climatology are used as inputs of the deep learning models. The forecast quality is quantitatively assessed by the continuous ranked probability and interval scores. The results indicate this architecture improves the forecast quality and is computationally efficient to be incorporated in an intraday decision-making tool for robust optimization.


I. INTRODUCTION
The Intergovernmental Panel on Climate Change special report 1 on the impacts of global warming of 1.5°C above preindustrial levels and related global greenhouse gas emission pathways' presents several scenarios of decarbonisation of the electricity sector with a renewable share that should reach in 2030 the interquartile range [47, 65] ([69, 86] in 2050). Therefore, the development of renewable generations, typically from wind and photovoltaic (PV) sources has been facilitated by policy makers. However, the intermittent and uncertain nature of these sources is challenging the traditional operation of electricity networks. Market players require reliable decisionmaking tools to deal with uncertainty based on forecasts of renewable generation.
In contrast to point predictions, probabilistic forecasts aim at providing decision-makers with full information about potential future outcomes [1]. The various types of probabilistic forecasts range from quantile to density forecasts, and through prediction intervals. This paper focuses on quantile forecasts that provide probabilistic information about future renewable power generation, in the form of a threshold level associated with a probability [1]. Quantile regression [2] is one of the most famous non-parametric approaches. It does not assume 1 https://www.ipcc.ch/sr15/ the shape of the predictive distributions and is implemented with neural networks, linear regression, gradient boosting, or any other regression techniques.
The following papers have gained our attention in PV probabilistic forecasting. At the Global Energy Forecasting Competition 2014 [3] solar forecasts were to be expressed in the form of 99 quantiles with various nominal proportions between zero and one. A systematic framework for generating PV probabilistic forecasts is developed by [4]. A non-parametric density forecasting method based on Extreme Learning Machine is adopted to avoid restrictive assumptions on the shape of the forecast densities. A combination of bidirectional Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) resulting in Bidirectional LSTM (BLSTM) is proposed by [5]. It has the benefits of both longrange memory and bidirectional processing. The BLSTM is trained by minimizing the quantile loss to compute quantile forecasts of aggregated load, wind and PV generation, and electricity prices on a day-ahead basis. Finally, an innovative architecture, referred to as encoder-decoder (ED), is developed by [6] to generate reliable predictions of the future system imbalance used for robust optimization.
In this study, the forecast quality of the models is evaluated. It corresponds to the ability of the forecasts to genuinely inform of future events by mimicking the characteristics of the processes involved. We follow the framework proposed by [7], [8] for evaluating the quality of solar irradiance probabilistic forecasts based on visual diagnostic tools and a set of scoring rules. Overall they indicate that two main attributes, reliability and resolution, characterize the quality of quantile forecasts.
This work exploits recent breakthroughs in the field of data science by using advanced deep learning structures, such as the encoder-decoder architecture [6], and quality metrics [8], [9] to develop a tailored deep learning-based multi-output quantile forecaster. The goal is to capture the time correlation between time periods and to use this forecaster as input of a robust optimization model. For instance, to address the energy management system of a grid-connected renewable generation plant coupled with a battery energy storage device [10].
Overall, the contributions can be summarized as follows. First, a deep learning-based multi-output quantile architecture is used to compute prediction intervals of PV generation on a day-ahead and intraday basis. Specifically, the goal is to implement an improved probabilistic intraday forecaster, the encoder-decoder, to benefit from the last PV generation observations. This architecture is compared to a feed-forward neural network that is used as the benchmark model. Second, the weather forecasts of the MAR climate regional model [11] are used. It allows to directly take into account the impact of the weather forecast updates generated every six hours. Finally, a proper assessment of the quantile forecasts is conducted by using a k-fold cross-validation methodology and probabilistic metrics. It allows computing average scores over several testing sets and mitigating the dependency of the results to specific days of the dataset.
The remainder of this paper is organized as follows. Section II provides the non-parametric quantile forecasting framework considered. Section III presents the forecasting techniques used to compute the quantile forecasts. Section IV details the different metrics used to evaluate the quality of PV quantile forecasts. Section V describes the case study and presents the results. Finally, Section VI summarizes the main findings and highlights ideas for further work.

II. NON-PARAMETRIC QUANTILE FORECASTING
Let y t be the PV power generation measured at time t, which corresponds to a realization of the random variable Y t . Then let f t and F t be the probability density function (PDF) and related cumulative distribution function (CDF) of Y t , respectively. Following the definition of probabilistic forecasting of [1], a probabilistic forecast issued at time t for time t + k consists of a prediction of the PDF (or equivalently, the CDF) of Y t+k , or of some summary features. Various types of probabilistic forecasts have been developed, from quantile to density forecasts, and through prediction intervals. This study focuses on quantile regression [2], which is the most widely used type of probabilistic forecasting method. It is a non-parametric approach with no restrictive assumption on the shape and features of the predictive distributions. Indeed, empirical investigations [4] showed that PV power forecast errors do not follow common, e.g. Gaussian, Beta, distributions. Following the definition of [1], a (model-based) forecastŷ t+k|t of PV power generation is an estimate of some of the characteristics of the stochastic process Y t+k given a model g, its estimated parametersΘ t and the information set Ω t gathering all data and knowledge about the processes of interest up to time t.

A. Point forecasting
A point predictionŷ t+k|t is a single-valued issued at time t for t + k. It corresponds to the conditional expectation of Y t+k given g,Θ t , and the information set Ω t A multi-output point forecast computed at t for t+k 1 to t+k T is the vector ŷ t+k1|t , · · · ,ŷ t+k T |t of size T .

B. Quantile forecasting
t+k|t with nominal level q is an estimate, issued at time t for time step t + k of the quantile y (q) t+k|t for the random variable Y t+k|t given a model g, its estimated parametersΘ t and the information set Ω t or equivalentlyŷ , withF the estimated cumulative distribution function of the continuous random variable Y . Finally, a multi-output quantile forecast computed at t for t + k with Q quantiles per time period.

C. Quantile loss function
Quantile regression consists of estimating quantiles by applying asymmetric weights to the mean absolute error. Following [2], the quantile loss function is For a given time period t, ρ q is summed over the forecasting time periods k 1 ≤ k ≤ k T and quantiles q ∈ Q to compute multi-output quantile forecasts at t for t + k Finally, the model g is trained by minimizing 1 |Ωt| t ∈Ωt L t . Note in the case of perfect prediction, the quantile loss cannot be differentiated, and a smooth approximation of (4) using the Huber norm [6] is built.

D. Prediction intervals (PIs)
Prediction intervals (PIs) define the range of values within which the observation is expected to be with a certain probability, i.e., its nominal coverage rate [9]. Formally, a prediction intervalÎ (α) t+k|t issued at t for t+k, defines a range of potential values for Y t+k , for a certain level of probability A central PI consists of centering the PI on the median where there is the same probability of risk below and above the median. A central PI with a coverage rate of (1 − α) is estimated by using the quantiles q = (α/2) and q = (1−α/2). Its nominal coverage rate iŝ For instance, central PIs with a nominal coverage rate of 90 %, i.e., (1 − α) = 0.9, are defined by quantile forecasts with nominal levels of 5 and 95 %.

A. Gradient boosting regression (GBR)
Gradient boosting builds an additive model in a forward stage-wise fashion [12]. It allows for the optimization of arbitrary differentiable loss functions. In each stage, a regression tree is fit on the negative gradient of the given loss function. The Scikit-learn [13] Python library is used to implement a gradient boosting regressor (GBR) with the quantile loss function. The learning rate is set to 10 −2 , the max depth to 5, and the number of estimators to 500. There is a GBR model per quantile as the library does not support multi-output quantile regression.

B. Multilayer perceptron (MLP)
A description of the most widely used "vanilla" neural network, the Multilayer perceptron (MLP), is provided by [12]. A MLP with a single hidden layer is considered for the day-ahead forecasts and as the benchmark for the intraday forecasts. MLPs with two and three hidden layers did not provide any significant improvement. The activation function is the Rectified Linear Unit (ReLU). The number of neurons of the hidden layer is n input +(n output −n input )/2, with n input and n output the number of neurons of the input and output layers, respectively. The learning rate is set to 10 −2 and the number of epoch to 500 with a batch size of 8. It is implemented using the PyTorch Python library [14].

C. Encoder-decoder (ED)
Several technical information about recent advances in neural networks is provided by [5], [6]. In particular, recurrent neural networks, have shown a high potential in processing and predicting complex time series with multi-scale dynamics. However, RNNs are known to struggle in accessing time dependencies more than a few time steps long due to the vanishing gradient problem. Indeed, back-propagated errors during the training stage either fades or blows up over time. Long Short-Term Memory and Gated Recurrent Units networks tackle this problem by using internal memory cells [6]. A neural network composed of a LSTM and feed-forward layers, referred to as LSTM in the rest of the paper, is implemented for the day-ahead and intraday forecasts. The number of LSTM units is n input + (n output − n input )/3, and the number of neurons of the feed-forward layer n input + 2 × (n output − n input )/3.
An innovative architecture, referred to as encoder-decoder [6], is composed of two different networks and has recently shown promising results for translation tasks and speech recognition applications and imbalance price forecasting. The encoder-decoder processes features from the past, such as past PV observations, to extract the relevant historical information that is contained into a reduced vector of fixed dimensions, based on the last hidden state. Then, the decoder processes this representation along with the known future information such as weather forecasts. A version of the encoder-decoder architecture (ED-1) is implemented with a LSTM as the encoder and a MLP as the decoder. In a second version (ED-2) the decoder is a LSTM followed by an additional feedforward layer. Both versions of the encoder-decoder are used as intraday forecasters. In ED-1, the encoder has 2×n input units with n input the number of neurons of the encoder input layer, features from the past. Then, the encoder output is merged with the weather forecasts becoming the decoder input layer that has n output /2 neurons. In ED-2, the decoder has the same number of cells as the encoder, and the feed-forward layer is composed of n output /2 neurons. The LSTM, ED-1, and ED-2 models are implemented using the TensorFlow Python library [15]. The activation functions are the ReLU, the learning rate is set to 10 −3 , the number of epoch to 500 with a batch size of 64 for the three models.
A sensitivity analysis has been conducted to select the hyperparameters: number of hidden layers, neurons, epochs, and learning rate. Overall, increasing the number of hidden layers and neurons increases the model complexity. It can enhance the accuracy, but only up to a limited number of layers and neurons due to overfitting issues. In addition, the hyperparameter solution is closely related to the size of the historical database [5]. A deep learning model with a larger amount of hidden layers and neurons requires a large amount of data to accurately estimate the parameters. In the case study considered, there are only 157 days of data with a 15 minutes resolution. Thus, we decided to restrict the number of layers and neurons to select a smaller model that performs better with the available information.

IV. PROBABILISTIC FORECASTING QUALITY ASSESSMENT
For predictions in any form, one must differentiate between their quality and their value [1]. Forecast quality corresponds to the ability of the forecasts to genuinely inform of future events by mimicking the characteristics of the processes involved. Forecast value relates, instead, to the benefits from using forecasts in a decision-making process such as participation in the electricity market. This section proposes quality metrics based on the framework proposed by [8]. The value assessment is not in the scope of this paper as it would require to consider a decision-making process.

A. Continuous rank probability score (CRPS)
A score is said to be proper if it ensures that the perfect forecasts should be given the best score value [16]. It is the case of the Continuous Rank Probability Score (CRPS) that penalizes the lack of resolution of the predictive distributions as well as biased forecasts. For deterministic forecasts, the CRPS turns out to be the Mean Absolute Error (MAE). The energy form [16] of the CRPS (NRG) is selected in this study where X and X are two independent copies of a random variable with distribution functionF t+k|t and finite first moment. A CRPS estimator (eNRG) of the energy form is provided by [17] when the CDF is only known at t + k through a Qensemble of quantile forecasts Q t,k = {ŷ Finally, CRPS eNRG (k) is the mean over the evaluation set for a given forecasting time period k, and CRPS eNRG is the average over all k with k 1 ≤ k ≤ k T .

B. Interval score (IS)
The Interval Score (IS) is a proper score proposed by [16] to specifically assess the quality of central (1 − α) prediction interval forecasts. The IS rewards narrow prediction intervals but penalizes, with the penalty term that depends on α, the forecasts for which the observation is outside the interval. The averaged IS over an evaluation set of length M and ∀k,

A. Case study description
The ULiège case study is composed of a PV generation plant with an installed capacity of 466.4 kW. The PV generation has been monitored on a minute basis from April 4, 2020 to September 14, 2020, 157 days in total, and the data is resampled to 15 minutes. The set of quantiles is Q = {0.1, 0.2, . . . , 0.9} for both the day-ahead and intraday forecasts. Numerical experiments are performed on an Intel Core i7-8700 3.20 GHz based computer with 12 physical CPU cores and 32 GB of RAM running on Ubuntu 18.04 LTS.

B. Numerical settings
The MAR regional climate model [11] provided by the Laboratory of Climatology of the Liège University is forced by GFS (Global Forecast System) to compute weather forecasts on a six hours basis, four-time gates per day at 00:00, 06:00, 12:00, and 18:00 with a 10 day horizon and a 15 minutes resolution. The solar irradiance and air temperature at 2 meters are normalized by a standard scaler and used as inputs to the forecasting models.
A k-fold cross-validation is strategy is used to compute average scores over several testing sets to mitigate the dependency of the results to specific days of the dataset. The dataset is divided into k parts of equal length, and there are k possible testing sets 1 ≤ i ≤ k. For a given testing set i, the models are trained over the k − 1 parts of the dataset. Eleven pairs of fixed lengths of 142 and 15 days are built. One pair is used to conduct the hyperparameters sensitivity analysis, and the ten others for testing where the scores are averaged. The Mean Absolute Error (NMAE) and Root Mean Squared Error (NRMSE) are introduced to evaluate the point forecasts. The MAE, RMSE, CRPS, and IS are normalized by the PV total installed capacity with NMAE and NRMSE the normalized MAE and RMSE.
The day-ahead models, MLP, LSTM, and GBR compute forecasts at 12:00 for the next day. Four intraday time gates are considered at 00:00, 06:00, 12:00, and 18:00. The intraday forecasts of time gate 00:00 are computed by the day-ahead models using only the weather forecasts. Then, the next three intraday forecasts are computed by intraday models where the MLP, ED-1, and ED-2, models use the weather forecasts and the last three hours of PV generation.
The day-ahead and the first intraday predictions are delivered for the 96 quarters of the next day from 00:00 to 23:45 indexed by time steps 0 ≤ k ≤ 95. The prediction horizons span from 12 to 36 hours, for the day-ahead gate 12:00, and 0 to 24 hours, for the intraday gate 00:00. The prediction horizon is cropped to 11 ≤ k ≤ 80 because the PV generation is always 0 for time steps 0 ≤ k ≤ 10 and 81 ≤ k ≤ 95 on the ULiège case study. The next three intraday predictions are performed for the 72, 48, and 24 next quarters of the day corresponding to the gates 06:00, 12:00, and 18:00. Therefore, the prediction horizons span from 0 to 18 hours, 0 to 12 hours, and 0 to 6 hours. The intraday forecasting time periods are 24 ≤ k ≤ 80, 48 ≤ k ≤ 80, and 72 ≤ k ≤ 80. Table I compares the mean and the standard deviation of the computation times, over the ten learning sets, to train the point and quantile forecast models 2 .

D. Intraday results
Table III provides the averaged NMAE, NRMSE, and CRPS per gate of intraday models. The LSTM achieved the best NMAE and NRMSE for the 06:00 gate and the ED-1 achieved the best NMAE and NRMSE for the 12:00 gate and the best CRPS for both gates. Figure 2 compares the CRPS per forecasting time periods k of the intraday models. The ED-1 benefits from the last PV generation observations. Indeed, some CRPS values for both 06:00 and 12:00 gates are below the ones of 00:00 gate. Table IV provides the Interval score of intraday models for 80 %, 60 %, 40 %, and 20 % width of central intervals. The ED-1 model achieved the best results for both 06:00 and 12:00 gates and all prediction intervals except for the 06:00 gate and the prediction interval width of 80 % where it is ED-2. The LSTM achieved close results to the ED-1. Figures 3b, 3d, and 3f compare the ED-1, LSTM, and ED-2 intraday quantile and point forecasts (black line named intra 6) of 06:00 gate on August 2, 2020 with the observation in red. Generally, one can see that the predicted intervals of ED-1 and LSTM models better encompass the actual realizations of uncertainties than ED-2.

VI. CONCLUSION
An encoder-decoder architecture is implemented on the intraday scale to produce accurate forecasts. It efficiently captures the contextual information composed of past PV observations and future weather forecasts, while capturing the temporal dependency between forecasting time periods        over the entire forecasting horizon. The models are compared by using a k-fold cross-validation methodology and quality metrics on a real case study composed of the PV generation of the parking rooftops of the Liège University. The best dayahead model for both point and quantile forecasts is a neural network composed of a LSTM cell and an additional feedforward layer. Then, the encoder-architecture composed of a LSTM-MLP yields accurate and calibrated forecast distributions learned from the historical dataset in comparison with the MLP and LSTM-LSTM models for the intraday point and quantile forecasts. However, the LSTM produced similar    results. Several extensions are under investigation. First, considering a larger dataset of at least one full year to take into account the entire PV seasonality. Second, developing a PV scenario approach based on the encoder-decoder architecture.