{'fraction_non_alphanumeric': 0.023441662226957913, 'fraction_numerical': 0.00811933937133724, 'mean_word_length': 5.045606802370523, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Artificial Intelligence Framework for Bidding Optimization with Uncertainty in Multiple Frequency Reserve Markets

The global ambitions of a carbon-neutral society necessitate a stable and robust smart grid that capitalises on frequency reserves of renewable energy. Frequency reserves are resources that adjust power production or consumption in real time to react to a power grid frequency deviation. Revenue generation motivates the availability of these resources for managing such deviations. However, limited research has been conducted on data-driven decisions and optimal bidding strategies for trading such capacities in multiple frequency reserves markets. We address this limitation by making the following research contributions. Firstly, a generalised model is designed based on an extensive study of critical characteristics of global frequency reserves markets. Secondly, three bidding strategies are proposed, based on this market model, to capitalise on price peaks in multi-stage markets. Two strategies are proposed for non-reschedulable loads, in which case the bidding strategy aims to select the market with the highest anticipated price, and the third bidding strategy focuses on rescheduling loads to hours on which highest reserve market prices are anticipated. The third research contribution is an Artificial Intelligence (AI) based bidding optimization framework that implements these three strategies, with novel uncertainty metrics that supplement data-driven price prediction. Finally, the framework is evaluated empirically using a case study of multiple frequency reserves markets in Finland. The results from this evaluation confirm the effectiveness of the proposed bidding strategies and the AI-based bidding optimization framework in terms of cumulative revenue generation, leading to an increased availability of frequency reserves.


Introduction
Power grid frequency deviations occur at any point in time when electricity production does not match consumption. Frequency reserves exist for the purpose of reacting to these deviations in real time, by modifying the electricity consumption or production behaviour of resources participating in the reserve. Frequency reserves are an ancillary market with significantly different characteristics and behaviours compared to the primary day-ahead markets [1]. Frequency reserves markets are different from spot markets as the transaction is not the exchange of energy. Instead, it is the capacity to react to the grid under frequency situations by increased generation, reduced consumption or trading the frequency reserve capacity to the Transmission System Operator (TSO) [2]. For example, an electric vehicle (EV) charging station can participate on frequency reserves markets by curtailing the charging current during underfrequency. The TSO compensates the station operator for being in readiness to react to underfrequency situations. The compensation is proportional to the market price for that hour and the capacity (in kW) which the station is ready to curtail. Although numerous research endeavours have been conducted on spot market prediction and analysis [3,4], only a limited number have focused on these ancillary markets [5]. The need for new reserves is expected to increase due to the penetration of solar and wind power generation [6] and the fact that reserves have traditionally been implemented with fossil fuel burning spinning reserves, which are now being eliminated in the push towards a carbon-neutral society [7]. Recent literature reports many innovative solutions for developing frequency reserve capability for distributed energy resources such as renewable power generation [8], electricity storage [9], electric vehicles [10] and smart loads [11]. However, in order to exploit such reserves in the most effective manner, it is necessary to trade these on frequency reserves markets. The statistical properties of frequency reserves markets are significantly different from spot markets, due to zero price days and exceptionally high price peaks [2]. Thereby it is evident that several technical challenges need to be addressed in order to effectively trade frequency reserves capacities in multiple markets to generate value for all stakeholders. Firstly, the paper addresses the challenge of rescheduling the reserve capacity [12]. Using electric vehicles as an example, since reserve provision is achieved by curtailing charging cur-rent, reserve market participation is only possible on market intervals in which the vehicle is plugged in and charging. In this case, rescheduling the charging is also rescheduling the market participation possibilities. For example, in Finland, bidding on primary reserves markets is done day-ahead for each hour of the next day [13], so the charging plans would need to be known at this time in order to profit from rescheduling. The market price often has large variations from one hour to the next [2]. This implies that significant value can be generated by shifting frequency reserve capacity from low price hours to high price hours, based on quality market predictions available at the time of making the bid. Secondly, many of the resources could be pre-qualified for more than one market, namely primary, secondary and tertiary frequency reserves. However, most research addresses only one market and a limited number of studies have focused on two markets [14,15,16,17]. An optimised bidding schedule for multiple markets depends on the national market rules, but weekahead, day-ahead and hour-ahead market mechanisms are common. When frequency reserves have been pre-qualified for all accessible markets, the maximum value can be generated by selecting the market with the highest price forecast for each bidding period. Finally, frequency reserves markets, unlike spot markets, are uniquely characterized by high price peaks and zero price days. It has been difficult for state-of-the-art forecasting methods to predict these irregularities accurately [2]. This means bidding very high reserve capacities during forecasted price peaks and avoiding market participation during forecasted low prices may not be profitable if the bids were made based on uncertain price forecasts. We report three research contributions in this article. The first contribution is the study of global frequency reserves markets that informs the design of a generalised model of the operation of a frequency reserves market. The second contribution is the formulation of three bidding strategies for multi-stage markets, based on this generalised model. The first two strategies are for non-reschedulable loads, while the third is for reschedulable loads with four alternative rescheduling schemes. These strategies are devised to account for market rules and data-driven predictions with uncertainty metrics. The third contribution is the use of Artificial Intelligence (AI) to forecast the multi-stage frequency reserve market prices, and to quantify the uncertainty of these forecasts, so that this information is available to the bidding strategies, followed by a demonstration of optimized revenue generation and predictive accuracy by empirically evaluating this AI-based bidding optimization framework on a realworld frequency reserves setting of multiple markets in Finland. The rest of the paper is organised as follows. Section 2 reports current literature in frequency reserve markets, highlighting the need for a bidding optimisation framework for multimarket scenarios. Section 3 presents the study of frequency reserves markets in several countries, leading to the generalised frequency reserves market model. The three bidding strategies are delineated in Section 4, and Section 5 presents the AI-based bidding optimisation framework that implements these strategies using uncertainty metrics based on supervised and unsu-pervised machine learning techniques. Section 6 explicates the empirical evaluation on a real-world frequency reserves setting of multiple ancillary markets. Section 7 concludes the paper.

Related work
This section begins with a deliberation of recent research on frequency reserves of renewable energy sources, highlighting a research gap in bidding strategies for multiple frequency reserve markets. Subsequently, a critical analysis of the closest state of the art in frequency reserve markets is presented. The final subsection reports a review of time-series forecasting methods and uncertainty metrics that can be utilised to develop an AI-based bidding optimisation framework that addresses this gap.

Review of related work in frequency reserves
Traditionally, frequency reserves have been provided using fossil fuel based spinning reserves. Such reserves could be traded on either primary, secondary or tertiary markets as long as the reserve meets the market requirements on minimum capacity and response time [18].
However, recent years have seen a surge of innovations adapting renewable energy sources, battery storages and smart loads for the provision of frequency reserves.
Despite this surge of innovations, there has been limited research conducted on the diverse characteristics of the frequency reserve markets and the requirement for participation in these markets. Electric vehicles are used as an example in this paper, as they are targeted by many of the research efforts in frequency reserves, e.g. [19,20]. While some authors do not explicitly identify the type of reserve being targeted by their solution [21], most works are focused either on primary or secondary frequency reserves. The works referenced in this section do not specifically reference the technical requirements of any specific reserves market with the exception of Bokhari et al. [22], who consider technical requirements such as response time for several European and North American markets. Several authors have proposed electric vehicle charging solutions for primary frequency reserves. In [23], it was specifically recognize that a sufficiently fast response time is key to participation in primary rather than secondary reserves, but their solution treats the electric vehicle battery as if it were dedicated for the provision of reserves rather than driving. Studies that discuss the design of the frequency controller to meet the response time requirements for primary frequency reserves [24], considers the problem of achieving the primary frequency reserve response requirements within the framework of the incentives paid to vehicle owners [10], and recognizes that a single electric vehicle does not have sufficient capability or availability to meet the minimum capacity requirements, and that aggregation is needed for this purpose [25] have been reported in recent literature. A step further, the technical requirements of primary frequency reserves to achieve smooth frequency control [26], and an aggregation solution aiming to minimize the utilization of the electric vehicle batteries [27], have also been proposed.
A further body of research has been reported on electric vehicle charging solutions for secondary reserves. Falahati et al. [28] have studied the role of electric vehicles within a power system that has wind power generation, but does not consider how the battery capacity is divided between driving and the provision of frequency reserves. Another study [29] assumes that vehicle usage requirements are known in advance, so that they can be treated as constraints to the frequency control problem. A multi-objective optimization solution has also been presented [30] to balance the trade-off between providing the frequency reserves and satisfying the EVs energy demands under battery degradation constraints. None of these authors discuss response time requirements or the number of vehicles required to meet the technical requirements of the reserves. The approach proposed in [29] is claimed to be applicable to both primary and secondary reserves, but they do not discuss how it should be adapted to the different technical requirements in these reserves. The other authors do not assess the applicability of their solution for primary or tertiary reserves. In summary, the lack of research on tertiary reserves is evident, and none of the reported approaches take into account the changing price of the secondary reserves.

Analysis of the closest state of the art
Ferruzzi et al. [31] consider the problem of bidding on two markets: a spot market and a frequency reserve market. The paper identifies three types of frequency reserves (primary, secondary and tertiary) and their US counterparts (regulation, spinning and not spinning reserve). Only one reserve market is considered in the proposed bidding optimization solution, namely spinning reserve. Further, the proposed bidding optimization solution assumes that the reserve market price is known at the time of placing the bids, although this assumption is not valid for real world reserve markets, in which the market price is determined by an auction mechanism that is executed after bidding is closed [2]. Rey et al. [25] consider the problem of bidding aggregated reserve resources on a frequency reserve market. Specifically, the types of resources considered are electricity loads in buildings, such as electric vehicle chargers and heating, ventilation and air conditioning equipment. All of these resources have a primary function other than providing frequency reserves, which constrains the use of these resources on frequency reserve markets. These constraints are specifically related to the maximum power consumption adjustment that the reserve resource can undertake in response to a grid frequency deviation, without violating requirements related to the primary function of the resource. It is assumed that the reserve market price for the upcoming bidding period is known or that it can be easily and accurately predicted from previous market clearings; this assumption is not valid according to an analysis of frequency reserve market price behavior [2]. The option to trade on more than one frequency reserve market is not considered. It is also assumed that the adjustable power capacity will be the same for all of the hourly bids of the upcoming day-ahead bidding period, due to the policy in Switzerland and some other countries, which are not identified in the paper. However, this policy is not generally in place in Europe, and especially not in Finland, which is the case study for this paper. Thus, our paper considers the possibility to reschedule the utilization of reserve resources to achieve higher adjustable power capacity for the hours in which peak prices are predicted. Perninge & Eriksson [32] consider specifically tertiary frequency reserve markets in Northern Europe; the Finnish mFRR market in the case study of this paper is such a market. The objective of the paper is to propose a scheme for the tertiary reserve market operator (i.e. the transmission system operator) to optimally operate the reserve resources that the owners had bid on the tertiary reserve market for a specific hour. This is in contrast to the aims of our paper in two ways. Firstly, the possibility to use the resource on other reserve markets is not considered. Secondly, the profitability for the resource owner is not considered. In summary, all of these state of-the-art works ( [31,25,32]) are based on rigorous mathematical models, with simplifying assumptions. Due to these simplifications, these research studies neglect the possibility of capitalizing on high price peaks on frequency reserve markets, either by trading on several frequency reserve markets or by rescheduling adjustable power capacity to peak price hours. Thereby, we propose an alternative bidding optimization approach based on data-driven predictions and AI-based uncertainty metrics to address such assumptions and simplification. Beyond frequency reserve markets, in the broader context of energy markets, some research has been done on approaches that consider multi-stage markets such as day-ahead and hourahead, but the aims, assumptions or approach are different from the research presented in this article. Lahouar & Ben Hadj Slama [33] forecast electricity price on half-hour-ahead and day-ahead markets, but do not act on the forecasts. Caramanis et al. [34] propose a bidding strategy on hour-ahead and day-ahead electricity markets, but do not address the problem of forecasting the prices or using the forecasts to improve the bidding. Reddy & Singh [35] propose a demand response strategy on day-ahead and hour-ahead markets, but assume that the prices of the upcoming bidding period are known. Shafie-khah et al. [36] propose scheduling of wind power production on multi-stage markets; instead of trying to predict market price, the price uncertainties are handled by the Monte Carlo method. Zhong [37] propose a demand response scheme for trading in two settlement markets; the benefit for the market operator is reduced market clearing prices, whereas our work aims at increasing revenues for market participants. Zhao et al. [38] propose a trading scheme for a virtual power plant operating on multi-stage electricity markets, so that market price uncertainly is handled with scenario trees. Böhringer et al. [39] trade a factory's flexible energy consumption capability on multi-stage electricity markets, in which market price uncertainty is treated as a risk without trying to forecast the price. None of these approaches that focus on multi-stage markets have addressed the challenges of price prediction with uncertainty in volatile markets for generating a profit from price peaks. The focus of this paper is to address these challenges using an AI based bidding optimization framework that implements three bidding strategies that encompass real-time market conditions and datadriven predictions of price.

Time-series forecasting and uncertainty metrics
Price forecasting in energy markets has become a fast growing interest with the introduction of competitive markets. [40] There are five main categories of time-series forecasting methods use for price prediction: multi-agent models or game theoretic models [2], fundamental methods, reduced form models, statistical models and computational intelligence models [41], with statistical and computational intelligence methods are the most effective in practice. [42] Variations of statistical models like autoregressive moving average (ARMA) [43], autoregressive [44] integrated moving average (ARIMA) are commonly used for electricity price forecasting. In rapidly-changing markets, statistical models are ineffective and computational intelligence methods outperform others [45]. These computational intelligence methods include Support Vector Regression Machines (SVR) [46] and Artificial Neural Network (ANN) methods. ANN based methods have been accurate in predicting ancillary market prices with high variability [2]. Despite this accuracy of prediction, the high variability has led to uncertainty mainly due to the data scarcity and volatility of energy markets. Addressing this uncertainty of prediction has been researched in several other domains, and it can be theorised into two aspects, aleatory uncertainty and epistemic uncertainty [47]. Aleatory uncertainty is an uncertainty due to chance, also known as stochastic uncertainty, and it cannot be resolved with more data. In contrast, epistemic uncertainty can be considered as a source of uncertainty due to lack of knowledge of the environment [48], and it can be addressed with more data. e.g: [49] Uncertainty metrics are necessary when the data is scarce or there is inherent randomness in the process generating the data. Bayesian inference based methods have been traditionally used to measure uncertainty [49]. Bayesian neural networks (BNN) is a variant of neural networks used for the purpose of uncertainty estimation. A BNN generates a distribution over the output for a given input, in contrast to Artificial neural networks (ANN), which produce a point output for given input [50]. However, this variational inference is a costly process [51]. Recent advances in BNN approximation with dropouts give efficient estimates of uncertainty in neural networks. This is commonly known as Monte Carlo dropout (MC-dropout). [52].

Generalised model of frequency reserves market
Frequency reserves are divided into primary, secondary and tertiary reserves. Primary reserves are often called Frequency Containment Reserves (FCR), which are responsible for stabilizing the frequency rapidly after a frequency deviation occurs. Secondary and tertiary reserves are often called Frequency Restoration Reserves (FRR). FRR is activated automatically for secondary reserves (aFRR) and manually for tertiary reserves (mFRR). The terms FCR, aFRR and mFRR are broadly used in Europe, but they are not globally established. [22] present a thorough comparison of North American and European markets. A major research challenge related to predicting frequency reserves markets is related to generality: there is considerable diversity between the terminology, market rules and technical specifications between countries. The following factors should be taken into account in order to improve the generality of solutions (examples are based on analyzing the ENTSOE-E Transparency Platform https://transparency.entsoe.eu/ ): • Bidding interval: how often are auctions held? As an example of a week-ahead market, German FCR has the same price for the entire week and the auctions are held once a week to determine the price of the next week. As an example of a day-ahead market, in Finnish FCR the auctions are held daily and 24 different prices are determined for the hours of the next day. As an example of an hour-ahead market, in Finnish mFRR the auctions are held hourly to determine the price for the upcoming hour.
• Market changes: what was said in the previous bullet applied to the German FCR market until 2018. In 2019 the price changes daily.
• Number of markets: many European countries have one single FCR market, but Northern European countries have separate FCR markets for normal operation (FCR-N) and disturbances (FCR-D).
• Epochs: the most typical epoch is one hour, but e.g. French FCR, aFRR and mFRR have a 30min interval.
• Symmetry: in a symmetric market, participants must provide both down and up regulation. In an asymmetric market, these are separate markets with separate prices, and it is not required to participate on both. FCR are often symmetric and aFRR are often asymmetric, e.g. as is the case of Finland, but as one exception French aFRR is symmetric.
• Peak hours: some markets distinguish between peak and off-peak hours, but the details are country specific. For example in Finnish aFRR, the market is held only on peak hours on weekdays, with auctions determining a separate price for each hour. In German aFRR, the price changes every 4 hours.
Currently, an EU wide harmonization of frequency reserves markets is underway. Key observations from this harmonization include shortening of bidding intervals and epochs and moving from weekly to day-ahead markets [56]. Table 2 defines the terms used throughout this paper. Symbols and abbreviations are defined only when necessary to aid the formalization of the proposed methodology. Figure 1 shows a UML (Unified Modeling Language) class diagram that defines the relationships between the key concepts in Table 2. The class diagram is limited to capturing the relationships between the concepts used in this paper. It does not have the detail needed to support a virtual power plant implementation. Figure 2 presents a generic operating model of a frequency reserves market, using terms and   Table 2 to parameterize the characteristics that vary across the primary, secondary and tertiary reserves of the analyzed countries. Figure 1: UML class diagram of generalised frequency reserves market [53]. Refer Table 2 for terms and abbreviations  Table 2 for terms and abbreviations

Bidding strategies
The proposed bidding strategies are optimised using AI based market forecasts as well as uncertainty metrics of these forecasts. The rationale for this integration of technologies is as follows. As discussed in section 2, AI has emerged as a stateof-the-art approach for time series forecasting in the energy domain; however, none of the state-of-the-art research studies have forecasted price peaks with high confidence, and this is the motivation for developing the uncertainty metric. In contrast to mathematical optimization, bidding strategies based on price prediction and uncertainty metrics can incorporate knowledge of the trading opportunities on the multi-stage markets and the possibilities for rescheduling frequency reserve resources. Based on the analysis of global frequency reserves markets and the generalised model developed in the previous section, the following assumptions are broadly valid: 1. The primary and secondary reserves markets are dayahead markets. In several countries there are two different primary reserves markets, so there are several dayahead markets to choose from. 2. There is one tertiary reserves market, which is an epoch ahead market.
In the following subsections, three bidding strategies are proposed. Strategies 1 and 2 address the scenario of non-reschedulable reserve resources whereas strategy 3 addresses the scenario of reschedulable reserve resources. The difference between strategies 1 and 2 is in the way that they make decisions whether to trade on day-ahead or hour-ahead frequency reserves markets. In this paper, the profitability of strategy 1 versus strategy 2 is evaluated empirically with a case study. Strategy 1 tries to trade all of the available reserve resource capacity on day-ahead markets, and participates on hour-ahead markets only when bids for day-ahead markets were rejected. Strategy 2 refrains from bidding on day-ahead markets if the hour-ahead markets are forecasted to have a higher price. Strategy 3 reschedules the utilization of reschedulable reserve resources to hours on which high reserve market prices have been forecasted.

Bidding strategy 1
The acceptance decision for day ahead bids will come on the same day as when the bids were placed e.g. [34,33], so it is reasonable to formulate a two-phase bidding strategy. First bid will be on day-ahead markets, after these bids have been accepted or rejected, bid the remaining capacity on epoch-ahead markets. Algorithm 1 presents a straightforward strategy that selects, for each epoch, the day-ahead market with the highest price forecast. Only markets with reliable forecasts are considered. After the acceptance or rejection information for these bids have been received, the capacity of the rejected bids is offered onto the epoch ahead market, i.e. the tertiary reserves market. From Figure 2 it can be seen that all accepted bids receive the same capacity fee regardless of what the requested fee was. In this kind of auction, it is reasonable to make bids at the lowest requested fee which is better than no-deal. The concept of MPP was defined for this purpose. The actual value of MPP depends on opportunity costs resulting from other possible usage scenarios of the reserve resource. It also depends on any usage cost of the resource as it participates in the reserve: for example, if the resource is a battery, participating in frequency reserves will involve charging and discharging operations that would shorten the lifespan of the battery. Quantification of the MPP is thus specific to the type of reserve resource and the system that it belongs to, so such calculations are beyond the scope of this article.

Bidding strategy 2
Bidding strategy 1 will not perform well if the tertiary reserves market has a higher price than any other market for a specific epoch. In such a scenario, it would be more profitable to refrain from bidding on any day-ahead market and to bid all the capacity on the tertiary market. Bidding strategy 2 in Algorithm 2 implements this reasoning.

Bidding strategy 3
Previously presented bidding strategies assumed non-reschedulable reserve resources, i.e. a fixed C[e]. Bidding strategy 3 focuses on reschedulable reserve resources, where the task is to perform the rescheduling in order to determine the C[e] that will be traded on reserve markets for each epoch. As an example, consider an electric vehicle (EV) that is plugged in all night to a home charger, which is able to fully charge the EV at 22kW in 2h. The EV needs to be fully charged by 8AM, so a charging of 44kWh needs to be scheduled. If the charger has a frequency control feature, it can be used on frequency reserves markets during the hours that it charges the EV: it is possible to simply curtail the charging current in the event of frequency deviations. Starting from midnight, it is possible to participate in day-ahead reserve markets. Several charging schemes are possible: 1. Charge at a constant power for the 8 hour period from midnight to 8AM, resulting in a charging power of 44kWh/8h = 5,5W. 2. Charge the EV in two hours starting at midnight. 3. Charge the EV in two randomly chosen one-hour periods starting earliest at midnight and ending latest at 8AM.  Table 3 defines additional terms that are only relevant for reschedulable reserve resources. A universal model for a reschedulable reserve resource would be a research question in its own right, as it would need to capture aspects such as user preferences, unexpected events such as an EV being plugged out in the middle of the night, as well as device specific constraints such as battery recovery times. In this paper, the example from section 3.4.1 is generalized as follows. We consider a reschedulable reserve resource that will produce or consume a quantity of energy E. This action may be scheduled so that it starts earliest on epoch e earliest and is completed latest on epoch e latest , so that the power production or consumption is constant during the epoch. This power is the C[e] that the reserve resource can provide. In this case:

General formulation
Here roundU p() is a function to convert floating point value to nearest largest integer value. The schemes 1-4 can be formalized as follows: Schemes 1-3 are straightforward and will be implemented in order to provide a point of comparison to the performance of scheme 4.

The AI-based bidding optimization framework
The AI-based bidding optimization framework implements the three bidding strategies delineated above, the first two strategies for non-reschedulable loads, and the third for reschedulable loads with four alternative rescheduling schemes. The following subsections focus on the constituents of this framework, price prediction, supervised and unsupervised uncertainty metric calculation, and uncertainty metric evaluation.

Price Prediction
Price prediction is based on a three-layer neural network approach proposed in [2], where n = 64 with hidden layer of 32 and output of 24 corresponding to 24 hours. An optimum window size of 180 days was used. The three-layer neural network was converted into a Bayesian neural network for point prediction of 24 hour output as averaged over N = 500 iterations with the same dropout rate 40% at the testing time.

Uncertainty metric
Despite using AI, the prediction of price can still be uncertain due to the scarcity of data and volatility of renewable energy sources [57,58]. It is important to quantify the uncertainty of the price prediction for optimised bidding. We used a combination of two uncertainty measures, supervised and unsupervised for better representation of price prediction uncertainty in frequency reserve markets. For the former, we used MC-Dropout, a well-established uncertainty measure for deep learning [52]. For the latter, we introduced a novel unsupervised uncertainty measure which will independently evaluate the training data for epistemic uncertainty.

Supervised uncertainty metric
As the supervised uncertainty metric, we used the MC-Dropout method with the neural network evaluated N times for a single price prediction. This resulted in a distribution of predictionŝ p 1 t ,p 2 t ...p N t for a single price predictionp t . The predictionp t and the normalized uncertainty nu t are evaluated from the distribution as follows:p

Unsupervised uncertainty metric
We introduce a novel unsupervised uncertainty metric that is independent of the three-layer neural network used for prediction. The unsupervised uncertainty metric is based on the Growing Self Organizing Map (GSOM) algorithm. The GSOM algorithm is an unsupervised neural network variant of Self organizing maps(SOM), which starts with four nodes and grows nodes to represent the input data [59]. It has been used in many unsupervised applications related to clustering, dimensional reduction and abstract representation in the energy domain [60,61,62]. The GSOM algorithm generates a topological latent representation of the input vectors that can be utilised to identify clusters of nodes based on a distance measure. We evaluated the GSOM algorithm alongside the SOM algorithm, T-SNE, and K-means algorithm (all of which generate similar topological maps based on unsupervised learning). The GSOM algorithm possesses the most effective topology preserving measures for a non-linear dataset such as the data streams in this scenario. This uncertainty metric based on the unsupervised learning capability of the GSOM algorithm generates an independent assessment of data uncertainty. In contrast, MC-Dropout generates an uncertainty value for both the data and the predictive model. If both MC-Dropout uncertainty and GSOM uncertainty values are similar, it is an indication that uncertainty for the prediction is from lack of knowledge in the data. In terms of an uncertainty metric, when input vectors clustered to a GSOM node have low variance, it is an indication that price is accurately represented by these input vectors and thereby epistemic uncertainty is low for those values. In contrast when a GSOM node has high variance, this is an indication that the price is not completely represented by the data vector and epistemic uncertainty is high. Further, in the absence of any similarity, it is an entirely new pattern with high uncertainty. Using the above notion we define the GSOM uncertainty for price prediction as follows. If a new data point is clustered to a node with previous price points P 1 , P 2 ..P c with mean price of µ If a new data point is clustered to a node with no previous price point To obtain the normalized uncertainty value u i is normalized using Equation 3.

Uncertainty evaluation metric
MC-dropout and GSOM uncertainty metrics provide estimation for uncertainty, but there are no ground truth values for model prediction, and it is challenging to quantitatively evaluate the uncertainty values [49]. We extend the evaluation metric proposed in [63,49] for categorical value prediction, which will capture the properties we require our model to satisfy. The method is based on the assumption that the model is robust when it is accurate and also uncertain if the model is inaccurate. To calculate this metric we need to convert our predictions to binary classification from continuous values of price prediction and uncertainty value. We proposed the following approach for converting a price prediction for binary classification (accurate or inaccurate) in the training dataset: We consider S markets and for each hour t we predict pricep s t s ∈ S and actual price p s t s ∈ S Consider prediction is accurate if: accurate ← maxMarket(p s t ) = maxMarket(p s t ) Here maxMarket() is a function which iterate all s in S and find the max priced market. The certainty of the prediction is determined as follows: certain ← nu t < u th (8) Here u th is the uncertainty threshold where u th ∈ [0, 1]. u th is a pre-calculated constant value for each market, where optimum u th value for a market is calculated as a maximizing optimization problem of uncertainty accuracy (UA) the exact value was calculated using a hierarchical search algorithm through [0,1] range with a step 0.01 With the above model of classification, we can determine the confusion matrix with following components, accurate and certain (N ac , accurate and uncertain (N cu ), inaccurate, certain (N ic ) and inaccurate, uncertain (N iu ). We can determine the accuracy and two conditional probabilities P(accurate|certain) and P(inaccurate|uncertain) as follows, Uncertainty Accuracy(UA) = N ac + N iu (N ac + N au + N ic + N iu ) (11)

A real-world frequency reserves setting
The proposed AI-based bidding optimization framework was evaluated on the Finnish frequency reserve markets. We considered two day-ahead markets FCR-N and FCR-D and one hourahead market mFRR for the case study. We assumed that our resources had a 10MW capacity to react to frquency deviations at all times, so each bid was 10MW for bidding strategy 1 and 2. We used May-June time of the year for the evaluation as it has the most variation in price for FCR-N and FCR-D markets. We selected 30 days starting from 2018-05-10 for the analysis of the bidding optimization with price prediction with uncertainty.   The markets have similar technical requirements, with the main difference being that FCR-D needs to be activated more quickly.
The technical requirements [18] are such that many distributed energy resources, such as batteries, could be pre-qualified for both markets. Since bidding is on an hourly basis, and since both markets have the same bidding schedule [13], a Virtual Power Plant(VPP) could choose to bid on either of these markets on an hourly basis, always selecting the market on which the price is expected to be higher.

Secondary reserve:
The aFRR market in Finland is currently only active on certain morning and afternoon hours on weekdays, with very little price fluctuation. The TSO Fingrid intends to change this in the near future [64]. Thus, aFRR was not included in the case study.
Tertiary reserve: Fingrid confusingly uses the terms mFRR and balancing capacity markets interchangeable [53]. This is an hour-ahead market with separate up and down markets like aFRR.

Experiment 1:Uncertainty Evaluation
In this experiment, we predict the price for 30 days starting from 2018-05-10 with uncertainty evaluation based on both MC-dropout and GSOM methods for the three considered markets: FCR-N, FCR-D and mFRR. Figure 3 visualizes the three markets each sub-figure depict the actual price, predicted price and uncertainty measure. Further, We have used three uncertainty thresholds to identify uncertain price predictions for each market, uncertain price regions are shaded in red for each market.  Table 5 Uncertainty accuracy is quantified using the proposed uncertainty accuracy (UA) metric and uncertainty measure from MC-Dropout and GSOM. The highest accuracy generated is 78.88%. Compared to the other two markets FCR-D market has more uncertain price ranges corresponding to the highest UA value, as shown in Figure 3  Selection of uncertainty thresholds was resolve as a max-imizing optimization problem. For the optimization problem, we used uncertainty accuracy (UA) as the maximizing cost function and used hierarchical search method to find the optimum threshold values. Table 5 shows the selected threshold values for each market for the UA value of 78.88%.

Experiment 2:Bidding Strategy 1 and 2 analysis
This experiment analyses bidding strategy 1 and 2 using the predicted price, uncertainty measure and uncertainty threshold values for highest UA value identified in experiment 1. Each strategy selects the highest priced market from the three markets to bid for each epoch. Figure 4 and 5 show the selected market vs the actual price for the three markets. The selected market is depicted as a red dot in actual price graphs.   Table 6: Bidding Strategy 1 and 2 comparison with accuracy Experiment 2 is evaluated as a binary classification problem, for each epoch if the max selected market from bidding strategy is equal to the actual max priced market it is considered as a correct classification. Figure 6 shows the confusion matrix for the two bidding strategies for predicted and actual values for each market. According to the confusion matrix, both bidding strategy 1 and 2 has the highest accuracy for the mFRR market and lowest for the FCR-D. Table 6 shows the classification accuracy of the two bidding strategies against without bidding strategy with price prediction. Bidding strategy 2 outperforms the other two methods.  Further, experiment 2 is evaluated with and without uncertainty. For each method, revenue is calculated assuming the 10MW constant supply for each epoch. Total revenue of each method is compared in Table 7. According to the results, bidding strategy 1 has a significant increase in revenue with uncertainty measure, but bidding strategy 2 has a slight decrease with uncertainty measure compared to without uncertainty. With bidding strategy 1, the hour-ahead market mFRR is traded on only when there are high uncertainty values for both day-ahead markets FCR-N and FCR-D. Due to frequent high prices on mFRR, the use of uncertainty with bidding strategy 1 has resulted in increased revenue. This phenomenon does not occur with bidding strategy 2, since it refrains from bidding on dayahead markets if a higher price has been forecasted on the hourahead market. The difference between bidding strategies 1 and 2 is that strategy 1 relies more on AI whereas additional knowledge of multi-stage markets has been incorporated to bidding strategy 2. According to Table 7, the uncertainty metric is useful to overcome the lack of knowledge on multi-stage markets in the case of bidding strategy 1.

Experiment 3:Bidding strategy with reschedulable loads
This experiment evaluates bidding strategy 3 for reschedulable loads. This experiment is also based on the price prediction, uncertainty measure and uncertainty thresholds acquired in experiment 1 in 7.1 Reschedulable loads are evaluated according to the EV vehicle scenario mentioned in the Section 4.3 Bidding Strategy 3. 4 Schemes are compared including the novel method based on predicted price and uncertainty value. Figure 7 shows the results for each scheme of this bidding strategy. The red dots indicate the bidding market for each selected epoch with amount C[e]. Compared to the scheme 1,2 and 3, scheme 4 capture most of the price peaks in all three markets.

Bidding strategy with reschedulable loads comparison
Experiment 3 is evaluated by comparing the revenue generated from each scheme. Figure 8 and 9 depict the comparison of the scheme 1 to 4 revenue.  Figure 8 shows the revenue of each epoch of scheme 1,2 and 3 compared to scheme 4. According to the graphs, scheme 4 has significant difference in individual epoch revenue compared to scheme 1 and 3. Scheme 2 has slight changes in epoch times. Figure 9 (cumulative revenue graph) further confirms this, according to this graph at the start all four schemes perform equally but after date 2018-05-17 there was a significant increase in the revenue in scheme 4. Moreover, this clearly shows the advantage of slight changes in time between scheme 3 and 4. The reason for this is evident from Figure 7: scheme 4 is the only scheme that is able to identify and bid all the capacity on high FCR-N price peaks using the AI-based price forecast-ing. Even though some volatile high price points are missed due to the high uncertainty values in scheme 4, it significantly reduces the risk of bidding on low-confidence epochs, leading to a competitive advantage compared to the other three schemes when combined with price forecasting.

Conclusion
Bidding optimization on multiple frequency reserve markets is a significant innovation that enables our collective clean energy future. In this paper, we presented the design of a generalised model for frequency reserves markets based on an extensive study of such markets in several countries, followed by the formulation of three bidding strategies based on this generalised model. We implemented these three strategies as an AI-based bidding optimization framework, with data-driven price prediction supplemented with uncertainty metrics as inputs to the bidding optimization process in multiple markets. To the best of our knowledge, this is the first study to model the bidding process on frequency reserves markets, formulate bidding strategies for each variation and implement these using an AI framework for optimal bidding outcomes. The framework and corresponding bidding strategies were empirically evaluated on the real-world case study of the Finnish frequency reserves markets. Out of the three bidding strategies, the third with reschedulable loads significantly outperforms the others. In multi-market bidding optimization when a tertiary market has a higher price, all markets with forecasted price should be considered instead of sequential consideration of day-ahead markets followed by hour-ahead markets -this conclusion is supported by the significantly higher revenues obtained by bidding strategy 2 opposed to bidding strategy 1. In summary, these results confirm the effectiveness of the proposed framework for optimized bidding on multiple frequency reserve markets.

Bidding interval
The interval between two successive auctions that are held for a specific reserves market. E.g. for day-ahead markets the interval is one day and for hourly markets, the interval is one hour A i Auction An auction that is held for bidding interval "i" of a specific reserve market in which a TSO procures the necessary amount of reserves by accepting or rejecting bids submitted to that auction Bid m

Bidding deadline
The deadline by which bids must be submitted in order for them to be considered for a specific auction on market "m".

Reserve Unit
A unit that fulfils the requirements concerning the providing of the reserve. A Reserve Unit can consist of one more Reserve Resources. Reserve Resource an individual resource capable of control; a power plant, consumption facility or energy storage facility. Epoch The shortest time period for which a reserve unit can be offered on a frequency reserves market N epochs Epochs per interval The number of epochs in one bidding interval S P Service Provider a party that has a valid agreement on the delivery of reserve services.

C[e]
Capacity Capacity is the power consumption/production that can be controlled in response to a frequency deviation according to the technical specifications of the reserve market.  Best market based on forecast The market that had the highest price with acceptable prediction uncertainty on epoch e Table 3: Terms and Abbreviation specific to bidding strategy 3