{'fraction_non_alphanumeric': 0.03970848658220951, 'fraction_numerical': 0.012872905598692277, 'mean_word_length': 4.711923750243144, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction-Free, Real-Time Flexible Control of Tidal Lagoons through Proximal Policy Optimisation: A Case Study for the Swansea Lagoon

Tidal Range Structures (TRS) have been considered for large-scale electricity generation for their potential ability to produce reasonably predictable energy without the emission of greenhouse gases. Once the main forcing components for driving the tides have deterministic dynamics, the available energy in a given TRS has been estimated, through analytical and numerical optimisation routines, as a mostly predictable event. This constraint imposes state-of-art flexible operation methods to rely on tidal predictions to infer best operational strategies for TRS, with the additional cost of requiring to run optimisation routines for every new tide. In this paper, a Deep Reinforcement Learning approach (Proximal Policy Optimisation through Unity ML-Agents) is introduced to perform automatic operation of TRS. For validation, the performance of the proposed method is compared with six different operation optimisation approaches devised from the literature, utilising the Swansea Bay Tidal Lagoon as a case study. We show that our approach is successful in maximising energy generation through an optimised operational policy of turbines and sluices, yielding competitive results with state-of-art optimisation strategies, with the clear advantages of requiring training once and performing real-time automatic control of TRS with measured ocean data only.

we propose the usage of Deep Reinforcement Learning (DRL) methods, more specifically Proximal Policy Optimisation (PPO) through the Unity ML-Agents package [10], which enables state-of-art energy generation of TRS (on par with best optimisation routines) through the real-time control of turbines and sluices.
DRL was chosen among machine learning techniques due to the nature of our problem, which involves sequential decision-making of a reactive environment (lagoon water levels vary depending on the operation of hydraulic structures) with the goal of maximising expected return (energy), and also because a target optimal operation of the tidal lagoon is not known "a priori" -a requirement for supervised learning techniques. After training, our method shows consistent performance, regardless of test data used, not requiring future tidal predictions or re-training the DRL agent. To date, this is the first flexible operation optimisation approach in the literature that can maximise TRS energy generation without such constraints. A 0D model of the Swansea Bay Tidal Lagoon is utilised to compare our DRL method with six optimisation baselines devised from the literature [11,12].
After related work (Section 2), this paper is divided in four parts. In the first part, we explain how tidal barrages extract energy from the tides through the lenses of classical and variant operation approaches. In the second part, we cover the theory behind DRL, more specifically on the PPO algorithm that is used in this study. In the third part, we cover our agent-environment setup, modelled with the Unity ML-Agents package (Table S1, Supplementary Material). In the final part, an experimental study contrasting our results with six baselines is presented and discussed.

Related Work
State-of-art optimisation methods for TRS estimate the available energy in such systems by operating the tidal lagoon hydraulic structures through flexible operational strategies (adaptive, according to tidal amplitudes and lagoon water levels for sequential tidal cycles [12,11]). With the assumption of well predictable tides, flexible operation of turbines and sluices can be inferred by "looking-ahead" through harmonic or numerical tidal prediction methods [13] and applying the acquired operation to the real, measured ocean -a procedure that needs to be repeated for every new tide. In fact, and to the best of our knowledge, the requirement of accurate future tidal predictions (concurrent with measured data and up to a multiple of half-tidal cycles into the future) has been the basis for all optimisation routines developed for enabling flexible operation [14,12,11,15,16,2,17]. This constraint can be a problem when future tidal predictions are unavailable, unreliable, have some associated validation cost [18] or are regulated by private companies or government agencies.
State-of-art optimisation routines in the literature utilise either grid search (a brute-force approach), gradient-based, global optimisation [12,11,16] and, more recently, genetic algorithm methods [17] to optimise the operation of TRS.
As a basis of comparison with our DRL agent, two non-flexible and four flexible state-of-art baselines devised from [11,12] are modelled utilising grid search and global optimisation methods.

Tidal Power Overview
TRS extract power by artificially inducing a water head difference between the ocean and an impounded area. By allowing water to flow through the hydraulic structures into an artificial impoundment, the incoming tide (flood tide) is confined within the lagoon at high level (holding stage). Then, during the receding tide (ebb tide), power generation begins when a high operational head (H start ) is established between the basin and ocean [19]. Power generation then stops when a minimum operational head (H min ) is achieved. A sluicing sequence immediately follows, where idling turbines and sluices allow water to flow in order to increase lagoon tidal range for the next operation. Following the same procedure, generating energy is also possible during the flood tide, although with reduced efficiency due to turbines usually being ebb-oriented [11]. From the literature, operational strategies that allow for power generation to occur during flood and ebb tides are called "two-way scheme" operation [19].
Operational modes for controlling TRS in a "two-way scheme" are shown in  Table 1. TRS turbines can be operated to either generate energy or to increase flow rates through the barrage during sluicing stage (idle operation of turbines).
Also, a minimum head H mt , usually in the [1 m − 2 m] range [20], is required for the turbine to generate energy (H mt = 1 m in this study). Considering that the holding stage begins automatically when the difference between ocean and lagoon is negligible, and that power generation is not possible with head differences below H mt , the classic operation of tidal lagoons [19] is reduced to two variables: H start and H min . As seen in Fig. 1, pairs H start and H min occur every half-tide period, when ocean oscillates between its valleys and peaks.
A slight modification of the discussed classical operation allows for opening the sluice gates at the end of "flood" and "ebb" generation stages [21,11] independently of H min , with the possibility of increasing power generation (in-   Fig. 1. Tables 2 and 3 show all possible combined operations of turbines and sluices, with resulting power generation, for each control stage in classic and variant operations, respectively. Classic and variant approaches to operate TRS are used in the optimisation routines of our baselines in Section 6.1.

Tidal Lagoon Simulation -0D Model
In order to estimate the available energy of two-way operational strategies, analytical or numerical models (0D to 3D) can be considered. When the goal is the optimisation of TRS operation for maximising energy generation, 0D models are usually chosen, given their computational efficiency, and the fact that for "small-scale" projects, such as the Swansea Bay Tidal Lagoon, 0D  [22,11,12,2,23]. 0D models are derived from conservation of mass: where L is the water level (in meters) inside the lagoon, Q T is the total directional water flow rate (m 3 /s) from both sluices and turbines and Al(L) is the variable lagoon area (m 2 ). From Eq. 1, the lagoon water level at the following time-step (L t+1 ) can be calculated by a backward finite difference method: where L t is the water level at time-step t and ∆t the discretized time (60 s in this work).

Turbine and Sluice Parametrization
From 0D to 2D models, studies show that flow rate and power from turbines can be approximated with the parametrization of experimental results [24,20,25]. For this work, the equations describing flow and power for low head bulb turbines were based on experimental results from Andritz Hydro [25]. The edited Andritz chart shown in Fig. 2 demonstrates how turbine unit speed n 11 and specific unit discharge Q 11 (obtained experimentally) are related. The graph also shows wicket gate and running blade openings (α and β, in degrees), and iso-efficiency curves E f . adapted from [25]. The blue line represents the parameterized maximum power output curve.
By specifying the parameters of the turbine: diameter D, number of generating poles G p and grid frequency f , the turbine rotation S p (rpm) is obtained from S p = 120f /G p . Furthermore, unit speed n 11 , turbine flow rate Q t and power output P are calculated as: H e is the head difference between ocean and lagoon, ρ the seawater density (1024 kg/m 3 ), g the gravity acceleration (9.81 m/s 2 ) and C E is the product of other efficiencies shown in Table 4.
When H e is available, n 11 is estimated directly from Eq. (3). For calculating Q t and P , Q 11 and E f are obtained experimentally by adjusting the opening of the wicket gates (α), the pitch angle of the runner blades (β) and crossing the values with the obtained n 11 (see Fig. 2). In order to choose appropriate values for α and β, a parameterized curve of maximum power output was drawn over and E f = (−0.0019)n 11 + 1.2461.
For simulating sluice gates, the barrage model utilises the orifice equation, so that the flow rate Q o is a function of H e [19,21]: where C d is the discharge coefficient for sluices (equal to one in this study, following [11]), and A S the sluice area.
When generating energy, turbines use Eq. (4) for estimating flow rate through the barrage. On the other hand, when operating in "idling" mode, turbines use equation Eq. (8), with C d = 1.36 [11].
When starting or stopping either turbines or sluices, the literature has used sinusoidal ramp functions r(t) for simulating the smooth transition of flow output as r(t) = sin[(π/2)(t − t m )/t r ] [11,22], where t ∈ [t m , t m + t r ], t r = transition time (around 15 min to 20 min [11,26]), and t m is the time when the current operation was triggered. Since this is a heuristic method, a simpler transition function (named "momentum ramp") is proposed in this work: Q t+1 is the estimated flow rate at the next time-step, Q c is the total flow rate calculated from turbines and sluice equations (Eq. (4, 8)), ζ a dimensionless hyperparameter that controls the intensity of flow rate update per time-step, Q t is the flow rate at time-step "t" and ∆Q = Q c − Q t .
The "momentum ramp" is applied every time-step during simulation. This not only simplifies the code, but facilitates training, since sluice opening is treated as a continuous control problem (Section 5.2). In this work we set ζ = .4, which guarantees a precision of 10 6 for a 15 min time interval with ∆t = 60s.

Reinforcement Learning Overview
As shown in the work of Sutton and Barto [27], a reinforcement learning where, s i , a i , r i+1 are instances of the random variables (S t , A t and R t ). The sequence of state-action pairs defines a trajectory τ : Also, in an MDP, we can say that the probabilities of R t+1 and S t+1 are completely conditioned on the preceding state and action (S t and A t ), that is: The probability distribution of Eq. (12) defines the dynamics of the MDP. It can also be manipulated to yield the state-action-transition probability distribution (which is just the sum of probabilities over all possible future rewards): For estimating Eq. (13) for a given state, we also need to condition an action.
In non-deterministic scenarios, the selection of possible actions by the agent is a stochastic process, defined by a conditional probability distribution (known as policy) of the form: π(a t |s t ).
Using Eq. (13,14), the probability distribution of starting in a state s t and ending in s t+1 , given a policy, can be estimated as: With a defined policy, we can sum the observed rewards for each state-action pair (as shown in Eq. (10)) and calculate a total return G t at time-step t: where γ is a discount factor between 0 and 1.
The objective of reinforcement learning problems is to find an optimal policy π * , that maximises the expected return of rewards E[G t ] conditioned on any initial state, i.e.

Proximal Policy Optimisation (PPO)
Once the reinforcement learning problem is formalised as an MDP, several algorithms can be used for finding an optimal control policy π * . In this work, the process of finding π * has been achieved through Proximal Policy Optimisation (PPO) [28], built in the Unity ML-Agents package. PPO was shown to outperform several other "on-policy" gradient methods [28] and is one of the preferred methods for control optimisation when the cost of acquiring new data is low [29]. Furthermore, a revised and up-to date PPO algorithm is already implemented in Unity ML-Agents. Nevertheless, for sake of completeness, we present in this section the mathematical derivation of the PPO method. The designing of the TRS operation as an MDP is shown in Section 5.2.
Differently from approaches that try to infer the policy through state-value or action-value functions (e.g. Deep Q-Network) [30], PPO uses an "on-policy" approach that maximises the expected sum of rewards by improving its current policy -smoothly shifting the probability density function estimate of the policy towards π * . The PPO algorithm is an updated form of Policy Gradients. As TRPO (Trust Region Policy optimisation) [31], it tries to increase sample efficiency (re-using data from previous policies), while constraining gradient steps to a trust region. It is also actor-critic, since it utilises an estimate of the statevalue function for its baseline [32]. An overview of Policy Gradients and PPO is presented below.

Policy Gradients
Policy gradient methods rely on the fact that a stochastic policy can be parameterized by an "actor" neural network with weights θ (simplified as θ going forward). As represented in Fig. 4, this neural network receives a vector state representation of s t . For the case of discrete actions, the neural network outputs the probabilities of each possible action in that state using a softmax layer. For continuous actions, each node in the last layer outputs the moments of a multivariate Gaussian distribution of the form [33,34]: where µ and Σ are the parameterized mean and co-variance matrices, respectively. While training, actions are randomly sampled from the distribution to favour exploration. During testing, µ is taken as the optimum action for each input state s t .
Considering a trajectory τ , the expected return of following a parameterized and R(s t , a t ) is the reward from taking action a t from state s t . We also note that G (τ ) represents the undiscounted return following a sampled trajectory τ for a time horizon H. With these considerations, finding an optimal policy can be viewed as tuning θ to maximise U (θ), i.e. to perform gradient ascent of θ: A sample based estimate for ∇ θ U (θ) assumes the form: where m is the number of sampled trajectories from the "actor" neural network.
For vanilla policy gradient methods [35], following a trajectory τ (i) , we get V π (s (i) t ), parameterized by a "critic" neural network, is the estimate for the value function of being in state s t and following policy π thereafter; G f uture is the discounted future return of following the chosen trajectory, from time k = t; andÂ t is the advantage estimate of taking this trajectory in respect to the current estimate of V π (s (i) t ). A complete derivation of ∇ θ U (θ) can be seen in [35].

Clipped Surrogate Loss derivation for PPO
In order to increase sampling efficiency [36], importance sampling can be used to rewrite the gradient term in Eq.( 21) as: Eq.( 24) importance sampling form allows for re-utilising samples from an older policy to perform gradient ascent steps, when refining a new policy. It is obtained when differentiating the Surrogate Loss: where r t (θ) = π θ (a|s)/π θold (a|s) is a probability ratio.
where is a hyperparameter that limits large policy updates.
To further reduce variance when estimating the advantage, [28,32] utilise a truncated version of generalized advantage estimation [37], whereÂ t is esti- where t is a time index within the sampled trajectory time horizon [0, H], λ is a hyperparameter that performs the exponential weighted average of k-step estimators of the returns [37], and δ t = r t + γV (s t+1 ) − V (s t ).
When utilising shared parameters for the "actor" and "critic" neural networks (as is the case with this work), the loss function needs to be augmented with a value function error term [28]. To ensure exploration, an entropy term "S" is also added. Finally, the loss function to be maximised at each iteration becomes: For this study, Unity ML-Agents package fixes c 1 = 0.5 [38], β is a hyperparameter controlling the entropy bonus magnitude S, and L V F t is a clipped, squared-error loss between the estimate of the state-value function V π (s t ; θ) and the actual return value obtained when following a trajectory τ [28]. The implementation of L V F t in Unity ML-Agents is seen in [38].
Additionally, parallel training can also be implemented as a way of substituting experience replay by running the policy on multiple instances of the environment. By guaranteeing that each environment starts in a random initial state during training, this parallelism helps decorrelate the sampled data, stabilising learning [39].

Unity ML-Agents
The Unity3D graphics engine is a popular game developing environment that has been used to create games and simulations in 2D and 3D since its debut in 2005. It has received widespread adoption in other areas as well, such as architecture, engineering and construction [10].
Unity ML-Agents is an open-source project that allows for designing environments where a smart agent can learn through interactions [10,38]. It has been chosen in this project due to ease of implementation, built-in PPO algorithm and visual framework for visualising real-time control of TRS.

Agent-Environment MDP Modelling and Training
To formalise TRS operation as a RL problem (and subsequently solve the RL problem through PPO) we need to design an MDP in Unity ML-Agents with environment, agent, actions, states and reward components.
By creating simple representative 3D models for turbines, sluices, ocean and lagoon, a training environment simulating TRS for our MDP is created in Unity3D and then imported to a Unity ML-Agents project. In this environment, the equations for simulating flow operation through the lagoon, when operating sluices and turbines, are extracted from the 0D model representation, detailed in Section 3.2. In order to choose appropriate parameters for operating our environment, we follow literature representations suggested by [11,22,12], for the Swansea Bay Tidal Lagoon project. The chosen parameters are shown in Table 5. A variable lagoon surface area, digitized from [12], is also utilised.  during ebb generation, with the representative models for sluice and turbines in offline and power generation modes, respectively. Ocean and Lagoon surface level motion are also represented.   Tables 6 and 7, respectively.  The actions selected by the agent are a function of the input states s t . In this work, these states are the water levels of ocean and lagoon, plus current operational mode of turbines and sluices, for current and previous MDP time-steps (Table 8). Finally, the reward received by the agent equals the accumulated energy generated by the turbines, for every 15 min.   is sufficient for enabling the agent to converge to an optimal policy. A major advantage of this approach is the fact that we can generate any amount of input data required for training the agent.
The tide constituent's amplitudes of the simulated ocean utilised in this work (Table 9), were obtained from a numerical simulation, at the location of Swansea Bay Tidal Lagoon, by [11]. The periods for each constituent were obtained by [40]. The final equation for simulating the ocean can be seen in Eq. (29).
where ω M 2 , ω S2 , ω N 2 and ω K1 are angular frequencies (rad/s) of each tidal component, and φ M 2 , φ S2 , φ N 2 and φ K1 are random phase lags in the range [0, 2π], generated for each environment instance during parallel training when starting an episode, which allow for learning more generalised scenarios. Ocean tide constituent Amplitude (m) Period (hr) The designed MDP is solved through the PPO algorithm (Section 4.1). A flow-chart, illustrating the agent's training stage through the PPO algorithm in Unity ML-Agents, is presented in Fig. 7. Training occurs until a "maxsteps" number of observations are sampled. For reproducibility, Table S2   input, performing real-time flexible control of turbines and sluices.

Experiments
In this section we compare our DRL trained agent performance against stateof-the-art optimisation routines. Codes for reproducibility can be made available under request to the corresponding author.

Test Data and Baselines optimisation
For comparing our DRL agent performance against conventional optimisation routines, we model six baselines devised from the recent literature [11,12] and compare the energy generated in a month for each method. All baselines in this work consider the operation of the Swansea Bay tidal lagoon either through classic or variant "two-way scheme" methods, as detailed in Section 3.
Regarding test data for baselines and trained agent, we utilise all tide gauge ocean measurements available from the British Oceanographic Data Centre (BODC) at Mumbles Station [41], located at the edge of Swansea Bay. The obtained measurements of ocean elevation are recorded every 15 min in a table, for the years of 1993 and the range [1997−2019]. Before utilising the data, a preprocessing step is performed so that data flagged as "improbable", "null value" and "interpolated" by BODC are not considered. After this step we retain 26 months of usable, non-overlapping, test data. The preprocessing step ensures a conservative comparison between baselines and our trained agent, since it considers scenarios where tidal predictions had a good match with measured data.
Tidal predictions for the same 26 months are also provided by BODC in the same data-set. For each month, baseline optimisation routines utilise tidal predictions for capturing operational head values H start , H min and HS start (when considered) that optimise power generation. These operational head values are then applied to the measured ocean test data, so that comparisons between baselines and trained agent can be made. Baselines, in increasing order of optimisation complexity, are described next: • CH (Constant Heads): Best, constant H start and H min are picked for extracting energy during a whole month [14].
• CHV (Constant Heads, with variant operation): Best, constant heads H start , H min and HS start are picked for extracting energy during a whole month.
• EHT (Every Half-Tide): optimised pairs of H start and H min are picked for every consecutive half-tide. Proposed by [12].
• EHTV (Every Half-Tide, with variant operation): optimised H start , H min and HS start are picked for every consecutive half-tide.
• EHN (Every Half-Tide and Next): optimised H start and H min are picked for every half-tide, considering the best H start and H min for the next half-tide as well. Proposed by [12].
• EHNV (Every Half-Tide and Next, with variant operation): optimised H start , H min and HS start are picked for every half-tide, considering the best H start , H min and HS start for the next half-tide as well.
All variant optimisation methods are augmented through the addition of independent sluice head operation HS start . This modification we are introducing is inspired by the work of [21,11]. CH and CHV perform non-flexible operation, while EHT, EHTV, EHN and EHNV perform state-of-art flexible operation.
A summary detailing each baseline operational heads and method is shown in  Scipy package [43], with COBYLA as a local minimizer [44]. Basin-hopping was chosen for its efficiency when solving smooth function problems with several local minima separated by large barriers [45]. The local minimizer COBYLA is a nonlinear derivative-free constrained optimisation that uses a linear approximation approach. Even though basin-hopping is not guaranteed to converge to a global optimum, EHNV is shown to be, on average, the best baseline method for energy generation.

Agent Performance Evaluation
Following hyperparameter tuning, we trained the agent for 8E7 steps, until convergence. The cumulative reward (energy) per month (episode) during parallel training, averaged for the 64 instances of the lagoon environment, is shown in Fig. 8. The log-representation insert highlights the two-step plateau that is observed when converging to an optimal strategy. After starting in a total random strategy, the cumulative reward received by the agent increases until  For test data, we utilise 26 months of real ocean measurements from BODC.
These months are presented and numbered in Table S3, while Table S4 (Supplementary Material) compares the amount of energy obtained in the numbered months between our trained agent (performing real-time flexible control) and the baselines. A block diagram representing the trained agent test stage for this work, while using ocean water level (WL) measurements, is shown in Fig. 9.
The averaged monthly energy attained for all methods is shown in Fig. 10.
For the baselines, CH and CHV present the worst performance, since constant operational heads cannot account for the varying ocean amplitudes in a month (about ≈ 2 m to ≈ 4.5 m in our test set). Furthermore, baselines with variant operation outputted more energy in average than their classical counterparts. Finally, "half-tide and next" approaches showed very small improvements (< 0.2%) when compared to "half-tide" methods, while requiring much greater ( 20×) computational time (Tables S5 and S6, Supplementary Material).
For the trained agent, Fig. 11 show operational test results of power generation and lagoon water levels for one month of measured ocean data (starting with initial lagoon water level at mean sea level). We note that the agent quickly converges to an optimal energy generation strategy for sequential tidal cycles, independent of tidal range input -a characteristic of state-of-art flexible operation [12]. Furthermore, Figs    Our agent managed very competitive energy outputs, staying on average within 1.4% of the best baseline (EHNV). Indeed, for all months tested, our agent performed optimally, outputting better results than the state-of-art EHN method for 22 out of 26 months (Table S4, Supplementary Material), within a 1.1% margin in worst scenarios. We note that this novel result was obtained by training the agent once with a simple artificial ocean input, in contrast with the baselines that require future tidal predictions and being re-run for every new tide.
(a) Ocean (in blue) and lagoon water levels.
(b) Flow rate from the 16 turbine units.

Conclusions
In this work, we have shown that Proximal Policy Optimisation (a DRL method) can be used for real-time flexible control of Tidal Range Structures after training with artificially generated tide signals. Our DRL agent implementation was then compared against state-of-art optimisation approaches devised from the literature, yielding competitive results for all test data utilised.
Our results were obtained in a conservative setting, i.e., when available tidal measurements had good agreement with tidal predictions (a requirement for state-of-art approaches).
We have chosen the Swansea Bay Tidal Lagoon for our analysis, given its status as a pathfinder project for larger tidal lagoon projects. We show that our novel approach obtains optimal energy generation from measured tidal data only, through an optimised control policy of turbines and sluices, which are operated independently by the trained DRL agent. Our method shows promising advancements over state-of-art optimisation approaches since it (i) performs real-time flexible control with equivalent energy generation, (ii) does not require future tidal predictions, and (iii) needs to be trained a single time only.
Owing to its characteristic features, the method introduced here shall be of wide applicability to achieve optimal energy generation of TRS in cases where future tidal predictions are unreliable or not available. Possible extensions of this work include more sophisticated cases, such as through the introduction of pumping capabilities and/or demand-oriented operation (where the goal is maximising revenue). For such cases, the increase in both the degrees of freedom for the agent's actions and stochasticity from the input signal could further emphasise the advantage of a DRL real-time operation of TRS over current state-of-art methods.