{'fraction_non_alphanumeric': 0.03302126089563252, 'fraction_numerical': 0.017248535719227045, 'mean_word_length': 4.715340010542962, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Model Predictive Control of Stochastic Wastewater Treatment Process for Smart Power, Cost-Effective Aeration

Abstract Wastewater treatment is an essential process to ensure the good chemical and environmental status of natural water bodies. The energy consumption for wastewater treatment represents an important cost for water utilities. Meanwhile has the increasing fraction of renewable energy sources in the electricity market created the possibility of exploiting cheaper (and greener) electricity. This paper proposes model predictive control driven by stochastic differential equations and genetic optimization to prioritize aeration in periods with low electricity prices thereby reducing costs and empowering smart use of green electricity. This is without violation of legislation and equipment constraints. The method is tested with real plant data and electricity market prices to demonstrate efficiency and feasibility.


INTRODUCTION
Wastewater treatment plants (WWTP) play a vital role in modern societies. They treat polluted water from domestic and industrial sources before it is discharged to the environment. Thereby they protect recipients from large nutrient loads and reduce human health hazards related to exposure to faecal matter. Unfortunately, this treatment comes at a high cost. Municipal wastewater treatment accounts for 0.8% of total electricity consumption in the US (Pabi et al., 2013) and a similar picture is drawn in many other countries (Longo et al., 2016). This implies; (1) a large economical cost for water utilities and (2) an environmental cost in terms of air pollution and green house gas emissions related to the electricity production method. Consequently, we need to balance between relaxed treatment of wastewater before discharge and increased costs related to electricity consumption.
A key energy demanding process in a typical WWTP is the aeration. Aeration is used to reduce nutrient concentrations such as nitrogen, N , by up to 95% before treated water is discharged. Nitrogen removal requires in fact different species of specialized bacteria living in different conditions. Aerobic conditions (O 2 present) are needed to reduce ammonium (N H 4 ) to nitrate (N O 3 , nitrification) while anoxic conditions (O 2 absent) are needed to reduce nitrate to nitrogen gas (N 2 , denitrification). This is the Alternating Activated Sludge Process (ASP). In WWTPs it is operated in large engineered tanks with aeration equipment that is turned on and off in specially designed/controlled cycles to secure good treatment (e.g. (Zhao et al., 2004;Nielsen andÖnnerth, 1995)). Typically 50-60% of the electricity used by a WWTP is aeration (Longo et al., 2016). Electricity is though not the only cost related to aeration as the discharge of nutrients is object to taxation, e.g., 30 DKK/kg discharged N in Denmark.
The wastewater treatment processes can be predicted online as a function of aeration signals as suggested by Stentoft et al. (2018) or Halvgaard et al. (2017). These methods build on Stochastic Differential Equations (SDEs) derived from well established, deterministic Activated Sludge Models (ASM) (Henze et al., 2000(Henze et al., , 1987. The process being predictable suggests that nonlinear Model Predictive Control (MPC) can be implemented to optimize aeration 24 hours ahead with respect to legislational effluent requirements, equipment constraints, taxes, and electricity prices/sources. However, this is not trivial as the aeration signal contains integers and real numbers and the ASP is affected by uncertainty of the nonlinear, bio-chemical processes.
One way to deal with complex control problems is by the application of Evolutionary Algorithms (EA) for the optimization (Fleming and Purshouse, 2002). Here we suggest a setup that uses a genetic optimization algorithm suggested by Mebane and Sekhon (2011) to find a good solution to the predictive control problem. The solution satisfies constraints given uncertainty and prioritizes cheap (typically green) electricity. It also avoids periods with expensive electricity such as in the evening. The strategy is data-driven with respect to online data from the WWTP and data from electricity market. Ultimately, this strategy can help WWTPs to enhance the use of green electricity from, e.g., wind turbines while saving operational costs.

Plant and data
The WWTP of Nørre Snede serves a catchment with a population of about 4000 Inhabitant Equivalents (IE) and two small industries producing about 500 IE. The plant contains the standard treatment steps including pretreatment, desanding, grease trap, aeration tank and secondary clarifer. The aeration tank is 3500 m 3 and water coming into the tank has a residence time of 2-3 days in dry weather. The aeration is currently controlled with a rule-based control strategy where aeration set points are determined from newest ammonium and nitrate measurements as described in Zhao et al. (2004) ;Nielsen andÖnnerth (1995).
The control is designed to fulfill two sets of constraints; (i) Equipment/Process constraints and (ii) Legislational constraints. These constraints affect the aeration set point, Os, that describes how much air should be released into the water.
(i) Equipment/Process constraints are related to the performance limits of the equipment (actuators and blowers) and good conditions for the biological removal processes. This results in bounds on the actuator settings and bounded N-time (aeration on, nitrification-time) and DN-time (aeration off, Denitrification-time). Ultimately, these constraints govern whether a sequence of aeration setpoints is valid for control, i.e., whether a sequence can actually be executed. The constraints for Nørre Snede WWTP are shown in Table 1 where it is indicated, that setpoint values, Os, should be 0 during DN-time and between 1 and 3 during N-time. (ii) Legislational constraints are related to the acceptable effluent concentrations decided by authorities. Hence, these dictate the maximum acceptable concentrations measured on a 24-hour weighted average for N H 4 and total-N concentrations in the effluent. These must be below 2 and 8 mg-N/L respectively where the weighted average is weighted with the waste water flow through the plant. This means that a control sequence of setpoints Os must result in average concentrations that are below these limits.
The plant is well equipped with sensors and computers that log data into a database. Hence, historic and real-time data is available online. The data consists of online sensor measurements of ammonium (M sN H) and nitrate (M sN O) taken directly in the aeration tank every 5 minutes. Furthermore the aeration signal, Os, is logged every two minutes and inflow to the plant is measured by a flow meter every two minutes.

Aeration costs: electricity and tax
The plant (and wastewater aeration in general) has two cost components that are related to the specific design of a control sequence, namely taxation and electricity costs. In Denmark, the tax is decided by authorities and it covers the price of discharging different nutrients to the environment. Currently (2018), the tax for Nitrogen, N tax , is 30 DKK/kg-N. The total amount is calculated by multiplying the discharged Nitrogen, N t , with the tax and the flow through the plant, Q t , as shown in (1) The electricity cost, E t , is the electricity used for aeration at the WWTP. It is calculated as a linear function of the set point, Os t , the electricity price, Ep t and a correction factor β which is related to the size and type of aeration equipment installed at the plant. For Nørre Snede WWTP this parameter is approximately 1.4. This cost is shown in (2).
As it can be seen from the equations, a rule of thumb is that increased aeration leads to increased electricity cost but a reduced tax as more nutrients are removed. However, due to the nonlinear dynamics of the bio-chemical processes this will not always hold.

Electricity price data
Electricity prices are available from the North European electricity market (Nordpool, 2018). This data consists of hourly electricity prices from the first 5 months of 2018 for Scandinavia and the Baltics. Nørre Snede WWTP is located in the market "DK1" which covers the western part of Denmark (Jutland and Funen). Like in real control, we do not know future prices and account for this by creating price "scenarios", S, which take different price developments (but with similar overall patterns) into account. A price scenario is here defined as 5 ensembles that each describe the price development over 24 hours as given by data. The ensembles are all chosen from the same weekday. Consequently, to create a scenario, we choose a start date, and sample the four remaining ensembles by shifting the start time 1, 2, 3 and 4 weeks ahead. The generated scenarios are: S 1 Contains the minimum price in the period, Sundays starting from 28-1-18 S 2 Arbitrarily chosen, Wednesdays starting from 14-02-18 For a more quantitative evaluation of the strategy, we sample all Tuesdays, Wednesdays and Thursdays in the Nordpool dataset. This supplies a total of 59 ensembles (24 hours of hourly prices) which is shown in Figure 1 together with the two scenarios.  Fig. 1. The five price ensembles of the two different electricity price scenarios ("S1" and "S2") as given by data from the Nordpool market and ("All prices") 59 electricity price ensembles corresponding to the actual prices for all Tuesdays, Wednesdays and Thursdays in the Nordpool DK1 market in the first 20 weeks of 2018 (Nordpool, 2018).

Prediction Model
The ASP is well described by the family of Activated Sludge Models (ASM) (e.g. (Henze et al., 2000)). The ASMs consist of at least 13 nonlinear Ordinary Differential Equations (ODE) based on Monod-kinetics and massbalances. Based on the ASMs, a stochastic ASM (SASM) (Stentoft et al., 2018) is developed to predict nitrogen removal based on online measurements. The SASM contains 3 coupled Stochastic Differential Equations (SDE) which estimate ammonium (S N H ), nitrate (S N O ) and available oxygen (S M O,O ) in the aeration tank as a function of the aeration control signal. The parameters of the SASM are estimated with 4 days of online measurements of ammonium and nitrate and the realised control signal. The estimation of parameters in the SDEs is done by maximizing a likelihood function. The noise on the online measurements and the model noise are split in two terms and managed by an Extended Kalman Filter (EKF). The predictions are performed following a numerical integration scheme. This methodology is specified in Juhl et al. (2016) and Kristensen et al. (2004) and the implementation is more thoroughly described in Stentoft et al. (2018).
From this model, we can also define the mean and standard deviation of total-N, S N and σ N . As we exclude organic-N in the estimate of total-N, this becomes (3).
To make sure, that a control sequence leads to satisfaction of the legislational constraints, the outputs from the SDEs must be evaluated with respect to the 24 hour average. This implies, we have to evaluate (4) In this case, E[S N H,t ] and E[S N,t ] refer to the weighted average over 720 time steps (which corresponds to 24 hours in this setup), at time t. This average is calculated as a weighted average with weights, W t , that are based on the normalized flow.
When we wish to predict future expected values, say x steps ahead, we need to extend Equation (4) with the uncertainty terms. Here the uncertainty is calculated as the fraction, α, of a standard normal distribution multiplied with the standard deviation related to the nutrient estimates from the SDEs. The uncertainty on the output estimate of the SDEs is described in Juhl et al. (2016) and Kristensen et al. (2004). The predicted average fractions, P N H,t+x|t (α) and P N,t+x|t (α) are presented in Equation (5).
In the results and further investigation, α is set to 0.95 in all optimizations.

Objective functions
The optimization is performed on a lexicographic objective function as shown in (6).
Where the notation lexmin(...) refers to lexicographic minimization of the objectives in order left to right. The four objectives, C i , C ii , C iii and Cost are presented in (7).
The first and most important objective, C i , is the ammonium requirement. If an aeration control strategy results in too large ammonium effluent concentrations, the other objectives are downgraded until this is reduced to an acceptable level (which for Nørre Snede WWTP is below 2 mgN/L on a 24 hour average).
The second objective, C ii is the total-N requirement. This is similar to the ammonium requirement but considers total-N instead (which for Nørre Snede is 8mgN/L on a 24 hour average).
The third objective, C iii , is the ammonium endpoint requirement. This is securing that we do not consider aeration strategies that result in large ammonium concentrations after the optimization period (24 hours). This is because an increase in ammonium towards the end might fulfill legislation, but it will at the same time make it more difficult (if not impossible) to fulfill legislation in the following day if the concentration starts out very high.
The last objective, Cost t , is only optimized if the other three objectives are equally good (in practice they should all be zero). This objective represents the operational costs. The cost at time t ∈ [0; 720], Cost t is given in (7).
Where i ∈ [0, 720]. The tax, T ax t (..) and electricity cost E t (...) from aeration are given in (1) and (2). However, to account for the uncertainty in the electricity price forecasts we rewrite (2) to use more price ensembles in the cost calculation. The electricity cost at time t now becomes E t calculated as (8) Where Ep t,i is the electricity price at time t in ensemble i ∈ S which has probability s i of realization. This corresponds to optimizing with respect to the weighted mean electricity price in each time step.

Genetic optimization
We wish to optimize the four objectives with respect to aeration set points, Os. The aeration signal consists of one set point for each 2 min timestep and, consequently, determining future control 24 hours ahead requires 720 setpoints to optimize. However, it is noted that many setpoints will be determined due to constraints (i.e. when DNtime starts, the aeration must be off for atleast 20 minutes and hence the next 10 setpoints must be 0). Therefore, a parametrization of the oxygen signal is preferable as this will help to avoid calculation times for infeasible solutions without compromises regarding the solution. Here, we choose a simple parametrization where we divide each aeration cycle into three parameters and add a fourth parameter which describes the continuation of the cycle setup. This is illustrated in Figure 2. By using this parameterization, we reduce the number of required parameters to a minimum of four parameters. For optimization over 24 hours this parameterization is repeated multiple times to ensure that the cycle designs can change during the day. Furthermore, this parameterization allows us to directly apply the bounds from Table  1 as bounds on P 1 , P 2 and P 3 and hence we have the constraints in (9) 3 ≤ P 1 ≤ 30 10 ≤ P 2 ≤ 60 1 ≤ P 3 ≤ 3 P 1 ∈ Z P 2 ∈ Z P 3 ∈ R The minimization problem in 6 can now be solved using the parameterization of aeration and the bounds in (9). The last parameter, P 4 , is set to (10) We allow up to 8 different cycle-setups in the 24-hour optimization, and hence we have 4 × 8 = 32 parameters to optimize. For this task we use a genetic optimization algorithm suggested by Mebane and Sekhon (2011). This algorithm is implemented in the R-package "Genetic Optimization Using Derivatives" (rgenoud). This is to ensure an optimization that manages the non-differentiable, mixed integer aeration signal in a robust way and making use of multiobjective optimization. The algorithm creates a new generation of parameters by using 8 different heuristic methods for generating new parameter sets. These are; Cloning, Uniform mutation, Boundary mutation, Nonuniform mutation, Polytype crossover, Simple crossover, Whole non-uniform mutation, and Heuristic crossover. The operators are thoroughly described in Sekhon and Mebane (1998) and hence will not be repeated here. For this study all operators are weighted equally during the optimization. The population (i.e. the number of parameter sets in each generation) is set to 5000 which is considered sufficient for a good optimization. The optimization is considered as an integer optimization as three of the four parameters are integers. To account for the third parameter being a real number, we simply optimize the integer 100 ≤ P 3 int ≤ 300 which is P 3 int ≈ 100P 3. The termination criterion is set to 10 generations without improvement of at least 0.001 DKK/day.

RESULTS AND DISCUSSION
The results are generated using R version 3.2.2 (2015-08-14). The CPU is an Intel Core i7-6600 with 2.60 GHz. This results in a runtime of the genetic optimizations of 40-60 minutes (20-30 generations) before genetic convergence is reached.

Example: Optimizing scenario 1 prices
To illustrate the dynamics of the strategy we optimize different situations based on electricity prices from scenario 1 (Figure 1). We run 4 optimizations with different price inputs described below as Variable, Reverse, Constant and Unaware. For comparison we include Rule-based which is the currently implemented rule-based control strategy.
The results are summarized in Table 2 and a control example is shown in Figure 3.
• "Variable" Optimization based on the scenario 1 data with probability s i = 0.2. This is shown in Figure 3 • "Reverse" Optimization based on the reverse scenario 1 data with probability s i = 0.2. • "Constant" Optimization based on the mean price from scenario 1 data. • "Unaware" Price estimate found by not paying attention to electricity prices in control optimization (electricity prices equal 0 in optimization) and evaluation with scenario 1 prices • "Rule-Based" Price estimate based on the implemented rule-based control and evaluation with scenario 1 prices In Figure 3, we see that the relative amount of aeration used is reduced by about 50 % in periods where electricity is more expensive compared to the mean (which also corresponds to the constant price scenario). In Table 2, we see that the variable prices and the constant averaged price are similar. However the unaware MPC is 22.6 % more expensive than the MPC with variable electricity price. Likewise the Rule-based control is 18.6 % more expensive. This is because these do not necessarily find the optimal balance between electricity consumption and taxation for which the price aware predictive controls aims.

Investigation of multiple price realizations
To get a quantitative measure of the efficiency of the strategy compared with the currently implemented rulebased control, we estimate the best control given price scenario 2. This control is then used to estimate costs given the true electricity price turned out to be one of the 59 prices in Figure 1. This is repeated for all ensembles, and the statistics of the costs are compared. In Figure 4, a histogram and a cumulative plot show the total costs. In Table 3 a summary of the 59 estimates is shown. Table 3. Summary statistics of the costs for the stochastic MPC with "variable" electricity prices and the currently implemented rulebased control. The "difference" is the statistics of "Variable" subtracted from "Rule-based" results for the same price realization. All numbers are in DKK/day. In Table 3 is is noted that the rule-based control is on average 20.9% more expensive than the variable (MPC). Furthermore, the standard deviation is larger indicating that it is more uncertain how the total costs will turn out. From Figure 4 it can be seen that it is more likely to get a cheaper total cost using the variable strategy.

CONCLUSION
A MPC strategy for smart electricity use in municipal wastewater treatment aeration is developed. The summarized results are: • The strategy uses a genetic optimization algorithm to optimize a parameterized aeration signal 24 hours ahead with respect to weighted mean of multiple electricity price outcomes • The strategy performs well in prioritizing electricity consumption in timeslots with cheap electricity and deprioritize when electricity is expensive • Comparison with rule-based control shows a reduction in cost on Nørre Snede Wastewater treatment plant for 59 price realizations based on real electricity market data Consequently, we consider this a step towards integrating wastewater treatment in the electricity markets. Ultimately, our approach can help wastewater treatment operation adjust to changes in electricity supply/prices and thereby make them more resilient to increasing amounts of renewables in electricity grids.
Last, it should be noted that the strategy is only estimated for dry weather periods. During wet weather periods it is expected that the predictions from the model will be more uncertain, and hence it will be more difficult to use the flexibility in the processes. Consequently, wet-weather will probably lead to lower cost reductions. On the other hand, the strategy might perform better in satisfying legislation compared to the rule-based control. However, wet weather scenarios should be investigated to determine this. Other nutrients such as phosphor, P should also be included in the model, as P is also influenced by aeration and is also target of taxation and legislative requirements.