{'fraction_non_alphanumeric': 0.03296404540202542, 'fraction_numerical': 0.014952762862774418, 'mean_word_length': 4.5501697472651825, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Epidemic Control Modeling using Parsimonious Models and Markov Decision Processes

Many countries have experienced at least two waves of the COVID-19 pandemic. The second wave is far more dangerous as distinct strains appear more harmful to human health, but it stems from the complacency about the first wave. This paper introduces a parsimonious yet representative stochastic epidemic model that simulates the uncertain spread of the disease regardless of the latency and recovery time distributions. We also propose a Markov decision process to seek an optimal trade-off between the usage of the healthcare system and the economic costs of an epidemic. We apply the model to COVID-19 data from New Delhi, India and simulate the epidemic spread with different policy review times. The results show that the optimal policy acts swiftly to curb the epidemic in the first wave, thus avoiding the collapse of the healthcare system and the future costs of posterior outbreaks. An analysis of the recent collapse of the healthcare system of India during the second COVID-19 wave suggests that many lives could have been preserved if swift mitigation was promoted after the first wave.


Introduction
Since the beginning of 2020, the COVID-19 pandemic has been challenging decision-making around the globe. Reinfection, multiple viral strains and inconsistent reporting are just some of the issues that decision-makers ought to consider [4,14,40]. Another important component is the geographical scope of the mitigating policies. Although the pandemic is a globally interconnected event and despite calls for unified response efforts [35,36], mitigating measures are generally planned at a national or local level.
To properly prescribe containment measures, decision makers must consider not only the current spread, but also the future consequences of such measures. These consequences include immediate economic losses, as well as the costs and effects of further measures that might result from ineffective containment. An ineffective mitigation may lead to high prevalence which, in turn, requires stronger and therefore more costly mitigation [4,36]. Furthermore, under high prevalence, mitigation measures become increasingly expensive and difficult to manage and enforce [36]. Hence, it is clear that mitigating policies should be designed to manage both the infection levels, as well as the mitigation costs and future consequences. Since all of these components involve a large amount of uncertainty, this paper will use a stochastic approach to model the spread and the effects of mitigation.
Vital to the modelling of outbreaks, classical epidemic models [39,21] use deterministic differential equations to describe the spread and are sensitive to uncertain disease-specific parameters [42]. Stochastic models were introduced later [1] to consider the underlying uncertainties in the spread and in the duration of the disease cycle [7]. However, analytical and computational tractability is vital to append optimisation under uncertainty into the framework and provide decision support for mitigation.
Making use of queuing systems with infinitely many servers, we introduce a parsimonious yet realistic stochastic epidemic model that considers general latency and infection times. We then introduce a Markov decision process to derive optimal control actions [25] that seek a balance between the occupation of the healthcare system and economic costs, depending on the current number of infected and exposed individuals. We apply the approach to COVID-19 data from New Delhi and investigate the controlled output of the epidemic model. The results illustrate the importance of acting swiftly to contain the epidemic and avoid multiple waves, as well as the importance of continuously monitoring the outbreak as larger control review intervals imply higher optimal control levels.
The remainder of this paper is organised as follows. Section 2 contains a brief literature review and underscores the proposed contributions. Section 3 introduces the model and the optimal control formulation. Section 4 features experimental discussion based on the COVID-19 epidemic in New Delhi. Finally, Section 5 concludes the paper.

Literature Review
Early in the COVID-19 pandemic, an influential study helped shape public policy by championing non-pharmaceutical interventions (NPI) such as social isolation, home quarantine and lock-downs [16]. Intended to preserve the healthcare system, these measures also have a profound economic impact [43], which may lead policy makers to costly ill-timed interventions [32,41].
Understanding the spread is vital for decision support. To that end, authors employed artificial intelligence to assess the epidemic threshold, predict the spread and reconcile classical epidemic models with the wealth of epidemic data made available. The same approach was applied to evaluate specific mitigation measures in distinct localities [12,8]. While useful, data-based models are hindered by the quality of the reported data [22], which is often inconsistent as it depends on testing and reporting policies that differ widely around the globe [41,14]; furthermore, even a quality testing strategy can be compromised by the uncertain and reportedly large percentage of asymptomatic infections generally not captured in the reports [29].
The trade-off between societal and economic consequences motivated discussions on the ideal duration and intensity of lock-downs [34,41]. An analytic approach is to use optimal control within a classical epidemic model to seek a balance between societal and economic costs [20,33,4]. While useful, these models provide only a predefined open-loop strategy that disregards the stochastic variations of the spread.
Time series predictive models can incorporate observed fluctuations in the data [27], but do not benefit from analytic epidemic models. To emulate uncertain propagation within a network, a Monte Carlo Markov chain approach considers social media interactions. Similar network models can evaluate on-off quarantine strategies [28], or predict the geographical spread of a disease with the dynamics underpinned by a classical epidemic model. Perhaps because of their complexity, these models do not embed optimization in the design of intervention strategies. To include optimization, [31] integrated reinforcement learning and agent-based simulation within a virtual environment. The model is computationally expensive and therefore limited to small population sizes.
Parsimonious models such as SEIR (Susceptible, Exposed, Infected, Removed) effectively described the 1918 flu epidemic and the COVID-19 epidemic in the US [10,9]. Computationally effective and analytically tractable, these models provide an ideal framework for optimal control [20,33]. To derive a closed-loop strategy whilst considering an uncertain disease spread, we utilize the stochastic SEIR framework [1,11]. Markov processes underpin stochastic SEIR models [6,24,2] that assume exponentially distributed latency and infection periods. [6] studies the outbreak's duration, [24] derives the transmission rate per infected person and [2] estimates the epidemic's size. General infection periods demand less tractable semi-Markov models [13,19]. These are intended to predict the outbreak and do not include optimization.
The contributions of this paper are twofold. Firstly, we introduce a parsimonious stochastic model of epidemic progression that seamlessly incorporates general latency and infectious periods. In contrast to the complex and computationally expensive neural network designs in [31], we use a simple M t /G/∞ whose output at any time is a Poisson variable [15]. Although as general as the semi-Markov models in [13,19], the model remains tractable for large population sizes. Secondly, we use Markov decision processes [37] to introduce a stochastic optimal control formulation that seeks an optimal trade-off between the healthcare system's occupation and the economic impacts of mitigation measures. Since the model is based on parsimonious stochastic formulations, it is tractable, easy to use and guaranteed to converge. We use the approach to derive optimal control strategies based on COVID-19 data from New Delhi and discuss implications of the optimal control in view of the controlled epidemic trajectories derived from the model.

Mathematical Modelling
We use a classical compartmentalised SEIR model that divides the total population in susceptible (S), exposed (E), infected (I), and removed (R) individuals [1,11]. Susceptible individuals have not been afflicted by the disease and can acquire it; exposed individuals acquired the disease but it is still latent, i.e. it is yet to manifest and become contagious; infected individuals have already manifested the disease and can transmit it; finally, removed individuals have undergone whole disease cycle and can no longer be affected by the condition. At a given day k ≥ 0, the total number of new contagions, i.e. the increase in the exposed population, is a Poisson variable with rate where β > 0 is the rate of transmission per encounter. In addition, S(k) and I(k) are the numbers of susceptible and infected individuals at the outset of day k ≥ 0.
Let σ be a discrete random variable representing the length of the latency period, in days. Similarly, γ is a discrete variable representing the duration of the infectious period, in days. As infections evolve independently and since there is no limit on the number of concomitant infections, the exposition-toinfection phase can be modelled as a M t /G/∞ queue [5]. Let δ e (k) be the number of new contagions on day k ≥ 0. It follows from [15] that δ e (k) is a Poisson variable with rate: where Ω σ = {0, . . . , M σ } is the sample space of σ, E is the expected value operator, δ e (k) = E(δ e (k)) is the expected value of δ e (k), and s ≥ 0 is a discrete time delay. Note that M σ is the maximum possible value assumed by the variable σ, i.e. the maximum length of the latency period in days. A similar reasoning yields that the infection-to-removal queue is also M t /G/∞. Therefore, the number of new removals on day k ≥ 0 is a Poisson variable δ i (k) with rate [15]: , and s ≥ 0 is a discrete time delay. Note that M γ is the maximum possible value assumed by variable γ, i.e. the maximum length of the infectious period in days.
The proposed model evolves in two levels. The input and output rates in Eq. (1), (2) and (3) vary daily. At each day, the respective rates drive a time-varying stochastic process X t , t ≥ 0 that evolves in continuous time and monitors each individual jump in the population. X t = (S t , E t , I t , R t ), t ≥ 0 is a four-dimensional stochastic process that monitors the susceptible (S t ), exposed (E t ), infected (I t ) and removed (R t ) populations in time.
Let N = {0, 1, 2, 3, ..., N } and Ω = N 4 , where N is a finite non-negative integer and Ω is the state space of the SEIR populations. Additionally, let Z + be the set of non-negative integers. At any time t ≥ 0, X(t) = ( S(t), E(t), I(t), R(t) ) ∈ Ω is the state of process X t , t ≥ 0. By definition, X t , t ≥ 0 will be subject to random jumps at rate: . .} be the sequence of jumps in the system, with τ 0 ≡ 0 and τ m+1 > τ m , ∀m ≥ 0. At time t = τ m , process X t jumps to state X(t + ) according to the following probability distribution: where X(t + ) is the state of the system just after time t. The first expression on the right-hand side of Eq. (5) is the probability of a new contagion; the second represents the manifestation of a previously latent infection and the third corresponds to a new removal.

Controlled dynamics
To control the epidemic's spread, assume that on day k ≥ 0 we apply a mitigation 0 ≤ u(k) ≤ 1 to prevent new contagions. The mitigation prevents a proportion u(k) of new transmissions and produces a new controlled rate of contagion: This affects the dynamics of the system. Let U be a discrete set of feasible mitigating actions in the interval [0, 1) and let π = {u(k) ∈ U, k ≥ 0} be a feasible mitigation policy. Then, the epidemic is characterized by a controlled Markov chain X t , t ≥ 0. Under the controlled dynamics, we have: and with k = t representing the number of days elapsed since the epidemic's outset, and w represents a positive time delay.
Let {τ 0 , τ 1 , . . .} be the sequence of jump times of process X t , t ≥ 0. Then, assuming k = t at a given jump time t ≥ 0, we have the controlled probability distribution P π as:

Near-optimal strategies
To control the dynamics in Eq. (10), we need the information of the input rates for the whole disease cycle -the past M σ + M γ days, as in Eq. (7)- (8). For practical purposes, however, this often results in an intractable Markov decision process. For example, if the maximum disease cycle is 7 days, the state of the Markov decision process would be S = R 7 + ∪ Ω, where the seven positive continuous variables represent the transmission-rate history over the last 7 days.
Let us now consider the following hypotheses: Hypothesis 1 is due to the limited number of hospital beds in most countries [30,41], and additional access barriers in some settings [17,3]. It ensures that effective mitigation is enforced when healthcare systems are close to nominal capacity. Effective mitigation imposes a contagion rate that is always surpassed by the removal rate. The recent COVID-19 outbreak demonstrated that healthcare systems around the globe tend to collapse in the early stages of an epidemic with moderate hospitalization rate [38,23]. Hence, the limited healthcare capacity will force early mitigation (Hypothesis 2, see also [41]) and limit the decrease of susceptible population. This motivates Hypothesis 2.
Considering Hypotheses 1 and 2, we introduce a simplified Markov decision process to control the evolution of the exposed and infected populations. The controlled process Z k , k ≥ 0 is discrete and evolves in state space S = {0, 1, . . . , z m }, that represents the total number of infections and expositions at any given period k ≥ 0; z m denotes the maximum allowed number of expositions and infections. Letσ = E(σ) denote the expected values of the latency period and infectious periods and γ = E(γ) be the expected value of the infectious period. For each state-action pair (i, u) in S × U , the one-step transition probabilities p u ij for all j ∈ S are defined as: where R 0 = βSγ and W = R 0 z m σ +γ + z m σ +γ is a normalizing factor, to allow each transition to occur within the same expected time interval. Fig.  1 illustrates the transition probabilities for a fixed u ∈ U . The first expression on the right-hand side of Eq. (11) is the application of Eq. (6), with I(k) =γ σ E(k) and i = E(k) + I(k), subjected to normalization factor W . This approximation is based on Little's law [18], that yields: whereλ is the average rate of contagion. The second expression in the right-hand side of Eq. (11) is the normalized removal rate of the current infected population, equivalent to At each step, the system incurs a cost c : S × U → R + , depending on the current state-action pair (i, u): where 0 <z < z m is a proxy for the target bed occupation level, c 1 , c 2 , c 3 are positive scalar parameters. Let π : S → U be a stationary closed-loop control policy as defined in Section 3.1 and let Π be the set of all feasible stationary closed-loop control policies. The cost-to-go V π of policy π ∈ Π is given by: The objective is to find an optimal policy π * ∈ Π such that: Since the state space is limited and the cost function c(·) is positive and bounded there exists an stationary policy [37] that satisfies Eq. (14). Let V be the space of real-valued functions in S. To find such a policy, we use the classical value iteration Algorithm 1, that has guaranteed convergence under the present settings [37].

Algorithm 1: Value iteration
Input: An arbitrary initial solution V 0 ∈ V and an arbitrary tolerance Output: Optimal solution V * , and optimal stationary policy π * k ← 1;

Experimental Results
We model the experiments for the COVID-19 situation in New Delhi, the capital of India. Considering very low testing in the initial stages of the pandemic and unawareness of the disease, R 0 varies. We take R 0 as given in [26]. Since R 0 varies with time and is higher during initial stages of the pandemic, we conduct the experiments using R 0 = {2.5, 3.5}. The Julia and R language source code is available here.
The parameters of the model are given in Table 1. Table 2 depicts the simulation parameters as per the statistics in https://populationstat.com/india/delhi.
We assume the distribution of the latency and the recovery periods (σ and γ, respectively) to be discrete to cope with the typical daily collection of the data. Tables 3 and 4 show the distributions of the latency and recovery periods, respectively. Table 5 introduces the cost parameters used in the experiments as in Eq. (12).     We conduct the experiments with no control measures for R 0 = 2.5 and R 0 = 3.5, then we introduce control measures that change after every single, 7, 14, and 28 days.
We study the system's behaviour without control measures and for different values of R 0 , for a period of two years from the pandemic's outset. Fig. 2 depicts the output of the stochastic model without control measures for R 0 = 2.5. On the 300 th day, the infected population reaches approx 12% and the exposed population approx 6% of the total population, thereafter the pandemic stabilises. By applying some control measures, we can reduce the peak of expositions and infections. Such measures will also stabilise the pandemic earlier than it would otherwise happen without control measures. We use Fig. 2 as a benchmark to evaluate the effect of the optimal control. On the 300 th day, the infected population reaches approx 12% and the exposed population approx 6%. The pandemic stabilises afterwards. Fig. 3 shows the output of the stochastic model with R 0 = 3.5 and all remaining parameters unaltered from the previous experiment. As R 0 increases, we observe higher peaks for the exposed and infected populations. The former reaches approximately 13%, whereas the latter peaks at 24% of the total population. We conclude that higher values of R 0 lead to an earlier stabilisation of the pandemic as the cases accumulate faster. For both values of R 0 , it is clear that the high level of infection will challenge the healthcare resources and demand mitigating measures when these resources become insufficient. The number of individuals in each SEIR block is shown as a percentage of the total population. On the 180 th day, the infected population reaches approx 24% and the exposed population approx 13%. The pandemic stabilises afterwards.
We consider applying control measures to some extent. We assume in all the test cases that the control measures are not 100% effective in order to facilitate the essential services, accommodate variation in individual behaviours, and limit the effect on the economy. We assume that the maximum achievable control (mitigation) effect is 90%. The experiments can support policy makers by indicating suitable and timely control measures.
We propose Algorithm 1 to find an optimal control strategy for each R 0 and apply this strategy in the original controlled model in Section 3.1 to assess its effect on the epidemic. We assume that the controls are applied only after the sum of infected and exposed individuals reach 0.1%; this is to simulate the delay in the epidemic detection. Below we find the optimal controlled dynamics for different cases. For each experiment, we simulated the stochastic system 100 times and plotted the mean values.
We have the sum of exposed and infected population vs control plot for each R 0 in Fig. 4 and 5, which depict the approximate control. We start the experiments with R 0 = 2.5. Fig. 6 depicts all four population counts with control measures revised daily for Case 1. We observe significant decreases in both the infected and exposed populations. Fig. 7 enhances the result for the E and I populations. The control measures reduce the peak of infections to only 0.12% and decrease the peak of expositions to less than 0.05%. This case is considered to be the baseline, but changing control measures daily may not be feasible in practice.  In Case 2 we revise the control measures after every 7 days. We expect an increase in infections and expositions due to the delay in control change; higher control levels may be needed in the middle of the 7-day interval, but the control will not be revised before the next 7-day period. Fig. 9 shows the SEIR plot for R 0 = 2.5 and control measures changing every 7 days. We observe in Fig. 10 that the maximum infected population remains virtually unchanged, but the peak of the exposed population increases slightly with respect to Case 1. It is noteworthy that the sum of exposed and infected individuals increases slightly if we delay the change in control for 7 days.  11 presents the CI for the infected and exposed populations, as well as the optimal control. We observe spikes in the control trajectory, which are necessary to stabilise the system after a 7day interval in which sub-optimal actions were taken. We need to adjust for overshooting and/or undershooting in the previous interval every 7 days, leading to spikes in the control and population trajectories. It is noteworthy, however, that the control is able to stabilise the system despite the delayed response to change. Fig. 12 shows the output of Case 3 when R 0 = 2.5 with control measures changing every 14 days. As in the previous case, we expect a slight increase in the exposed and infected populations. Fig. 13 shows a slight increase in the exposed and infected populations with respect to Case 2. Fig. 14 shows the CI plot for the exposed and infected populations for Case 3, changing controls every fortnight. We notice the increased fluctuations and spikes due to control corrections performed every other week to compensate for sub-optimal actions in the previous fortnight. We observe that the upper bound for the infected population reached 0.15%, whereas that value remained close to 0.15% in Case 2. After 91 days, the infected population is very low and accordingly control takes a low value. As we can only change the control every fortnight, the infected population increases exponentially. If by the time the control is revised, the infected population has increased too steeply, the control will be high and will lead to a swift decrease in infection. Otherwise, the control level will administer the infection levels more smoothly. As a matter of fact, we will have sufficient time to monitor the effect of the control and this may also decrease the value of epidemic monitoring. This case is very useful in some of the countries where everyday or weekly monitoring is difficult and expensive to perform. The confidence interval chart for the Delhi population shows that we can reduce the infected population with this approach efficiently. Fig. 15 depicts the results for Case 4, when the control measures are revised every 4 weeks. We observe that the delay in policy adjustment causes a significant increase in the peak of infections, which ramp up to approximately 0.18% in contrast to the 0.12% peak with daily control adjustment. The exposed population peaks at about 0.09%. This experiment has a practical interest because most developed countries use similar intervals to review their epidemic mitigation measures whilst monitoring the progress of the disease. In developing countries, however, policy adjustments are often unplanned and monitoring is not systematic. Furthermore, poor communication may hinder policy changes, as the population may be unaware of the current guidelines.  15 illustrates the effect of large control review intervals. Since it will take 28 days to react to fluctuations, we expect a more erratic behaviour. Because the optimal control tends to increase with time in the epidemic outset, a large wait may lead to high levels of infection until the control is revised. With low R 0 this can be problematic, since the optimal control levels tend to be smaller in the outset of the epidemic and may lead to insufficient mitigation before the next adjustment. That will lead to higher control levels in the next adjustment, as illustrated by the steep increase in control observed after four and eight weeks. This situation is similar to that of India, where low control levels were kept without review for a long time, which resulted in a powerful second wave of infections that required much higher control levels. The results illustrate the importance of properly monitoring the outbreak and adjusting the control levels as often as possible to avoid fluctuations and unintended increase in infections, which may lead to the collapse of the healthcare system. Fig. 16 shows the CI for the exposed and infected populations in Case 4. It shows that over a 28−day interval with initially low control levels, the pandemic has spread and reached an upper bound of ≈ 0.25% infected individuals. Hence, we are forced to keep high control levels over a large time interval.

Experiments with
This section covers the set of experiments for R 0 = 3.5. We notice that a large value of R 0 warrants high control levels from the outset, as it is more difficult to control a highly contagious outbreak. A byproduct of this is that the mitigation becomes less sensitive to the control review period. infected population peaks at about 0.11% while the exposed population reaches approx 0.05% of the total population.  19 shows the CI plot for the exposed and infected populations. We have high control levels and a smooth behaviour as a result of changing the control every day.   Fig. 22 shows that the length of the CI increases with respect to Case 1 for both expositions and infections, as a result of the increased fluctuation due to the 7-day delay in control adjustment.
Next we conduct the experiments for Case 3, with R 0 = 3.5 and control measures changing after every 14 days. Fig. 23 and Fig. 24 show a decrease in the peak of infections with respect to Case 2.
Since high levels of control are enforced from the beginning, the number of infections remain limited until the policy is revised in the next period.  25 shows the CI plot and we observe that the 14-day review leads to a large variation in the infected and exposed populations across different realisations of the system. We see that the large review interval leads to increased volatility.
The last set of experiments is for Case 4, with R 0 = 3.5 and control measures changing after every 28 days. Fig. 26 and Fig. 27 show the controlled trajectories. The control sequence curbs the epidemic and maintains reduced infection levels, similar to those in Cases 1 to 3. Since the control levels are high from the beginning, they are kept high for the first 28 days to stabilize the epidemic. The result is a decrease in the variance of the trajectories. It suggests that high control strategies are robust and can be used to ensure a decreasing epidemic when the control review period is large. Indeed, we can see that the trajectories with 14-day and 28-day review periods are rather similar and this is due to the high levels of control from the outset of the epidemic. This underscores the value of continuously monitoring an outbreak and revising the control as frequently as possible.
Finally, it is worth highlighting that, although large review periods do not compromise the ability to stabilise the epidemic, they force the decision maker to apply higher levels of control to conquer the outbreak. Indeed, by comparing the control levels for Cases 1 to 4, we can clearly notice an increase in the levels of control.

Conclusions
This paper explored a parsimonious stochastic model to describe the evolution of an epidemic by means of the classical SEIR framework. Based on queues with infinite service capacity, the model is tractable and seamlessly incorporates general latency and infectious periods. It introduces a stochastic optimal control formulation that seeks an optimal trade-off between the occupation of the healthcare system and the economic impacts of mitigation measures. Since the model is based on parsimonious stochastic formulations, it is tractable, easy to use and guaranteed to converge.
We tested the formulation on the epidemic data from New Delhi, including experiments with R 0 = 2.5 and R 0 = 3.5 to account for data uncertainty. The results show that an optimal control policy acts swiftly in the first wave of the epidemic, therefore avoiding successive waves as observed in countries where control was delayed or relaxed too soon. They also highlight that continuous monitoring is vital, as larger policy review periods imply higher levels of control to curb the epidemic. It is apparent that failing to act swiftly in the first wave is more costly, since myopically sacrificing the healthcare system to preserve the economy in the outset of the epidemic will require higher levels of control to contain a larger outbreak. Furthermore, failing to act swiftly may lead to the collapse of the healthcare system, thereby incurring additional economic and societal costs.