In this section, we present numerical contributions by which we shed light on the value of information and optimization in infectious disease control with uncertain resources. The conclusions drawn are based on a validated simulation model of the 2014 Ebola epidemic in Sierra Leone and the results show that improvement regarding patient death is significant. Recall that in order to quantify the value of information in this context, we consider three settings: \begin{enumerate*}[label=\arabic*.]
\item The decision maker has little available data (20 days in our implementation) about the state of the epidemic and has a probability distribution over uncertainty regarding future budget. She creates and validates the simulation and solves the optimization problem, the solution of which provides first and second stage decisions. She then applies these solutions in practice.
\item The decision maker follows the same steps as in Setting 1, except that in the ``middle" of the epidemic when the extra resources become available (60 days in our implementation), she re-optimizes and finds possibly new second stage decisions for implementation. Note that the first stage decisions are irrevocable.
\item The decision maker retrospectively analyzes the epidemic and allocation decisions in the first and second stages. That is, the data is available for the whole period and the goal is to find the best decisions that could have been taken conditioned on the observed epidemic trajectories. Note that this approach is common in data-driven analysis because most analysis is post-implementation, i.e., a dataset becomes available from past behavior of a system and an analysis is run over the data set to draw conclusions. Our approach goes one step further and considers an online process for this application.
\end{enumerate*}

For each of these three settings we report the results of a series of analysis. First, we report the results of simulation calibration/validation. We emphasize that the validation step is important because conclusions are based on the results of the simulation. Second, we describe the procedure to estimate the coefficients of logistic curves for stages one and two and report quantities of interest to validate their quality. Third, using the simulation model, we report the number of infections that could have been averted if our proposed approach had been used. We also report the bed allocation for each setting and conduct sensitivity analysis over the initial budget.  All of the experiments are run on a high performance cluster computer setup in an environment with 2.4 GHz Intel Xeon E5 processor with 20 cores, and 62GB RAM, with Python 3.7.

\subsection{Validation of the Simulation Model}\label{validation}
Since we use the simulation model in three settings as described above, we calibrate and validate it for each setting separately. Data from the 2014 Ebola epidemic in Sierra Leone with 11 districts (two districts are omitted because of the low number of cases in the first 20 days) are used to calibrate the simulation model with  $80\%$ of the available data for training and the rest for validation purposes. Table \ref{calibration_validation} shows the average percent difference between the simulated and real data for calibration and validation periods in each setting in Sierra Leone at the national level. Our validation results show good fit in each setting. Figure \ref{calibration_figures} shows the corresponding time series for cumulative number of infected individuals for each setting. The simulation model reasonably follows the trajectories of the epidemic in all of our settings. Therefore, we can use these simulated curves for our curve fitting process. We report the full results from our validated simulation model for every district in Appendix \ref{districts_validated}.


\begin{table}[H]
	\centering
	\caption{Percent difference between the real and simulated cumulative number of infected individuals.}
	\vspace{0.2cm}
	\large
	{\begin{tabular}{|c|c|c|}
			\hline
			& Calibration & Validation \\ \hline
			Setting 1& $9\%$ & $6\%$ \\ \hline
			Setting 2 & $16\%$ & $8\%$ \\ \hline
			Setting 3 & $11\%$ & $-4\%$ \\ \hline
	\end{tabular}}
	\label{calibration_validation}
\end{table}


\begin{figure}[H]
	\centering
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.7]{calibration_20}
		\caption{Setting 1}
	\end{subfigure}
\end{figure}

\begin{figure}[H]\ContinuedFloat
	\centering
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.7]{calibration_60}
		\caption{Setting 2}
	\end{subfigure}
	\\
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.7]{calibration_150}
		\caption{Setting 3}
	\end{subfigure}
	\caption{Calibration and validation of the simulation model with respect to different data availability.} \label{calibration_figures}
\end{figure}

\vspace{-4mm}
\subsection{Functional Estimations for Cumulative Number of Infected Individuals}
In the previous section, we reported the validation results for the simulation model. In this section, though, we report the validation results for fitting the logistic curves in the first and second stages for all three settings. In particular, we report the percent difference between $I(\tau)$ and $I(T)$, which comes from the fitted curve, with the corresponding cumulative number of infected individuals at times $\tau$ and $T$ in simulated data. We use Python's Scipy library to find optimal parameters for the logistic curves. Table \ref{s_shape_errors} reports the percent difference between the simulated and fitted values for different settings. We sum up the cumulative number of infected individuals in every district to obtain the values for Sierra Leone. Overall, we obtain good fits for the values of cumulative number of infected individuals. 
 

\begin{table}[H]
	\centering
	\caption{Percent difference between the simulated and fitted cumulative number of infected individuals at $\tau = 60$ and $T=188$.}
	\vspace{0.2cm}
	\large
	{\begin{tabular}{|c|c|c|c|}
			\hline
			Cumulative Infected & Setting 1 & Setting 2 & Setting 3 \\ \hline
			$I(\tau)$ & $22.56\%$ & $-3.98\%$ & $5.49\%$ \\ \hline
			$I(T)$ & $2.78\%$ & $13.72\%$ & $-0.37\%$ \\ \hline
	\end{tabular}}
	\label{s_shape_errors}
\end{table}

Recall that we estimate coefficients $a_n^1$, $b_n^1$, $a_n^2$ and $b_n^2$ in Equation \eqref{logistic_curves} for each population first for each stage. Then, in order to include the effect of initial infected population and resources allocated, we construct a regression for $K_n^1$ and $K_n^2$. To estimate the coefficients of these regressions, we generate data points by generating random initial states and resources. Specifically, the initial infected number of individuals are generated from the initial infected number of individuals observed from real data plus a normally distributed noise with mean zero and standard deviation 50 to obtain a wide range of values. Then, we find the initial number of susceptible individuals by subtracting the number of infected individuals from the total population in every district. We also generate random first and second stage capacity vectors for every initial state by using the real capacity vectors. We add a normally distributed random noise term to the total capacity with mean zero and standard deviation 50. Then we order the districts randomly and allocate the total number of beds by a uniform distribution between zero and the remaining capacity. The remaining capacity is found by subtracting random number of beds from the total available beds until all the districts are covered. The regression coefficients for all districts are found using lasso regression from the Scikit-learn library in Python. The average $R^2$ values for the two regression models are given in Table \ref{r_squared}. Our results yield high quality approximations for the cumulative number of infected individuals which we then use in the TSSP model. We provide the detailed analysis for the curve fit and regression model results in Appendix \ref{data_analysis}.


\begin{table}[H]
	\centering
	\caption{$R^2$ values for the regression models for Stages 1 and 2 for all settings.}
	\vspace{0.2cm}
	\large
	{\begin{tabular}{|c|c|c|c|}
			\hline
			& Setting 1 & Setting 2 & Setting 3 \\ \hline
			Stage 1 & 0.93 & 0.91 & 0.92 \\ \hline
			Stage 2 & 0.88 & 0.87 & 0.89 \\ \hline
	\end{tabular}}
	\label{r_squared}
\end{table}

\subsection{Comparison of Policies}
The goal of this section is to compare the results of policies produced by our optimization framework with that of what actually happened during the 2014 Ebola outbreak in Sierra Leone. The results can be interpreted as improvement that could have been achieved by implementing the proposed framework. Of course, these interpretations are based on the assumptions considered in this study. To that end, how we estimate the parameters of the model from available data from 2014 Ebola outbreak in Sierra Leone is explained next.

There are 1544 beds in ETUs across Sierra Leone \citep{EbolaBeds}. We assume that the 634 beds with an $operating$ status are available in the first stage, and the remaining 910 beds with a $pending$ status become available in the second stage. To estimate the opening costs we use data from \cite{Johnson2016} for the Western Area, and calculate the total construction cost per individual. Then, we use this information to calculate the ETU opening costs for the other districts proportional to their total population. The operating costs include medical and transportation costs. The medical costs are estimated by \cite{Bartsch2015}. They include treatment and personal protective equipment costs per patient in Sierra Leone for the 2014 Ebola epidemic. The essential items needed in ETUs are delivered from depots in Bombali, Kenema, and Port Loko, which constutite the transportation costs. We assume that every district is served by its closest depot, and then we calculate distance based costs considering the fuel prices. We used the fuel prices from Sierra Leone in 2015 \citep{fuel_prices}. Using this cost structure, the cost of bed allocations in Sierra Leone correspond to $\$505,000$ in the first stage and $\$470,000$ in the second stage. We consider three budgetary scenarios for the second stage: low, medium, and high. The $\$470K$ available for the second stage is used as the ``medium" value. The ``low" and ``high" values are set to $\$235K$ ($50\%$ less) and $\$705K$ ($50\%$ more), respectively. The probabilities of having low, medium, and high budgets are set to $0.25$, $0.5$, and $0.25$, respectively. The integrated TSSP model is solved using Gurobi 8.0. 

Table \ref{compare_results} reports the improvement due to optimized decisions compared to real data in terms of the cumulative number of infected individuals in each of the three settings. The following observations are in order. First, even with 20 days of data, our approach can lead to improvements of about $1.5\%$, which translates to averting $220$ Ebola cases. Second, if the decision maker uses the updated data when the extra budget is revealed and re-optimizes bed allocation decisions for the second stage, then the improvement can be almost doubled (i.e., 206 more cases can be averted). This suggests that value of information in a data-driven approach for infectious disease control is significant. Finally, a retrospective analysis shows that up to $5\%$ improvement is possible by following different allocations, which is the focus of our next discussion. 
Note that the improvements are calculated for a period of 188 days and one may expect larger improvements over longer periods.
In a data-driven setting, one may interpret the $5\%$ improvement as the value of perfect information. A similar conclusion can be drawn when comparing the $5\%$ improvement in Setting 3 with $1.5\%$ improvement in Setting 1 where we can quantify the value of information on the cumulative number of infected individuals.


\begin{table}[H]
	\centering
	\caption{Percent improvement due to optimization.}
	\vspace{0.2cm}
	\large
	{\begin{tabular}{|c|c|c|}
			\hline
			Setting 1 & Setting 2 & Setting 3 \\ \hline
			$1.49\%$ & $2.89\%$ & $4.37\%$ \\ \hline
	\end{tabular}}
	\label{compare_results}
\end{table} 

The proposed optimization approach does lead to improvements in terms of reducing the number of infected individuals, but it is also interesting to analyze the decisions made by the proposed optimization and compare them to the actual decisions made during the 2014 Ebola epidemic. Figure \ref{allocation_settings} shows the bed allocations obtained by the optimization approach for the three settings. The lower and upper parts of the bars show the first and second stage bed allocations for every district, respectively. Recall that, the first stage decisions are the same for Settings 1 and 2, whereas the second stage decisions for Setting 2 are updated using new information. As can be seen in Figure \ref{allocation_settings}, under Setting 2 the resource allocation is shifted from Pujehun to Bombali. Initially, Bombali has low number of infected individuals. However, the epidemic grows fast in Bombali in the later time periods. Therefore, with the new information available, the resources are shifted to address this change in trends. In Setting 3, the resources are allocated in some districts that do not even have treatment units opened in Settings 1 and 2. In addition, in Setting 3, more resources are allocated to Western Area. As a result of this change in resource allocation, Setting 3 results in lower cumulative number of infected individuals in Sierra Leone. This observation suggests that the epidemic control strategies should prioritize more populated areas, while covering other districts with lower levels of resources.


\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{allocation_settings}
	\caption{First and second stage bed allocations obtained by settings 1, 2, and 3.}
	\label{allocation_settings}
\end{figure}

Figure \ref{allocation_real} compares the allocation strategies obtained by perfect information in Setting 3 to the real bed allocations. Similar to Figure \ref{allocation_settings}, the lower and upper parts of the bars correspond to first and second stage allocations, respectively. In both cases, Western Area receives the most allocation, but our optimization model allocates more to this district. Especially, in the second stage, when the growth of the epidemic in Western Area is faster than the other districts, our model takes appropriate precautions, and this results in a lower cumulative number of infected individuals by $11\%$ in Western Area compared to the cumulative number of infected individuals obtained by the real bed allocations. In Kono, Moyamba, and Port Loko, our model assigns the beds in the first stage as opposed to real bed allocations where more resources are allocated in the second stage. In Kono and Port Loko, our allocation results in a higher cumulative number of infected individuals. However, in Moyamba by allocating more resources, $128\%$ improvement in the cumulative number of infected individuals is achieved. In the real setting, the first stage budget is used in opening more treatment units in more districts. On the other hand, in our optimization setting, the budget is used in allocating more beds with fewer number of ETUs. 


\begin{figure}[H]
	\centering
	\includegraphics[width=\textwidth]{allocation_real}
	\caption{First and second stage bed allocations obtained by Setting 3 and real bed allocations.}
	\label{allocation_real}
\end{figure}

\subsection{Sensitivity Analysis}\label{sensitivity}

We conduct a sensitivity analysis by changing the first stage budgets using Setting 3 with perfect information in our simulation model. In this analysis, we keep the set of second stage budget the same. The first stage budgets used are $\$250,000$, $\$375,000$, $505,000$, and $\$625,000$. Figure \ref{scenario_budget} shows the difference in the cumulative number of infected individuals with respect to different budgets.


\begin{figure}[H]
	\centering
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.85]{low_scenario}
		\caption{Low level extra budget.}
	\end{subfigure}
	\\
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.85]{med_scenario}
		\caption{Medium level extra budget.}
	\end{subfigure}
\end{figure}

\begin{figure}[H]\ContinuedFloat
	\centering
	\begin{subfigure}{\textwidth}
		\centering
		\includegraphics[scale=0.85]{high_scenario}
		\caption{High level extra budget.}
	\end{subfigure}
	\caption{Cumulative number of infected individuals in Sierra Leone with respect to different first stage budgets.}
	\label{scenario_budget}
\end{figure}


As expected, when we increase the first stage budget, the number of cumulative infected individuals is decreased at time $T=188$ for every second stage extra budget level. However, the improvement is not linear. This is because after a critical level of resources is achieved, the infection progression slows down, and the additional resources have only marginal effect as the need for these resources has diminished. Therefore, depending on the first stage budgets, decision makers might choose to allocate some of these resources to other activities.
{"config": "arxiv", "file": "2006.01743/results.tex"}