{'fraction_non_alphanumeric': 0.03487651588917412, 'fraction_numerical': 0.010356731875719217, 'mean_word_length': 5.0579088471849865, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bayesian deep neural networks for low-cost neurophysiological markers of Alzheimer's disease severity

As societies around the world are ageing, the number of Alzheimer's disease (AD) patients is rapidly increasing. To date, no low-cost, non-invasive biomarkers have been established to advance the objectivization of AD diagnosis and progression assessment. Here, we utilize Bayesian neural networks to develop a multivariate predictor for AD severity using a wide range of quantitative EEG (QEEG) markers. The Bayesian treatment of neural networks both automatically controls model complexity and provides a predictive distribution over the target function, giving uncertainty bounds for our regression task. It is therefore well suited to clinical neuroscience, where data sets are typically sparse and practitioners require a precise assessment of the predictive uncertainty. We use data of one of the largest prospective AD EEG trials ever conducted to demonstrate the potential of Bayesian deep learning in this domain, while comparing two distinct Bayesian neural network approaches, i.e., Monte Carlo dropout and Hamiltonian Monte Carlo.


Introduction
Alzheimer's disease (AD) is the most common form of dementia and highly prevalent among elderly individuals. As societies around the world are ageing, the number of patients affected is rapidly increasing. AD is associated with an enormous disease burden regarding morbidity, mortality, and financial expenses. It impairs basic bodily functions such as walking and swallowing, has devastating impact on memory and cognition, and is ultimately fatal. The costs of care for individuals with AD or other dementias are already enormous [1], dementia being one of the costliest conditions to society [2].
Since AD progresses over time, early accurate diagnosis and precise clinical monitoring would be essential. Yet, in daily clinical routine, AD assessment is usually done by subjective clinical interpretations once there is a strong prior that the disease is already at a progressed stage.
To date, no low-cost, non-invasive biomarkers have been established to advance the objectivization of diagnosis and disease progression assessment. For a wide use, such markers should not rely on expensive equipment (e.g., scanners for magnetic resonance imaging (MRI), or positron emission tomography (PET)), or require invasive procedures (injection of a positron emitter (PET), or lumbar puncture (analysis of cerebrospinal fluid)). Properties like non-invasiveness, cost-effectiveness and a wide availability make electroencephalography (EEG) a potential candidate modality for such markers. Various studies have shown associations between quantitative EEG (QEEG) markers of slowing, complexity, and functional connectivity and AD (for reviews, see [3,4]).
Here, we utilize Bayesian neural networks [5] to develop a multivariate predictor for AD severity, as measured by the Mini-Mental State Examination (MMSE). We use 820 distinct QEEG markers as features, all belonging to the aforementioned categories of slowing, complexity, and functional connectivity.
Bayesian techniques allow for an adjustment of predictions through varying priors. Therefore potential variations in diagnostic settings (distinct locations) or distinct disease prevalence (varying age groups) can be accounted for. Furthermore, Bayesian approaches provide probabilistic predictions quantifying predictive uncertainty -which is of utmost importance for medical practitioners. Finally, the Bayesian treatment of neural networks prevents over-fitting to sparse data sets as typically found in clinical neuroscience.
To demonstrate the potential of Bayesian deep learning techniques, we investigate the performance of two different Bayesian approaches, i.e., (1) Monte Carlo dropout [6] (MC dropout BNN) and (2) Hamiltonian Monte Carlo [7] (HMC BNN), and compare their results to traditional (non-Bayesian) networks (NN). All models use identical QEEG features from one of the largest prospective AD EEG trials ever conducted.

Experimental data
One hundred and eighty-eight AD patients (133 with probable, 55 with possible AD diagnosis according to NINCDS-ADRDA criteria; 100 females; mean age 74.86 ±8.06 standard deviation (SD); mean MMSE score 22.84 ±3.70 SD; mean years of education 11.14 ±3.04 SD; mean duration of illness (months) 26.88 ±24.11 SD ; 115 (61.17%) with anti-dementia medication) were considered for this investigation. They were recruited prospectively at the tertiary-referral memory clinics of the neurological departments at the Medical Universities of Graz, Innsbruck and Vienna, as well as Linz General Hospital as part of the PRODEM (Prospective Dementia Database Austria) cohort study. Assessment of disease severity was done using the MMSE [8]. Continuous EEG (alpha trace EEG recorder, 10-20 electrode placement) was analyzed for an eyes-closed (180 sec) and an eyesopen resting condition (180 sec). For details on the entire PRODEM experimental protocol and preprocessing pipeline, see [9][10][11].
Features were normalized and projected into a lower-dimensional space (five components) using principal components analysis, before passing the data to the models.

Bayesian neural networks
In this paper we use Bayesian neural networks (BNNs) as our model for regression [13,5]. These are neural networks, where we define the prior p(ω l ) for each layer l ∈ L as a product of multivariate normal distributions L l=1 N (0, I/λ l ) (where λ l is the prior length-scale) and the Gaussian likelihood p(y | ω, x) as N (f ω (x), σ 2 ), where σ 2 is the noise variance. Assigning probability distributions over the weights leads to a function approximation f ω (x) that is also a distribution. Therefore, we are also provided with important uncertainty estimates of our predictions.
In the presence of data {X, Y}, we can perform inference to get the posterior distribution over the weights, which requires the normalizing distribution, otherwise known as the marginal likelihood, The inferred posterior distribution is then used to form the predictive distribution over a new test point x * , The integrations in Equations (2) and (3) are analytically intractable, which is the bottleneck in BNNs. To perform these integrations there are two common solutions. One solution is to replace the true posterior with a variational approximation that is either cheaper to sample from [14] or conjugate to the likelihood [13]. The other option is to use a Markov Chain Monte Carlo technique such as Hamiltonian Monte Carlo [7] to sample from the posterior.
In this work, we make a direct comparison between two BNN implementations: 1. MC dropout [6], which is a variational approximation.

Hamiltonian Monte Carlo [7], which is a Markov Chain Monte Carlo technique that utilises
Hamiltonian dynamics to explore the parameter space of networks.
In addition, we also implement a standard (non-Bayesian) neural network as a useful baseline.
MC dropout is a variational approximation which implements dropout at test time of a neural network, as well as during training. A dropout mask is drawn from Bernoulli distributed random variables and sets a proportion of the weights in the network to zero. Dropping these weights at test time gives an approximation to the predictive distribution by sampling T new dropout masks for each forward-pass through the network. This results in T samples that represent the predictive distribution.
Hamiltonian Monte Carlo can be used to infer the predictive distribution in Equation (3) by introducing a Hamiltonian, that defines the total energy. This introduces an additional momentum variable p, which is used to form the kinetic energy K(p). However, our interest lies in the potential energy E(ω) as Given this Hamiltonian, we define a distribution over the phase space p(ω, p) ∝ exp (−H(ω, p)). Therefore in sampling from this distribution and ignoring the momentum parameters, we can retrieve the posterior distribution over ω, by using the Hamiltonian dynamics to sample from the phase space. For further details, we refer to Neal [5].

Metrics for comparison
To compare results across the model we use both the mean squared error (MSE) and the standardized mean squared error (SMSE) [15], where for N test points, we define The SMSE is the MSE normalised by the predictive variance σ 2 i and we also define the predictive mean for a given data point x i asf ω (x i ). This is an appropriate metric for regression tasks, where we are also interested in the model predictive variance.

Nested cross-validation and grid search
A two-level nested cross-validation (5 × 5) was used to determine generalization performance. We included a nested loop to perform a grid search to select the hyperparameter settings with minimal expected generalization error. For the HMC BNN, we optimized for the prior weight precision (see [5] for an intuition behind the weight precision) and for the MC dropout network, we optimized for dropout, which is directly related to the weight precision (see [14] for details of the relationship). We also used the nested loop to optimize for the dropout rate and early stopping for the standard NN. In addition to selecting these hyperparameters in our grid search, we also searched through a range of small neural network architectures, with the smallest having a hidden layer of 100 units to the largest having two hidden layers of 300 units.

Results and Discussion
For each of the five outer-fold test sets, we first selected the optimal hyperparameters for each type of network (HMC BNN, MC dropout BNN and NN) based on the performance in the five inner folds of the nested cross-validation loop. Then, we used the corresponding settings to compute the results in the outer folds, as reported in Table 1.
Overall, HMC BNNs significantly (p < 0.05) outperformed MC dropout BNNs and standard NNs. Differences between model performances were assessed by statistically comparing squared errors of test set predictions.
All Bayesian models selected for the outer folds had two hidden layers of 100 units. The BNNs showed an average lower MSE than the best standard NN across the test sets. This highlights the importance of combining the function approximating power of neural networks with Bayesian techniques. This is especially true for sparse data sets. In the present study, the data set consisted of 188 patients, which allowed us to directly compute the derivatives in the Hamiltonian. As such, we saw the HMC BNN outperforming the MC dropout BNN, which is often seen in practice [16]. The comparably high SMSE values for the MC dropout model is due to the underestimation of the predictive uncertainty. This is a common limitation of variational inference techniques as the approximate posterior is optimised through an objective that is mode-seeking and can fail in modelling probability mass far away from the mode. In larger data sets, approximate inference (such as MC dropout) is currently the most practical way to implement BNNs.
Overall, we have shown how BNNs can be used to facilitate the development of low-cost, noninvasive markers of AD severity. To the best of our knowledge, this is the first article reporting a BNN approach for building QEEG markers of AD severity.