{'fraction_non_alphanumeric': 0.03579032165256919, 'fraction_numerical': 0.016801887335289718, 'mean_word_length': 5.106816584680253, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploring linguistic feature and model combination for speech recognition based automatic AD detection

Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and delay progression. Speech based automatic AD screening systems provide a non-intrusive and more scalable alternative to other clinical screening techniques. Scarcity of such specialist data leads to uncertainty in both model selection and feature learning when developing such systems. To this end, this paper investigates the use of feature and model combination approaches to improve the robustness of domain fine-tuning of BERT and Roberta pre-trained text encoders on limited data, before the resulting embedding features being fed into an ensemble of backend classifiers to produce the final AD detection decision via majority voting. Experiments conducted on the ADReSS20 Challenge dataset suggest consistent performance improvements were obtained using model and feature combination in system development. State-of-the-art AD detection accuracies of 91.67 percent and 93.75 percent were obtained using manual and ASR speech transcripts respectively on the ADReSS20 test set consisting of 48 elderly speakers.


Introduction
Alzheimer's disease (AD), the most common form of dementia often found in aged people, is characterized by progressive degradation of the memory, cognition, and motor skills, and consequently decline in the speech and language skills of patients [1,2]. Currently, there is no effective cure for AD [3], but a timely intervention approach can delay its progression and reduce the negative physical and mental impact on patients [4]. To this end, speech-based automatic AD diagnosis has advantages of being non-intrusive, more scalable and less costly, when compared with conventional approaches based on clinical tests. In recent years there has been a burgeoning interest in developing such systems, notably represented by the recent ADReSS challenges [5,6].
One major challenge confronting the development of automatic speech-based AD detection systems is the scarcity of the required specialist data due to the difficulty in collecting large quantities of transcribed AD assessment speech recordings and shortage of clinical professionals. The resulting "small data" learning task leads to model uncertainty encountered in multiple stages of system development. These include, but not limited to, the following considered in this paper: a) cross-domain transfer text embedding DNNs, for example, state-of-the-art BERT [33] and Roberta [34] pre-trained models, by parameter fine-tuning on limited in-domain AD detection data, leading to over-confident parameters; b) possible errors in the transcripts produced by ASR systems; and c) the suitable form of back-end classifier design and parameter estimation.
To this end, feature and ensemble approaches are adopted in this paper as overarching solutions to address the above model uncertainty issues. First, the robustness of domain fine-tuning of BERT and Roberta pre-trained text encoders on limited AD detection speech transcripts is improved using model averaging. Second, the uncertainty over the suitable form of text embedding features is accounted for using both BERT and Roberta text embedding representations for back-end classifier construction. Third, the uncertainty over ASR transcript's quality is further handled using two-pass decoding based speech recognition system combination, or later fusion of their respective AD detection outputs. Lastly, the uncertainty over the back-end classifier is addressed using decision voting among an ensemble of classifiers.
Experiments conducted on the ADReSS20 Challenge dataset suggest consistent performance improvements were obtained using model and feature combination in system development. State-of-the-art AD detection accuracies of 91.67% and 93.75% were obtained using manual and state-of-the-art hybrid CNN-TDNN ASR system produced speech transcripts respectively on the ADReSS20 test set consisting of 48 elderly speakers.
The main contributions of this paper are summarized below. First, this paper presents a systematic application of feature and model combination approaches to address the inherent data sparsity and model uncertainty issues in speech recognition based AD detection systems. In contrast, prior research primarily investigated model combination over back-end classifiers [8,10,19,21,27,32], while its application to cross-domain fine-tuning of pre-trained text embedding and ASR system outputs fusion remains limited. Second, this paper presents a state-of-the-art AD detection accuracy of 91.67% obtained using manual transcripts and best published accuracy of 93.75% using ASR speech transcripts respectively on the ADReSS20 test set. To the best of our knowledge, the closest published accuracy performance to date on the same task are 91.67% [32] and 87.50% [22] using manual and ASR speech transcripts, respectively.
The rest of this paper is organized as follows. Section 2.1 introduces the AD detection task and the ADReSS20 [5] dataset used in this work. Section 2.2 briefly describes the hybrid CNN-TDNN and end-to-end Conformer ASR systems, BERT and Roberta text embedding models, and back-end classifiers. Section 3 suggests an approach to fine-tune the pre-trained text encoders. Section 4 presents AD detection systems with feature combination and classifier combination methods on manual transcripts, followed by Section 5 showing AD detection with those methods using ASR system outputs. Finally, the conclusions are drawn, and future works are discussed in Section 6.

Task Description
This subsection describes the audio and text data used in this paper and the baseline system structure.

Dataset
In this paper, the ADReSS20 Challenge dataset [5] is used for the AD detection system training and evaluation. It is selected from the Cookie Theft picture description part of the Pitt Corpus in the DementiaBank database [35] with balanced age, gender and AD label distribution among subjects. The data consists of speech recordings, corresponding manual transcripts annotated by the CHAT coding system [36] and binary AD labels. Both the participant and the interviewer parts are provided. It is divided into the training set of 2 hours 9 minutes from 108 subjects and the test set of 1 hour 6 minutes from 48 subjects.

Baseline System
A baseline tandem BERT text embedding and SVM classifier based system structure was used [22] to produce AD detection results with either ground truth manual transcripts or ASR system generated transcripts. ASR Systems: The hybrid CNN-TDNN system is based on a recently proposed lattice-free MMI trained factored TDNN system [37] with additional 2-dimensional convolutional layers being employed as the first 6 layers, and is further optimized with data re-segmentation, domain adaptation, and Bayesian LHUC speaker adaptation [38]. The end-to-end ASR system used is a Conformer specially developed for elderly speech by training with augmented elderly speech data, incorporating manual re-configuration of Conformer hyper-parameters and crossdomain adaptation. Two-pass decoding based system combination ASR outputs involving the CNN-TDNN ASR system to produce initial N-best recognition hypotheses before being rescored using the Conformer system using a 2-way crosssystem score interpolation [39,40] are also considered. BERT and Roberta text embedding: we combine all participant speech segments corresponding to a single picture description for extracting text features. The principal 768-dimensions of the last hidden layer BERT or Roberta model output are taken as the single vector representation for each participant's AD classification. This is then normalized by removing mean and scaling to unit variance before being passed into the classifiers. The BERT pre-trained model bert − base − uncased is based on the open-source model library 1 , with 12 transformer Figure 1: Example of (A) an ensemble of 3 fine-tuned BERT/Roberta pre-trained models on the ADReSS training data obtained at different update epochs (top, left) and being used to produce separate text embedding features for SVM based AD detection outputs majority voting (in green/yellow/red, bottom, left); and (B) AD detection system pipeline using manual, or one of three ASR systems (CNN-TDNN, Conformer, or their combination via two-pass recognition [39]) produced transcripts (green) for text embedding features extraction (blue) and back-end classification (orange). Feature combination feeds fine-tuned BERT and Roberta text embedding of (A) into a single back-end SVM classifier for outputs voting (top right). Classifiers combination feeds fine-tuned BERT or Roberta text features into five classifiers for outputs voting (bottom right). Table 1: AD detection performance evaluated by 10-fold CV on the ADReSS training set and test set using BERT/Roberta as the text feature extractor w/o model combination based fine-tuning of Fig. 1(A) using SVM classifier. All BERT/Roberta models fine-tuned on ground truth transcript (GT). All back-end classifiers trained and evaluated on ground truth transcript (GT). Sys.3-7 and Sys.9 show the effects of various update epochs selection schemes for model combination in Section 3. Random result (Sys.6) is averaged from 10 random selections. Finetuning without merging result (Sys.2) is averaged from performances of the selected single snapshot models. are used in all the following experiments of this paper.

Feature and Classifier Combination
This section investigates the feature combination of two text encoders (BERT and Roberta), and the back-end classifiers combination, with the manual transcriptions as inputs.

Feature Combination
The two types of fine-tuned text embedding DNNs, BERT and Roberta, can capture linguistic information with discrepancy. To further reduce the uncertainty underlying the resulting text embedding features, we exploited a late feature fusion via majority voting over decisions made by a single type of classifier based on either BERT and Roberta features respectively (Feature decision voting, Table 2), which is shown in Figure 1 (B, top right). The effectiveness of feature decision voting with different types of back-end classifiers is examined. We also explored alternative methods of fusing the two text embedding features by concatenation before feeding them into SVMs. Two trends are observed from the manual transcripts based results in Table 2 (column 5-6). First, the method of feature fusion by decision-level majority voting produced improvements using all 5 back-end classifiers on the CV data over the comparable baselines using either BERT or Roberta features alone (Sys.13-17 vs. Sys.1-5 & Sys.7-11). On the test set, similar improvements were obtained for SVM and LDA classifiers (Sys.13 vs. Sys.1&7, Sys.14 vs. Sys.2&8). Second, the decision-level fusion by majority voting performed better than early fusion by concatenation (Sys.13 vs. Sys.19, col.5-6, Table 2). Lastly, decision-level fusion among BERT and Roberta feature based SVM classifiers produced the best feature combination accuracy of 87.96% on the CV data, and 87.50% on the test (Sys.13 vs. 14-17, col.5-6, Table 2).

Classifiers Combination
Apart from the feature extractor stage, uncertainty could also manifest in architecture design and parameter estimation of back-end AD detection models on limited AD detection training data. To this end, we conducted classifier combination with majority voting over the binary AD detection decisions by a range of BERT (or Roberta) feature based back-end classifiers, as illustrated in Figure 1 (grey, bottom right). All possible feature-classifier combinations involving both the full set 2 × 5 combination and its various subsets were examined to maximize the diversity and complementarity among systems. The best performing feature-classifier combination configuration involves fine-tuned (with snapshot level model voting of Section 3) BERT or Roberta features trained LDA, GP and MLP classifiers (Sys. 18, col.5-6, Table 2).
The hyperparameters in the back-end classifiers are optimized by greedy search on CV, as mentioned in Section 2.2. More specifically, we used the SVM setting as in the baseline, LDA solved by SVD, GP with kernel 4.0 * RBF (5.0), XGB based on tree models with the number of boosting round set as 16, the maximum tree depth for base learners set as 2, learning rate η = 0.4, minimum loss reduction required to make a partition on the tree γ = 0.1 , and MLP with a single hidden layer of 256-dimension, L2 penalty parameter α = 1e − 5, optimized by limited memory BFGS. All other hyper-parameters were kept as default settings.
As shown in Table 2, the first general trend is that classifier combination system consistently surpasses, or at least matches, the performance of any single classifier system across the CV and test sets (Sys.6 vs. Sys.1-5, Sys.12 vs. Sys.7-11, col.5-6, Table 2). Secondly, conducting classifier combination on the top of feature combination produced no further improvements over pure feature combination (Sys.18 vs. Sys.13, col.5-6, Table 2). This suggests further cross-system diversity and complementarity need to be injected into individual back-end classifiers' design and training when feature combination is used.

ASR Output Based AD Detection
In this Section we apply the feature combination and classifier combination methods of Section 4 to AD detection using ASR system generated transcripts. Table 2: Accuracy of AD detection using different back-end classifiers with or without feature and/or model combination on manual transcripts, or one of three ASR systems (CNN-TDNN, Conformer, or their two-pass decoding combination [39] "CNN-TDNN − → Conformer") produced transcripts. "CNN-TDNN + Conformer" refers to separate AD detection using each ASR system's specific transcripts before final decision voting combination. "LDA" stands for linear discriminant analysis, "GP" for Gaussian Process, "MLP" for multilayer perceptron, and "XGB" for extreme gradient boosting.

AD Detection Using Single ASR System's Output
The AD detection performance using various feature and classifier combination configurations for a single ASR system produced transcripts are shown in  Table 2). In particular, using the CNN-TDNN ASR system produced speech transcripts, this decision-level feature combination based SVM system produced the best published speech recognition outputs based AD detection accuracy of 93.75% on the benchmark ADReSS test set.

AD Detection Using Multiple ASR Systems' Outputs
Compared with ASR performance measured on non-aged, healthy speech [41,42], significantly higher speech recognition error rates are often found on elderly speech data [2,15,22,23,31,[43][44][45]. In order to mitigate the impact of possible ASR transcript errors on the downstream AD detection task, ASR system combination approaches are also investigated in this paper to account for the uncertainty over of the quality of speech recognition system outputs produced by the hybrid CNN-TDNN and end-to-end Conformer ASR systems considered in this paper. In this regard, we studied the efficacy of two approaches to combine the hybrid CNN-TDNN and end-to-end Conformer ASR systems. One is detection based on their twopass decoding combined system produced outputs [39] ("CNN-TDNN − → Conformer", Table 2), while the other performs separate AD detection using either of the two ASR system's output transcripts before final decision voting combination ("CNN-TDNN+Conformer", Table 2). From results presented in Table 2 (col.9-10), we first confirmed that BERT plus Roberta feature decision voting system with the SVM classifier is again the best performing system among all (Sys.13 vs. Sys.1-12 & 14-19, col.9-10, Table 2), regardless of which ASR combination approach being used. However, using neither of the two ASR system combined outputs led to improvements over using the hybrid CNN-TDNN system transcripts only (Sys. 13, col.9-10 vs. col.7). This suggests further diversity and complementarity may be required in ASR system combination and integration with the downstream AD detection component. The performance of the best performing AD detection system in this paper (sys. 13, Table 2) are further contrasted in Table 3 with those that are obtained on the same ADReSS20 challenge task and report in the recent literature to demonstrate their competitiveness.

Conclusion
The development of a state-of-the-art speech-based AD detection system constructed using the ADReSS20 challenge dataset was presented in this paper. AD detection was performed based on either manual or ASR system transcripts. A series of model ensemble approaches were used in fine-tuning of BERT and Roberta pre-trained text encoders, text embedding feature and back-end classifier combination, as well as the integration with front-end ASR system to address the modeling uncertainty issue resulted from the sparsity of AD detection training data. Experiments conducted on the ADReSS20 Challenge dataset suggest consistent performance improvements were obtained using model and feature combination in AD detection system development. Tighter integration between the ASR system and AD detection components will be investigated in future research.