{'fraction_non_alphanumeric': 0.0389338907760648, 'fraction_numerical': 0.025921087013326367, 'mean_word_length': 4.8833205226748655, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Early evoked brain activity underlies auditory and audiovisual speech recognition deficits in schizophrenia

Highlights • Reduced N1 amplitudes reflect speech processing deficits in schizophrenia (SZ).• Crossmodal N1 amplitude suppression in audiovisual speech is preserved in SZ.• N1 amplitudes correlate with speech recognition performance in controls but not in SZ.


Introduction
Many patients with schizophrenia (SZ) show impaired language functions, such as disorganized speech (de Boer et al., 2020;DeLisi, 2001). Furthermore, electrophysiological and functional neuroimaging studies in SZ have found aberrant processing of speech at the neural level (Hirano et al., 2020;Li et al., 2007). The vast majority of studies, however, examined unisensory speech processing using, e.g., articulated words or written sentences as stimuli (Javitt and Sweet, 2015;Mohr et al., 2000). In everyday life, speech usually occurs in a multisensory contextviewing lip movements and hearing words. Thus far, studies investigating audiovisual (AV) speech processing in SZ have found evidence for impaired bisensory audiovisual speech processing in SZ (De Gelder et al., 2003;Roa Romero et al., 2016a, 2016bRoss et al., 2007b;Stekelenburg et al., 2013;Szycik et al., 2013;Szycik et al., 2009). However, the neural basis of these deficits, especially the temporal dynamics, are not well understood. While previous electrophysiological studies have primarily focused on alterations in late stimulus processing (Mathalon et al., 2002;Mohammad and DeLisi, 2013), much less is known about the role of early neural processing for speech recognition deficits in SZ.
Previous research in healthy individuals has shown relationships between the amplitudes of the early auditory evoked N1 and P2 components in event-related potentials (ERPs) and bisensory audiovisual speech processing (Baart et al., 2014;Besle et al., 2008Besle et al., , 2004Jaaskelainen et al., 2004; van Wassenhove et al., 2005). Analyzing data across various electroencephalography (EEG) experiments, Baart (2016) found a suppression of the N1 and P2 components for bisensory audiovisual compared to unisensory auditory speech stimuli. In particular, the suppression of the N1 component has been interpreted as an early multisensory facilitation of auditory speech processing (Brunellière et al., 2013; van Wassenhove et al., 2005). In SZ only one study has thus far examined audiovisual speech integration in early ERPs (Stekelenburg et al., 2013). The central finding of this study is the lack of an audiovisual N1 suppression effect in SZs. The study included a simple visual catch trial task to ensure that participant attended to the visual inputs, but no active speech processing was required. Therefore, it is possible that differences in attention allocated to the speech stimuli contributed to the observed group differences in N1 suppression effects. Moreover, there is not enough information on behavioral data to determine whether the lack of audiovisual N1 suppression in SZs was related to possible speech recognition deficits. To reveal a more accurate picture on the role of early neural processing for auditory and audiovisual speech perception SZ, an experiment with active speech-related task demands and variable levels of difficulty is required.
In this study, we examined SZ and healthy control (HC) participants by adapting a previously established paradigm that required active auditory and audiovisual speech processing under different levels of auditory noise (Schepers et al., 2013). Noise reduces the stimulus signal strength, which interferes with auditory and audiovisual speech recognition performance (Ross et al., 2007a). In contrast to a previous study in SZ, in which auditory and audiovisual words were overlayed by auditory background noise (Ross et al., 2007b), we added noise directly to the auditory stimuli. This is an important advantage because the noise in our study did not serve as a temporal cue for the onset of auditory speech. Taken together, our setup with variable levels of auditory noise that is directly added to the stimuli is well-suited to examine the effects of early neural processing on auditory and audiovisual speech recognition in SZ.

Sample and clinical data
A sample of 31 patients diagnosed with SZ, as defined by the diagnostic and statistical manual of mental disorders, fifth edition (DSM-5), were recruited at the outpatient units of the Charité -Universitätsmedizin Berlin. Of these, 7 had to be excluded from the final analysis (N = 5 for task performance problems, and N = 2 for EEG signal related problems, outlined below), leaving 24 patients with SZ. To minimize confounding effects of medication on the EEG (Aiyer et al., 2016), patients taking the following medications were not included in the study: Benzodiazepines, lithium, valproic acid, SSRIs, and haloperidol. A total of 27 HC participants were recruited from the general population, of which 6 were excluded from the final analysis (N = 2 for task performance problems, and N = 4 EEG signal related problems, outlined below). During recruitment, the two groups were matched for handedness (Oldfield, 1971), education (measured in years), smoking Table 1 Demographics, medication status, neuropsychological assessment and symptomatology of study participants. Group differences were calculated by either parametric independent t-tests (t) or non-parametric Wilcoxon tests (w), where assumptions for parametric tests were violated. Antipsychotic dose is calculated as the percentage of a standard daily dose of Olanzapine, which is 10 mg. BACS raw scores were converted to z-scores, normalized according to age and gender. m = mean, med = median, sd = standard deviation.  Kay et al., 1987). Items from this were grouped according to a five-factor model (Wallwork et al., 2012). It is established that cognitive capacity, including executive function, working memory and motor function are generally reduced in SZ (Dickinson et al., 2008;Kustermann et al., 2018;Sheffield et al., 2018). This has been linked to perceptual deficits (Haenschel et al., 2007), and could thus be an instrumental factor affecting performance in the perceptual tasks. Therefore, it is important to have a psychometrically detailed picture of cognitive deficits. In this study, the participant's cognitive capacity was measured with the Brief Assessment of Cognition in Schizophrenia (BACS; Keefe et al., 2004). Written informed consent was obtained from all participants, including information regarding European Union data protection laws. All participants had normal hearing and normal/corrected to normal vision. People with neurological disorders or head injury with loss of consciousness were excluded at the early recruitment stages, as were participants in the HC group who reported immediate family members with psychiatric or neurological disorders. Drug screening (Drug-Screen Multi 5 Test, Nal von Minden, amphetamine, benzodiazepine, cocaine, opioids, and cannabis) was carried out for all participants and all screenings were negative. The study was conducted in accordance with the 2008 Declaration of Helsinki and approved by the ethics commission of the Charité -Universitätsmedizin Berlin (Approval number: EA1/ 169/11).

Setup and procedure
Participants sat in an electrically and acoustically shielded chamber with low lighting. The experiment was programmed in PsychToolBox (Brainard, 1997;Pelli, 1997) and there were six experimental conditions: Three unisensory auditory (A) conditions (no, low, and high noise) and three bisensory audiovisual (AV) conditions (no, low, and high noise; see Fig. 1). Three syllables were used as stimuli: /da/, /ga/, and /ta/. At the beginning of each block, the participants were instructed to respond to a particular target syllable. Within each block, the target probability was lower than the probability of the other two standard syllables (~23% and 38.5%, respectively). This ensured a larger number of standards for the final analysis, with ~ 160 trials for each of the 6 conditions (noise: no, low, high; mode: AV, A). The visual clips had a total duration of 600 ms and they were made of 17 different frames with a duration of ~ 35 ms, (CRT monitor refresh rate was 85 Hz).
The three syllables were voiced by a female speaker. The signal had a length of 270 to 290 ms (sampled at 44.1 kHz). The program 'Cool Edit Pro (Adobe Systems)' was used to edit the sound files. The syllables were presented over in-ear air pressure headphones (66 dB, E-A-R tone Gold, Auditory Systems, Indianapolis, USA). For the conditions involving auditory noise, the syllable sounds were degraded with additional noise. This was generated by estimating the temporal power signal as the smoothed, rectified auditory signal (convolution with a Hanning window of a temporal width of 5.7 ms) and spectral power distribution as the smoothed power spectrum (convolution with a Hanning window of spectral width of 630 Hz). Each syllable was edited separately. Following this, syllable-specific noise with matching spectral and temporal power distributions were generated. In a final step, stimuli with different levels of degradation were generated by altering noise and audio signal with different weights whilst maintaining a constant power level. To minimize the chance that a certain pattern of noise would become recognizable by participants, ten different realizations of the noise were randomly presented for each of the two different noise levels.
Since the different syllables are differentially affected by noise levels, we altered the relative weight of the noise for each syllable based on our earlier experiment (Schepers et al., 2013) as follows: /da/ low noise: M Fig. 1. Illustration of a unisensory A and a bisensory AV trial. In unisensory A trials (left) a static picture is presented throughout the trial. In bisensory AV trials (right) visual lip movements commenced 230 ms before auditory stimulus onset. Auditory stimuli, which were presented at three noise levels (no, low, high) consisted of three different syllables (/da/, /ga/, /ta/). Participants were required to identify one of these in each block. = 0.54, /da/ high noise: M = 0.62; /ga/ low noise: M = 0.58, high noise: M = 0.66, and /ta/ low noise: M = 0.66, high noise: M = 0.78. In contrast to our earlier experiment, in which we individually adapted the noise levels, we kept the auditory noise levels with the different noise conditions constant for all participants. This was important because it allows a direct comparison of behavioral and EEG data between groups.
The visual stimuli consisted of short video clips of a female speaker, showing the mouth and nose. The size was 4x4 degrees visual angle, presented against a grey background. The viewer was approximately 120 cm distant from the monitor. The images generated were averaged for each frame separately over the three syllables (/da/, ta/, /ga/). Thus, the same visual input was provided for each of the three syllables and it contained no syllable-specific information. In bisensory AV trials the onset of the lip movements always preceded the sound onset by 230 ms. In unisensory A trials a static picture of a nose and a closed mouth was presented instead of a video clip (Fig. 1). There was a pause of 0-400 ms (mean 200 ms) on the first frame and a 1400 ms pause on the last frame to allow for responses. Thus, each trial was on average 2200 ms. Varying the time interval of the initial pause was important to ensure that the onset of the first frame did not serve as a temporal cue for the onset of the auditory syllable. There were a total of 1248 trials presented, divided into a total of twelve blocks (104 trials per block) of about 3 min and 48 s each.

EEG recordings
Data were recorded with a 128-channel passive EEG system (Easy-Cap). This included two electrooculography (EOG) electrodes (online: 1000 Hz sampling rate with a 0.016-250 Hz bandpass filter; offline. 49-51 Hz, 4th order Butterworth notch filter, 125 Hz 24th order finite impulse response (FIR) lowpass filter, downsampled to 500 Hz, 0.3 Hz 1500th order FIR highpass filter). The offline nasal reference was rereferenced online to the average of all the EEG electrodes. Nonstationary artifacts were identified and cut with visual inspection. There was no difference between HC and SZ groups in number of remaining trials ( However, during preprocessing 6 participants (2 SZ, 4 HC) had to be removed from further data analysis due to excessive eye-blinks or sweat artifacts.

Analysis of behavioral data
The analysis of behavioral data focused on d-prime values and hitrates. The d-prime values for behavioral responses were calculated from hit-rates and false-alarm rates. Hit rates were defined as the proportion of correct responses to the relevant target within each of the 6 conditions (3 noise levels by 2 mode levels). The false-alarm rate was the proportion of false responses under a given target condition within each of the 6 conditions. Behavioral responses outside of the range of 100-900 ms following the onset of the auditory syllable were removed from the d-prime calculation. Across groups, the probability of a response outside of this time range was relatively low (SZ: Med = 1.74 % [range: 0.34-7.64 %] and HC: Med = 0.69 % [range: 0-4.17 %], W = 133.5, p = 0.018). Seven participants were cut owing to performance problems (HC = 2, SZ = 5). In these participants the d-primes were uniformly at chance levels owing to either difficulties understanding the task or technical problems with the online registration of stimulus triggers. The d-prime values and hit-rates were analyzed separately with linear mixed effects models (Bates et al., 2015), specifically using the lmerTest package (Kuznetsova et al., 2017) in R (3.6.3), which offers approximations of p-values and degrees of freedom via Satterthwaite's method. For both models, the independent factors Group (SZ vs. HC), Mode (AV vs. A), and Noise (no, low, high) were entered as fixed variables, with all two-way and three-way interactions. As random effects we made intercepts for individual participants. Follow-up Bonferroniadjusted contrasts were calculated with the emmeans package.

Analysis of evoked brain activity
For the calculation of ERPs, high-pass filtering (2 Hz, 4th order Butterworth) and low pass filtering (45 Hz, 14th order Butterworth) was applied offline to remove low frequency drifts and better define our components of interest, i.e., N1 and P2. The baseline was set at between − 430 ms and − 330 ms relative to auditory stimulus onset. This baseline was necessary to avoid overlapping with visual movement onset in the AV conditions, which began at − 230 ms relative to auditory stimulus onset. Based on previous studies examining audiovisual speech processing in ERPs (Baart et al., 2014;Schepers et al., 2013;Shahin et al., 2012;van Wassenhove et al., 2005), an a priori defined mediofrontal region of interest (ROI), comprising 14 electrodes, was used for the statistical analysis. To define the times of interest (TOIs) all 6 experimental conditions were combined to create group means. Then, a weighted overall mean was calculated that accounts for the unequal sample size between SZ and HC. Fig. 2 illustrates the ROI and the weighted grand-average over groups and conditions. For the grandaverage ERP the minimum amplitude of the N1 was observed at 100 ms and the maximum amplitude of the P2 was found at 198 ms post auditory onset. Accordingly, the TOI for the statistical analysis of the N1 component was defined as 85-115 ms, and the TOI for the broader P2 component was defined as 178-218 ms.
LMEs (ImerTest package) were applied to the mean amplitudes at these ROIs and TOIs with fixed factors Group (SZ vs. HC), Mode (AV vs. A), and Noise (no, low, high), and all interactions, with random intercepts applied for individual participants. The mean amplitudes at these ROIs and TOIs were extracted and analyzed against independent factors Group (SZ vs. HC), Mode (AV vs. A), and Noise (no, low, high), using the lmer package in R (3.6.3) and follow-up Bonferroni-adjusted contrasts with the emmeans package. Where the interpretation of a null result was critical, Bayes Factors (BF10) were calculated to further specify the relative evidence for H0 and H1 (Aczel et al., 2017;Rouder et al., 2009). BF10s between 1 and 3 indicate anecdotal support for the alternative hypothesis (H1); BF10s from 3 to 10 suggest moderate support; and BF10 above 10 indicates strong support for H1. A BF10 of 1 indicates equal support for H1 and H0, BF10 between 1/3 and 1, 1/ 10-1/3, and below 1/10 suggest respectively anecdotal, moderate, and strong support for H0.
To more precisely test the relation between behavioral performance and ERPs, LMEs were applied for N1 and P2 amplitudes with d-prime, mode and group and their interactions as fixed factors and random effect of participants modelled as individual intercepts. We elected to leave noise out of the analysis in view of the sample size, and instead focus on the interactions between multisensory processing and our behavioral parameter of interest. In the same way, potential influence of SZ symptomatology, quantified through PANNS, as well as that of cognitive performance level, via BACS, upon the ERPs was also tested with LMEs.
The amplitudes of event-related potentials (ERPs) were correlated with potential confounding factors (olanzapine equivalent dose, Fagerström test). Critical P-values were adjusted for multiple comparisons with Benjami-Hochberg corrections (Benjamini and Hochberg, 1995).

Evoked brain activity
An LME examined N1 amplitudes against factors Mode, Noise, and Group. There was a main effect of Mode (F(1,215) Table S1c). This replicates the N1 suppression effect for audiovisual compared to auditory only speech stimuli (Jaaskelainen et al., 2004;van Wassenhove et al., 2005). There was also a main effect of Noise (F(2,215) = 10.99, p < 0.0001), with a decline in N1 amplitude with increasing noise level (Fig. 6A)  between P2 amplitudes to bisensory stimuli of the low and high noise condition (t(2 1 5) = 0.88, p = 1.00). Moreover, no P2 amplitude differences in response to unisensory A stimuli were observed between the different noise levels (all p-values > 0.248). Hence, group differences in P2 amplitudes were specifically observed for bisensory AV stimuli, but did not depend on the noise level.
LMEs of d-prime against N1 showed that overall greater deflections in N1 were associated with overall better d-prime performance (F(1, 237.30) = 7.20, p = 0.008). A significant group*d-prime interaction (F (1, 237.30) = 5.72, p = 0.0176) suggested that the N1 association was specific to the HC group, with the SZ group showing an undifferentiated response (Fig. 6). Follow-up LME analyses for groups separately supported this, showing that higher d-prime values were associated with greater deflections for HC (F(1, 110.18) = 18.65, p < 0.0001)), but not for SZ (F(1,126.89) = 0.044, p = 0.835). An LME of P2 on d-prime together with group and mode factors showed an overall association of higher d-primes with greater deflections (F(1, 243.30) = 5.98, p = 0.015). No interactions were significant.
A LME of N1 with the independent variables mode, group, and BACS total z-scores, showed a main effect of BACS (F(1,39) = 4.51, p = 0.040, β = -0.37). This shows that greater cognitive capacity was related to generally higher N1 amplitude deflections (Fig. 7). There were no main effects of group (F(1,39) = 0.28, p = 0.599, β = -0.09) or mode (F(1,39) = 3.27, p = 0.078, β = -0.09) and no significant interactions. Two separate LMEs of N1 and also P2 on the five dimensions of the PANSS did not show any significant relations (p > 0.064). Finally, potential confounds psychopharmaceutical medication (olanzapine equivalent dose) and smoking (Fagerström test) were correlated with ERP amplitudes (N1 and P2), with critical p-values adjusted for multiple comparisons. There were no significant correlations between medication and neural parameters (adjusted p > 0.416, BF10 ranged between 0.47 and 0.64, with the exception of AV high noise and P2, with BF10 = 2.88 suggesting Fig. 3. Speech recognition performance differs between bisensory AV vs. unisensory A stimuli, noise levels and study groups. Panel A: d-prime values for bisensory AV (left) and unisensory A (right) stimuli. d-prime values declined with increasing noise levels. Moreover, healthy controls (HC) showed higher d-prime values than SZ, specifically in the no noise condition. No significant group differences were observed for the low and high noise conditions. Panel B: Hit-rates for bisensory AV (left) and unisensory A (right) stimuli. Hit rates declined with increasing noise levels. In addition, hit-rates were higher for bisensory AV compared with unisensory A stimuli. Finally, the hit-rate was lower in SZ than in HC. This difference was most pronounced in no noise and low noise conditions. The box plots illustrate the median (horizontal black line), the interquartile range (showing the range between the 25th and 75th percentile), the range from minimum to maximum scores (excluding outliers; thin vertical lines) and outliers. n.s. p > 0.05, * p < 0.05, ** p < 0.01, *** p < 0.001. anecdotal support for H1). There was no relation between smoking and neural parameters (adjusted p > 0.63, most BF10 scores ranged between 0.52 and 1.03. There were two exceptions in the HC group, suggesting anecdotal support for H1 in P2: AV low BF10 = 2.16, AV high BF10 = 1.79).

Discussion
In this study we examined early auditory and audiovisual speech processing under different noise levels in HC and SZ. We observed reduced speech recognition performance in SZs in the no noise condition, while the performance differences were diminished or absent in the low and high noise conditions, respectively. Mirroring the behavioral results, we observed reduced N1 amplitudes to auditory and audiovisual speech stimuli in the SZ group, specifically in the no noise condition. In the HC group the N1 amplitudes were related to the behavioral performance, whereas no such relationship was found in the SZ group. In relation to multisensory processing, we observed similar N1 suppression effects for bisensory compared to unisensory speech stimuli in SZs and HCs. Finally, we observed generally positive relationships between the N1 amplitudes and cognitive performance, as obtained by the BACS, in both study groups.

Noise influences auditory and audiovisual speech recognition deficits in schizophrenia
As expected, increasing the auditory noise level led to a decline in speech recognition performance. In addition, a better performance was found for bisensory audiovisual compared to unisensory auditory speech stimuli. These observations replicate our previous findings in healthy young adults (Schepers et al., 2013). The effects of sensory stimulation, i.e., bisensory vs. unisensory stimulation, on speech recognition did not significantly differ between groups, which indicates a relatively intact multisensory benefit in SZs (Moran et al., 2021). Across experimental conditions, patients showed a poorer performance than controls, which supports the notion of a speech recognition deficit in SZ (DeLisi, 2001).
The main finding in the behavioral data was that that the speech recognition deficit in SZ was primarily observed in the no noise condition. In SZs, compared to HCs, we observed strong group differences in the no noise condition, small group differences in the low noise condition and no group differences in the high noise condition. Thus far, studies examining the effects of auditory noise on auditory and audiovisual speech recognition are sparse (Ross et al., 2007b). In a behavioral study investigating the effects of auditory background noise on auditory and audiovisual speech recognition, Ross et al., (2007b) observed a diminished multisensory benefit of viewing lip-movements in SZ compared to HC at an intermediate noise level, which fits with the principle of inverse effectiveness (Senkowski et al., 2011;Stein and Meredith, 1993;van de Rijt et al., 2019). Critically, the background noise in this study always started at a fixed interval of 520 ms prior to the auditory onset, i.e., around the onset of the lip movements. Thus, the noise in this study served as a reliable temporal cue for the auditory speech onset. Given the evidence that temporal multisensory processing is compromised in SZ (Stevenson et al., 2017), it is possible that group differences in temporal processing contributed to the findings by Ross et al. (2007b). In our study, the noise was directly added to the auditory syllable and did therefore not provide temporal information about the auditory speech onset. There are also differences in speech recognition performance between the previous findings and those of our study. In Ross et al. (2007b) the hit rate for no noise stimuli in both study groups was above 90% for bisensory stimuli and around 85% for unisensory auditory stimuli. In our study, the hit-rate for combined bisensory and unisensory stimuli in the no noise condition was on average 80% in HCs and 57% in the SZ group. Thus, in the bisensory noise condition in which SZs showed multisensory gain deficits in Ross et al. (2007b), the hit rate was comparable to the hit rate observed in the no noise condition in our study. This implies comparable processing demands, which could have contributed to the speech recognition deficits in the Ross et al. (2007b) and in the present study. In summary, our study revealed aberrant speech recognition for auditory and audiovisual speech stimuli in SZ, which, however, declined with increasing noise level.

Early evoked brain activity relates to speech recognition deficits in schizophrenia
In agreement with previous studies emphasizing the role of early evoked brain activity for auditory and audiovisual speech processing in healthy individuals (Baart, 2016;Baart et al., 2014;Bernstein et al., 2008), our analysis focused on the N1 and P2 components triggered by the onset of the auditory syllables. In parallel with our behavioral results, we observed group differences in N1 amplitudes for both auditory and audiovisual speech stimuli SZ, primarily in the no noise condition. Fig. 4. Traces and topographic maps of N1 and P2 components for bisensory AV and unisensory A stimuli. Panel A: Grand average ERP traces of the mediofrontal ROI. For the N1 component, significant smaller amplitudes were observed for bisensory AV compared with unisensory A stimuli. This crossmodal N1 suppression effect was comparable between conditions and groups. Significant group differences in N1 amplitudes were specifically found in the no noise condition, with smaller amplitudes in SZs compared to HCs. No significant group differences were observed in the low and high noise conditions. For the P2 component, reduced amplitudes were found in SZs compared to HC, specifically in response to bisensory AV stimuli, but not in response to unisensory A stimuli. Panel B: Topographic maps of the N1 (upper panel) and the P2 components (lower panel). The plots highlight the mediofrontal topographies of both components.
In contrast, group differences in N1 amplitude were attenuated in the low and high noise conditions. Previous studies using non-speech unisensory auditory stimuli (Gallinat et al., 2002;Leicht et al., 2010) as well as speech stimuli (Ford et al., 2001;Ford and Mathalon, 2004;Mathalon et al., 2019;Perez et al., 2012) have shown reduced N1 amplitudes in SZ. Given that alterations in N1 amplitudes in response to non-speech auditory stimuli have been found in first-episode SZs as well as clinically unaffected first-degree relatives, it has been hypothesized that reduced auditory N1 amplitudes are an endophenotype of SZ Salisbury et al., 2010). However, a study by Ford et al. (2013) has found reduced N1 amplitudes in response to auditory speech stimuli in SZ patients, but not in relatives. Further research is required to examine speech-related auditory N1 amplitudes in SZ relatives.
Our observation of reduced N1 amplitudes in the no noise condition in SZs raise questions regarding the neural basis of this finding. The majority of studies reporting reduced N1 amplitudes in SZ have used salient auditory stimuli like sinusoidal tones (Rosburg et al., 2008). As such, our finding of reduced N1 responses to salient no noise auditory and audiovisual speech stimuli fits with these previous studies. Enhanced trial-by-trial fluctuations of neural activity in sensory and higher order cortical areas have been consistently observed in SZ (Javitt and Sweet, 2015;Rolls et al., 2008). Furthermore, it is likely that excitation/inhibition (E/I) imbalance in neural circuits contributes to the enhanced fluctuations in neural activity (Anticevic and Lisman, 2017;Uhlhaas and Singer, 2015), which, in turn, could be a factor contributing to reduced evoked brain responses (Hall et al., 2015;Winterer et al., 2000). It is possible that enhanced trial-by-trial fluctuations in neural activity primarily affected the processing of the salient speech stimuli in SZs. In our study, we found positive relationships between the N1 amplitudes and behavioral performance in the HC group but not in SZs. Hence, it may be that the N1 amplitudes in SZs reflect a stronger bias of non-speech related factors, such as variations in attention or memoryrelated processing. The correlations with speech recognition performance also indicate a better decoding of the speech signals in the N1 Fig. 5. Amplitudes of the N1 and P2 components in response to bisensory AV and unisensory A stimuli. Panel A: Amplitudes of the N1 component. HCs showed higher N1 amplitudes than SZ, specifically in the no noise condition. Across groups and conditions, there was also an N1 suppression effect, with smaller amplitudes for bisensory AV compared to unisensory A stimuli. Moreover, N1 amplitudes declined with increasing noise level. Panel B: Amplitudes of the P2 component. HC showed significantly higher amplitudes than SZ, specifically in the AV condition. Amplitude increased with noise levels specifically in the AV condition. n.s. p > 0.05, * p < 0.05, ** p < 0.01, *** p < 0.001. amplitudes, which are generated in the auditory cortex and to some extent in higher order areas in HCs compared to SZs (Bernstein et al., 2008;Besle et al., 2008;Campbell, 2008) . Future research implementing multivariate decoding approaches could provide further insight into this interesting question (Bae et al., 2020). Finally, in both groups we observed positive relationships between the N1 amplitudes and cognitive performance, as obtained by the BACS total score. In a recent study, positive relationships with the BACS total score have been found for the auditory discrimination threshold but not for clinical symptoms in SZs with psychosis (Ramsay et al., 2020). This suggests a link between basic auditory processing, as reflected in the evoked N1 component and auditory discrimination threshold, with general cognitive performance that is not explained by clinical symptoms. Taken together, the findings of reduced N1 amplitudes for no noise auditory and audiovisual stimuli in SZs mirror the behavioral results. This indicates that the early speech processing deficits in SZs primarily relates to salient stimuli.
The findings for the P2 component were more complex than for the N1 component and they were confined to bisensory audiovisual stimuli. In both study groups, the amplitude of the audiovisual P2 component was larger for the no noise compared to the two noise conditions. This observation resembles our previous finding in young healthy adults (Schepers et al., 2013). Moreover, for all three noise conditions, we observed larger audiovisual P2 amplitudes for HCs compared to SZs.
Given that no such effects were found for the unisensory stimuli, this indicates later group differences in the processing of audiovisual speech. In addition, we did find positive relationships between the P2 amplitudes and speech recognition performance. However, these relationships did not differ between groups or stimulation conditions. This indicates that they could reflect more general differences in attention across participants, which has a general effect, i.e., independent of sensory modality and noise level, on auditory and auditory speech recognition performance. Finally, we did not find any relationships between the P2 amplitudes with cognitive functions. Hence, the observed group differences on the bisensory P2 were relatively unspecific. Our study also revealed evidence that the P2 amplitudes could reflect more general attention-related processing that equally affects speech recognition performance in patients and controls.

Multisensory benefits are comparable in SZs and HCs
Another important aspect of our study is the question regarding multisensory integration differences between SZs and HCs. Reduced N1 and P2 amplitudes for bisensory compared to unisensory stimuli are consistent findings in audiovisual speech research (Baart et al., 2014;Besle et al., 2004;Jaaskelainen et al., 2004;van Wassenhove et al., 2005). Previously, Stekelenburg et al. (2013) examined the N1 Fig. 6. Interaction plot of d-prime against N1 amplitudes across mode and group. The overall main effect suggests that higher behavioral performance scores are associated with greater N1 deflections in response to auditory and audiovisual speech stimuli. This effect was specifically observed in the HC group (left), but not the SZ group (right).

Fig. 7.
Interaction plot of BACS total z-scores against N1 amplitudes across modality and group. BACS total scores were transformed to z-scores taking into account age and gender. The overall main effect suggests that higher cognitive performance scores are associated with greater N1 deflections in response to bisensory AV and unisensory A stimuli.
suppression effect for no noise audiovisual vs. auditory stimuli in SZs and HCs. In contrast to our study, the authors did not find an N1 suppression effect in SZs. Recently, we observed a close interplay of topdown attention and multisensory integration in SZ (Moran et al., 2021), which fits with the notion that top-down mechanisms can influence basic integrative sensory processing (Senkowski et al., 2005;Talsma et al., 2010;Talsma et al., 2007). In line with this notion, we proposed that differences in attentional demands could have contributed to previous seemingly contradictory findings on multisensory integration in SZs. As such, it could be that the differences in findings between Stekelenburg et al. (2013) and our study relate to differences in attention demands.
In our study there are several findings that suggest a relatively intact multisensory benefit in SZs. First, across the different noise levels the multisensory benefit in speech recognition performance was comparable for SZs and HCs. Second, across the different noise levels there were no differences in bisensory N1 amplitude suppression effects between groups. Third, the behavioral deficits and N1 reductions in the no noise condition in SZs were comparable for unisensory and bisensory stimuli. In summary, our study suggests a relatively intact multisensory benefit in SZ. This implies that the observed deficits in audiovisual speech processing mainly relate to aberrant processing of the auditory syllables.

Limitations
We tested for evidence of the impact of nicotine consumption or current medication on N1 and P2 findings. Although no results were significant even with a less conservative Benjamni-Hochberg adjustment for multiple comparisons, Bayes Factor calculations suggested anecdotal support for a relation in some P2 correlations. In particular, between nicotine consumption and the two higher noise conditions in the HC group, and the two higher AV noise conditions and medication in the SZ group. It should be noted here, that they are not related to our central findings relating to N1. We found no relation between SZ psychopathology, as measured by the PANSS, with behavioral or EEG parameters. It is possible that any such relations require a larger sample size than that of our experiment, which was primarily designed to uncover differences between groups. Our measurement of the PANSS was prior to the EEG measure, rather than during the measurement, so any potentially interesting relation between, e.g., auditory hallucinations and task performance and EEG, could not be assessed. Future studies could employ the Audio-Visual Abnormalities Questionnaire (AVAQ; Nikitova et al., 2019), to relate experimental findings to this standardized measure of everyday experiences of audiovisual abnormalities in people with SZ. This interesting instrument was not available at the time of data collection. Finally, it is possible that long-term medication effects in SZs contributed to our findings. Thus, as with other experiments examining chronic SZ patients, a replication of findings in a sample of first-episode and ideally unmedicated patients would be desirable.

Conclusion
Our study demonstrates that impaired early neural processing, as reflected in reduced N1 amplitudes, relates to auditory and audiovisual speech processing deficits in SZ. Our findings also suggest that this impairment is confined to salient speech input. In our study we did not find evidence for alterations in integrative multisensory processing in SZs. In fact, bisensory N1 suppression effects and comparably better speech recognition performance for bisensory compared to unisensory stimuli was consistent across both groups. This suggests a relatively intact multisensory benefit in SZs, which indicates that the observed deficits in auditory and audiovisual speech recognition were primarily related to aberrant processing of auditory speech.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.