{'fraction_non_alphanumeric': 0.03422928422928423, 'fraction_numerical': 0.011805761805761806, 'mean_word_length': 4.899693385895751, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An algorithm for the automatic deglitching of x-ray absorption spectroscopy data

Analysis of x-ray absorption spectroscopy (XAS) data often involves the removal of artifacts or glitches from the acquired signal, a process commonly known as deglitching. Glitches result either from specific orientations of monochromator crystals or from scattering by crystallites in the sample itself. Since the precise energy or wavelength location and the intensity of glitches in a spectrum cannot always be predicted, deglitching is often performed on a per spectrum basis by the analyst. Some routines have been proposed, but they are prone to arbitrary selection of spectral artifacts and are often inadequate for processing large data sets. Here we present a statistically robust algorithm, implemented as a Python program, for the automatic detection and removal of glitches that can be applied to a large number of spectra. It uses a Savitzky-Golay filter to smooth spectra and the generalized extreme Studentized deviate test to identify outliers. We achieve robust, repeatable, and selective removal of glitches using this algorithm.

extreme Studentized deviate test to identify outliers. We achieve robust, repeatable, and selective removal of glitches using this algorithm.

Introduction
In x-ray absorption spectroscopy (XAS), glitches correspond to artifacts in a confined energy range, usually a few data points, manifesting as a sharp variation in measured absorption (Stern & Lu, 1982;Bunker, 2010;Calvin, 2013). Glitches often arise from multiple-diffraction events within the crystal monochromator used to set the energy of the x-rays, resulting in a significant decrease or rise in the intensity of the beam delivered to the sample (Bauchspiess & Crozier, 1984). Alternatively, diffraction from other crystalline phases, for instance in diamond anvil cells, may also create glitches, impacting the measurement of the absorption coefficient (µ(E)) (Sapelkin & Bayliss, 2002).
Analysts can minimize the impact of glitches by using best practices at the time of data collection, which includes ensuring uniform sample preparation, confirming the linearity of equipment at the beamline, and detuning the monochromator to reduce the intensity of the multiple diffraction events; however, these practices do not completely eliminate glitches (Abe et al., 2018). Collecting data free from significant artifacts is particularly challenging in some energy ranges, such as at the Fe K-edge using a Si(111) monochromator (Pickering, 1999). Leaving spurious points in place may interfere with the analysis of XAS data; for instance, glitches may introduce error either in the normalization of the spectrum or the transformation of the extended x-ray absorption fine structure (EXAFS) data into R-space (Abe et al., 2018), and spectra may be improperly clustered. As such, the processing of spectra may require a deglitching step, where data points corresponding to glitches are removed.
Given the erratic nature of glitches, their removal is commonly based on the judge-ment of the analyst through visual inspection of the data. For example, the program Athena offers a graphical user interface for glitch removal, where one may either conduct point-and-click identification of spurious data points or set a threshold value based on the post-edge normalization curve outside of which data points are removed (Ravel, 2016). Other programs include the capability to remove data points at specified indices (Wellenreuther & Meyer-Klaucke, 2009) or the option to compare µ(E) with respect to the incoming incident beam intensity I0 (Aberdam, 1998). As a result, these methods require the analyst to manually inspect each XAS spectrum to remove the aberrant values, an approach that is readily applicable to large glitches and small datasets.
However, the growing relevance of time-resolved XAS (Bak et al., 2018) and continuous scanning "quick EXAFS" modes (Prestipino et al., 2011) along with increasing accessibility to laboratory-source XAS instruments (Anklamm et al., 2014;Jahrman et al., 2019) promises large datasets where manual deglitching of spectra becomes impractical. A rapid and robust method for automated deglitching of XAS spectra would directly address this shortcoming. Previous methods with greater potential for automation used the first derivative of the absorption coefficient to identify glitches (Zhuchkov et al., 2001). While useful, such strategies can result in points adjacent to glitches being erroneously identified, and applicability to the x-ray absorption near edge structure (XANES) region is limited. Moreover, a manual threshold must still be set by the analyst to identify glitches based on visual inspection of the data.
Here, we propose a method for the high-throughput deglitching of XAS data.
Outliers are identified in the normalized residuals between original and low-passfiltered data. A second iteration of this process minimizes the occurrence of type I errors. Through this, we achieve accurate and repeatable removal of glitches from full-spectrum XAS data. We present deglitching examples on XANES and EXAFS IUCr macros version 2.1.11: 2020/04/29 spectra, as well as a negative control. Finally, we discuss potential limitations of our method as well as strategies for improvement.

Methods
The deglitching algorithm was implemented as a computer program written in the Python 3.7 programming language. The packages Scipy (Virtanen et al., 2020) and Numpy (Oliphant, 2006) were used for all calculations, and Larch (Newville, 2013) was used for XAS data processing in a Jupyter Notebook environment (Kluyver et al., 2016).
The deglitching program was designed to be compatible with the data structure used by Larch for XAS analysis; the only required input for the deglitching program are data channels corresponding to the energy (E) and the absorption coefficient (µ(E)) within a Larch group. Absorption data may or may not be normalized prior to deglitching. Additional parameters may be specified here to tune the deglitching program, though default parameters should work in most circumstances.
All spectra presented here were collected at the bending magnet beamline of the Dow-Northwestern-Dupont Collaborative Access Team (DND-CAT), Sector 5 at the Advanced Photon Source at Argonne National Laboratory. Fe K-edge data is presented for drinking water treatment residual samples collected from drinking water treatment plants in the United States. The other samples include a denture adhesive cream and cobalt (II, III) oxide (Aldrich). Energy was set using a Si(111) double crystal monochromator. Intensity values for the incident beam (I 0 ), the transmitted beam (I T 1 ), and the secondary transmitted beam (I T 2 ) were collected using Oxford ionization chambers with 29.6 cm path lengths. Fluorescence data were captured using Vortex ME4 silicon drift detectors.

Algorithm
What follows is a description of the deglitching process, also depicted graphically in

δµ(E)
Given that the Savitzky-Golay filter acts as a low-pass filter, δµ(E) A will reflect the contribution of high frequencies to µ(E). One expects a normal distribution of residuals should Gaussian noise be the only source of misfit between the data and a fit aiming to represent the true values of the data (Trutna et al., 2003). Glitches, being aberrant high-frequency spectral features, will result in residuals that are outliers compared to the rest of the residuals. In simple cases, only glitches will be outliers in these residuals. However, glitches may impact δµ(E) A at any point within (w SG /2)−1 points of a glitch, increasing the variance of these residuals relative to those outside the Savitzky-Golay window. Moreover, the residuals in the absence of the glitch will also vary slightly based on regional features of the XAS spectrum. The window size for the median calculation is selected such that, should a glitch be contained within w m , more than half the points in w m will not have included the glitch in their Savitzky-Golay filter fitting window w SG . This may be defined as: Where w m is the window length for calculating the median absolute deviation, w SG is the Savitzky-Golay filter window length, and L g is the maximum number of points corresponding to a single glitch. Normalized residuals r are computed by dividing δµ(E) A by the local median absolute deviation of residuals to account for regional variability in the fit (Figure 1.4): Where r i corresponds to the normalized residual value at E i . At the beginning and end of the data, calculations are performed using truncated windows. Provided Savitzky-Golay parameters that do not filter true signal, these normalized residuals will be Outliers identified in this first stage are taken as candidate points for glitches. A sufficiently large glitch may result in a poor fit from the Savitzky-Golay filter on points within the same filter window as the glitch (Figure 1.6), so outliers in the first pass (candidate points) may not all be glitches. Therefore, candidate points are removed from a copy of the data, and µ(E) at these points is interpolated using a cubic spline (Figure 1.6). The resulting µ(E) B only differs from µ(E) at the candidate points. A second set of Savitzky-Golay filtered data, µ SG (E) B , is generated based on this copy of the data using the same parameters as before (Figure 1.7A). From here, the process is repeated, finding the residuals (δµ(E) B ) between the original data (µ(E)) and the second Savitzky-Golay filter (µ SG (E) B ), defined as: As before, these residuals are normalized by the regional median absolute deviation, this time calculated on δµ(E) B . The window for this second normalization shrinks to (2 * L g ) + 1 points, effectively limiting the maximum number of points in a glitch to L g . This adjustment is made possible by the interpolation of candidate points, which minimizes the impact of glitches on the filter's fit for other points within w SG .
Outliers are identified in δµ(E) B using the generalized ESD. The outliers identified in this second pass that are within one half the Savitzky-Golay window length, w SG , of a candidate point are taken to be glitches and removed.

Results and Discussion
The deglitching algorithm was tested on a range of elements and data collection modes, as shown in Figure 3. For all data, the deglitching procedure was applied across the full spectrum and used a Savitzky-Golay window length of 9, a significance level of 0.025, and a maximum glitch length of 4, the default values for the program. In each of these circumstances, the algorithm successfully identified and removed glitches across the full spectrum of data without removing non-glitch points. Given the diverse datasets included in these examples -which include XANES glitches, minor EXAFS glitches, and data for various elements collected in both absorbance and fluorescence mode, the default parameters should be adequate for most applications.  This algorithm provides repeatable and robust removal of glitches. As such, for glitches that are subtle or in relatively noisy data, parameters may need to be adjusted to identify and remove these points. Several strategies may be used in these situations.
For EXAFS glitches, the low-frequency, low-intensity features of the region may allow stricter low-pass filters to be implemented. A longer Savitzky-Golay filter window will have the effect of filtering more high frequencies from the data. For glitches anywhere in the spectrum, the significance level may be increased for more aggressive outlier identification. For both of these changes, it is recommended to limit the deglitching algorithm to a region of interest that includes the glitch; otherwise, non-glitch points may be removed.  (3). These residuals are normalized (4, Panels A and C), and outliers are identified (5, Panel C). Next (Panel D), the outlying points are interpolated (6), and the Savitzky-Golay filter is repeated with this new data (7A). Finally, outliers from the normalized residuals (7B) of the new fit and the original data are identified as glitches and removed. Panel E shows the resulting EXAFS data after deglitching (8).

Fig. 2.
A series of XAS spectra are deglitched with our algorithm using default parameters. The left column shows the full spectrum of normalized absorption data, the location of any removed glitches, and the incoming beam intensity I0, with the y-axis scaled to show the range from 77% to 132% of the median intensity of I0 for the scan. The right column shows the k 2 -weighted EXAFS for each spectrum. Row A shows Fe K-edge data collected in fluorescence mode on a drinking water treatment residual sample. Significant monochromator glitches are present in both XANES and EXAFS regions (A1). One point corresponding to a monochromator glitch in the XANES region is removed by the algorithm (A1 inset), and four points corresponding to two separate glitches are removed in the EXAFS region (A2). Row B shows Zn K-edge transmission data collected on a denture adhesive cream (B1). Only one point, corresponding to a monochromator glitch, is removed at approximately 13Å 1 in the EXAFS region (B2). Finally, Row C shows transmission data collected on a cobalt (II-III) oxide sample. Though there is some variation in I0 (C1), there are no obvious effects on either the XANES or the EXAFS (C2), and the algorithm removes no data points.