{'fraction_non_alphanumeric': 0.024056360616301047, 'fraction_numerical': 0.005498596712297382, 'mean_word_length': 4.461370034407257, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Flash ADC data processing with correlation coefficients

The large growth of flash ADC techniques for processing signals, especially in applications of streaming data, raises issues such as data flow through an acquisition system, long-term storage, and greater complexity in data analysis. In addition, experiments that push the limits of sensitivity need to distinguish legitimate signals from noise. The use of correlation coefficients is examined to address these issues. They are found to be quite successful well into the noise region. The methods can also be extended to Field Programmable Gate Array modules for compressing the data flow and greatly enhancing the event rate capabilities.


Introduction
Data acquisition (DAQ) electronics in nuclear and particle physics have been shifting from analog to digital methods for many years. As part of this process, flash ADC (FADC) chips and Field Programmable Gate Array (FPGA) modules are having ever increasing use. Although these chips and modules provide a much greater range of data processing options along with the flexibility of firmware programming, considerably more data bits often must be passed through the electronics hardware, processed in computers, and stored for the long term. Data analysis methods must also be able to handle the increased size and complexity of the raw data. In some situations, only a small fraction of the total data flow may contain meaningful information relevant to the physics being pursued.
A common situation is that in which the signal from a detector component or a preamplifier is digitized in a FADC. The signal contains information about the energy deposited or collected as well as the time. It may have a shape that depends on the particle, or it may represent overlapping pulses. Although the desired information (e.g., energy) may need only a fraction of the data bits of the digitized pulse, sophisticated processing is often not possible on the fast time scale of the signals. So the digitized pulses are passed on and possibly stored for later analysis.
An additional problem with such signals is that of distinguishing a real signal from a noise excursion, perhaps one that is many standard deviations from the average. Precision experiments that push to the limits of sensitivity might falsely record such excursions while, due to noise fluctuations, miss good signals.
To address such issues, we have been developing methods based on correlation coefficient (CC) algorithms.
Such CC methods can be used to pick out signals from a stream of input data, to scan through atomic or nuclear spectra to identify peaks, to obtain initial estimates of the yield and location of a peak for use in a detailed fitting program, for pedestal evaluations in the presence of noise [1], among other things. Although the context for our work is the KOTO experiment at J-PARC [2], benefits can also extend to many other experiments.
Our initial and primary interest in developing CC methods was for use in offline data analysis. We explore here the ability of CC algorithms to identify FADC signals and to distinguish them from noise.
Very good estimates of the energy and time of FADC signals can be obtained quickly and easily for use in event reconstruction. Consideration is also given to ways in which the methods might be extended to online applications where only a small fraction of the data contains meaningful information, and/or the event rate through a DAQ system or the storage of data are limited.

Correlation Functions
Consider a set of data {y i } and a reference peak {x i }, each with N points. A simple correlation function is that of the cosine similarity function [3] where each sum is over N points. This normalized, N -dimensional scalar product expresses how well two vectors are 'aligned' in the space.
More generally, the correlation coefficient ρ is defined as [4] Each sum is again over N values. The value ρ 2 is actually more meaningful in that it expresses the fraction of the variance in the data {y i } which is accounted for by the hypothesis of the reference peak {x i }. For example, if ρ = 0.7, about half of the variance in the data would need to be ascribed by something other than the reference peak.
A reference peak can be shifted through a data spectrum to identify candidate peaks for which ρ at some location exceeds a selection criterion. The sums x i , x 2 i , and ( x i ) 2 are all known and fixed. The sums involving y i alone can be adjusted at each step by dropping off the contributions from the first (oldest) data location and adding the contributions of the new location. However, the correlation sum x i y i must be recomputed at each step.

KOTO Data Acquisition
To provide a context for the discussion, we consider the KOTO experiment which seeks to obtain the first observation of the K 0 L → π 0 νν decay and a measurement of the CP-violating parameter in the Standard Model (SM). A full description of the KOTO DAQ system, along with an overview of the experiment, is provided in Ref. [2]. Only some items relevant to the discussions in this paper are noted here.
The heart of the detector system is a roughly circular array of 2716 small and large CsI crystals, approximately 90 cm in radius, for detecting photons. The array has a hole of 20 cm by 20 cm, partly filled with some veto detectors, for the beams to pass through. In front of and surrounding the CsI array are numerous detectors to veto events with charged particles and/or photons that miss the array.
Signals from each CsI crystal are sent to custom-made FADC modules, each with 16 inputs [5]. The signals are passed through a low-pass filter that is designed to convert the pulse into a quasi-Gaussian shape about 45 ns full width at half maximum, and with a long and relatively flat tail. This shape is digitized by  The signals in each module are placed in a 4-µs pipeline, sufficient to retain all data. Upon receipt of a suitable trigger pulse, the FADCs then transmit the pipelined data for all time samples of the event to Level 2 (L2) modules. The L2 modules prepare the data for routing through 1-Gb ethernet links to a computer array for event building, followed by ethernet transfer from J-PARC to the nearby KEK laboratory for long-term storage. The DAQ hardware is fully synchronized to an 8-ns clock.

Offline Data Analysis
For each event, the goal of data analysis is to identify the 'hit' crystals, their energy deposits, and their times so that clusters corresponding to the incident photons can be constructed. Kinematic analyses are then applied to reconstruct the K 0 L decay modes and their properties. It is assumed for the discussion here that suitable methods are applied to determine pedestals. Our analyses typically involve an algorithm the utilizes the averages of the first 8 and the last 8 time samples, with checks on possible accidental peaks in those regions [6]. The discussion is also focused on the CsI crystals as they determine the kinematics of an event.

Application of the correlation coefficient
For this work, a model reference peak {x i } was constructed by averaging the shapes obtained from a large number of peaks in the data of an early KOTO test run. A representation of the shape is shown in To find hits, a pedestal-subtracted time spectrum is prepared for each crystal (extended with samples of zero at each end to accommodate the samples before the maximum of the reference peak). The reference peak is then scanned through a time spectrum, a correlation coefficient is calculated at each step, and values that exceed the user's criterion are placed in a table along with a label of the time sample. In addition, the CC routine automatically calculates the area A = i y i of the signal peak and the weighted time sum T = i i y i , and includes them in the candidate table. The user's code selects the highest CC value from the table, if any, to record a hit, along with the area and mean time T /A. The area is converted to a value for the peak maximum by a fixed constant. The energy is obtained from this peak maximum by applying experimental calibration constants.
It is to be noted that the information returned to the user is obtained directly from the data and is not the result of a fitting procedure. The CC value is simply a measure of the quality of the correlation with respect to a reference shape. Statistical precision is improved by using a sum over the peak distribution rather than a single sample at the maximum.
Our standard analysis code contains some other useful features. For example, a time spectrum can be scanned prior to the CC analysis to see if it satisfies a tight criterion for having no energy, at which point it is removed from further processing. A simple method is to find the maximum and minimum values of all time samples and test if their difference is less than a specified value (e.g., 4-6 σ noise ) [6]. Also, because the peaks of all the CsI signals are located near a fixed time sample by design, it is not necessary to scan over the full time distribution but only over a region around the nominal time sample. This option can eliminate crystals that have only out-of-time accidentals. Two CC criterion levels are available for test and development purposes: one in the CC routine to fill its table of hit information for the user, and a second, higher one for final user selection. Finally, an energy cut can be applied, commonly near the noise floor.

Monte Carlo Single Photon Studies
Real data are often not the best source for testing a correlation coefficient method. The energies and times have uncertainties from calibration methods, pedestal values that must be determined empirically, non-linearities, noise from known and unknown sources, and many other issues. In short, the true values are unknown and the purpose of the data analysis codes is to estimate the best values. The best tests are against Monte Carlo (MC) simulations, even with their own uncertainties.
Extensive MC studies were made to test the ability of the CC method to properly detect photon events in the CsI array, and especially to explore its sensitivity to small signals down to the noise region. In these studies, pseudo-FADC data were generated and then processed through the offline analysis code in the same way as real data.
In generating the pseudo-FADC data, a pedestal level was generated randomly for each crystal. The values could be selected for the analysis, or an algorithm could be used to extract a pedestal from the first Single photons for ten energies between 1 and 1000 MeV were directed to the CsI crystals. Except for the lowest energies, the photon energy is deposited over several individual crystals. The individual deposits and the summed total energy were recorded by the MC code. The analysis code with a CC value ρ = 0.7 was used to select hits. A window of about ±4 time samples about the expected location was used, and there was no energy cut. The results from the analysis were compared against the original MC energy and time values, crystal by crystal for each event, to find matched, missed, and false hits. Full details are given in a separate Tech Note [7], with key features summarized here.
Even though the CsI crystals have a radiation length of 27X 0 , there is some inefficiency in collecting all of the energy of an electromagnetic shower. Separate MC studies have shown that e − and e + from the showers (as well as neutrons from photonuclear processes) can be emitted and carry energy back upstream.
In addition to the intrinsic loss of energy through physical processes, analysis codes may not be able to identify all of the deposited energy of a photon due to ambiguities arising from photo-statistics, noise, and/or limitations of the detection algorithm. At some point, no method that reports energy deposits below the noise floor can be considered to be reliable. Hence, with such a cut, it would be feasible on average to correct the energy of a photon cluster obtained from data by using an energy-dependent factor determined from MC simulations.
On average, the CC method provides very good agreement with the energies and times of the hits. All of these quantities were obtained with the use of known pedestals. If a pedestal algorithm is used, the summed energy differences were about 0.07 MeV with a RMS spread of 0.23 MeV, while the summed time differences and RMS spreads were unaffected. The CC method is not intended to provide high precision for these quantities. Yet it provides excellent estimates for the quantities as starting points if needed for detailed fitting routines.

Sensitivity in the noise region
In the previous section, events were considered "matched" if they had the same crystal ID, although the energies were generally close as well. But the energy regions for the missing and false hits also need to be examined.
The fraction, in percent, vs. low-energy deposits within a cluster of matched, missed, and false hits between MC and analyzed pseudo-data distributions are shown in Fig. 2 for four of the pure photon energies.
The percent distributions are similar in each case, with the lines for the matched and missed hits crossing at 50% near 0.6 MeV (in our model). This value can be interpreted as having an equal probability for either choice: below this value, noise prevails; above this value, there is an improving likelihood of having a real hit. The missing MC hits are nearly all gone by 1 MeV, with a few reaching perhaps 1.5 MeV. Hence, one may define the hard noise region E hard < 0.6 MeV, and a gray region 0.6 MeV < E gray < 1.0 MeV. These values agree well with separate studies of the noise regions based on pseudo-FADC events with zero energy.  It is not necessary to know the actual CC value ρ to make a decision about whether to retain some data or not, but only that it exceeds a preset value. In that case, it will be convenient to square Eq. (2) and rearrange it in a slightly different form. To ensure a positive correlation, require that the numerator of Eq.
(2) before squaring be greater than zero. Then, defining X i = N x i , As mentioned in Sec. 4.1, x i in our model is normalized to 512 (2 9 ), so X i = 10, 752. This scaling is needed to obtain sufficient accuracy with integer arithmetic for small signals. The factor in curly braces can be precomputed once. The selection criterion is met if the left-hand side is greater than the right-hand side.
It is not necessary to complete the calculations over the full dynamic range of the signals because those that exceed some level will certainly be passed. If the test fails, a compression bit for the corresponding ADC input to the module can be set, and referenced by subsequent data stages to recognize missing data.
The Monte Carlo studies have shown that a value of ρ = 0.7 is generally successful. It has the ability to reach well into the noise region without picking up an excessive number of false hits. A more conservative value of 0.6 may be better for FADCs, to preserve marginal cases for further analysis. For the MC data, the extra hits with ρ = 0.6 all had energies below 1 MeV. Values below 0.6 tended to be excessively sensitive to noise. As with the offline case, a time window can be used for the CC scans to suppress accidentals.
Because the correlation coefficient function is complex and can take up considerable space in a FPGA, replacing it with the similarity function Eq. (1) may be satisfactory. In offline comparisons, very little difference has been found between the results of the two functions.

Summary and Conclusions
Many experiments need to be sensitive to signals that border on or are embedded in the noise region.
A common approach is to test for an excursion that is a few noise standard deviations above the pedestal level to define a signal. This method, however, can be compromised by additional random fluctuations in the data, such as accidentals, or the pedestal level. Also, excursions a few time samples from the correct time can occur.
If the signals have an approximately fixed shape apart from magnitude (or can be transformed to such a shape), a correlation comparison against a reference peak is a far better way to identify a real signal. To dig a signal pattern out of noise fluctuations with some level of confidence, there is little choice but to use such comparisons. At the lower values of ρ, the CC method is effectively a 'bump' detector for low signals.
Detailed studies with simulated data have established that the correlation-coefficient (or similarity) method is very successful in identifying energy deposits (hits) in detector elements well into the noise region. It provides great benefits especially for offline analysis. It easily provides very good estimates of the signal energy and time that are needed for subsequent detailed fitting, and optionally for reducing the amount of stored data.
For online use in FADC modules, the application of the method should be sufficiently conservative to preserve possible hits of small signals for later analysis, but also to provide a high level of data compression.
As with other possible methods of working in the noise region, there is some risk of losing a real signal. The use of the method must be carefully tailored to the design requirements of the experiment, especially the balance between the required signal range, photo-statistics, and other sources of noise.