{'fraction_non_alphanumeric': 0.021257847780592577, 'fraction_numerical': 0.011124573190880054, 'mean_word_length': 4.685660613650595, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Preprocessing of Raman Lidar Signal Over a High Altitude Station in India: Practical Considerations

Lidar can provide the range-resolved information about the vertical profiles of optical properties of aerosol content and clouds present in the atmosphere. However, before obtaining the information from received lidar signal we ought to use a few pre-processing techniques such as range-correction, temporal and spatial averaging, dead time correction of the photon counts (PC) signal, correction due to overlap effect, Simulation of molecular Rayleigh signal and gluing analog & PC signals. In this work, we have discussed some of the initial pre-processing techniques ought to be performed before lidar inversion.


INTRODUCTION
Lidar system has proved to provide rangeresolved measurements of the vertical profiles of optical properties of aerosol content and clouds present in the atmosphere [1]. Before obtaining this information about aerosols and clouds through signal inversion, it is required to perform a number of preprocessing techniques so as to achieve better and improved signal such as signal averaging, dead time correction, bin shift correction, background correction, overlap correction, gluing both analog and photon counts (PC) signals etc. Here in this paper we have tried to provide a practical approach to some of these preprocessing techniques.

METHODOLOGY
A Raman Lidar system has been deployed at the CSIR-Institute of Himalayan Bioresource Technology (IHBT), Palampur, Himachal Pradesh, India. The station is situated at an altitude of 1347 m (amsl). This lidar system is currently being used to obtain the vertical distribution of clouds and aerosol over Palampur. Before deriving the optical properties from this data, a number of preprocessing techniques mentioned above can be performed. The data preprocessing includes the techniques that are applied on the raw lidar data before performing any inversion algorithm. The corrections or the techniques used in this work are mentioned below in sequential order.

Signal Averaging
The preprocessing of the raw lidar data starts with the temporal and spatial averaging. Both temporal and spatial averaging is done to increase the signal to noise ratio of the lidar measurements. The selection of time for averaging is chosen in such a way so that the variability in the lidar signal can be minimized. For our lidar data analysis we have taken the 30 minutes and 30m time and space averaged data respectively to increase the SNR.

Dead time Correction
The next correction that is applied on the averaged PC data is known as the dead time correction of PC signals. The dead time of any detector system is referred as the minimum amount of time that is required by the detector to distinguish two consecutive events or photon counts. If there is not enough time separation between two events, then second event may not be counted. Hence, the photons which have time separation more than dead time, in that case both the photons would be counted. In case of not enough separation, they may be counted as single photon. To correct this, we must apply the dad time correction on the received PC signals. For a non-paralyzable system the dead time corrected signal is determined as follows [2], [3]:  (1) Here, Ntrue is the dead time corrected signal, Nobserved is the observed signal and τ is the dead time i.e. 4ns prescribed by the manufacturer [4].

Background Subtraction
After performing the dead time correction on the PC signals, the background noise or the solar/atmospheric radiation is to be removed from both analog and PC signals. The background noise is inherent in the received signal and it can be considered to be almost constant throughout the whole range in the signal. To remove the noise from the signals, first we have to calculate the noise offset value. For that we select a region in the higher heights where aerosol content can be considered to be negligible and the backscattered signal from such heights will mainly be due to the background radiation. The average of the lidar signal in this altitude range can be considered as noise and this is subtracted from the lidar signal throughout the whole range. For our study we have selected the area above 20km and calculated the noise value for both analog and PC channel and subtracted it from the respective signals.

Bin Shift Correction
After the noise removal from both signals there is a need to remove the time delay between both the signals. The analog signal is received at the detector side with some time delay as compared to the PC signal. This delay is caused mainly due to two reasons i). Due to the presence of antialiasing filter in analog channel and ii). ADC pipelining [5]. So before gluing we have to correct the analog signal for bin shift or we have to shift the analog signal by some memory bins. The amount of shift can be calculated by selecting a cloud profile and the peak of the signal should be at the same time in both the signals.

Overlap Correction
The lidar observations are significantly affected by the incomplete overlap found between the beam of the laser and the field of view of the telescope particularly at lower altitudes. To correctly measure the vertical profiles of the aerosol optical properties at lower altitude, it becomes extremely prominent to take overlap correction into account. To find out the overlap function and further correct it, we have utilised the approach mentioned by Wandinger and Ansmann [6] which demonstrates that the comparison of the inversion algorithms i.e. klett inversion (elastic inversion) and Raman inversion (inelastic inversion) can yield us the information about the incomplete overlap. The lidar signal is corrected for the overlap effect as mentioned in the equation 2 below, (2) here O(R, λ) represents overlap function, Pmeasured denotes the received power and Pcorrected is lidar signal after overlap correction.

Gluing of Analog and PC signals
The lidar data collected in analog mode exhibits high linearity at lower heights whereas the PC signal provides good sensitivity at higher altitudes. The SNR is high for analog at lower heights and for PC at higher heights. Due to this reason, there is a need to find a signal which shows higher SNR at both lower and higher heights. As a solution, we combine both the signals into a single signal through a process known as gluing. Although there are numerous strategies to achieve this but we have followed a method proposed by [7].

RESULTS
The lidar data of 1 month has been taken in this study after performing temporal (30 minutes) and spatial (30 meters) averaging on the raw lidar data to improve the SNR. The preprocessing of the signal has been shown from figure 1 to 4. The averaging was followed by the dead time correction of the raw PC signal with 4ns dead time. We can see from figure 1(b) that pulse pileup effect was reduced in the dead time corrected PC signal (pink). Later, both analog and PC signals were corrected for the background noise caused due to atmospheric radiation. The results of the background corrected data for both analog and PC signal can be seen in figure 1(a) and 1 (b) respectively. Figure 2 shows the cloud profile in both analog and PC mode, we can observe that there exists a shift of 9 bins between both analog and PC data as EPJ Web Conferences 237, 02026 (2020) ILRC 29 https://doi.org/10.1051/epjconf/202023702026 analog data is delayed by 9 bins compared to PC data because of the reasons mentioned in section 2.4. Results reveal that before gluing, the shift correction of the analog data becomes mandatory.  Next, we determined the overlap function for the lidar system using analog data as analog signal shows good linearity at lower heights. We can see from figure 3(a) that there was incomplete overlap below 300 meters and the complete overlap was achieved beyond 300 meters. Below 300 meters, the value of overlap function was less than 1 and after 300 meters, the value was close to 1. Figure  3(b) shows the relative deviation of overlap factor calculated for each day with respect to mean overlap factor. The deviation above 300 meters was less than 5%, however for lower heights i.e. below 300 meters the deviation was high. Figure 4 depicts the comparison of the scaled analog signal, dead time corrected PC signal and glued signal. It can be seen that the transition region for gluing between dead time corrected PC signal and scaled analog (red) was found from 1.7km to 3.2km. So, the scaled analog signal was completely replaced upto 1.7km by the glued signal (cyan) whereas dead time corrected PC signal was replaced beyond altitude of 3.2km by glued signal. So, after applying all these corrections this glued signal was used to further derive the optical properties of aerosols and

CONCLUSIONS
This paper reports the techniques which we follow to preprocess the lidar signal before deriving optical properties from it. Results reveal that the lidar signal is improved to an optimal level after applying these preprocessing steps.