{'fraction_non_alphanumeric': 0.02970083627898957, 'fraction_numerical': 0.010949219760324166, 'mean_word_length': 4.915094339622642, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Denoised Labels for Financial Time-Series Data via Self-Supervised Learning

The introduction of electronic trading platforms effectively changed the organisation of traditional systemic trading from quote-driven markets into order-driven markets. Its convenience led to an exponentially increasing amount of financial data, which is however hard to use for the prediction of future prices, due to the low signal-to-noise ratio and the non-stationarity of financial time series. Simpler classification tasks -- where the goal is to predict the directions of future price movement -- via supervised learning algorithms, need sufficiently reliable labels to generalise well. Labelling financial data is however less well defined than other domains: did the price go up because of noise or because of signal? The existing labelling methods have limited countermeasures against noise and limited effects in improving learning algorithms. This work takes inspiration from image classification in trading and success in self-supervised learning. We investigate the idea of applying computer vision techniques to financial time-series to reduce the noise exposure and hence generate correct labels. We look at the label generation as the pretext task of a self-supervised learning approach and compare the naive (and noisy) labels, commonly used in the literature, with the labels generated by a denoising autoencoder for the same downstream classification task. Our results show that our denoised labels improve the performances of the downstream learning algorithm, for both small and large datasets. We further show that the signals we obtain can be used to effectively trade with binary strategies. We suggest that with proposed techniques, self-supervised learning constitutes a powerful framework for generating"better"financial labels that are useful for studying the underlying patterns of the market.


Introduction
The notion of economic noise was introduced in 1986 by Fischer Black [4], who describes the term as information with opposite effects on pricing, such as hype and inaccurate data. In financial markets, noise is considered as a form of market movement and can be the result of unknown interactions between the many actors within (e.g., algorithmic trading, large investment institutions). Noise can not only distort the market trend and increase the possibility of trading on false signals, but also damage an investor's long-term portfolio performances due to possible over-reaction. Once the underlying trend is identified, the future price movement can be predicted more efficiently. Conventional strategies to reduce noise exposure include applying different technical indicators, including, for example moving averages, before making trading decisions.
More recent approaches to predict future price movement are concentrating on using supervised learning algorithms, where labels are required for training the model. Labels are defined based on certain properties such that the useful features of the financial time-series can be learned and then generalised on unseen samples. Labelling is not too controversial in other domains (e.g., a picture either contains or not a cat) and, in fact, approaches in the computer vision area are well defined and extensively studied, including semantic segmentation (e.g., [5]) and panoptic segmentation (e.g., [6]). On the contrary, financial time-series is hard to label universally and effectively. The current labelling methods for financial data can be divided into two groups used for static or dynamic markets, depending on whether or not they include volatility. For example, the binary labelling method and the fixed-time horizon method assume that the market remains static, whereas the triple-barrier method and quantised-labelling method are used to process the dynamic changes in the market [7].
Self-supervised learning has been widely adopted in natural language processing (e.g., [2,8,9]) and computer vision (e.g., [10,11]). The algorithm is proposed to extract useful features from large-scale unlabelled datasets so that labels can be generated without human annotation. The self-supervised learning process is divided into two successive tasks, namely, pretext task and downstream task, and the model works by training the unlabelled dataset through pretext tasks and then transferring the learned parameters into downstream tasks. In computer vision, pretext tasks are some unsolved pre-designed tasks so that useful representations can be learned by solving the tasks. Downstream tasks are algorithms that can examine the quality of representations learned during pretext task [2]. A denoising autoencoder is a paradigm adopted for pretext tasks to reconstruct corrupted images (e.g., [12]). Other popular methods used for image recognition are image colourisation (e.g., [13]), context encoder (e.g., [14]) and contrastive learning (e.g., [15]).
After seeing the efficient bottleneck network structure of denoising autoencoder in image recognition to automatically generate labels, it is natural to ask whether self-supervised learning can be applied to produce denoised labels for financial time series. Given the non-convergent nature of financial data, one key question is how to interpret the results. What does it really mean for the labels to remove (or less ambitiously reduce) noise? In our study, we will make this question more precise, as follows: 1. Will labels make prediction easier?
2. Can we reliably take positions on these predictions?
To answer these questions, we design a contrastive experiment evaluating the performance of having denoising autoencoder as a pretext task versus naive labels for a classification downstream task, for which we use a simple Support Vector Machine (SVM).

Main Contributions
To the best of our knowledge, we are the first to apply principles of self-supervised learning to financial time series. Technically speaking, our contrastive experiments test the advantages of considering the raw financial time-series data as corrupted inputs for an autoencoder, that outputs reconstructed time series. Our results allow us to give answers to the two questions above.
1. The labels generated by the autoencoder generally make prediction of future price movement easier for the SVM classifying downstream, with an increase of 20% or more in terms of F1 score, depending on the classes we adopt.
2. The labels generated by the autoencoder generally preserve the trading signals of a class of binary trading strategies [3]. They seem to avoid taking unnecessary positions, and make execution cheaper.
The benchmark we use in our experiments is a naive labeling method, whereby market prices are taken at face value and used to define the classes up/stationary/down. Despite its simplicity, this method is widely used in the literature and by practitioners. Interestingly, the validity of the output of the autoencoder can also be assessed visually by comparing the original financial time series with the reconstructed time series, the latter being smoother and less bumpy.
The observations at point (1) above are also robust to different latencies and asset classes, as we prove them valid on different datasets (see below for more details).
Ultimately, our experiments indicate that self-supervised learning can be used to produce high quality denoised labels that can be used to improve the use of machine learning techniques in finance.

Related Work
This work aims at analysing financial time-series data with a widely used algorithm from computer vision. Classical approaches of classifying financial time-series adopt the principles from physics perspective and focus on mathematical derivation. For example, wavelets (e.g., [16,17]) and Fourier transform (e.g., [18]) are used to identify the signals in the frequency domain. Distance-based methods coupled with similarity metrics are another popular approach in time-series classification, such as the nearest neighbour classifier with the dynamic time warping distance function [19].
In [20], the authors present the empirical study of the performance of different deep neural networks for classifying the univariate and multivariate time-series dataset. In addition to this, deep learning algorithms, such as recurrent neural networks with Long Short-Term Memory (LSTM) units [21,22] and stacked autoencoders with LSTM units [23], are adopted for predicting the future price movement from large-scale high-frequency datasets.
Recent research suggests to solve the time-series classification problem by encoding the time-series data as a form of images and then adopting the techniques from computer vision (e.g., [24,25,26]). There are examples of financial time-series, where the candlestick charts were encoded into images to replicate the human's trading strategies [1] or to identify the candlestick patterns [27]. Another work focused on transforming financial ratio data into images before feeding them into the convolutional neural network [28]. Furthermore, [29] proposed to detect the anomalies in time-series by making use of an encoder-decoder architecture with skip connections.

Data and Methods
We start with the daily close price of S&P 500 index from Yahoo finance for the period 2017-2019 to represent the case with a small dataset available (i.e., 502 observations in total). We then use Bitcoin 1-minute price [30] for the period from Jun 2020 to Jun 2021, and high-frequency limit order data of Google on the day Jun 21st 2012 [31] to represent the case with a large and noisy datasets.
We let X be the feature matrix of n daily close prices x(t) of the given dataset with t = 1, ..., n. The simple return of this asset will be To analyse the relative variations between two successive business days, the log return [18] is defined as following: The observation x(t) is then assigned a label y t ∈ {−1, 0, 1} [7], which is computed by: where τ is a threshold value between (min(r), max(r)). When τ equals to zero, the setup becomes that of a binary classification problem with labels {-1,1} only. We call the labeling method in (1) naive; it is a special case of fixed-horizon methods that is often used in practice [7].
The contrastive experiment is set up with two different workflows as described below.
Workflow 1. The first workflow benchmarks the performance of financial time-series classification by using SVM only.
The default values of parameters C and "gamma" are used for SVM with the kernel of Radial Basis Function (RBF). The model is trained on the first 80% of observations X with corresponding labels and tested on the last 20% of the dataset.
Workflow 2. The second workflow implements self-supervised learning on real financial time-series data by using a denoising autoencoder for the pretext task and SVM with the same hyperparameters as Workflow 1 for the downstream task. We expand the feature matrix X from a 1-dimensional column array to a multi-dimensional matrix, so that the datasets are ready for the denoising autoencoders without overfitting [32]. The denoising autoencoder is designed to have two types of input, the original input and the corrupted input, which are then trained through the 1-d convolutional neural networks. To avoid confusion, we named the new feature matrix as "pure input", D, and corrupted input as "noisy input",X. The pure input is composed of moving averages with different lookback windows as follows, D is a L × n matrix, where l i ∈ (2, n] is the length of lookback window for i = 2, ..., k, and L = 2(l k − l 2 + 1) is the total number of different moving averages. We use moving averages because of their higher degrees of smoothing. The pure input matrix can be structured with simple moving averages (SMA) only, exponential moving averages (EMA) only or a combination of both. Based on our experiments, the combined structure performs better than the other two options. The corrupted input,X, is composed of L identical daily close price, which is due to the fact that financial time-series data is noisy enough and noise signals are random. It is built up as follows: Figure 1 describes the framework of pretext task, where the noisy inputX in (3) is fed into the denoising autoencoder. The neural network is composed of two Conv1D layers as an encoder, two Conv1DTranspose layers as a decoder, and one Conv1D layer with the output, the Sigmoid activation function and the padding as an output layer. It is trained by minimising the reconstruction loss, which measures the difference between the pure input (2) and outputX. The output matrix,X, has the same dimension as pure input (2) and it can be expressed as follows: Since the outputX represents the denoised outcome of noisy inputX and the noisy input is built up with L identical daily close price, we break down the outputX as L identical column matrices X of n denoised daily close pricex(t) with t = 1, ..., n. The bottom part of Figure 2 illustrates the framework of downstream task. The training and testing labels are both updated during the downstream task since the labels are computed based on the reconstructed daily close price, X , and the same parameters are used in SVM.
We examine the performance of the denoised daily close price by applying three commonly used trading indicators and we compare the triggered buying signals only between the denoised daily close price and the original daily close price. The buying signals are defined as the following: • Moving Average crossover: Moving Average (MA) Crossover is used to indicate the change in trend and trading opportunities where the shorter simple moving averages cross the longer simple moving averages. It has greater degrees of lag comparing to single moving averages but reduces the loss of changing in directions. The indicator is composited of two simple moving averages with a shorter period and a longer period, respectively. For example, the shorter moving average could have 5-,10-, or 25-day period while the longer moving average commonly have 50-,100-or 200-day period. A buying signal is generated when the shorter moving average crosses the longer moving average as it suggests upward momentum [3].
• Moving average convergence divergence: Moving average convergence divergence (MACD) is similar to MA crossover, which is used as a trend following indicator by measuring the difference between two exponential moving averages. The indicator consists of a faster line and a slower line, named MACD line and signal line correspondingly. The faster line (i.e., the MACD line) measures the price velocity [33] by subtracting the 26-day exponential moving average away from the 12-day exponential moving average. The MACD is positive when the 12-day moving average is above the 26-day one and it indicates upward momentum. On the contrary, the negative MACD represents downward momentum when the 26-day moving average is above the 12-day one. The slower line (i.e., the signal line) is used intentionally to smooth the MACD line further, and the 9-day exponential moving average of the MACD line is commonly used by analysts. A buying opportunity is triggered when the MACD line crosses above the signal line [3].
• Bollinger Bands: Bollinger Bands (BB) is a technical indicator widely used for momentum trending following and detecting overbought and oversold conditions. It can describe the price volatility over time by including a graphical band on a two-dimensional graph. The indicator consists of an upper band, a middle band and a lower band, where upper and lower bands are resistance line and support line of the financial time-series, respectively [33]. The middle band is normally the 20-day simple moving average and it is enveloped by the upper and lower bands. The upper band is twice a 20-days standard deviation above the middle band and conversely, the lower band is twice a 20-day standard deviation below the middle band. A buying signal is generated when the price line crosses above the lower band since it indicates a potentially oversold situation [3].

Results
This work intends to analyse whether or not self-supervised learning is useful for financial time-series classification so that noise signals can be removed efficiently. We conduct empirical studies to verify the effectiveness of the denoised close price in predicting the future price movement and to investigate whether or not the true signals are smoothed out at the same time.

Small low-frequency dataset
We begin by discussing the results for the S&P 500 dataset.
Visual results. Figure 3 describes the comparison between the original S&P 500 daily close price (i.e., the orange curve) and the denoised price (i.e., the blue curve). It is clear to see that the denoised price is smoother than the original one, which shows a positive sign that the denoising autoencoder can recognise the noise signals from the corrupted input. In addition, the denoised price curve has shorter lags compared to simple moving averages, but higher degrees of smoothing compared to exponential moving averages. This graph supports our conclusion that the autoencoder in Workflow 2 goes beyond conventional approaches to denoise the data.
We next take a more quantitative approach in assessing the quality of the reconstructed time series. Effect on downstream task. Figure 4 illustrates the F1 scores of both workflows. Under the same threshold τ , Workflow 2 achieves higher F1 score and growth with faster speed than Workflow 1. This result is explained by Figure  5: there are more samples in Workflow 2 being classified into the "no signal" group at the same threshold τ level. This indicates that many labels are misclassified into either of the other two groups under the same threshold τ due to the noise and the denoising autoencoder successfully recognised and removed the noise. Therefore, the prediction of the future price movement has higher accuracy scores and performs better. It is worth mentioning that we label the prices using the fixed-time horizon method [7] as the performance of this method is negatively affected by the noise and so is gradually replaced with volatility-based methods. Especially for the returns between two consecutive days, the fixed-time horizon method may cause confusion when analysing the underlying trend and hence, the buying and selling positions changes frequently and costly. For example in Table 2, the BB indicator suggests buying on 2018-02-05 and 2018-02-08 based on the original close price and buying on 2018-02-08 only based on the denoised close price. Importantly, this proves that the denoised close price can also be beneficial for traders in decision-making by reducing the number of false signals.
Trading opportunities and costs. Figure 6 demonstrates the examples of applying technical indicators on S&P 500 daily close price, where the buying opportunities are marked as green and red triangles, blue and purple reversed triangles, yellow and black stars for workflow 1 and 2 by using MA crossover, BB and MACD correspondingly. For the MA crossover indicator, the shorter simple moving average has a 10-day lookback window and the longer one has a 50-day lookback window. The lookback windows for both moving averages are chosen due to the small range of   Table 1 that the indicator triggers 5 buying opportunities based on the original close price and denoised close price, and there are three buying signals generated from the denoised close price having lower positions (highlighted in red colour). For the MACD indicator, there are 14 out of 22 buying signals produced based on the denoised price. It can be seen from Table 3 that there are 3 buying signals (highlighted in red colour) from the denoised close price which propose lower buying positions, 1 buying signal (highlighted in green colour) proposes the same buying position and another 2 buying signals (highlighted in blue colour) propose positions that are higher but within 5 dollars. Though this indicator does not perform as well as we expected, there are still approximately a half of buying signals from the denoised close price proposed within a reasonable range. For the BB indicator, there are 8 buying signals generated based on the denoised close price among 14 signals based on the original close price. It can be seen from Table 2 that 4 buying signals (highlighted in red colour) from the denoised close price propose lower buying positions and 2 of them (highlighted in green colour) suggest buying at the same position with the generated from the original close price. We thus can conclude that the majority of buying signals are correctly created by using the reconstructed price, and more than half of them suggest buying at lower prices. Moreover, the important buying opportunities are not accidentally smoothed out with noise.

Large high-frequency dataset
We also examine the performance of the self-supervised learning model where the dataset is large and noisy. The same contrastive experiment is performed on 1-minute Bitcoin prices within a year and high-frequency limit order data of Google with a day. The comparison plot between the original price curve and the reconstructed price curve is hard to check visually due to the scale, but the F1 scores in Figure 7 and Figure 8 clearly reveal that self-supervised learning outperforms naive labeling with large and noisy datasets.

Conclusions
In this paper, we are motivated by the success in image recognition and deep neural networks from recent studies and we show that these techniques can be used for financial time-series classification problems. We also verify that the reconstructed price is meaningful since it can improve the accuracy of classification problems in financial labelling. The contrastive experiments are performed on both small and large noisy datasets, and we demonstrate the advantages of using self-supervised learning in identifying the trading opportunities on S&P500 daily close price. Our study suggests that self-supervised learning can outperform the traditional models on real financial time-series, and the denoised close price can be adopted to create new strategies for backtesting.
Several directions for future research exist. It remains to compare the trading signals triggered by the original 1-minute Bitcoin close price and high-frequency limit order data and their denoised prices. The challenge of designing a trading algorithm that performs well on the denoised prices and can be backtested in a variety of conditions is also an interesting problem. More generally, the model could be further improved. Possible directions include: 1) replace the naive labelling method with volatility-based approaches, such as the triple-barrier method [7]; 2) transfer the financial time-series into 2-dimensional data to use 2-d convolutional neural network for denoising autoencoder [34]; 3) examine the model's performance with other pretext tasks or downstream tasks, for example, using stacked autoencoder as the pretext task while remaining the downstream task unchanged. It is also worth exploring different combinations between pretext tasks and downstream tasks.