{'fraction_non_alphanumeric': 0.035360520315478636, 'fraction_numerical': 0.014353530445183718, 'mean_word_length': 4.941372418387742, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modelling Temporal Information Using Discrete Fourier Transform for Video Classification

Recently, video classification attracts intensive research efforts. However, most existing works are based on framelevel visual features, which might fail to model the temporal information, e.g. characteristics accumulated along time. In order to capture video temporal information, we propose to analyse features in frequency domain transformed by discrete Fourier transform (DFT features). Frame-level features are firstly extract by a pre-trained deep convolutional neural network (CNN). Then, time domain features are transformed and interpolated into DFT features. CNN and DFT features are further encoded by using different pooling methods and fused for video classification. In this way, static image features extracted from a pre-trained deep CNN and temporal information represented by DFT features are jointly considered for video classification. We test our method for video emotion classification and action recognition. Experimental results demonstrate that combining DFT features can effectively capture temporal information and therefore improve the performance of both video emotion classification and action recognition. Our approach has achieved a state-of-the-art performance on the largest video emotion dataset (VideoEmotion-8 dataset) and competitive results on UCF-101.


Introduction
As technology advances, millions of video clips are uploaded to the Internet every day. From video sharing websites (e.g. YouTube and Flicker), people can easily access hundreds of video clips. It is an essential and urgent demand to develop intelligent algorithms for classifying these videos, which form the basis of various potential applications, such as video retrieval, recommendation and annotation.
Video classification is to automatically assign a label to a video clip. Recent research classify video sequences into either emotional classes [1,2,3] or action categories [4,5,6]. Compared to static image classification, video classification is a complex task. This is mainly because of three reasons: 1) Video classification demands high computation cost since a short video can contain hundreds of or even thousands of frames. 2) A video sequence contains multiple frames which are grouped together to reflect a dominant theme. Within a video sequence, some frames might not convey (or even convey an opposite meaning of) the dominant theme. 3) Spatial and temporal information should be jointly considered for video classification. Static image features have been extensively studied by researchers over the decades. Static image features can be utilised to represent video at a frame-level. In addition to static image features, temporal information is an import clue for video classification. Since the length of video clips are different, it is a challenging task to generate uniform video representations which can jointly model spatial and temporal information.
Most conventional approaches of video classification involve mainly three stages: First, local features or frame-level features are extracted. Then these features are quantised to a fixed length representation using a visual dictionary usually learned by K-means algorithm. Later, video-level representations are obtained by pooling method, such as max-pooling and average-pooling. Lastly, a classifier is trained on the video-level representations to differentiate different classes of video cat-

egorises.
However, during the process of feature quantisation and pooling, the temporal information (i.e., information accumulated along time) of videos has not been well treated. Without modelling temporal information, the performance of classifiers are restricted. Taking activity recognition as an example, without considering temporal information, a classifier cannot differentiate activities of standing up v.s. sitting down and opening doors v.s. closing doors.
Most recently, realising the importance of temporal information, researchers have started exploring how to take temporal information into account for video classification. One dominated approach in recent years is Dense Trajectories [7], which tracks densely sampled image patches over time and calculates conventional local features, e.g. HOG [8], HOF [9] and MBH [10], based on the calculated trajectories. However, the application of Dense Trajectories is restricted to relatively small scale dataset since the method needs intensive computation.
In [11], Long Short Term Memory (LSTM) [12], which can preserve information for a long time, was adopted to model temporal information for video classification. However, training an LSTM is a timeconsuming task. Although these initiatives can some-how capture temporal information, they might failed to consider the accumulated information along time.
Inspired by Discrete Fourier Transform (DFT), which can transform a discrete signal form time domain to frequency domain, this paper proposes to analysis features in frequency domain to model temporal information for videos. To some extent, signal characteristics along time can be accumulated and represented through sampling in frequency domain [13]. In this work, we integrate CNN features (features extracted from a pretrained convolutional neural network) and DFT features for video representation by leveraging feature encoding method.
The proposed method consists of five steps as shown in Figure 1. The first step is to extract CNN or C3D (convolutional 3D) features from videos. Secondly, considering each feature dimension as a discrete signal over time, we apply DFT to transform the signal to frequency domain. Thirdly, an interpolation method is adopted to generate a fixed length representation for every dimension of DFT features. Fourthly, feature encoding methods (i.e. average pooling, LLC, FV and VLAD) are applied to aggregate CNN features and DFT features. The combination of the aggregated CNN features and DFT features can be regarded as a video-level representation. Finally, with the features generated in the fourth step, an SVM is trained for video classification. Different from existing methods for temporal information analysis, our method has the following advantages: • In order to capture video temporal information, we propose to analyse features in frequency domain transformed by discrete Fourier transform.
• Video clips having different length places a challenge for obtaining a uniformed feature representation. Using DFT is resilient to signal length variation. Moreover, compared to DFT, applying fast Fourier transform (FFT) can significantly reduce the computational complexity.
• Experiments on two tasks, i.e. video emotion classification and action recognitions, demonstrate that combining DFT features can effectively capture temporal information and therefore improve the performance of video classification.
The remainder of this paper is organised as follows. In section 2, we review the related work on video representations using temporal information. In section 3, DFT-based temporal information modelling is introduced in details. Experimental results are presented and discussed in section 4. Finally, we conclude this paper in section 5.

Low-level Representations
Although video classification has been researched for many years, it is still a challenging task that attracts much research interests over the decades. Early works on video classification focus on employing efficient image representations. Local image features (such as SIFT [14], HOG (histograms of oriented gradients) [8] and HOF (histograms of optical flow) [9]) are extracted from video frames. These features are further encoded to generate a uniform video-level representation. As one of the feature encoding method, bag of visual words (BoVW) [15] is the most commonly used model for generating a uniform representation. Sparse coding, which uses sparse constraints, is an extension of the BoVW model. Sparse coding achieves less quantisation error than BoVW. In [16], sparse coding was proposed for image classification. In addition to sparse constraints, LLC utilises locality constraints and can further reduce quantisation error. In [17], LLC was proposed for image classification. Fisher vector, which was derived from fisher kernel [18] , was firstly introduced for large-scale image classification in [19]. However, the dimension of features generated by FV is much higher than BoVW and LLC. Vector of locally aggregated descriptors (VLAD) is anther popular feature encoding approach, which was firstly proposed by Jegou in [20] for image representation.

High-level Representations
Due to the limited discriminative capacity of lowlevel representations to video semantics, high-level representations were introduced. In high-level feature representations, an image is represented as a response map of a large number of pre-trained detectors. In [21], Object Bank was proposed for scene classification and semantic feature sparsification. SentiBank, which consists of 1,200 concepts and associated classifiers, was constructed for sentiment prediction in images in [22]. Action Bank, which is comprised of many individual action detectors sampled in semantic space and viewpoint space, was proposed for action recognition in [23]. Jiang et. al. [1] applied ObjectBank [24] and Sentibank [22] for video emotion categorisation.

Deep-learned Features
In contrast to hand-crafted features, the last few years have witnessed the success of deep features. Deep features extracted from the activation of a convolutional neural work (CNN) pre-trained on a large image dataset (e.g. ImageNet [25]) have proved to be more discriminative than hand-crafted features [26]. CNN features have achieved state-of-the-art results on many benchmarks [27] and are widely used in image classification, object detection and attribute detection [28,27]. Recently, researchers started applying CNN features to video classification [4,11,29]. Xu et. al. [29] proposed a video representation method through leveraging frame-level features extracted by CNN with FV and VLAD as feature encoding method. In [4], two stream convolutional networks was proposed on the top of video frames and stacked optical flows to capture spatial and motion information.

Temporal Information Modelling
Researchers started to model temporal information using motion features. Dense Trajectories [7], which was inspired by dense sampling method in image classification, was proposed for action recognition. As a derived version of DT, improved Dense Trajectories (IDT) [30] were proposed to improve the performance of DT by taking camera motion into consideration and tested on a number of challenging dataset (e.g. HMDB51 [31], Sports-1M [32]). However, motion features can only capture temporal information over a couple of consecutive frames. As a special architecture of recurrent neural network (RNN), long short term memory (LSTM) was specifically designed with memory cells to store, modify and access its internal states, and can persist long time information. LSTM was successfully used to capture temporal information for sequence learning tasks, such as speech recognition [33] and machine translation [34]. In [5], an unsupervised learning approach was proposed for video representation using the LSTM encoder-decoder architecture.
Conventional CNN are only limited to handle framelevel inputs. As an extension of CNN, 3D convolutional neural network was proposed for videos classification in [35], which extract features from both spatial and temporal dimensions by performing a 3D convolution and pooling. In [6], a 3D convolutional neural network was trained to extract spatial-temporal features. However, 3D CNN architecture can only take video clips with a short length (usually, 16 frames) as inputs. This stops it from capturing long term accumulated temporal information.

The proposed DFT-based Temporal Information Modelling
The proposed algorithm involves five steps and is introduced in details step by step in this section.

Feature Extraction
CNN Features As shown in [27], deep features extracted from a convolutional neural network which is pre-trained on a large image dataset can be used as a powerful feature representation for many visual analysis tasks. In this paper, we leverage a deep convolutional neural network [36] pre-trained on ImageNet [25], which contains 1.2 million images categorised into 1000 classes, to extract frame-level descriptors for all video clips. The network consists of five convolution layers and three fully connected layers with a final 1000-way softmax. All input images are resized to 256×256 without considering its original aspect ratio before feeding into the network. Considering features extracted from fully connected layers can capture semantic information from the input image, activation from the fully connected layer are extracted as the frame descriptor. Following [29], fc 6 and fc 7 refer to the activation of the first and second fully-connected layers. Then ℓ 2 normalisation is adopted to all frame-level descriptors.
C3D Features C3D (Convolution 3D) features refers to features extracted from a pre-trained 3D convolutional networks [6]. Unlike convolutional networks, 3D convolutional networks takes a short video clip (usually 16 frames) as input and leverages on 3D convolution and pooling.
Let f denote the set of frame-level descriptors of a video clip which has N frames, then f can be described as ) T represents the descriptor of the i-th frame with dimension D. In this work, D equals 4096, which is the dimension of fc 6 . The value of N can be different for different video clips.

Discrete Fourier Transform of CNN Features
The aim of discrete Fourier transform (DFT), which is widely used in the field of signal processing, is to transform a discrete signal from time domain to frequency domain. At this step, we present how DFT is applied to CNN features.
As described in 3.1, f = (f 1 , ..., f N ) represents the set of frame-level CNN features extracted from a video clip with N frames. Let the k-th dimension of f be denoted can be considered as a discrete signal which has N sample points with equal sampling time intervals ∆t. We transform f[k] to frequency domain using the following equation ) T , termed as a DFT feature in this paper.

Interpolation
As mentioned in section 3.1, the number of sample points N is different duo to the various video length. For two video clips u and v with N and M frames re- have the same frequency range from 0 to S with sampling interval S /N and S /M respectively.
From equation 1, we know that the number of points obtained in frequency domain is as same as that in time domain. Therefore signals with more sample points in time domain are more compactly spaced in frequency domain than signals with less sample points. Like image resizing, we use cubic interpolation method [37] to generate a fixed length (L) represen- ). By this way, different video clips have the same frequency sample interval from the frequency range from 0 to S , as shown in Figure 2.

Feature Pooling and Fusion
So far, the CNN features are extracted at frame-level. In order to generate a uniform video-level representation, we need to aggregate the obtained CNN features and DFT features separately. We apply four most commonly used pooling methods, i.e. average pooling, locality-constrained linear coding (LLC), Fisher vector (FV) and vector of locally aggregated descriptors (VLAD) to aggregate both CNN features and DFT features in our experiments, which are briefly reviewed in this section.

Average Pooling
Average pooling is simply to calculate the mean value of the feature vector. Suppose that f = (f 1 , ..., f N ) represent the set of frame-level features extracted from a video clip, the video-level features generated using average pooling can be represented as The dimension of the video-level features generated by average pooling is same as the frame-level features. The calculation of average pooling is easy. The disadvantage is that the temporal information between frames is totally lost.

LLC Encoding
LLC, which utilises locality constraints, selects knearest codewords from a dictionary learned by Kmeans algorithm, and generates a sparse representation for the input vector. Given an input vector x, which can be a frame-level CNN feature or a DFT feature in our case, LLC code can be obtained by solving the following fitting problem: where ⊙ denotes element-wise multiplication, and d i is the locality adaptor. The solution of LLC can be derived analytically [17] by: After that, max-pooling strategy is applied to aggregate LLC-based features.

Fisher Vector Encoding
Fisher vector representation does not require as many visual words as LLC. In Fisher vector [38,29] encoding, the vocabularies of visual words are represented by the means of a Gaussian mixture model (GMM), which is learned in an unsupervised manner. Let a GMM model with K components be denoted as Θ = {(µ k , Σ k , π k ), k = 1, 2, .., K}, where µ k , Σ k and π k represent the mean, variance and prior parameter of the k-th component, respectively. Let f = (f 1 , ..., f N ) denote the set of frame-level CNN descriptors extracted from a video clip with N frames, then the mean and covariance deviation vectors for the k-th component are computed as: where q ki represents the posterior probability. The concatenation of u k and v k of all the K components lead to the final Fisher vector representation. The dimension of video-level features generated by Fisher vector is 2DK, where D indicates the dimension of frame-level CNN descriptor.

VLAD Encoding
Vector of locally aggregated descriptors (VLAD) [20] can be viewed as a simplification of the Fisher vector representation. Same as LLC encoding, a visual dictionary C = {c 1 , ..., c K } of K visual words is learned by K-means method. Let f = (f 1 , ..., f N ) denote the set of frame-level CNN descriptors extracted from a video clip with N frames. Each vector f i is associated with its nearest visual word c i = NN(f i ). Different vector regarding center c k can be obtained by: The dimension of video-level features generated by VLAD is KD, where D represents the frame-level feature dimension. Compared with FV, the cost for calculating VLAD can be significantly reduced.

Feature Fusion
After obtaining aggregated CNN features and DFT features. We then adopt late feature fusion. The linear combination of the aggregated CNN features and DFT features lead to the final video-level representation, which is denoted asx.

Video Classification
After obtaining all video-level features, an SVM is trained by optimising the following equation [39] for video classification. arg min Where x i and y i represent video-level feature and its corresponding label, C and T indicate penalty parameter and the number of training features respectively.

Experiments
Recently, video emotion classification and action recognition have attracted intensive research efforts. In this section, in order to evaluate the effectiveness of the proposed DFT-based temporal information model, two sets of experiments were conducted: video emotion classification and action recognition.

Video Emotion Classification
In our experiments, we intended to evaluate: (1) the performance of CNN features with different pooling methods; (2) the performance of DFT features with different pooling methods; and (3) the overall performance of combined CNN features (with different pooling methods) and DFT features (with different pooling methods). Moreover, to prove the efficiency of the proposed method, we compared our results with the most recent three works [1], [2] and [3].

Video Emotion Dataset
VideoEmotion-8 dataset It contains 1,101 usergenerated videos labelled with 8 basic human emotion categories. There are at least 100 videos in each category. The average duration of the 1,101 videos is 107 seconds. Currently, this is the largest dataset available for recognising emotions in user-generated videos. These videos were collected from popular video sharing websites, i.e. Youtube and Flicker. Similar as [1], we randomly selected 2/3 data from each category for training and the rest for testing. Experiments were conducted ten times. The average accuracy of the ten times was calculated to evaluate the classification performance. For computation efficiency, we sampled a frame every 15 frames.

Implementation Details
The activation of fc 7 were extracted as frame-level features using the Caffe toolkit [26], and were further ℓ 2 normalised. We implemented LLC according to [17]. FV and VLAD representation were generated by utilising vlfeat [40].
Fast Fourier transform (FFT) [41] was adopted to compute DFT. The dimension of CNN features were reduced from 4,096 to 1,024 using principal component analysis (PCA). At the interpolation step, L (mentioned in section 3.3) was set to 500 experimentally. For CNN and DFT features, different feature pooling methods were applied and compared. We trained a vocabulary with 1,024 codewords for LLC, 16 codewords for FV, and 16 codewords for VLAD. In our experiments, the aggregated CNN features were normalised to 3/5 and the aggregated DFT features were normalised to 2/5.
We applied the LibLinear toolbox [39] for SVM classification. The penalty parameter C was set to 100 experimentally.

Experimental Results and Discussion
Evaluation of CNN Feature and DFT Features The performance of CNN features and DFT features, each with four pooling methods, are shown in Table 1. From Table 1, we find that the performance of CNN features are better than DFT features, with the four pooling methods. For both CNN features and DFT features, FV achieves the best performance, with the accuracy of 65.5% and 50.2% respectively, although the dimension of video-level features aggregated by FV is the highest among the four pooling methods. The performances of LLC and VLAD are similar as average pooling.

Evaluation of Combining CNN features and DFT features
The results of combining CNN features and DFT features, as listed in Table 2, indicate that, by concatenating DFT features, the classification accuracy can be improved. CNN features and DFT features complement each other to achieve satisfactory results. The concatenation of CNN features with FV and DFT features with FV achieves the best performance 70.2%.

Comparison with State-of-the-arts Results
In order to demonstrate the effectiveness of our approach, we compare our results with the most recent three works, namely [1,2,3]. Comparison results are shown in Table 3, from which we can find: (1) In comparison to [1], [2] and [3], the accuracy of using CNN features with FV encoding improves 19.4%, 15.6% and 14,1% respectively. In [1], [2] and [3], the authors used low-level visual features, audio features and attribute features, whereas CNN 20.3% and [3] 19.1%, which is a significant improvement. To the best our knowledge, our method achieves the best performance at the moment on the VideoEmotion-8 dataset.

Action Recognition 4.2.1. Action Recogntion Dataset
UCF-101 dataset It consists of 13,320 videos categorised into 101 human action categories with an average of 180 frames per video and a total of 27 hours of video data. Downloaded from YouTube, these videos have fixed frame rate (25 FPS) and resolution (320 × 240). Currently, UCF-101 dataset is one of the most challenging datasets for action recognition, due to its large number of categories. Following the original evaluation scheme in [42], we use three train/test splits. The average accuracy over the three splits is used to measure the final performance.

Implementation Details
For UCF-101 dataset, two types of features were extracted in our experiments.

CNN features Similar as video emotion recognition,
the CNN model pre-trained on ImageNet was adopted for feature extraction. Unlike video emotion recognition, the activation of fc 6 were extracted as frame-level features, followed by ℓ 2 normalisation. The dimension of fc 6 is 4,096.
C3D features For C3D feature extraction, we utilised the public available deep 3-dimension convolutional networks (3D ConvNets) [6], which was pretrained on I380K and fine-tuned on Sports-1M. "To extract C3D features, a video is split into 16 frame long clips with a 8-frame overlap between two consecutive clips." The activation of fc 6 were extracted as the features, followed by ℓ 2 normalisation. The dimension of fc 6 is 4,096.   We trained a vocabulary with 1,024 codewords for LLC, 32 codewords for FV, and 32 codewords for VLAD. The value of L was set to 200 for CNN features and 50 for C3D features. ℓ 2 normalisation was applied for the aggregated CNN features and DFT features. Linear SVM was applied for action recognition. The cost parameter C was set to 1.
(3) The combination of CNN Avg , DFT Avg (CNN), C3D Avg , DFT Avg (C3D) improves 0.4%, compared to combining CNN Avg with DFT Avg (CNN). The results demonstrate the effectiveness of DFT features. The improvement of adding DFT features for action recognition is not as significant as that for video emotion recognition. One of the possible reasons might be that combining CNN and C3D features had already achieved satisfactory accuracy. Only small space was left for further improvements. (4) Compared with the state-of-the-art action recognition results, our best result (84.1%) is competitive. Our result performs 0.2% worse than composite LSTM model. However, composite LSTM model uses both spatial image features and optical flow features, whereas we only use CNN features and C3D features without using optical flow features. Our method also outperforms LRCN [44] 1.2% and temporal stream ConvNet [4] 0.4% respectively. Our method achieves lower performance than [45]. A possible reason might be that we directly apply teh CNN model pre-trained on ImageNet for feature extraction without any finetuning. The main purpose of this work is to prove the effectiveness of DFT features rather than challenging the best performance.

Conclusions
In this paper, we have proposed to analyse features in frequency domain transformed by DFT. In our approach, CNN and DFT features are adopted to jointly model spatial and temporal information for video classification. Capturing temporal information, DFT features have been proved to be efficient and effective for both video emotion classification and action recognition. The combination of CNN and DFT features achieves the state-of-the-art performance on VideoEmotion-8 dataset and competitive results on UCF-101 dataset.

Acknowledgments
Haimin Zhang is supported by UTS-CSC international research scholarship.