{'fraction_non_alphanumeric': 0.01944528188121797, 'fraction_numerical': 0.011832981609888453, 'mean_word_length': 4.759114583333333, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multiple View Generation and Classification of Mid-wave Infrared Images using Deep Learning

We propose a novel study of generating unseen arbitrary viewpoints for infrared imagery in the non-linear feature subspace . Current methods use synthetic images and often result in blurry and distorted outputs. Our approach on the contrary understands the semantic information in natural images and encapsulates it such that our predicted unseen views possess good 3D representations. We further explore the non-linear feature subspace and conclude that our network does not operate in the Euclidean subspace but rather in the Riemannian subspace. It does not learn the geometric transformation for predicting the position of the pixel in the new image but rather learns the manifold. To this end, we use t-SNE visualisations to conduct a detailed analysis of our network and perform classification of generated images as a low-shot learning task.


INTRODUCTION
View synthesis has been a long-standing problem with works such as [1,2] notable in giving good qualitative solutions. However,majority of these works [3,4] involve the use synthetic data which is unlimited in quantity and can be created with desired azimuth, elevation, degree variation and background setting. ShapeNET [5] dataset which consists of over 8000 synthetic car and chair models and SURREAL dataset [6] which contains synthetic human motion sequences are two such widely used datasets. We make the problem challenging by dealing with natural infrared images of MWIR (medium wave) frequency in the 'DSIAC-ATR Image database' . These images are inherently complex and carry intrinsic details such as background , clutter , moving targets, day and night variations which are difficult to comprehend and predict.Collection of such data is also a big hassle due to which we show that our generated images can also be used in the training corpus for various image processing tasks including classification. Since infrared data is extremely scarce, our study assists in creating data as close to real world natural images. Most of the related view generation studies have been categorized in two categories. Geometry based where the weights of the neural net-work learn the geometric transformation of a pixel between two views, the input and output view. Second, where view prediction is seen as a learning problem and the network is trained to understand the 3D representation of objects. Few works have exploited the use of both approaches [1] to come up with hybrid methods of using geometry as well as learning for view morphing. We however adopt the learning based approach and assert via our results that our approach is not just learning the 3D representation of objects but rather its manifold. We believe the pixels in the input and output view are highly correlated and if observed on a manifold lie at the same point. We make use of this assumption and do novel view synthesis as a manifold learning task and not as a geometry based task. We do not propose to explicitly learn the 3D information of our model but rather we want to exploit the latent space embedding of autoencoders to learn the underlying semantic information and pixel relation between input and novel output views. We show via experiments that our network learns the underlying feature subspace in the nonlinear domain. Our approach is entirely unsupervised and we can say that while learning the manifold, our network also learns to classify objects and becomes a classification model as suggested in [7]. We are able to generate novel views both during day and night, which is significant in infrared imagery. We make following contributions: • We present a novel study of generating unseen views for infrared imagery. • We propose a method that creates realistic looking training images to assist in low-shot learning and classification. • We perform experiments that show view generation is a manifold learning task that exploits latent space embeddings.

Deep Convolutional Neural Network based Autoencoder framework
Feature extraction and feature selection(a NP hard problem) when done manually pose a lot of problems. Are the features good or variable enough to assist in network learning are  questions that need addressing. Autoencoders have emerged as an alternative to feature extraction and selection in the nonlinear feature subspace [8]. We propose a deep convolutional neural network based autoencoder with emphasis on exploiting its latent space and learning the manifold. Our proposed method is illustrated in Figure 1.Block 1 denotes a vanilla autoencoder [9] which acts as a guide and paves way to formulate the first term of our loss function. This block is trained separately and is used to supply the ground truth embeddings for the desired viewpoints.Block 2 denotes the predictor network which is an exact replica of the first block except that it receives an input view and produces an unseen output view. We adapt the network proposed in [2] to create the autoencoder blocks. The autoencoders have 4 convolutional blocks in encoder and decoder with consecutive layers having 32, 32,64 and 64 filters respectively.The filter size is 5 × 5 for the first layer and then 3 × 3 for the following layers.We use an image size of 64 × 64 for our network. We tried using image size of 128 × 128 as well but obtained better results with the former. The encoder's output is 4 × 4 × 64 dimensional which we flatten to create a 1024 long embedding representing the input view. We introduce feature fusion in the network and pass a 5 dimensional long vector through a 64 dimensional fully connected layer to be processed independently.In an unsupervised setting, the 1024 embedding of our input view which is of varying orientation, azimuth and time of day is concatenated with our independent feature vector. This latent embedding which is now 1088 dimensional long is jointly processed using two further fully connected layers as shown in the detailed architecture in Figure 2. We use leaky relu activation.For the last layer, we use hyperbolic tangent since it produces steeper slope and stronger gradients as stated by LeCun et al. [8,10].

Objective function
Our objective function is a sum of two separate loss terms. The first term aims to guide the latent space embedding to a set of values which best describe our desired viewpoint. This also means, we must have the ground truth for the embeddings. Section 2.1 describes how we obtain these ground truth embeddings. Once we obtain them, we compute our first term in the loss function L e .
Here e 2 is the ideal embedding which we generate using our vanilla autoencoder and e 1 is the embedding which we obtain while training our predictor network. This is depicted in Figure 1.We use mean square error to calculate the loss. The second term in our objective function aims to reduce the loss between our predicted output view and its ground truth. It is given by : where y 2 and y 1 are the ground truth and generated views respectively. We use L e and L o to compute our final reconstruction loss L t which is a sum of these two error terms as observed below.

Dataset
We work on DSIAC-ATR Image Dataset. The Automated Target Recognition (ATR) Algorithm Development Image database [11] Table 1. A quantitative comparison -average error with different methods,input dimensions and objective functions. distance which makes our task of view prediction extremely difficult and painstaking.Most of the space in the images is taken up by background and clutter with the object of interest appearing very small and distorted. The 10 object categories consist of SUVs, pickup trucks, battle tanks and anti-aircraft missile launchers.Since these are infrared images; all vehicles are grayscale and show uneven heat distribution. Our work as a result is very different from other view prediction research studies [2,4,1,6,3,12] which use evenly distributed colored objects, have unlimited training samples and also have the ability to create their own azimuth, elevation and range parameters. We on the contrary are dealing with annotations supplied with the dataset. Hence parameters like range, azimuth, elevation , light intensity are all beyond our control.

Training and Implementation details
We first convert the videos from arf format to avi format which are then converted to 640 × 580 dimensional images [13]. These images are used to obtain the objects of interest which are cropped to a size of 64 × 64 removing background as much as possible. The dataset has 5 ranges, we work with objects that are at a range of 1000 m or less so that the object is visible. This threshold gives us approximately 180 images per object of different azimuth. We create our dataset of views which are all 5 degrees apart and contain images both from day and night time.These are used to create training pairs with varying input and output viewpoints.We end with 30k training pairs and 10k test pairs.We use Adam optimizer [14] and adaptive learning rate methods. We use learning rate of 1e-3 initially and decrease it to 1e-4 for the last few epochs. We reach convergence after 80 epochs.We extract ground truth embeddings from block 1. These embeddings are then fed with the ground truth images for view 2 in block 2 of our proposed method. Our predictor network uses our new loss function as defined in section 2.2 to compute the final reconstruction error given in Table 1. We perform multiple experiments; baseline based on [2,4,12,1], our new architecture with just simple mean square error loss and then with our new objective function. We use a batch size of 64 for all our experiments and keras with tensorflow backend for implementation. Figure 3 illustrates our obtained qualitative results.

Classification and Low-shot learning
Studies have shown that it is difficult to judge the generated images both qualitatively and quantitatively . Visual analysis depends on how the reader observes and assess the image. A good evaluation metric in terms of quality is their classification. This has given rise to the concepts of low-shot [15], zero-shot and one-shot learning.These techniques are becoming popular because it is difficult to collect and annotate real world datasets.We perform such a classification experiment using 8 classes and replace class 2-BRDM2s training images with only our network generated images. We have already shown in Figure 3 that our network is capable of generating multiple novel views using a single image. We use a pretrained VGG-16 [16] network and place our 8-class classifier on top. We test using only real-world images for all classes even BRDM2. We obtain 68% classification accuracy with very little mispredictions for class 2-BRDM2 shown in Figure 4 as compared to a 79% accuracy when we use real images for training affirming that our generated images are of good quality and can be used for experimental research.

Manifold learning
Manifold representations are not only convenient for visualizing high dimensional data, but also provide insights about how different classes inter-relate. A lower dimensional embedding of the manifold ensures that the geodesic relations between the data points is preserved. The problem of predicting new views from a few original views may be viewed as learning the behavior of the manifold, and the transition from one view to another as a movement along its surface. Thus, we interpret our results in terms of how the original and predicted views behave in the latent space. We verify that the predicted views and the ground truth images are close on the underlying manifold. The t-SNE [17] method models each high-dimensional object by a two-or three-dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points with high probability. We visualize two embeddings from our trained network, one obtained from the output of the encoder, second obtained after we inject our orientation parameters and process the concatenated features. Figure 5 (a) shows our embeddings before feature fusion. We can see different object classes, for each object class, we observe two clusters, one showing the night and the other, the day view. The data points however seem scattered and do not show uniformity. Figure 5 (b) shows the results for our embeddings after feature fusion and what the network will predict given the unseen viewpoints. It clearly depicts uniformity and precise clustering which affirms that the network learnt the manifold of the objects.

CONCLUSION
We have performed a novel study of predicting unseen views on infrared imagery. The use of deep learning and a CNN based autoencoder for exploiting the latent space embeddings and predict novel views has shown promising results. Our proposed method has provided us an effective modeling of entities present in our dataset. The method is able to understand the difference in object shape and class without any reliance on color. We are able to show that the network learns the manifold and operates in riemannian feature space. Moreover our experimental evaluations show the quality of our generated images and the effectiveness of our technique.