{'fraction_non_alphanumeric': 0.03450603855674743, 'fraction_numerical': 0.008476483384592304, 'mean_word_length': 4.564273789649416, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tensorizing Generative Adversarial Nets

Generative Adversarial Network (GAN) and its variants demonstrate state-of-the-art performance in the class of generative models. To capture higher dimensional distributions, the common learning procedure requires high computational complexity and large number of parameters. In this paper, we present a new generative adversarial framework by representing each layer as a tensor structure connected by multilinear operations, aiming to reduce the number of model parameters by a large factor while preserving the quality of generalized performance. To learn the model, we develop an efficient algorithm by alternating optimization of the mode connections. Experimental results demonstrate that our model can achieve high compression rate for model parameters up to 40 times as compared to the existing GAN.


INTRODUCTION
Generative Adversarial Networks [1] demonstrate high performance within the class of generative models. The success of GAN is accomplished not only by algorithmic advance but also by recent growth of computational capacity. For example, the state-of-the-art generative adversarial models such as [2,3,4] utilize enormous computational resource by constructing deep adversarial models with a large number of model parameters and train them on powerful Graphics Processing Units (GPUs). The dependency on such expensive GPUs to train GANs limits the flexibility to deploy deep generative models on standard computational systems and mobile devices.
Another problem we address in this paper is of the multimodality of dataset. Natural datasets often possess multimodal structure. For instance, a video can be described as 4-dimensional tensor (width × height × channel × time). When samples are fed to multilayer perceptron (MLP), it is necessary for the input data to be in a linear space. Thus, vectorization operations are applied to each sample from datasets before feeding them to a network. This flattening process will lead to losing the rich multi-modal information of such high-dimensional data and curse of dimensionality.
Several tensor decompositions such as Tensor-Train (TT) decomposition [5] and Tensor Ring (TR) decomposition [6] overcome the curse of dimensionality. These tensor decompositions have linear growth in space complexity as the dimension of the original tensor increases. By enforcing low-rank constraints, the low-rank tensor approximation algorithm is able to preserve the flexibility to represent high-dimensional tensors with moderately small number of model parameters.
Some studies utilize such low-rank tensor approximations for artificial neural networks. In particular, Novikov et al. and Kossaifi et al. propose to use low-rank tensor approximation for weight parameters [7,8], where it shows that applying lowrank tensor approximation will lead to reduction of the space complexity of a model. Although applying TT decomposistion to dense matrices demonstrates a large factor of compression rate [7], finding optimal TT-ranks still remains a difficult problem. For other related works, see [9,10,11].
In our proposed framework, we represent all hidden, input and output layers as a tensor structure. We treat a multidimensional input without vectorization operation, aiming to preserve its original multi-modal information. We also replace each hidden layer with multi-way arrays. To reduce the number of model parameters, and improve computational efficiency, the fully connected layers are modified by a multilinear contraction operation, which also has the theoretical grounding. We call the resulting layer a tensor layer and networks consisting of tensor layers FT-Nets which stands for fully tensorized networks. In addition to tensor layer and FT-Nets, we call an adversarial networks with FT-Nets a TGAN.
We empirically demonstrate that our proposed framework can achieve high compression rate for model parameters with both real and synthetical datasets. In our synthetical experiment, we observed that our model converges to the ground truth faster than GAN with MLP, given approximately same number of model parameters.
In section 2, we review several preliminary knowledge of basic tensor arithmetics and generative adversarial nets. In section 3, we introduce the tensor layer. In section 4 we demonstrate that the state-of-the-art learning algorithms are swiftly applicable to the FT-Net. In section 5, we present experimental result followed by the conclusion in section 6. There are multiple operations necessary to construct a tensor layer. In particular, the n-mode product, Kronecker product and Hadamard product are described below. For more comprehensive review of tensor arithmetics and notations, see [12,13].
The order of a tensor is the number of dimensions. Vector, matrix and tensor with order three or higher are denoted by a, A and A respectively. Given an N th-order tensor The mode-n fibers are the vectors obtained by fixing every index but the n-th index of a tensor. The mode-n matricization or mode-n unfolding of a tensor X is denoted by X (n) . It arranges the mode-n fibers to be the columns of the resulting matrix. Given matrices A and B, both of same size R I×J , their Hadamard product is denoted by A * B. The resulting matrix is also of the size I × J.
The n-mode product of a tensor X ∈ R I1×I2×···×I N with a matrix U ∈ R J×In is denoted by X × n U and an entry of the resulting tensor is denoted by The Kronecker product of matrices A ∈ R I×J and B ∈ R K×L is a matrix of size IK × JL, denoted by A ⊗ B. It is defined by We use one of the properties of the Kronecker product in this paper. Given a tensor X ∈ R I1×I2×···×I N and a set of matrices denoted by A (n) ∈ R Jn×In for n ∈ [1, N ],

Generative adversarial nets
Generative adversarial nets consist of two models called generator and discriminator, which usually are represented by MLPs. The task of the discriminator is to correctly identify whether the input belongs to the real data distribution p data or the model distribution p model . Given a prior z ∼ p z , the generator tries to produce indistinguishable samples to deceive the discriminator. By alternatively training discriminator and generator, GAN aims to implicitly learn two distributions p data and p model . Goodfellow et al. adopt cross entropy for the cost function of the discriminator. [1] Given a generator G and a discriminator D, the cost function for the discriminator and generator are Although the cost function of the generator theoretically fulfills its purpose, the training process can be stabilized by maximizing log D(G(z)) instead of minimizing log (1 − D(G(z))). The reason behind such alternation is because of the saturated gradient which is caused by discriminator outperforming the generator in the early stage of training.

TENSOR LAYER
In this section, we introduce the Tensor layer using tensor algebra which we reviewed in section 2.1. Intuitively, a tensor layer is a layer without vectorization. When we train an artificial neural network, it is common to flatten the inputs then feed them to the network. In particular, given an input vector x, the following transformation is applied in neural networks: In the proposed model, the number of weight matrices per each connectionx between two layers is as same as the order of the input tensor. We establish the transformation between two tensor layers by applying combinations of mode product operations to the input tensor and an addition of a bias tensor which lies in the same space as input space. Given an N-way tensor X ∈ R I1×I2···×I N as an input, the transformation from X to the tensor layer Y ∈ R J1×J2×···×J N is formulated as: where B ∈ R J1×J2×···×J N and U i ∈ R Ji×Ii for i ∈ [1, N ]. Fig. 1: Visualization of a three-way tensor layer. Given an input three-way tensor X ∈ R I×J×K and three weight matrices U 1 ∈ R L×I , U 2 ∈ R M ×J and U 3 ∈ R N ×K , the output is Y ∈ R L×M ×N . We call the resulting tensor Y a tensor layer and U 1 , U 2 and U 3 weight matrices.
Tensor layers demonstrate equivalent flexibility as fully connected layers do because of the set of weight matrices within a connection. The only constraint imposed on the tensor layer is the size of weight matrices. The size of weight matrices has to be correlated with the size of input tensor layer. The transformation applied to tensor layer is equivalent to the transformation of each component of Tucker Decomposition in which given a core tensor G, it casts multiplication and tensor foldings on the core tensor to shape a new tensor.
In this paper, a network consisting of tensor layers is referred as TF-Nets. In the following section, we demonstrate the application of state-of-the-art gradient-based optimization methods to TF-Nets.

TRAINING FT-NET
Multilayer models are often trained with gradient-based optimization algorithms. In this section, we demonstrate the application of such optimization algorithms and the backpropagation [14] to FT-Net.
Given a FT-Net with an input tensor X ∈ R I1×···×I N ×C and two tensor layers H 1 ∈ R J1 ×···×J N ×C and H 2 ∈ R K1×···×K N ×C , we can build a FT-Net as follows. function g(A) is a component wise activation function applied to a tensor A. Note that the last mode of the input tensor X denotes the size of each batch, thus we do not apply any mode product to the N + 1 th mode when performing the multilinear operation to X . Given a tensor layer H 2 , we can derive the gradient of H 2 with respect to each weight matrix U i for i ∈ [1, N ] using equation (4).
The gradient of H 2 with respect to O 1 can be derived as (13) Note that the Kronecker products of weight matrices U i for i ∈ [1, N ] are in descending order. The ordering is dependent upon whether the model adopts column-major order or rowmajor order. In this paper, we adopt column-major ordering, however, both orders are adaptable to learning.
We can derive the gradient of a loss function f with respect to W i for i ∈ [1, N ] as follows.

Given a neural network with three fully-connected layers of size
the number of model parameters of the network is J (I + K) + J + K. The model complexity of such network can be greatly reduced by converting the network to a FT-Net. After such conversion, the number of model parameters of the FT-Net is It is not difficult to see that the difference in the number of parameters between tensorized and un-tensorized networks can grow exponentially as the dimension of the input tensor increases.

MNIST
In this section, we describe the experimental details of both TGAN and GAN on handwritten digits [15]. The purpose of this experiment is to demonstrate both computational and space efficiency of our model when applied to real dataset. We conducted experiments with TGAN and two GANs to MNIST. In particular, GAN (1) has larger amount of model parameters compared to TGAN and GAN (2) has approximately as (a) Samples from TGAN, which is compressed by a factor of 40 compared to (b) in terms of the number of model parameters (b) Samples from GAN (1) (c) Samples from GAN (2) with approximately the same number of model parameters of (a) We collected the samples in the figure without cherry-picking. We used the same prior z ∼ U (−1, 1) to sample randomly for both TGAN and GANs. From left to right column-wise, each sample from both TGAN and GAN is at training step of 2k, 7k, 17k, 25k, 35k and 47k respectively.   same number of model parameters as TGAN does. For the result of our experiment, see Figure 2. Comparing samples generated from TGAN and two GANs, it is observable that the samples from TGAN demonstrate more curvilinear feature than samples generated by GANs do. We report the factor of the compression rate in the number of model parameters between TGAN and GAN (1) to be 40 times. See Table 1 for the detailed description of each layer and total number of model parameters of these models.

Synthetic Data
In this section, we empirically compare TGAN with GANs using synthetic data. In our experiments, TGAN and GANs are trained to capture bivariate normal distributions. In particular, we collected 10000 data points sampled from 6 clusters located circularly around the point (0.5, 0.5). Each cluster is populated by bivariate normal distribution of σ 2 = 0.2.
We note that all frameworks use approximately same number of trainable units. TGAN and three GANs at each training step. The experiment result demonstrates that; 1) our model converges to original distribution more quickly than GAN does and 2) the appropriate low-rank constraints assigned on TGAN is useful when capturing low level spaces.

CONCLUSION
We present a new generative adversarial model with tensorized structure, TGAN. Two major advantages of TGAN are; 1) significant reduction in consumption of computational resource and 2) capability to capture multi-modal structure of datasets. We empirically demonstrated that there is negligible impact on the quality of samples generated by our framework compared to samples generated by GAN, while achieving 40 times of reduction in the number of model parameters. Future work we consider is to apply various tensor decomposition algorithms such as [5,6] to each tensor layer. Tensor-Train decomposition could be applied to each layer for further reduction in the number of model parameters.