{'fraction_non_alphanumeric': 0.031765197603678075, 'fraction_numerical': 0.015232433938605862, 'mean_word_length': 4.848451928299837, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Low-Resource Text Classification using Domain-Adversarial Learning

Deep learning techniques have recently shown to be successful in many natural language processing tasks forming state-of-the-art systems. They require, however, a large amount of annotated data which is often missing. This paper explores the use of domain-adversarial learning as a regularizer to avoid overfitting when training domain invariant features for deep, complex neural networks in low-resource and zero-resource settings in new target domains or languages. In case of new languages, we show that monolingual word vectors can be directly used for training without prealignment. Their projection into a common space can be learnt ad-hoc at training time reaching the final performance of pretrained multilingual word vectors.


Introduction
Text classification is the generic term to describe the process of assigning a document x to a class y (Aggarwal & Zhai, 2012). Depending on the semantics of the class label y, text classification can be used for a variety of tasks, including sentiment analysis (Liu & Zhang, 2012), spam filtering (Sahami et al., 1998) or topic labeling (Wang & Manning, 2012).
Traditional approaches use sparse, symbolic representations of words and documents, such as the bag-of-words model (Collobert et al., 2011). Then, linear models or kernel methods are used for the classification (Wang & Manning, 2012). This approach has obvious disadvantages. While the symbolic representation of words can not model similarities and relations between words (Bengio et al., 2003), the bag-of-words approach removes any order and thus relation between words which is particularly important for sentiment detection in longer documents (Pang et al., 2002).
Current state-of-the-art natural language processing (NLP) models often use distributed representations of words (Mikolov et al., 2013;Pennington et al., 2014) which are then fed into complex neural network models such as convolutional neural networks (CNNs) (Kim, 2014) or recurrent neural networks (RNNs) (Tai et al., 2015). While these approaches proved to be successful for many NLP tasks (Collobert et al., 2011;Bahdanau et al., 2014), they often require large amounts of labeled data. However, there is often only little training data or even no data available when a NLP application is needed for new domains (Glorot et al., 2011) or new languages (Agić et al., 2016;Fang & Cohn, 2017;Hao et al., 2018;Wan, 2008;Salameh et al., 2015). Moreover, in the case of adaptation to new languages, multilingual word vectors are considered to be required. They are, however, not trivial to obtain due to the lack of parallel data (Faruqui & Dyer, 2014) or multilingual dictionaries (Ammar et al., 2016). Therefore, the ability to use these state-of-the-art architectures in such low-resource scenarios is investigated in this work.
In this work we therefore, concentrate on the low-resource setting, where only little data is available to train such a potentially complex network. For this, we explore domain adversarial training and its interpretation as a regularizer for the training of complex model architectures avoiding overfitting in low-resource and zero-resource settings. In the case of transfer learning from one language to another low-resource one, our experimental results reveal that a projection of word vectors from different languages into a common space can be automatically learnt during training, which suggests that the multilingual word vectors are no longer needed. We also extend the domain-adversarial neural network architecture to multiple source domains and show that this benefits the overall model performance. Lastly, a qualitative and quantitative analysis of the produced features and word vectors by different model architectures and hyperparameters is presented. All code necessary to reproduce our experimental results is made available 1 .
The remainder of the paper is organized as follows: In Section 2, we shortly summarize the most related work and emphasize the differences between our work and the previous one. Section 3 describes the model architecture and various feature extractors. In Section 4, we introduce the three datasets and present our experimental setups. Section 5 reports the results and our analysis on the experiments including low-resource domain and language scenarios. The study is concluded in Section 6.

Related Work
Previous work in low-resource text classification includes feature engineering in the low-resource domain (Tan & Zhang, 2008) with the obvious disadvantage of needing an expensive manual process to adapt the technique to a new dataset.
Another approach, that is suitable for multilingual settings, is to use machine translation (MT) techniques to convert the data that should be classified into a language where lots of training data is available (Wan, 2008;Salameh et al., 2015;Mohammad et al., 2016) and use a model that is trained on this plentiful data source. This comes at the cost of relying on the quality of the automatic translation and can only work across different languages but not domains. Søgaard & Goldberg (2016) present a method for training domain-adaptive NLP models by using a multi-task learning (MTL) approach that relies only on lower-level supervision in the target domain (e.g. part-of-speech (POS) tags) which may be more readily available compared to the high-level labels. The disadvantages of these previous methods substantiate the desire for machine learning methods that are able to leverage knowledge of resource-rich corpora without the need for any manual work and without relying on any prior knowledge.
One approach that tries to solve the disadvantages of the previously mentioned methods is the pretraining of a general neural language model (Mikolov et al., 2013;Pennington et al., 2014) that is then either fine-tuned for a downstream supervised task (Dai & Le, 2015;Howard & Ruder, 2018;Devlin et al., 2018) or enriched with additional context information in an unsupervised fashion (Peters et al., 2017).
Another approach for cross-lingual classification where training data is only available in one language is the use of delexicalized treebanks, using only the POS tags, which was demonstrated to perform well on syntactically similar languages (Zeman & Resnik, 2008) and, using data point selection techniques, even on syntactically less similar languages (Søgaard, 2011).

Adversarial Training of Neural Networks
In this work, we focus on adversarial training (Goodfellow et al., 2014) of a domain classifier, which has recently gained considerable interest in NLP research community (Chen et al., 2016;Ganin et al., 2016b;Gulrajani et al., 2017;Devon Hjelm et al., 2017;Li et al., 2017;Press et al., 2017;Rajeswar et al., 2017;Zhao et al., 2017). The most related works to our research are presented in Ganin et al. (2016b) and Chen et al. (2016). Ganin et al. (2016b) proposed "Domain-Adversarial Neural Networks" (DANN), a general approach for domain adaptive classifiers using the reversal of the gradients in an adversarial domain classifier. They showed that domain-adversarial training using gradient-reversal enables a feature-extractor to learn domaininvariant representations of an input.
In this general architecture, an input x is fed into a feature extractor F (more details in Subsection 3.2), which should learn to produce a representa-tion z with two distinctive properties; being discriminative to the class y of x while also being indiscriminative to the domain d it stems from. To achieve this, the architecture trains 2 separate classifiers: a label predictor P(z) ⇒ y and a domain discriminator Q(z) ⇒ d. Both get their input from the single feature extractor F(x) ⇒ z and are jointly trained. While P is trained to minimise the loss on the label classification, the domain classifier Q is adversarially trained to minimise the loss on the domain classification. At a high level, the intuition is that F will learn a joint feature space where elements are invariant for their domain d but distinguishable in their class label y. For the adversarial training, a gradient-reversal-layer (GRL) is used that inverts the gradients of Q during back-propagation. Thus, minimising the loss L(Q) effectively trains F to produce features that hinder Q from learning a good domain discriminator.
Formally, this can be explained by the fact that L(Q) approximates the divergence of the space of hyperplanes H of F that separates the training elements by their domain association. See Ganin et al. (2016a) for a full mathematical elaboration. To weight the impact of the two branches, the gradients of Q are multiplied with the hyperparameter λ during training. Therefore a higher λ value puts more emphasis on learning F to produce domain-invariant features, while a low value for λ will put more emphasis on learning features that are easily discriminated to their class label. One important property of this architecture is that P and Q can be trained independently while jointly optimizing F. Thus, training samples from the target domain that are not labeled for the output of P can be used to train Q by utilizing the inherent domain association d that is available for any (labeled or unlabeled) training sample. As this means that a model can be trained without any y-labeled data points from the target domain, this architecture is suitable for zero-shot learning of the task x tgt ⇒ y, given training data for x src ⇒ y and x ⇒ d (the latter of which is trivial to obtain). The experiments conducted by (Ganin et al., 2016b) show that training the label predictor only on elements from the source domain and using unlabeled data from the target domain to aditionally train the domain discriminator often yields better results than training on the source domain alone without any adversarial training. Moreover, it is also possible to use any available data from the target domain for the training of P, realizing few-shot learning. This makes the architecture suitable for no-and low-resource settings. Chen et al. (2016) extended this architecture by using the Wasserstein approximation over categorical cross-entropy as the loss function of Q and used the general DANN architecture for multilingual sentiment classification by averaging multilingual word vectors to represent a document as input x to the model and using a simple fully-connected feed-forward architecture for the featureextractor F. They call this architecture "Adversarial Deep Averaging Networks" (ADAN) which is a special case of the architecture presented in this paper using the averaging feature extractor (more details follow in Section 3.2).
They use a pretrained multilingual embedding and show that this architecture can outperform some baselines even with a fully randomized embedding. Chen et al. (2016) report more stable results for ADAN across different hyperparameters when using the Wasserstein approximation over categorical cross-entropy as the loss function of Q. Conneau et al. (2017) described an unsupervised method to project separately trained embeddings into a shared space. They learnt projection matrices by training a model to discriminate between the source language of a projected vector. They then adversarially trained the projection matrices to hinder the discriminator from learning a good classification rule.
In comparison with these works, we will explore DANN when training more complex networks in low-resource scenarios. Furthermore, we will address the question of whether multilingual word vectors are necessary for transfer learning to low-resource languages. the documents x src and x tgt are from different languages, diverging from the ADAN architecture we introduce an additional layer with shape K × K for each domain to project each word vector into a common space. As this projection is learnt during training of the classifier, it optimises the alignment of the embeddings to the objective of the network. We argue this presents an advantage over the use of a prealigned embedding which may have been optimised to a different objective (i.e. using the distributional hypothesis (Harris, 1970) or parallel corpora).

Architecture
The label predictor P is trained to minimise the cross-entropy between the output and the document labels. As the domain discriminator effectively tries to predict from which distribution P F a document vector z is drawn from, we follow  and approximate the Wasserstein distance by using the Kantorovich-Rubenstein duality (Villani, 2008) with the output of Q to avoid saturating the gradients and thus giving F good feedback during training: where P src

Multi-Domain Training
In some cases, there may be more than one source of labeled documents that could be used during training. We hypothesise that it would be advantageous in these cases to train Q to discriminate between all domains rather than combining all train sets into a single "source"-domain. For this, the output of Q is extended from the binary classification task (F(x) ∼ P src F or F(x) ∼ P tgt F ) using a single output neuron to a multi-class capable output with multiple neurons.

Feature Extractors
We implement several feature extractors F with an increasing level of complexities in this work. The idea is to explore the effectiveness of domain adversarial learning as a regularizer when training complex networks with a small amount of annotated training data or even without any training data.
word vector Average. The first simple feature extractor maps a document to a single vector by averaging all embedded word vectors in the document w i ∈ x.
The document vector is then fed into a subsequent fully connected layer with 100 neurons and rectified linear unit (ReLU, Nair & Hinton (2010)) activations.
tf-idf weighted Average. The second method extends F avg by weighting all the word vectors by its term frequency-inverse document frequency (tf-idf) (Leskovec et al., 2014, p.7).
Convolutional Neural Network. We also use convolutional neural networks (CNNs) following Kim (2014). Each document x is modelled as a N × K matrix, where N = max(|X|) is the maximum number of words in the set of documents X, and K is the dimensionality of the used word-embedding. Shorter documents are padded with zero vectors. This input representation is fed into a set of filters with widths 3, 4, 5 each with 100 feature maps. The feature maps are max-over-time pooled (Collobert et al., 2011), which naturally deals with the zero-padding. For regularization, dropout with a constraint on l 2 -norms (Hinton et al., 2012) is applied to the flattened and pooled feature maps. Hierarchical where u it is the hidden representation of h it for the attention mechanism and α it is the softmax-normalized attention for the current word as a similarity measure of the hidden representation with a word-context vector u w that is learned during training. s i is then the weighted sum of all word representations and used as input to the sentence-encoder. The sentence-encoder uses the sentence vectors s i as an input and has the same general structure as the word-encoder.

Datasets
We evaluate the model across four datasets with a focus on sentiment classification:

Amazon Reviews
This dataset contains 142.8M text reviews including a 5-star rating includ-

. Arabic Social Media
The BBN Arabic Sentiment Analysis dataset (Salameh et al., 2015) was

Yelp Reviews
TheYelp Open Dataset Challenge 2 contains over 4.7M reviews that have a 5 star rating. A subset of 600.000 reviews (120, 000 entries per rating) was selected for training. To match the polarity labels of the Arabic Social Media dataset, reviews with a rating of 1 and 2 were given thelabel, 4 and 5 rated reviews were assigned to the + category and a rating of 3 was labeled neutral. • For the low-resource language settings, in the case of the BBN Sentiment Analysis task, Arabic was used as the low-resource target language, while the English Yelp reviews were used as the source.
• For the CLS-10 Dataset, again, the English reviews are used as the source domain, while German reviews comprise the target domain. To simulate the low resource setting, the model has access to 2.000 labeled training elements in the target domain to get comparable results to the work of Prettenhofer & Stein.

Hyperparameters
Intuitively, using a simple architecture for the label predictor P puts more relevance to the fact that the feature extractor F produces features that are easily differentiable for their label but indistinguishable for their domain. Thus, P was chosen to only contain a single, fully-connected layer that takes the input of F and outputs the likelihood that an input document x belongs to class y.
Cross-entropy is used as the loss that is minimized using Adam (Kingma & Ba, 2014) with a learning-rate of 0.01.
The domain discriminator Q's first layer is the GRL that is implemented by

Analysis of the Feature Space
In Section 3 we explained the objective of F to learn features with a low variance between domains but a high variance between classes. To analyze and quantify the performance of the feature extractor, we analyze the output of F.
For a qualitative analysis, the high-dimensional output of the feature extractor after training is transformed into a 2-dimensional space using t-SNE (van der Maaten & Hinton, 2008). Only a single t-SNE transformation is performed on all models to be compared. The projected features can then be visualized and analyzed.
To allow for a quantitative analysis of the effect the λ hyperparameter has on different model configurations, a distance measure between n different classes or domains F n is defined as: with dist being the Euclidean distance between two documents projected into the feature space (dist(a, b) = dim(a) i=0 (a i − b i ) 2 ) and n C 2 being the set of 2-combinations in the set of feature-groups. This metric measures the average minimum distance between each point in one group to any point in the other group, thus giving a measure of separation between the groups. This metric is based upon the Silhouette Score (Rousseeuw, 1987), however the compactnessterm of the groups is ignored. 5 https://gitlab.mi.hdm-stuttgart.de/griesshaber/dann-evaluation       Table 3 shows the comparison in classifier performance when Q is trained for a binary-or multiclass classification. As one can see, treating the domains separately (|D| = 5) improves the performance on the source and target domains

Low-resource Domain
considerably. This suggests that the general DANN architecture is capable of learning indifferent feature spaces for multiple (|D| > 2) domains.  Table 4 shows the results with different feature extractors using monolingual and multilingual word vectors. Note that the monolingual word vectors are trained independently and therefore are (initially) unaligned in different subspaces. It should also be mentioned that in the case of the BNN Arabic Sentiment task, the columns using the pretrained MUSE embeddings show the results obtained by the model when training on the same data that was used to evaluate the ADAN model (Chen et al., 2017). When using the averaging feature-extractor architecture (F avg ) the results resemble those obtained by the ). Using the embeddings before and after projection into the common space of the source and target domain (X src and X tgt respectivley), a lower Hausdorff distance between the two sets indicates a better alignment of the two embedding spaces.  These results also show that this configuration is able to leverage the context vectors trained on documents in one particular language to classify documents that are available in another language. This is expected because, as demonstrated, the aligned embedding spaces project words close to their translations across languages.    in domain-separation may generally be beneficial as it indicates that the feature extractor produces domain-invariant features, a higher separation between classes will help the label predictor P to learn a good classification rule. Thus this result demonstrates the tradeoff between weighting the two branches P and Q.

Analysis of the Feature Space
The results also show, that the models that use categorical cross-entropy (CE) to train the domain predictor, are more sensitive to varying values of λ when compared to the models using Wasserstein distance as loss function in Q.
This supports the results by Chen et al. (2016) who also reported more stable results for various values of λ when using the Wasserstein distance.

Conclusion
We evaluated different feature extractors for the domain-adversarial training of text classifiers in low-and zero-resource scenarios. Our experimental results reveal that adversarial training of a domain discriminator works as a regularizer across different architectures ranging from simple to complex networks.
All tested feature extractors were able to learn a domain-invariant documentrepresentation.
We then demonstrated that the DANN architecture is able to profit from multiple source-domains for training. In those cases, it is advantageous to learn the domain discriminator to differentiate between all domains rather than combining all training data into a single "source" domain.
We also showed that learning a projection of word vectors into a common space during training can improve classification performance and renders the use of pretrained multilingual word vectors unnecessary.