{'fraction_non_alphanumeric': 0.0347822707576317, 'fraction_numerical': 0.013621436760390961, 'mean_word_length': 4.651037663335895, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comparison of Word2Vec, HMM2Vec, and PCA2Vec for Malware Classification

Word embeddings are often used in natural language processing as a means to quantify relationships between words. More generally, these same word embedding techniques can be used to quantify relationships between features. In this paper, we first consider multiple different word embedding techniques within the context of malware classification. We use hidden Markov models to obtain embedding vectors in an approach that we refer to as HMM2Vec, and we generate vector embeddings based on principal component analysis. We also consider the popular neural network based word embedding technique known as Word2Vec. In each case, we derive feature embeddings based on opcode sequences for malware samples from a variety of different families. We show that we can obtain better classification accuracy based on these feature embeddings, as compared to HMM experiments that directly use the opcode sequences, and serve to establish a baseline. These results show that word embeddings can be a useful feature engineering step in the field of malware analysis.


Introduction
Malware detection and analysis are critical aspects of information security. The 2019 Internet Threat Security Report [46] claims an increase of 25% in one year in the number of attack groups using malware to disrupt businesses and organizations. According to the 2016 California Data Breach Report [13], malware contributed to 54% of all breaches and 90% of total records breached, with a staggering 44 million records breached due to malware in the years 2012-2016. Statistics such as these imply that malware is an increasing threat.
In this paper, we apply machine learning classification techniques to engineered features that are derived from malware samples. This feature engineering involves machine learning techniques. In effect, we apply machine learning to higher-level features, where these features are themselves obtained using machine learning models. The motivation is that machine learning can serve to distill useful information from training samples, and hence the classification techniques may perform better on such data. In this research, we consider the effectiveness of using these derived features in the context of malware classification.
Specifically, we use word embeddings based on opcodes to derive features for subsequent classification. We consider three distinct word embedding techniques. First, we derive word embeddings from trained hidden Markov models (HMM). We refer to this technique as HMM2Vec. We then consider and analogous technique based on principal component analysis (PCA), which we refer to as PCA2Vec. And, as a third approach, we experiment with the popular neural network based word embedding technique known as Word2Vec. In each case, we generate word embeddings for a significant number of samples from a variety of malware families. We then use several classification techniques to determine how well we can classify these samples using word embeddings as features.
The remainder of this paper is organized as follows. We provide a selective survey of relevant related work in Section 2. Section 3 contains an extensive and wide-ranging discussion of machine learning topics that play a role in this research. In Section 4, we provide details on the word embedding techniques that form the basis of our experiments. Section 5 gives our experiments and results, while Section 6 provides our conclusion and some paths for future work.

Related Work
Malware analysis and detection are challenging problems due to a variety of factors, including the large volume of mlaware and obfuscation techniques [10]. Every day, thousands of new malware are generated-manual analysis techniques cannot keep pace. Obfuscation is widely used by malware developers to make it difficult to analyze their malicious code.
Signature-based malware detection methods rely on pattern matching with known signatures [47]. Signature detection is relatively fast, and it is effective against "traditional" malware. However, extracting signatures is a labor intensive process, and obfuscation techniques can defeat signature scanning.
Anomaly-based techniques are based on "unusual" or "virus-like" behavior or characteristics. An example of anomaly detection is behavior based analysis, which can be used to analyze a sample when executed or under emulation [47]. When an executable file performs any action that does not fit its expected behavior, an alarm can be triggered. Such a method can detect obfuscated and zero-day malware, but it is slow, and generally subject to excessive false positives.
Recently, machine learning techniques have proven extremely useful for malware detection. The effectiveness of machine learning algorithms depends on the characteristics of the features used by such models. In malware detection and classification, a sample can be represented by a wide variety of features, including mnemonic opcodes, raw bytes, API calls, permissions, header information, etc. Opcodes are a popular feature that form the basis of the analysis considered in this paper.
In [7], the author experiments with opcodes and determines that such features can be successfully used to detect malware. The paper [11] achieves good results using API calls as a feature. Such features can be somewhat more difficult for malware writers to obfuscate, since API calls relate to the essential activity of software. However, extracting API calls from an executable is more costly than extracting opcodes.
Another example of malware research involving opcodes can be found in [33]. This paper features opcode -grams, with a Markov blanket used to select from the large set of available -gram. Classification is based on hidden Markov models, and experiments are based on five malware families.
In [3], malware opcodes are treated as a language, with Word2Vec used to quantify contextual information. Classification relies on -nearest neighbors ( -NN). The research in [34] also uses Word2Vec to generate feature vectors based on opcode sequences, with a deep neural network employed for malware classification. In this latter research, the number of opcodes is in the range of 50 to 200, and the length of the Word2Vec embeddings range from 250 to 750.
Word2Vec embeddings are used as features to train bi-directional LSTMs in [20]. The experiments achieve good accuracy for malware detection, but training is costly. In [14], the author proposed a word embedding method based on opcode graphs-the graph is projected into vector space, which yields word embeddings. This technique is also computationally expensive.
In comparison to previous research, we consider additional vector embedding techniques, we experiment with a variety of classification algorithms, we use a smaller number of opcodes, and we generate short embedding vectors. Since we use a relatively small number of opcodes and short embedding vectors, our techniques are all highly efficient and practical. In addition, our experiments are based on a recently collected and challenging malware dataset.

Background
In this section, we present background information on the various learning techniques that are used in the experiments discussed in Section 5. Specifically, we introduce neural networks, beginning with some historical background and moving on to a modern context. We also introduce HMMs and PCA, which form the basis for the word embedding techniques that we refer to as HMM2Vec and PCS2Vec, respectively. Finally, we introduce four classification techniques, which are used in our experiments.
In Section 4, we discuss HMM2Vec, PCA2Vec, and the neural network based word embedding technique, Word2Vec, in detail. For our experiments in Section 5, we use these three word embedding techniques to generate features to classify malware samples.

Neural Networks
The concept of an artificial neuron [12,49] is not new, as the idea was first proposed by McCulloch and Pitts in the 1940s [22]. However, modern computational neural networks begins with the perceptron, as introduced by Rosenblatt in the late 1950s [37].

McCulloch-Pitts Artificial Neuron
An artificial neuron with three inputs is illustrated in Figure 1. In the original McCulloch-Pitts formulation, the inputs ∈ {0, 1}, the weights ∈ {+1, −1}, and the output ∈ {0, 1}. The output is 0 (inactive) or 1 (active), based on whether or not the linear function ∑︀ exceeds the specified threshold . This form of an artificial neuron was modeled on neurons in the brain, which either fire or it do not (thus, ∈ {0, 1}), and have input that comes from other neurons (thus, each ∈ {0, 1}). The weights specify whether an input is excitatory (increasing the chance of the neuron firing) or inhibitory (decreasing the chance of the neuron firing). Whenever ∑︀ > , the excitatory response wins, and the neuron fires-otherwise the inhibitory response wins and the neuron does not fire.

Perceptron
A perceptron is less restrictive than a McCulloch-Pitts artificial neuron. With a perceptron, both the inputs and the weights can be real valued, as opposed to the binary restrictions of McCulloch-Pitts. As wth the McCulloch-Pitts formulation, the output of a perceptron is generally taken to be binary.
Given a real-valued input vector = ( 0, 1, . . . , −1), a perceptron can be viewed as an instantiation of a function of the form that is, a perceptron computes a weighted sum of the input components. Based on a threshold, a single perceptron can defines a binary classifier. That is, we can classify a sample as "type 1" provided that ( ) > , for some specified threshold , and otherwise we classify as "type 0." In the case of two dimensional input, the decision boundary of a is of the form which is the equation of a line. In general, the decision boundary of a perceptron is a hyperplane. Hence, a perceptron can only provide ideal separation in cases where the data itself is linearly separable. As the name suggests, a multilayer perceptron (MLP) is an ANN that includes multiple (hidden) layers in the form of perceptrons. An example of an MLP with two hidden layers is given in Figure 2, where each edge represent a weight that is to be determined via training. Unlike a single layer perceptron, MLPs are not restricted to linear decision boundaries, and hence an MLP can accurately model more complex functions. For example, the XOR function-which cannot be modeled by a single layer perceptron-can be modeled by an MLP.
To train a single layer perceptron, simple heuristics will suffice, assuming that the data is actually linearly separable. From a high level perspective, training a single layer perceptron is somewhat analogous to training a linear support vector machine (SVM), except that for a perceptron, we do not require that the margin (i.e., minimum separation between the classes) be maximized. But training an MLP is clearly far more challenging, since we have hidden layers between the input and output, and it is not obvious how changes to the weights in these hidden layers will affect each other or the output. As an aside, it is interesting to note that for SVMs, we deal with data that is not linearly separable by use of the "kernel trick," where the input data is mapped to a higher dimensional "feature space" via a (nonlinear) kernel function. In contrast, perceptrons (in the form of MLPs) overcome the limitation of linear separability by the use of multiple layers. With an MLP, it is as if a nonlinear kernel function has been embedded directly into the model itself through the use of hidden layers, as opposed to a user-specified explicit kernel function, which is the case for an SVM.
We can view the relationship between ANNs and deep learning as being somewhat akin to that of Markov chains and hidden Markov models (HMM). That is, ANNs serve as a basic technology that can be used to build powerful machine learning techniques, analogous to the way that an HMM is built on the foundation of an elementary Markov chain.

Hidden Markov Models
A generic hidden Markov model is illustrated in Figure 3, where the represent the hidden states and all other notation is as in Table 1. The state of the Markov process, which we can view as being hidden behind a "curtain" (the dashed line in Figure 3), is determined by the current state and the matrix. We are only able to observe the observations , which are related to the (hidden) states of the Markov process by the matrix .

Notation and Basics
The notation used in an HMM is summarized in Table 1. Note that the observations are assumed to come from the set {0, 1, . . . , − 1}, which simplifies the notation with no loss of generality. That is, we simply associate each of the distinct observations with one of the elements 0, 1, . . . , − 1, so that we have ∈ = {0, 1, . . . , − 1} for = 0, 1, . . . , − 1. The matrix is row stochastic, that is, each row satisfies the properties of a discrete probability distribution. Also, the probabilities are independent of , and hence the matrix does not vary with . The matrix = { ( )} is of size × , with ( ) = (observation at | state at ).
As with the matrix, is row stochastic, and the probabilities ( ) are independent of . The somewhat unusual notation ( ) is convenient when specifying the HMM algorithms. An HMM is defined by , , and (and, implicitly, by the dimensions and ). Thus, we denote an HMM as = ( , , ).
Suppose that we are given an observation sequence of length four, that is, Then the corresponding (hidden) state sequence is denoted as We let 0 denote the probability of starting in state 0, and 0 ( 0) denotes the probability of initially observing 0, while 0 , 1 is the probability of transiting from state 0 to state 1. Continuing, we see that the probability of a given state sequence of length four is Note that in this expression, the represent indices in the and matrices, not the names of the corresponding states.
To find the optimal state sequence in the dynamic programming (DP) sense, we simply choose the sequence (of length four, in this case) with the highest probability. In contrast, to find the optimal state sequence in the HMM sense, we choose the most probable symbol at each position. The optimal DP sequence and the optimal HMM sequence can differ.

The Three Problems
There are three fundamental problems that we can solve using HMMs. Here, we briefly describe each of these problems.
That is, we want to compute a score for the observed sequence with respect to the given model .

Problem 2 Given
= ( , , ) and an observation sequence , find an optimal state sequence for the underlying Markov process. In other words, we want to uncover the hidden part of the hidden Markov model.

Problem 3
Given an observation sequence and the parameter , determine a model = ( , , ) that maximizes the probability of . This can be viewed as training a model to best fit the observed data. This problem is generally solved using Baum-Welch reestimation [35,43], which is a discrete hill climb on the parameter space represented by , , and . There is also an alternative gradient ascent technique for HMM training [4,45].
Since the technique we use to train an HMM (problem 3) is a hill climb, in general, we obtain a local maximum. Training with different initial conditions can result in different local maxima, and hence it is often beneficial to train multiple HMMs with different initial conditions, and select the highest scoring model.

Example
Consider, for example, the problem of speech recognition which, not coincidentally, is one of the earliest and best-known successes of HMMs. In speech problems, the hidden states can be viewed as corresponding to movements of the vocal cords, which are not directly observed. Instead, we observe the sounds that are produced, and extract training features from these sounds. In this scenario, we can use the solution to HMM Problem 3 to train an HMM to, for example, recognize the spoken word "yes." Then, given an unknown spoken word, we can use the solution to Problem 1 to score the word against the trained model and determine the likelihood that the word is "yes." In this case, we do not need to solve Problem 2, but it is possible that such a solution (i.e., uncovering the hidden states) might provide additional insight into the underlying speech model.
English text analysis is another classic application of HMMs, which appears to have been first considered by Cave and Neuwirth [9]. This application nicely illustrates the strength of HMMs and it requires no background in any specialized field, such as speech processing or information security.
Given a length of English text, we remove all punctuation, numbers, etc., and converts all letters to lower case. This leaves 26 distinct letters and word-space, for a total of 27 symbols. We assume that there is an underlying Markov process (of order one) with two hidden states. For each of these two hidden states, we assume that the 27 symbols are observed according to fixed probability distributions.
This defines an HMM with = 2 and = 27, where the state transition probabilities of the matrix and the observation probabilities of the matrix are unknown, while the observations consist of the series of characters we have extracted from the given text. To determine the and matrices, we must solve HMM Problem 3, as discussed above. We have trained such an HMM, using the first = 50,000 observations from the Brown Corpus, 1 which is available at [8]. We initialized each element of and randomly to approximately 1/2, taking care to sure that the matrices are row stochastic. For one specific iteration of this experiment, the precise values used were Each element of was initialized to approximately 1/27, again, under the constraint that must be row stochastic. The values in the initial matrix (more precisely, the transpose of ) appear in the second and third columns of Table 2.
After the initial iteration, we find log with the converged appearing in the last two columns of Table 2. The most interesting part of an HMM is generally the matrix. Without having made any assumption about the two hidden states, the matrix in Table 2 shows us that one hidden state consists of vowels while the other hidden state consists of consonants. Curiously, from this perspective, word-space acts more like a vowel, while y is not even sometimes a vowel.
Of course, anyone familiar with English would not be surprised that there is a significant distinction between vowels and consonants. But, the crucial point here is that the HMM has automatically extracted this statistically important distinction for us-it has "learned" to distinguish between consonants and vowels. And, thanks to HMMs, this feature of English text could be easily discovered by someone who previously had no knowledge whatsoever of the language.
Cave and Neuwirth [9] obtain additional results when considering HMMs with more than two hidden states. In fact, they are able to sensibly interpret the results for models with up to = 12 hidden states. For more information on HMMs, see [43], which includes detailed algorithms including scaling, or Rabiner's classic paper [35].

Principal Component Analysis
Principal component analysis (PCA) is a linear algebraic technique that provides a powerful tool for dimensionality reduction. Here, we provide a very brief introduction to the topic; for more details, Shlens' tutorial is highly recommended [40], while a good sources for the math behind PCA is [39]. The discussion at [42] provides a brief, intuitive, and fun introduction to the subject.

Figure 4: A better basis
Intuitively, larger variances correspond to more informative data-if the variance is small, the training data is clumped tightly around the mean and we have limited ability to distinguish between samples. In contrast, if the variance is large, there is a much better chance of separating the samples based on the characteristic (or characteristics) under consideration. Consequently, once we have aligned the basis with the variances, we can ignore those directions that correspond to small variances without losing significant information. In fact, small variances often contribute only noise, in which cases we can actually improve our results by neglecting those directions that correspond to small variances.
The linear algebra behind PCA training (i.e., deriving a new-and-improved basis) is fairly deep, involving eigenvalue analysis. Yet, the scoring phase is simplicity itself, requiring little more than the computation of a few dot products, which makes scoring extremely efficient and practical.
Note that we treat singular value decomposition (SVD) as a special case of PCA, in the sense that SVD provides a method for determining the principal components. It is possible to take the opposite perspective, where PCA is viewed as a special case of the general change of basis technique provided by SVD. In any case, for our purposes, PCA and SVD can be considered as essentially synonymous.

Classifiers
In the research presented in this paper, we consider four different classifiers, namely,nearest neighbors ( -NN), multilayer perceptron (MLP), random forest (RF), and support vector machine (SVM). We have already discussed MLPs above, so in this section, we give a brief overview of -NN, RF, and SVM.

-Nearest Neighbors
Perhaps the simplest possible machine learning algorithm is -nearest neighbors ( -NN). In the scoring phase, -NN consists of classifying based on the nearest samples in the training set, typically using a simple majority vote. Since all computation is deferred to the scoring phase, -NN is considered to be a "lazy learner."

Random Forest
A random forest (RF) generalizes a simple decision tree algorithm. A decision tree is constructed by building a tree, based on features from the training data. It is easy to construct such trees, and trivial to classify samples once a tree has been constructed. However, decision trees tend to overfit the input data.
An RF combines multiple decision trees to generalize the training data. To do so, RFs use different subsets of the training data as well as different subsets of features, a process known as bagging [44]. A simple majority vote of the decision trees comprising the RF is typically used for classification [18].

Support Vector Machine
Support vector machines (SVM) are a class of supervised learning methods that are based on four major ideas, namely, a separating hyperplane, maximizing the "margin" (i.e., separation between classes), working in a higher-dimensional space, and the so-called kernel trick. The goal in SVM is to use a hyperplane to separate labeled data into two classes. If it exists, such a hyperplane is chosen to maximize the margin [44].
An example of a trained SVM is illustrated in Figure 6. Note that the points that actually minimize the distance to the separating hyperplane correspond to support vectors. In general, the number of support vectors will be small relative to the number of training data points, and this is the key to the efficiency of SVM in the classification phase. Of course, there is no assurance that the training data will be linearly separable. In such cases, a nonlinear kernel function can be embedded into the SVM process in such a way that the input data is, in effect, transformed to a higher dimensional "feature space." In this higher dimensional space, it is far more likely that the transformed data will be linearly separable. This is the essence of the kernel trick-an example of which is illustrated in Figure 7. That we can transform our training data in such a manner is not surprising, but the fact that we can do so without paying any significant penalty in terms of computational efficiency makes the kernel trick a very powerful "trick" indeed. However, the kernel function must be specified by the user, and selecting a (near) optimal kernel can be challenging.

Last Word on Classification Techniques
We note in passing that MLP and SVM are related techniques, as both of these approaches generate nonlinear decision boundaries (assuming a nonlinear kernel). For SVM, the nonlinear boundary is based on a user-specified kernel function, whereas the equivalent aspect of an MLP is learned as part of the training process-in effect, the "kernel" is learned when training an MLP. This suggests that MLPs have an advantage, since there are limitations on SVM kernels, and selecting an optimal kernel is more art than science. However, the tradeoff is that more data and more computation will generally be required to train a comparable MLP, since the MLP has more to learn, in comparison to an SVM.
It is also the case that -NN and RF are closely related. In fact, both are neighborhoodbased algorithms, but with neighborhood structures that are somewhat different [19].
Thus, we generally expect that the results obtain using SVM and MLP will be qualitatively similar, and the same is true when comparing results obtained using -NN and RF. By using these four classifiers, we obtain a "sanity check" on the results. If, for example, our SVM and MLP results differ dramatically, this would indicate that we should investigate further. On the other hand, if, say, our MLP and RF results differ significantly, this would not raise the same level of concern.

Word Embedding Techniques
Word embeddings are often used in natural language processing as they provide a way to quantify relationships between words. Here, we use word embeddings to generate higher-level features for malware classification.
In this section, we discuss three distinct word embedding techniques. First, we consider word embeddings derived from trained HMMs, which we refer to as HMM2Vec. Then we consider a word embedding technique based on PCA, which we refer to as PCA2Vec. Finally, we discuss the popular neural network based technique known as Word2Vec.

HMM2Vec
Before discussing the basic ideas behind Word2Vec, we consider a somewhat analogous approach to generating vector representations based on hidden Markov models. To begin with we consider individual letters, as opposed to words-we call this simpler version Letter2Vec.
Recall that an HMM is defined by the three matrices , , and , and is denoted as = ( , , ). The matrix contains the initial state probabilities, contains the hidden state transition probabilities, and consists of the observation probability distributions corresponding to the hidden states. Each of these matrices is row stochastic, that is, each row satisfies the requirements of a discrete probability distribution. Notation-wise, is the number of hidden states, is the number of distinct observation symbols, and is the length of the observation (i.e., training) sequence. Note that and are determined by the training data, while is a user-defined parameter. Suppose that we train an HMM on a sequence of letters extracted from English text, where we convert all upper-case letters to lower-case and discard any character that is not an alphabetic letter or word-space. Then = 27, and we select = 2 hidden states, and we use = 50,000 observations for training. Note that each observation is one of the = 27 symbols (letters plus word-space). For the example discussed below, the sequence of = 50,000 observations was obtained from the Brown corpus of English [8]. Of course, any source of English text could be used.
In one specific case, an HMM trained with the parameters listed in the previous paragraph yields the matrix in Table 2. Observe that this matrix gives us two probability distributions over the observation symbols-one for each of the hidden states. We observe that one hidden state essentially corresponds to vowels, while the other corresponds to consonants. This simple example nicely illustrates the concept of machine learning, as no assumption was made a priori concerning consonants and vowels, and the only parameter we selected was the number of hidden states . Thanks to this training process, the model has learned a crucial aspect of English directly from the data.
Suppose that for a given letter ℓ, we define its Letter2Vec representation (ℓ) to be the corresponding row of the converged matrix in the last two columns of Table 2. Then, for example, Next, we consider the distance between these Letter2Vec embeddings. However, instead of using Euclidean distance, we measure distance based on cosine similarity. The cosine similarity of vectors and is the cosine of the angle between the two vectors. Let = ( 0, 1, . . . , −1) and = ( 0, 1, . . . , −1). Then the cosine similarity is given by In general, −1 ≤ cos ( , ) ≤ 1, but since our Letter2Vec encoding vectors consist of probabilities-and hence are non-negative-we have 0 ≤ cos ( , ) ≤ 1 for the and under consideration.
For the vector encodings in (3), we find that for the vowels "a" and "e", the cosine simarity is cos ( (a), (e)) = 0.9999. In contrast, the cosine similarity between the vowel "a" and the consonant "t" is cos ( (a), (t)) = 0.0817. These results indicate that these Letter2Vec embeddings-which are derived from a trained HMM-provide useful information on the similarity (or not) of pairs of letters.
Analogous to our Letter2Vec embeddings, we could train an HMM on words (or other features) and then use the columns of the resulting matrix (equivalently, the rows of ) to define word (feature) embeddings.
The state of the art for Word2Vec based on words from English text is trained on a dataset corresponding to = 10,000, = 300 and = 10 9 . Training an HMM with such parameters would be decidedly non-trivial, as the work factor for Baum-Welch re-estimation is on the order of 2 .
While the word embedding technique discussed in the previous paragraph-we call it HMM2Vec-is plausible, it has some potential limitations. Perhaps the biggest issue with HMM2Vec is that we typically train an HMM based on a Markov model of order one. That is, the current state only depends on the immediately-preceding state. By basing our word embeddings on such a model, the resulting vectors would likely provide only a very limited sense of context. While we can train HMMs using models of higher order, the work factor would be prohibitive.

PCA2Vec
Another option for generating embedding vectors is to apply PCA to a matrix of pointwise mutual information (PMI). To construct a PMI matrix, based on a specified window size , we compute ( , ) for all pairs of words ( , ) that occur within a window of each other within out dataset, and we also compute ( ) for each individual word . Then we define the PMI matrix as = { } = log ( , ) ( ) ( ) We treat column of , denoted , as the feature vector for word Next, we perform PCA (using a singular value decomposition) based on these feature vectors, and we project the feature vectors onto the resulting eigenspace. Finally, by choosing the dominant eigenvalues for this projection, we obtain embedding vectors of length .
It is shown in [32] that these embedding vectors have many similar properties as Word2Vec embeddings, with the author providing examples analogous to those we give in the next section. Interestingly, it may be beneficial in certain applications to omit some of the dominant eigenvectors when determining the PCA2Vec embedding vectors [17].
For more details on using PCA to generate word embeddings, see [17]. The aforecited blog [32] gives an intuitive introduction to the topic.

Word2Vec
Word2Vec is a technique for embedding "words"-or more generally, any features-into a high-dimensional space. In Word2Vec, the embeddings are obtained by training a shallow neural network. After the training process, words that are more similar in context will tend to be closer together in the Word2Vec space.
Perhaps surprisingly, certain algebraic properties also hold for Word2Vec embeddings. For example, according to [30], if we let 0 = "king", 1 = "man", 2 = "woman", 3 = "queen" and we define ( ) to be the Word2Vec embedding of , then ( 3) is the vector that is closest to where "closest" is in terms of cosine similarity. Results such as this indicate that Word2Vec embeddings capture meaningful aspects of the semantics of the language. Word2Vec uses a similar approach as the HMM2Vec concept outlined above. But, instead of using an HMM, Word2Vec embeddings are obtained from shallow (one hidden layer) neural network. Analogous to HMM2Vec, in Word2Vec, we are not interested in the resulting model itself, but instead we make use the learning that is represented by the trained model to define word embeddings. Next, we consider the basic ideas behind Word2Vec. Our approach is similar to that found in the excellent tutorial [21]. Here, we describe the process in terms of words, but these "words" can be general features.
Suppose that we have a vocabulary of size . We encode each word as a "one-hot" vector of length . For example, suppose that our vocabulary consists of the set of = 8 words = ( 0, 1, 2, 3, 4, 5, 6, 7) = ("for", "giant", "leap", "man", "mankind", "one", "small", "step") Then we encode "for" and "man" as ( 0) = ("for") = 10000000 and ( 6) = ("man") = 00010000 respectively. Now, suppose that our training data consists of the phrase "one small step for man one giant leap for mankind" To obtain our training samples, we specify a window size , and for each offset we consider pairs of words within the specified window. For this example, we select = 2, so that we consider words at a distance of one or two, in either direction. For the sentence in (4), a window size of two gives us the training pairs in Table 3.
Consider the pair "(for,man)" from the fourth row in Table 3. As one-hot vectors, this training pair corresponds to the input vector 10000000 and output vector 00010000.
A neural network similar to that illustrated in Figure 8 is used to generate Word2Vec embeddings. The input is a one-hot vector of length representing the first element of a training pair, such as those in Table 3. The network is trained to output the second element of each ordered pair which, again, is represented as a one-hot vector. The hidden layer consists of linear neurons and the output layer uses a softmax function to generate probabilities, where is the probability of the output vector corresponding to for the given input.
Observe that the Word2Vec network in Figure 8 has weights that are to be determined via training, and these weights are represented by the blue lines from the hidden layer to the output layer. For each output node , there are edges (i.e., weights) from the hidden layer. The weights that connect to output node form the Word2Vec embedding ( ) of the word .  The state of the art in Word2Vec for English text are trained on a vocabulary of some = 10,000 words, and embedding vectors of length = 300, training on about 10 9 samples. Clearly, training a model of this magnitude is an extremely challenging computational task, as there are 3 × 10 6 weights to be determined, not to mention a huge number of training samples to deal with. Most of the complexity of Word2Vec comes from tricks that are used to make it feasible to train such a large network with such a massive amount of data.
One trick that is used to speed training in Word2Vec is "subsampling" of frequent words. Common words such as "a" and "the" contribute little to the model, so these words can appear in training pairs at a much lower rate than they are present in the training text.
Another key trick that is used in Word2Vec is "negative sampling." When training a neural network, each training sample potentially affects all of the weights of the model. Instead of adjusting all of the weights, in Word2Vec, only a small number of "negative" samples have their weights modified per training sample. For example, suppose that the output vector of a training pair corresponds to word 0. Then the "positive" weights are those connected to the output node 0, and these weights are modified. In addition, a small subset of the − 1 "negative" words (i.e., every word in the dataset except 0) are selected and their corresponding weights are adjusted. The distribution used to select negative cases is biased towards more frequent words.
A general discussion of Word2Vec can be found in [5], while an intuitive-yet reasonably detailed-introduction is given in [21]. The original paper describing Word2Vec is [30] and an immediate followup paper discusses a variety of improvements that mostly serve to make training practical for large datasets [31].

Experiments and Results
In this section, we summarize our experimental results. These results are based on HMM2Vec, PCA2Vec, and Word2Vec experiments. But, first we discuss the dataset that we have used for all of the experiments reported in this section.

Dataset
The experimental results discussed in this section are based on the families in Table 4, with the number of available samples listed. From to keep the test set balanced, from each of these families, we randomly selected 1000 samples, for a total of 7000 samples in our classification experiments. These families have been used in many recent studies, including [6] and [48], for example. The malware families in Table 4 are of a wide variety of different types. Next, we briefly discuss each of these families.
BHO can perform a wide variety of malicious actions, as specified by an attacker [25].
CeeInject is designed to conceal itself from detection, and hence various families use it as a shield to prevent detection. For example, CeeInject can obfuscate a bitcoin mining client, which might have been installed on a system without the user's knowledge or consent [24].
FakeRean pretends to scan the system, notifies the user of nonexistent issues, and asks the user to pay to clean the system [29].
OnLineGames steals login information of online games and tracks user keystroke activity [26].
Renos will claim that the system has spyware and ask for a payment to remove the supposed spyware [23].
Vobfus is a family that downloads other malware onto a user's computer and makes changes to the device configuration that cannot be restored by simply removing the downloaded malware [27].
Winwebsec is a trojan that presents itself as antivirus software-it displays misleading messages stating that the device has been infected and attempts to persuade the user to pay a fee to free the system of malware [28].
In the remainder of this section, we present our experimental results. First, we discuss the selection of parameters for the various classifiers. Then we give results from a series of experiments for malware classification, based on each of the three word embedding techniques discussed in Section 4, namely, HMM2Vec, PCA2Vec, and Word2Vec. Note that all of our experiments were performed using scikit-learn [38].

Classifier Parameters
For each of our word embedding classification experiments, we test the three classifiers discussed in Section 3.4, namely, -nearest neighbors ( -NN), random forest (RF), and support vector machine (SVM), along with the multilayer perceptron (MLP), which is discussed in Section 3.1.2. The features considered are the word embeddings from HMM2Vec, PCA2Vec, and Word2Vec. Note that this gives us a total of 12 distinct experiments. For each case, we performed a grid search over a set of hyperparameters using GridSearchCV [41] in scikit-learn. GridSearchCV performs five-fold cross validation to determine the best parameters for each embedding technique. The parameters tested are listed in Table 5. Observe that for each of the three different word embedding techniques, we tested 36 combinations of parameters for MLP, we tested 12 combinations for SVM, we tested 16 combinations for -NN, and we tested 400 RF combinations. Overall, we conducted 3 · (36 + 12 + 16 + 400) = 1392 experiments to determine the parameters for the remaining experiments.
The optimal parameters selected for each classifier and for each embedding technique are listed in Table 6. We note that overall there is considerable agreement between the parameters for the different word embedding techniques, but in two cases (learning rate and n estimators), a different parameter is selected for each of the three embedding techniques.

Baseline Results
First, we consider experiments based on opcode sequences and HMMs. These results serve as a baseline for comparison with the vector embedding techniques that are the primary focus of this research. We choose these HMM-based experiments for the baseline, as HMM trained on opcode features have proven popular and highly successful in the field of malware analysis [1,2,16,36,50].
Specifically, we train an HMM for each of the seven families in our dataset, using = 2 hidden states in each case. For classification, we score a sample against all seven of these HMMs, and the resulting score vector (of length seven) serves as our feature vector. We use the same classification algorithms as in our word embedding experiments, namely, -NN, MLP, RF, and SVM.
Note that ee use the same opcode sequences here as in our vector embedding experiments. Specifically, the top 20 most frequent opcodes are used, with all remaining opcodes deleted.
The confusion matrices for these baseline HMM experiments are given in Figure 9. The accuracies obtained for -NN, MLP, RF, and SVM are 0.92, 0.44, 0.91, and 0.78, respectively. We see that MLP and SVM both perform poorly, whereas the neighborhood based techniques, namely, -NN and RF, are both strong, considering that we have seven classes. In addition, -NN and RF give very similar results.

HMM2Vec Results
For these experiments, we train an HMM on each sample in our dataset. Recall that our dataset consists of 1000 samples from each of the seven families listed in Table 4. We train each of these 7000 models with = 2 hidden states, using the = 20 most frequent opcodes over all malware samples. Opcodes outside the top 20 are ignored.
As mentioned in Section 3.2.2, we often train multiple HMMs with different initial conditions, and select the best scoring model. This becomes more important as the length of the observation sequence decreases. Hence, when training our HMMs, we perform multiple random restarts-the number of restarts is determined by the length of the training sequence, as indicated in Table 7.

Observations Restarts
Greater than 30,000 10 10,000-30,000 30 5000-10,000 100 Less than 500 500 Each matrix is 2 × 20, where each row corresponds to one of the hidden states of the model. From each of these matrices, we construct a vector of length 40 by appending the two rows. Since the order of the hidden states can vary between models, we select the order of the rows so as to obtain a consistency with respect to the most common opcode. That is, the row corresponding to the state that accumulates the highest probability for MOV is the first half of the feature vector, with the other row of the matrix becoming the last 20 elements of the feature vector. This accounts for any cases where the hidden states differ.
Based on the resulting feature vectors, we use the parameters in the HMM2Vec column of Table 6 to classify the samples using -NN, MLP, RF, and SVM. The confusion matrices for each of these cases is give in Figure 10.
The accuracies obtained for -NN, MLP, RF, and SVM based on HMM2Vec features are 0.93, 0.91, 0.93, and 0.89, respectively. From the confusion matrices in Figure 10, we see that the greatest source of misclassifications is between FakeRean and Winwebsec families. In most-but not all-of our subsequent experiments, these two families will prove to be the most challenging to distinguish.

PCA2Vec Results
For our PCA2Vec experiments, we generate embedding vectors for each of the 7000 samples in our training set, as discussed in Section 4.2. We then train and classify the 7000 malware samples using these PCA2Vec feature vectors. The confusion matrices for these experiments are summarized in Figure 11.
As above, each model is based on the 20 most frequent opcodes, which gives us a 20 × 20 PMI matrix. For consistency with the HMM2Vec experiments above, we consider the two most dominant eigenvectors, and for consistency with the Word2Vec models below, we use a window size of = 10 when constructing the PMI matrix. The resulting projection into the eigenspace is 2 × 20, which we vectorize to obtain a feature vector of length 40.
The accuracies obtained for -NN, MLP, RF, and SVM based on PCA2Vec features are 0.84, 0.78, 0.88, and 0.76, respectively. From these numbers, we see that PCA2Vec performed poorly for each of the classifiers considered, as compared to HMM2Vec.

Word2Vec Results
Analogous to the HMM2Vec and PCA2Vec experiments above, we classify the samples using the same four classifiers but with Word2Vec embeddings as features. The confusion matrices for these experiments are given in Figure 12.  As with the PCA2Vec experiments above, to generate our Word2Vec models, we use a window size of = 10. And, to be consistent with both the HMM2Vec and PCA2Vec models discussed above, we use a vector length of two, giving us feature vectors of length 40. We use the so-called continuous bag of words (CBOW) model, which is the model that we described in Section 4.3.
The accuracies obtained for -NN, MLP, RF, and SVM based on Word2Vec features are 0.93, 0.91, 0.93, and 0.89, respectively. These results match those obtained using HMM2Vec.
In Section 5.8, we compare the accuracies obtained in our baseline HMM, HMM2Vec, PCA2Vec, and Word2Vec experiments. But first we discuss possible overfitting issues with respect to the -NN and RF classifiers discussed above.

Overfitting
As discussed above in Section 3.4.4, both -NN and random forest are neighborhood-based classification algorithms, but with different neighborhood structure. Thus, we expect that these two classification algorithms will generally perform in a somewhat similar manner, at least in a qualitative sense.
For -NN, small values of tend to result in overfitting. To avoid overfitting, the rule of thumb is that we should choose ≈ √ , where is the number of samples in the training set [15]. Since we use an 80-20 split for training-testing and we have 7000 samples, for our -NN experiments, this rule of thumb gives us = √ 5600 ≈ 75. However, for each feature set considered, our grid search yielded an optimal value of ≤ 3.
In Figure 13, we graph the accuracy of -NN as a function of for the baseline HMM, HMM2Vec, and Word2Vec feature sets. We see that all of these techniques perform more poorly as increases. In particular, for ≈ 75, the performance of each is poor in comparison to ≤ 3, and this effect is particularly pronounced in the case of the baseline HMM. This provides strong evidence that small values of in -NN results in overfitting for each feature set, and the overfitting is especially pronounced for the baseline HMM. For a random forest, the overfitting that is inherent in decision trees is mitigated by using more trees. In contrast, if the depth of the trees in the random forest is too large, the effect is analogous to choosing too small in -NN, and overfitting is likely to occur.
To explore overfitting in our RF experiments, in Figure 14, we give the misclassification results for the baseline HMM, HMM2Vec, and Word2Vec features, as a function of the maximum depth of the trees. In this case, Word2Vec performs best for smaller (maximum) depths, which indicates that the baseline HMM and HMM2Vec features are more prone to overfitting. In Figures 15 (a) and (b), we give misclassification results as a function of both the maximum depth and the number of trees for the baseline HMM and for HMM2Vec features, respectively. From these results, we see that the baseline HMM performs similarly as a function of the maximum depth, regardless of the number of trees. In contrast, the HMM2Vec features yield consistently better results than the baseline HMM (as a function of the maximum depth), except when the number of trees is very small. This indicates that, with respect to the maximum depth, overfitting is significantly worse for the baseline HMM, since the overfitting cannot be overcome by increasing the number of trees. From the discussion in this section, we see that all of our -NN experiments results suffer from some degree of overfitting, with this effect being most significant in the case of the baseline HMM. For our RF results, overfitting is a relatively minor issue for the HMM2Vec and Word2Vec engineered features but, as with -NN, it is a significant problem for the baseline HMM. Consequently, both the -NN and RF results we have reported for the baseline HMM are overly optimistic, as these represent cases where significant overfitting has occurred. Figure 16 gives the overall accuracy for each of our multiclass experiments using -NN, MLP, RF, and SVM classifiers, for our baseline HMM opcode experiments, and for each of the HMM2Vec, PCA2Vec, and Word2Vec engineered feature experiments. In general, we expect RF and -NN to perform somewhat similarly, since both are neighborhood-based algorithms. We also expect that in most cases, SVM and MLP will perform in a qualitatively similar manner to each other, since these techniques are closely related. We find that these expectations are generally met in our experiments, which can be viewed as a confirmation of the validity of the results.

Discussion
From our 16 distinct experiments, we see that HMM2Vec and Word2Vec perform best, with PCA2Vec lagging far behind. The baseline HMM does well with respect to the neighborhoodbased classifiers, namely, RF and -NN. However, as discussed in the previous section, these neighborhood based techniques overfit the training data in the baseline HMM experiments. Neglecting these overfit results, we see that using the HMM2Vec and Word2Vec engineered features with SVM and MLP classifiers, give us the best results. Furthermore, these HMM2Vec and Word2Vec results are substantially better than either of the reliable results obtained for the baseline HMM, that is, the baseline HMM results using SVM and MLP classifiers.

Conclusion and Future Work
In this paper, we have presented results for a number of experiments involving word embedding techniques in malware classification. We have applied machine learning techniques to raw features to generate engineered features that are used for classification. Such a concept is not entirely unprecedented as, for example, PCA is often used to reduce the dimensionality of data before applying other machine learning techniques. However, the authors are not aware of previous work involving the use word embedding techniques in the same manner considered in this paper. Our results show that word embedding techniques can be used to generate features that are more informative than the original data. This process of distilling useful information from the data before classifying samples is potentially useful, not only in the field of malware analysis, but also in other fields where learning plays a prominent role.
For future work, it would be interesting to consider other families and other types of malware. It would also be worthwhile to consider more complex and higher dimensional data-as with dimensionality-reduction techniques, such data would tend to offer more scope for improvement using the word embedding strategies considered in this paper.