{'fraction_non_alphanumeric': 0.03272284035504438, 'fraction_numerical': 0.01253281660207526, 'mean_word_length': 4.639231582657737, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TreeCaps: Tree-Based Capsule Networks for Source Code Processing

Recently program learning techniques have been proposed to process source code based on syntactical structures (e.g., Abstract Syntax Trees) and/or semantic information (e.g., Dependency Graphs). Although graphs may be better at capturing various viewpoints of code semantics than trees, constructing graph inputs from code needs static code semantic analysis that may not be accurate and introduces noise during learning. Although syntax trees are precisely defined according to the language grammar and easier to construct and process than graphs, previous tree-based learning techniques have not been able to learn semantic information from trees to achieve better accuracy than graph-based techniques. We propose a new learning technique, named TreeCaps, by fusing together capsule networks with tree-based convolutional neural networks, to achieve learning accuracy higher than existing graph-based techniques while it is based only on trees. TreeCaps introduces novel variable-to-static routing algorithms into the capsule networks to compensate for the loss of previous routing algorithms. Aside from accuracy, we also find that TreeCaps is the most robust to withstand those semantic-preserving program transformations that change code syntax without modifying the semantics. Evaluated on a large number of Java and C/C++ programs, TreeCaps models outperform prior deep learning models of program source code, in terms of both accuracy and robustness for program comprehension tasks such as code functionality classification and function name prediction


Introduction
Software developers often spend the majority of their time in navigating existing program code bases to understand the functionality of existing source code before implementing new features or fixing bugs , Evans Data Corporation, 2019, Britton et al., 2012. Learning a model of programs has been found useful for their tasks such as classifying the functionality of programs [Nix and Zhang, 2017, Dahl et al., 2013, Pascanu et al., 2015, Rastogi et al., 2013, predicting bugs [Yang et al., 2015, Li et al., 2017, Zhou et al., 2019, translating programs [Chen et al., 2018, Gu et al., 2017, etc. It is a common belief that adding semantic descriptions (e.g., via code comments, visualizing code control flow graphs, etc.) enhances human understanding of programs and is also helpful for machine learning. With the help of static code dependency analysis techniques [Nielson et al., 1999], for example, Gated Graph Neural Networks (GGNN) , Fernandes et al., 2019, Allamanis et al., 2018 learn code semantics via graphs where edges are added between the code syntax tree nodes to indicate various kinds of dependencies between the nodes. However, such edges may be noise due to inaccurate code analyses that are inherently unsound or incomplete, and too many such edges contribute to long training time and may hinder the learning techniques from achieving higher accuracies [Nghi et al., 2019]. Another issue with the GGNN method is that it relies on a synchronous message-passing mechanism to accumulate information from a node's neighbors to learn their embeddings and it usually uses as few as eight message-passing iterations due to computational concerns [Allamanis et al., 2018]. This mechanism hinders GGNN from capturing information from distant parts in large graphs. There also exist deep learning techniques that process code syntax trees or abstract syntax trees (ASTs) [Mou et al., 2016, Alon et al., 2019b. However, they are limited in how they represent and learn ASTs although ASTs entail code semantics precisely. The TBCNN [Mou et al., 2016] method shares the same computational principle with the GGNN method, i.e., information is accumulated from children to parent nodes only, which limits the number of iterations for a node to accumulate information from its distant descendants. Code2vec [Alon et al., 2019b] decomposes trees into a bag of path-contexts for learning, while ASTNN  splits big trees for programs and functions into smaller subtrees for individual statements. Then, they adapt different recurrent neural network models to learn the path-contexts or flattened subtrees but are likely to miss code dependency information that is not represented in the decomposed paths and subtrees. It is desirable to learn code semantics via ASTs directly because trees can be efficiently and precisely constructed from code without any inaccurate semantic analysis and the model should not be limited by a few numbers of message-passing steps when accumulating information. Towards this goal, this paper proposes a novel architecture called TreeCaps by fusing capsule networks [Sabour et al., 2017] with TBCNN [Mou et al., 2016] to process tree inputs directly. TreeCaps first adapts TBCNN to take in trees and extract (local) node features with its convolution capability and converts the node features into capsules in its Primary Variable Capsule (PVC) layer where the number of capsules can change for different tree inputs. It then adapts CapsNet by introducing two methods to route the dynamic number of capsules in PVC to a static number of capsules in its Secondary Capsule (SC) layer. Our first method inherits the dynamic routing algorithm [Sabour et al., 2017] for static numbers of capsules; it shares a global transformation matrix across every pair of capsules between the layers [Yang et al., 2018, Zhang and. Our second method is a novel Variable-to-Static (VTS) routing algorithm that selects the capsules with the most prominent outputs in the PVC layer and squeezes them into a fixed set of capsules. The method utilizes the common intuition that code semantics can often be determined by considering only a portion of code elements. Further, we apply a dynamic routing algorithm from the capsules in the SC layer to the final Code Capsule (CC) layer whose number of capsules is fixed according to a specific learning task, to get the vector representations of the trees for the task. Compared to the max-pooling method to combine node features in TBCNN, the pipeline of our routing methods (PVC=>SC=>CC) can learn more sophisticated combinations of features in the ASTs. Across codebases in C/C++ and Java with respect to commonly compared program comprehension tasks such as code functionality classification and function name prediction, our empirical evaluation shows that TreeCaps achieves better classification accuracy and better F1 score in prediction compared to other code learning techniques such as GGNN, Code2vec, ASTNN, and TBCNN. We have also applied three types of semantic-preserving transformations [Rabin et al., 2020, Zhang et al., 2020, Wang and Su, 2019] that transform programs into syntactically different but semantically equivalent code to attack the models. Evaluations also show that our TreeCaps models are the most robust, able to preserve its predictions for transformed programs more than other learning techniques.

Related Work
There has been a huge interest in applying deep learning techniques for software engineering tasks such as program functionality classification [Mou et al., 2016, function name prediction [Fernandes et al., 2019], bug localization [Pradel andSen, 2018, Gupta et al., 2019], code clone detection , program refactoring [Hu et al., 2018], program translation [Chen et al., 2018], and code synthesis . Fundamentally, a model of source code can be learned in two steps: (1) convert the source code into a suitable representation, and (2) design learning networks to process such representation. Mou et al. [2016] parse code into ASTs and design Tree-Based Convolutional Neural Networks (TBCNNs) as the learning networks. Allamanis et al. [2018] extend ASTs to graphs by adding a variety of code dependencies as edges among tree nodes, intended to represent code semantics, and apply Gated Graph Neural Networks (GGNN)  to learn the graphs, which indeed enhances the performance of TBCNN [Mou et al., 2016] for certain tasks. Code2vec [Alon et al., 2019b], Code2seq [Alon et al., 2019a], and ASTNN  are designed based on splitting ASTs into smaller ones, either as a bag of path-contexts or as flattened subtrees representing individual statements, and use various kinds of Recurrent Neural Network (RNN) to learn such code representations. Inst2vec [Ben-Nun et al., 2018] uses the RNN to model the Intermediate Representation of the binary code that is independent of the source programming language. Capsule networks (CapsNet) [Sabour et al., 2017, Hinton et al., 2018 use dynamic routing to model spatial and hierarchical relations among objects in an image. The techniques have been successfully applied imaging processing tasks such as image classification, character recognition, and text classification , Yang et al., 2018. However, none of the studies has considered complex tree data as input, which is natural for programs. Capsule Graph Neural Networks [Zhang and Chen, 2019] propose to classify biological and social network graphs, but not for code syntax trees or graphs. To the best of our knowledge, we are the first to adapt capsule networks for program source code processing to learn code semantics directly on syntax trees without the need for inaccurate static program semantic analysis techniques [Nielson et al., 1999].

Tree-based Capsule Networks
An overview of the TreeCaps architecture is shown in Fig. 1. The code snippet in the training data is parsed into an AST and vectorized. The node vectors are fed into the TBCNN to extract node features to be used as the input for the Primary Variable Capsule (PVC) layer where the number of capsules can change dynamically according to the input tree size. The capsules in the PVC layer are then routed and reduced to a fixed number of capsules in the Secondary Capsule (SC) layer. The outputs of the SC layer are routed to the final Code Capsule (CC) layer. The capsules in the CC layer can be seen as the vector representations for the input code, and can be trained with respect to different code comprehension tasks, such as code functionality classification and function name prediction, which are evaluated in the next sections.

Tree-based Convolutional Neural Networks
We briefly introduce the Tree-based Convolutional Neural Networks (TBCNN, Mou et al. [2016]) for processing tree-structured inputs used in TreeCaps. A tree T = (V, E, X) consists of a set of nodes V , a set of node features X, and a set of edges E. An edge in a tree connects a node and its children. Each node in an AST also contains its corresponding texts (or tokens) and its type (e.g., operator types, statement types, function types, etc.) from the underlying code. Initially, we annotate each node v ∈ V with a D-dimensional real-valued vector x v ∈ R D representing the features of the node. We associate every node v with a hidden state vector h v , initialized from the feature embedding x v , which can be computed from a simple concatenation of the embeddings of its texts and type [Allamanis et al., 2018]. The embedding matrices for the texts and types are learn-able in the whole model training pipeline. In TBCNN, a convolution window over an AST is emulated via a binary tree, where the weight matrix for each node is a weighted sum of three fixed matrices W t , W l , W r ∈ R D×D (each of which is the weight for the "top", "left", and "right" node respectively) and a bias term b ∈ R D Hence, for a convolutional window of depth d in the original AST with K = 2 d − 1 nodes (including the parent nodes) belong to that window with vectors [x 1 , ..., x K ], where x i ∈ R D , the convolutional output y of that window can be defined as follows: where η t i , η l i , η r i are weights calculated corresponding to the depth and the position of the nodes. A TBCNN model usually stacks M such convolutional layers to generate the final node embeddings, where output at layer m will be used as the input for the next layer m + 1. Each layer has its own W t , W l , W r ∈ R D×D and the bias term b ∈ R D with different initialization.

The Primary Variable Capsule Layer (PVC)
In the PVC layer, we use M tree-based CNN layers with different random initializations for W t , W l , W r and b. We group outputs of the convolutional layers together to form N pvc = |V | × D sets of capsules with outputs c i ∈ R Dpvc , i ∈ [1, N pvc ], where D pvc = M is the dimension of the capsules in the PVC layer. We apply a non-linear squash function [Sabour et al., 2017] to a capsule to produce u i , which represents the probability of existence of an entity by the vector length: Hence, the output of the primary variable capsule layer is X pvc ∈ R Npvc×Dpvc .

The Secondary Capsule Layer (SC)
As argued by Sabour et al. [2017], the capsule network tries to address the representational limitation and exponential inefficiencies of convolutions with transformation matrices. To this, the child capsules in the PVC layer will be routed to the parent capsules in the next capsule layer through a transformation matrix.

Sharing Weight across Child Capsules with Dynamic Routing (DRSW)
Since the number of capsules in the PVC is dynamic, a global transformation matrix cannot be defined in practice with variable dimensions. The solution for this problem is to defined a shared transformation matrix W s ∈ R Npvc×Dpvc×Dsc across the child capsules, where N pvc is the number of capsules in the PVC layer [Yang et al., 2018], D sc is the dimension of the capsules in the SC layer, and use the dynamic routing algorithm to route the capsules (as summarized in Algo.1).
For each capsule i in the PVC layer (layer l in Algo.1), and for each capsule j in the SC layer (layer l + 1 in Algo.1), we multiply the output of the PVC layer u i by the shared transformation matrix W s to produce the prediction vectorsû j|i = W s u i . The "prediction vectors" are responsible to predict the strength of each capsule in the PVC layer, then a weighted sum over all "prediction vectors"û j|i will produce the capsule j in the SC layer. The trainable shared transformation matrix learns the part-whole relationships between the primary capsules and secondary capsules, while effectively transforms u i 's into the same dimensionality as v j where each v j denotes the capsule output of the SC layer. The coupling coefficients β ij between capsule i and all the capsules in the SC layer sum to 1 and are determined by a "routing softmax" whose initial logits α ij are the log prior probabilities that capsule i in PVC layer should be coupled to capsule j in the SC layer. Then we use r iterations to refine β ij based on the agreements between the prediction vectorsû j|i and the secondary capsule Algorithm 1 Dynamic Routing return vj

Variable-to-Static Routing (VTS)
Sharing the transformation matrix reduces the ability to learn different features as each pair of capsules is supposed to have its transformation matrix. Because of this limitation, we offer the second solution to route the variable number of capsules in the PVC layer. It is based on an observation of source code that, in practice, not every node of the AST contributes towards a source code learning task. Often, source code consists of non-essential entities, and only a portion of all entities determine the code class. Therefore, we propose a novel variable-to-static capsule routing algorithm, summarized in Algo. 2. The intuition of this algorithm is that we squeeze the variable number of capsules in the PVC layer to a static number of capsules by choosing only the most important capsules in the PVC layer. The major difference between the VTS algorithm and the DRSW algorithm is that the DRSW needs to produce prediction vectors by multiplying the capsule outputs in PVC layer with the shared transformation matrix, and then the prediction vectors will be combined to produce the capsules for SC layer, while in the VTS, the capsule outputs in the PVC layer are selected and the prominent ones are used to initialize for the capsules in SC layer directly. We initialize the outputs of the SC layer with the outputs of the a capsules with the highest L 2 norms in the PVC layer. Hence, the outputs of the PVC layer, [u 1 , ..., u Npvc ], are first sorted by their L 2 norms, to obtain U sorted , and then the first a vectors of U sorted are assigned as v j , j ≤ a.
Since the probability of the existence of an entity is denoted by the length of the capsule output vector (L 2 norm), we only consider the entities with the highest existence probabilities for initialization (in other words, highest activation) following the aforementioned intuition. It should be noted that the capsules with the a-highest norms are used only for the initialization; the actual outputs of the static capsules in the SC layer are determined by iterative runs of the variable-to-static routing algorithm. It is the capsules with the most prominent outputs along with the capsules of the highest vector similarities to them that get routed to the next layer. In this way, rare capsules, when they have prominent outputs, are still preserved and routed to the next layer. Next, we route all b capsules in the PVC layer based on the similarity between them and the static capsule layer outputs. We initialize the routing coefficients as α ij = 0, equally to the b capsules in the primary variable capsule layer. Subsequently, they are iteratively refined based on the agreement between the current SC layer outputs v j and the PVC layer outputs u i . The agreement, in this case, is measured by the dot product, f ij ← u i · v j , and the routing coefficients are adjusted with f ij accordingly. If a capsule u in the PVC layer has a strong agreement with a capsule j in the SC layer, then f ij will be positively large, whereas if there is strong disagreement, then f ij will be negatively large. Subsequently, the sum of vectors u i is weighted by the updated β ij to calculate s j , which is then squashed to update v j .

The Code Capsules layer
The Code Capsule (CC) layer in TreeCaps outputs the vector representations for the code X cc ∈ R Ncc×Dcc , where D cc is the dimensionality of each code capsule and N cc is fixed with respect to a specific code learning task. Since the outputs of the Secondary Capsule layer X sc ∈ R Nsc×Dsc , where N sc is also fixed, It then produces the needed final capsule outputs X cc . In the following subsections, we explain how we set N cc and train the TreeCaps models for different code learning tasks.

Code (Functionality) Classification
This task is to, given a piece of code, classify the functionality class it belongs to. We want N cc capsules in the CC layer, each of which corresponds to a functionality class of code that appeared in the training data. As such, we let N cc = κ, where κ is the number of functionality classes. We calculate the probability of the existence of each class by obtaining L 2 norm of each capsule output vector. We use the margin loss [Sabour et al., 2017] as the loss function during training.

Function (Method) Name Prediction
This task is to, given a piece of code (without its function header), predict a meaningful name that reflects the functionality of the code. For this task, following Alon et al. [2019b]'s prediction approach, we let N cc of the CC layer be 1, and the output of the only capsule represents the vector for the given piece of code. In this case, the output capsules of the CC layer has the shape of X cc ∈ R 1×Dcc , which is also the code vector that represents for the code snippet C, denoted as v C . The vector embeddings of the function are learn-able parameters, formally defined as f unctions_vocab ∈ R |L|×Dcc , where L is the set of function names found in the training corpus. The embedding of f unction i is row i of f unctions_vocab. The predicted distribution of the model q (l) is computed as the (softmaxnormalized) dot product between the context vector v C and each of the function embeddings: where q (l i ) is the normalized dot product between the vector of l i and the code vector v C , i.e., the probability that a function name l i should be assigned to the given code snippet C. We choose l that gives the maximum probability for the snippet v C . For training the network, we use the cross-entropy as the loss function.

Empirical Evaluation
General Settings. We mainly use SrcML [Collard et al., 2013] to parse source code into ASTs; 1 we also use another parser PycParser 2 used by TBCNN and ASTNN to ensure a fair comparison and evaluate the effects of different parsers. For the parameters in our TBCNN layer, we follow Mou et al. [2016] to set the the size of type embeddings = 30, and size of text embeddings = 50 and the number of convolutional steps M = 8. For the capsule layers, we set N sc = 100, D sc = 16 and D cc = 16.
We have used Tensorflow libraries to implement TreeCaps. To train the models, we have used the Rectified Adam (RAdam) optimizer  with an initial learning rate of 0.001 subjected to decay, on an Nvidia Tesla P100 GPU. 3

Set up for Code Classification
Datasets, Metrics, and Models. We use datasets in two different programming languages. The first Sorting Algorithms (SA) dataset is from Nghi et al. [2019], which contains 10 algorithm classes of 1000 sorting programs written in Java. The second OJ dataset is from Mou et al. [2016], which contains 52000 C programs of 104 classes. We split each dataset into training, testing, and validation sets by the ratios of 70/20/10. We use the same classification accuracy metric as Mou et al. [2016] for comparing classification results. We compare TreeCaps with other techniques applied for the code classification task, such as Code2vec, TBCNN, ASTNN, GGNN, etc. Since TBCNN [Mou et al., 2016] and ASTNN  use PycParser to parse code into AST, we also compare TreeCaps with all the baselines by using both PycParser and SrcML. We also include a token-based baseline by treating source code as a sequence of tokens and using the 2-layer Bi-LSTM to process the sequence of tokens. We follow Allamanis et al. [2018] to use the concatenation of embeddings of node types and texts (tokens) for node initialization and representation. We also include an ablation study to measure the impact of different combinations of node initialization and representation. Table 1, TreeCaps models, especially TreeCaps-VTS, have the highest classification accuracy when combining node type and node token information, for both of the SA and OJ datasets. When only node token information is used, the simpler 2-layer Bi-LSTM models may achieve higher accuracy. The OJ dataset also shows that the choice of a parser affects the performance significantly. The models using PycParser all achieve higher accuracy than the models using SrcML. This is due to the reason that ASTs generated by PycPaser have only around 50 node types, while SrcML has more than 400 node types, which makes it harder for the networks to learn. Across the datasets, The TreeCaps-VTS performs consistently the best in terms of the F1 measure among the baselines under different settings.

Set up for Function Name Prediction
Datasets, Metrics, and Models. We have used the datasets from Alon et al.
[2019a] that contain three sets of Java programs: Java-Small (700k samples), Java-Med (4M samples), and Java-Large(16M samples). We measure prediction performance using precision (P), recall (R), and F1 scores over the sub-words in generated names, following the metrics used by Alon et al. [2019b], Fernandes et al. [2019]. For example, a predicted name result_compute is considered to be an exact match of the ground-truth name called computeResult; predicted compute has full precision but only 50% recall; and predicted compute_model_result has full recall but only 67% precision. We compare TreeCaps to the following baselines applied to the function name prediction task: Code2vec, TBCNN, GGNN, and a neural machine translation baseline that reads the input source code as a stream of tokens by using a 2-layered bidirectional encoder-decoder LSTMs (split tokens) with global attentions. The ASTNN is not designed for this task, so we exclude it in this evaluation. Function Name Prediction Results. As seen in Table 2, TreeCaps-DRSW and TreeCaps-VTS are comparable in term of F1 score, although the TreeCaps-VTS is slightly better. The TreeCaps models also achieve comparable or better results than all other models for most of the settings. In particular, TreeCaps are comparable or better than GGNN but without the need of additional code dependency analysis for constructing graphs.

Model Analysis
To better understand the importance of the different components of our approach, we evaluate the effect of various aspects of the TreeCaps models. This subsection provides a robustness analysis and a comparison between DRSW algorithm and VTS algorithm. We use the code classification task on the OJ Dataset for the experiments in this subsection. 4

Robustness of Models
We measure the robustness of each model by applying the semantically-preserving program transformations to the OJ Dataset's test set. We follow Wang and Su [2019], Rabin et al. [2020] to transform programs in three ways that change code syntax but preserve code functionality: (1) Variable Renaming (VN), a refactoring transformation that renames a variable in code, where the new name of the variable is taken randomly from a set of variable vocabulary in the training set; (2) Unused Statement (US), inserting an unused string declaration to a randomly selected basic block in the code; and (3) Permute Statement (PS), swapping two independent statements (i.e., with no dependence) in a basic block in the code. The OJ test set is thus transformed into a new test set. We then examine if the models make the same predictions for the programs after transformation as the prior predictions for the original programs. We use percentage of predictions changed (P P C) as the metric previously used by [Rabin et al., 2020, Zhang et al., 2020, Wang and Su, 2019 to measure the robustness of the models. Formally, suppose P denotes a set of test programs, a semantic-preserving program transformation T that transforms P into a set of transformed programs P = {p = T (p)|p ∈ P }, and a source code learning model M that can make predictions for any program p: M (p) = l, where l ∈ L denotes a predicted label for p according to a set of labels L learned by M , we compute the percentage of predictions changed as follows.
The lower P P C values for M suggest higher robustness since they can maintain more of correct predictions with respect to the transformation. As shown in Table 3, TreeCaps-VTS is the most robust model against the program transformations. Even though more kinds of program transformations could be applied to evaluate model robustness in our future work, the current analysis gives us the confidence that TreeCaps can be more robust against attacks via such adversarial examples [Ramakrishnan et al., 2020, Bielik andVechev, 2020].  (1) when DRSW is used, the loss decreases much slower than when VTS is used (in the right plot); and (2) VTS improves validation accuracy much faster than DRSW (in the left chart). An explanation for this is that DRSW has to learn an additional shared transformation matrix W s , resulting in slower convergence due to a larger number of parameters to be learned.

Conclusion
We propose TreeCaps, a novel neural network architecture that incorporates tree-based convolutional neural networks (TBCNN) into capsule networks for better learning of code directly on source code syntax trees. To handle dynamic numbers of capsules produced from TBCNN, we propose two methods to route the capsules in the Primary Variable Capsule layer to fixed number of capsules in the Secondary Capsule layer. We are the first to re-purpose capsule networks over syntax trees to learn code without the need for explicit semantics analysis. Our empirical evaluations have shown that TreeCaps can outperform existing code learning models (e.g., Code2vec, TBCNN, ASTNN, GGNN, etc.) for two different program comprehension tasks (e.g., code functionality classification and function name prediction) on C/C++/Java programs. It is our belief that the new method can be applied to other valuable software engineering tasks such as bug localization and clone detection. A limitation of TreeCaps is similar to that of the original capsule networks and many other neural networks: it still lacks the explainability. Software developers may require additional evidence before accepting the predication results, which suggests that relating TreeCaps outputs to certain visible patterns in code could help explain the predictions in our future work.

Broader Impact
TreeCaps excel in the two program comprehension tasks surprisingly well in our experiments while requiring no program semantics analysis. This outcome can directly benefit software developers, increasing their productivity by understanding large code bases better. This is not to say that program semantics analysis tools are no longer needed for specific programming tasks. It is just that latent routes among program elements learned from a large quantity of source code may play an important role in identifying those features neglected by certain specific program semantic analysis. As such, TreeCaps may be suitable for other multi-faceted source code learning tasks such as semantic clone detection, code summarization, code search, etc. The technique also has the potentials to be applied to security problems such as vulnerability detection [Zhou et al., 2019], where there is no single dominating semantics for the analysis. Broadly speaking, automating software development with the help of deep learning may eventually remove the needs of several mundane tasks of programming [Allamanis et al., 2018], but not the needs of experienced programmers. However, there are many obstacles to this goal. First, the state-of-the-art learning of program representations, as shown in this work, cannot yet reach near-perfect accuracy which is essential for mission-critical tasks. Secondly, the availability of semantically analyzed high-quality datasets is still in scarcity for various viewpoints, compared to those used in this work.
Without high-quality hierarchical datasets in other tasks, it is still uncertain whether TreeCaps would perform equally well compared to other learning methods. Therefore, it can be risky to apply the proposed method without having confidence in the quality of datasets.
In terms of robustness, we have seen that TreeCaps also have better performance compared to other semantic-specific models. We have considered only semantic-preserving transformations which are likely to preserve certain viewpoints more than the others. In programming practice, semantic non-preserving changes may happen frequently too. Although they are not reflected by the datasets, evaluating the impact of such arbitrary changes to robustness is an open question: Does one need to intervene developers during the edits even while the syntax of the code has not been settled? Since other learning methods such as Code2vec and GGNN require syntactically correct code samples, it is a common belief that such code assistant tools should not be in the way of frequent programmer edits. Aside from their applications in the domains of Software Engineering and Programming languages, TreeCaps may also apply to other domains that may benefit from tree-based learning. For example, in Natural Language Processing tasks such as text classification, sentiment analysis, etc., sentences can be parsed into trees, and TreeCaps may also perform better than other successfully applied tree-based learning techniques, e.g., TBCNN [Mou et al., 2016[Mou et al., , 2015, Recursive NN [Socher et al., 2011], and TreeLSTM [Tai et al., 2015].