{'fraction_non_alphanumeric': 0.026494184961751353, 'fraction_numerical': 0.008831394987250451, 'mean_word_length': 4.706174591909155, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tactile-ViewGCN: Learning Shape Descriptor from Tactile Data using Graph Convolutional Network

For humans, our"senses of touch"have always been necessary for our ability to precisely and efficiently manipulate objects of all shapes in any environment, but until recently, not many works have been done to fully understand haptic feedback. This work proposed a novel method for getting a better shape descriptor than existing methods for classifying an object from multiple tactile data collected from a tactile glove. It focuses on improving previous works on object classification using tactile data. The major problem for object classification from multiple tactile data is to find a good way to aggregate features extracted from multiple tactile images. We propose a novel method, dubbed as Tactile-ViewGCN, that hierarchically aggregate tactile features considering relations among different features by using Graph Convolutional Network. Our model outperforms previous methods on the STAG dataset with an accuracy of 81.82%.


I. INTRODUCTION
For humans, tactile information is essential to perform day to day activities. We can get information like shape, weight, inertial property of an object with just the interaction of our hand and an object, due to our "sense of touch". In an experiment conducted by Westling et al. [1], they injected anaesthesia in the hands of volunteers and noticed that for humans, without tactile feedback, their movement while grasping objects became inaccurate and unstable. This experiment shows that humans rely on vision along with tactile feedback to perform everyday manipulation tasks. This could also be true for robots, so robotic manipulation could be made much better if we include haptics information along with a vision. Understanding and modelling this tactile grasp information could improve AR/VR experience and robotic object manipulation. But to get such information from tactile data is difficult. There has not been much work focused on that. Being able to find the inertial properties of objects from tactile data will be helpful, especially for tasks such as robotic manipulation of an object with a complex shape.
In this work, we only focus on object classification using tactile data by getting a shape descriptor from multiple tactile images. One of the main problem is to aggregate features from multiple tactile data to get a shape descriptor. Previous work of Sundaram et al. [2] used to solve this problem by concatenating or pooling multiple features to get a single feature. But, in this method, any permutation of the tactile features still represents the same object. Also, this approach doesn't consider any relations between the feature vectors. Some other approaches view this as a sequential problem and use CNN-LSTM [3] or ConvLSTM [4]. In our work, we propose a novel method called Tactile-ViewGCN which is based on View-GCN [5] that considers the relationship between multiple tactile images. After evaluating our proposed method on STAG dataset [2], our model outperforms previous methods on the same dataset with an accuracy of 81.82%.

II. RELATED WORK
A. Learning human grasp from tactile data A deep convolutional model is used in the work done by Sundaram et al. [2] to get image features. Specifically, a modified ResNet-18 [6] is used to get features from tactile images. To better classify the objects, they increased the number of input images used and get better results, which seems sensible since having more data about the object's contact could give a better idea about object shape. However, when multiple tactile images are used to classify the objects, finding an appropriate way to aggregate features from all tactile images will help to get a better global shape descriptor feature that could be used for classification. In [2], they have concatenated tactile features from multiple inputs to obtain a global feature that was used for object classification, weight detection, etc. But their approach doesn't consider the relation between multiple tactile images.
Other approaches consider this a sequential problem and try to predict object class using sequential models like ConvLSTM [4], CNN-LSTM [3] variants, etc. However, these approaches didn't work well for this dataset. In work done by Wang et al. [7], they included IMU data along with tactile frame to include hand movement during object interaction. Still, no model configuration has provided any improvement over the model that uses only tactile data. They also didn't consider using multiple tactile frames to cover the whole surface of the object or the relation between multiple tactile frames.

B. 3-D shape recognition from multiple view images
The view-based approach is advantageous in 3D shape recognition from multi-view images. Some methods like MVCNN [8] get features from multiple images and aggregate  Fig. 1. Overview of our method, first a viewgraph is constructed from features from the Backbone CNN network (modified Resnet-18 for takking single channel input). Then, the viewgraph is hierarchically coarsened by multiple layers of view-GCN. Then, we concatenate features from all the levels to get a global shape descriptor which is passed to a classifier to predict the object class. them by taking max-pooling of all feature vectors to get a global shape descriptor. However, the relationship between features and discriminative properties of the views is left unexplored. Recent work [9], [10] has shown that viewpoint location plays an essential role in getting a better shape descriptor for 3D shape recognition. Models like GVCNN [11] has tried to include this by a hierarchical view-group-shape architecture of content descriptions. RotationNet [9] considers a predefined viewpoint as a latent variable and tries to optimize it during training to find the best viewpoint for finding shape descriptors.
In view-GCN [5], they use a viewgraph to represent the predefined viewpoints. The viewgraph representation enables them to use a modified GCN [12] to aggregate features from multiple inputs, considering the relation between predefined viewpoints. Recent work done in GCN [13] makes them perform spatial convolution by aggregating features in the local neighbourhood in the graph to update node features. So, in this work, we will be using the same method as view-GCN to aggregate multiple features from the tactile images with different backbone to see whether this method can give a better shape descriptor vector to classify objects from a tactile image.

A. Motivation
For classifying an object using multiple tactile data from different regions of an object, no previous work has considered using the information about which part of the object the tactile image belongs to or the relationship between multiple tactile images, which can tell a lot about the shape of an object. Since some view-based approaches in 3D shape recognition have solved this problem by having a better way to aggregate features from multi-view images. We proposed to use a model called View-GCN, which has a good method for feature ag-gregation from multi-view images using Graph Convolutional Network.

B. Model Architecture
To get image features from tactile data, we use a Modified ResNet-18 which takes a single channel tactile image as input rather than a 3-channel RGB image and gives a onedimensional feature vector as output. Then, we create a graph with nodes of the graph as a tactile image feature, obtained earlier for each input frame. A GCN is used to hierarchically combine features to get a shape descriptor of the object. View-GCN [5] has three parts: 1) Local graph convolution, 2) Nonlocal message passing, and 3) Selective view sampling for coarsening the graph.
In part first, local graph convolution, the node features are updated by considering neighbouring nodes, which are found by applying kNN of viewpoints coordinates. Updated features from local graph convolution are passed through non-local message passing to include long range relationship between the graph's nodes. Then, selective view sampling is used to coarsen the graph, and they learn to choose the best view from the neighbouring view.
1) Building the view-graph: View-Graph is built considering i th -node of the graph as i th vertex of a cube, where the centre of the palm is assumed to be placed, while facing the centre of the cube when interacting with the object. The adjacent matrix of the graph is calculated as follow, , and g ij represents the relation between two different viewpoints.
For Ω, we have used a three-layered MLP with LeakyReLU and has ten hidden units in the first two layers, and the output is a scalar S ij .
We then use kNN to find the nearest neighbour for each viewpoints and only keep edges between nodes which are n nearest neighbours of nodes, with edges being the distance between the viewpoints. Here, n is chosen as a constant.
2) Local Graph Convolution: Local graph convolution is defined as, where, A l is the learnable adjacent matrix of view graph at lth level (G l ). F l n are feature vector in row, and W l is a linear transformation. Ψ is a non-linear transformation consisting of linear and batch normalization layer, which is used to update the node features F l .
3) Non-Local Message Passing: To get pair wise relation between two nodes, we define, where, F l i is the feature vector of ith node and τ is a singlelayered MLP with LeakyReLU to find the relation between pairwise nodes. Then, all the nodes are updated by fusing the cumulated message with the feature vector F l .
where, r l i = N l j=1 m l ji and Ω is a fusion function with parameter θ l f . In the implementation, it is a single layer MLP with Batch normalization and LeakyReLU activation. The output is a fused feature vector that is updated considering the pairwise relation between nodes of the view graph.

4) Selective view Sampling:
Typically, Furthest Point Sampling (FPS) is used for coarsening the graph [14]. FPS samples each view that is furthest from other previously sampled views based on viewpoint coordinates. But this sampling method doesn't assure that the sampled views are representative for the downstream classification tasks. Because of that, our model uses selective view sampling.
In this, we first select a subset of viewpoints using FPS. Then, we choose a sampled view by view selector among the kNN viewpoints of sampled viewpoints. The maximum output of view selector in the local neighbourhood is the new sampled view. The viewpoint coordinate of sampled view is given by, where, V () is the view selector which output the probability of a tactile data from sampled view belonging to N c object shape and max value of the vector is taken by using a max operator. Here, for the view selector, we have used a single layer MLP with d hidden units. View selectors are trained to select discriminative views among neighbouring viewpoints. We derive a coarsened graph with nodes selected by selective view sampling using this method. This View-GCN is applied hierarchically to the graph G l and output graph G l+1 at each level, which is a coarsened graph with updated nodes with fewer nodes than the previous graph. At each level, max pooling is applied on the node features after being updated by local graph convolution to get a pooled descriptor. The final global shape descriptor is the concatenation of all the pooled features from each layer. 5) Training loss: While training, we consider two loss functions, i.e., shape loss and view loss. The total loss is the sum of both, computed as where, L shape is cross-entropy loss based on the output class we got and L view ensures that each view selector is able to classify object category based on view features from the local neighbourhood.

A. Dataset
For training this model, we used MIT-STAG dataset [2]. This dataset is collected with a scalable tactile glove designed by Sundaram et al. [2] covering the full hand with 1024 sensors. Using that, they created a large dataset with 135,000 frames, while interacting with 26 objects with a hand. After removing frames with no useful information, we have a training dataset of 36,531 frames and a testing dataset with 16,119 frames. The tactile image in the dataset is of size 32×32 with a single channel. To ensure that, these tactile signals correspond only to the interactions between the hand and objects, measured tactile signals are subtracted from the tactile signals obtained from an empty hand.

B. Implementation Detail
Tactile-viewGCN is trained considering that the tactile data are collected from predefined viewpoints. Since dataset doesn't have IMU data for respective frames, we won't be able to calculate the viewpoints for the tactile image. So, we have used kNN to cluster each object type in the dataset into 8 clusters, and each cluster is assumed to be data collected from viewpoints places at vertices of a cube. Training consists of two parts. First, a convolutional backbone (a modified Resnet-18) is used to learn features of the tactile images by training it from scratch to classify tactile image datasets without clustering the dataset. It is trained for 30 epoch with optimized learning rate, weight decay, and momentum as 5 × 10 −3 , 1 × 10 −4 , 0.9, respectively. The learning rate is scheduled to decay by half for every ten epoch.
Then, the whole model is trained for 15 epoch with a learning rate of 1 × 10 −4 , 5 × 10 −4 for backbone and view GCN, respectively, with weight decay and momentum the same as before and everything is implemented using PyTorch [15]. Since the View-GCN requires predefined viewpoints, we use a clustered dataset for training. This model is trained on Nvidia GeForce GTX 1650 GPU. The model is tested on a dataset [2] which is clustered using kNN methods into N clusters, where N is dependent on the number of input images used.

C. Comparative Analysis
The accuracy for our model fluctuated around 82%, which is expected since we are clustering the dataset and the test dataset distribution is different each time. It can also be due to our assumption of the predefined viewpoint position. So, we have taken an average of 10 trials to calculate the model's accuracy. We can see that using View-GCN to aggregate features from multiple tactile images gives better results than just concatenating features to get a shape descriptor as in [2], [7]. Our method has given 9.44% more accuracy than the previous method used. 1) Effect of changing predefined viewpoints: First, we chose the predefined viewpoints as vertices of a cube. We also tried a 12 view circular configuration with an elevation of 30 degrees around the upright direction. 2) Effect of not using clustered Dataset: When we trained the model on an unclustered dataset, the classification accuracy we got was 76.19%, which is 5.63% less than what we got using a clustered dataset. This decrease in accuracy could be due to the high correlation among input images since they are sampled from an unclustered dataset. Thus, the dataset cannot mimic the assumption that the collected images are from predefined viewpoints.
3) Failure cases: As we can see from the confusion matrix of our model from Fig. 3, our model performed well on all classes except scissors, battery and board eraser. Scissors and the battery have been mostly misclassified as Allen key set, which has decreased the overall performance of the model.

V. CONCLUSION
This paper assumed that object recognition from tactile data problems could be considered similar to 3D shape recognition from multiple view images. Our results show that finding a better method to aggregate features from multiple tactile images considering the relation between the features performs better than the previous methods.
The proposed model performance could be improved if the dataset contained any IMU data so that we might be able to calculate predefined viewpoints for each data, and thus eliminating the assumption that clustering the dataset might mimic data collected from predefined viewpoints.