{'fraction_non_alphanumeric': 0.02851228004140397, 'fraction_numerical': 0.0050813964430224895, 'mean_word_length': 4.665245202558635, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
One-Shot Image Segmentation with U-Net

Image segmentation is an important task in the field of computer vision research. Many applications need accurate and efficient segmentation mechanisms. However, most existing methods need a large support set and have struggled with dealing with new classes in solving computer vision problems. In order to deal with the problem, this paper modified the traditional model to adapt it to the task of few-shot segmentation. Especially, the part of encoders was changed into a Siamese neural network for support branch and query branch. Then, the decoded query image is compared with different levels of features from the encoded support image. Experimental results show that this proposed model achieved the best performance compared with several baseline methods, and the improvement is more than 8%.


Introduction
Deep Neural Network has shown its powerful ability in the area of image segmentation. Many popular convolutional networks have been proposed for computer vision tasks, such as U-Net [1], Mask R-CNN [2], FCN [3], SegNet [4], and DeepLab [5]. However, most of these networks require a great amount of data for training, which costs significant time and effort. What's more, these models also struggle to adapt to new classes in a data-efficient manner. In recent years, one-shot learning has been proposed to deal with the data-sparse problem and shown excellent performance in some tasks. In this paper, we take advantage of the one-shot learning and apply it to the image segmentation task, i.e., one-shot segmentation. One-Shot image segmentation aims to train a network based on a limited labeled dataset and an unknown large dataset.
In One-Seg [6], the task of One-Shot semantic Image Segmentation is proposed. In this paper, we followed the same configuration and task in One-Seg [6]. The task's goal is to predict a segmentation mask for a semantic class given in the supported image. With only one sample of the target class, the network will be able to segment on the query images. What's more, we leveraged the Siamese Network to make our model more efficient, for it is a simple and efficient structure that aims to build two branches. In Siamese network, the two branches can share the same parameters. These branches are made of the same network, which uses the same parameters. In actual structure, the two branches can be replaced by only one network. Inducing the Siamese Network into the encoding module makes training and optimization more efficient.
Furthermore, most convolution networks, such as VGG [7] and FCN [3], will lose much information in the convolution process. Since one-shot learning has limited data, losing information will decrease 2 accuracy significantly. In order to avoid so much information loss, we propose U-net as the backbone of our network. The U-net [1] is a convolutional neural network, which analyzes information from different levels to determine a mask. The U-net's contracting path follows the typical architecture of a convolutional network. However, its expansive path consists of an up-sampling of the feature map and concatenation with the corresponding cropped feature map from the contracting path. The U-net allows the up-sampling layers to have a large number of features. The network allows propagating context information, which helps module trained by limited dataset to make higher accuracy predictions.
In this work, we proposed a model based on U-net [1] with a Siamese Network as an encoding module. The encoders and the decoders are based on a double convolution module induced from U-net. We use the Siamese Network with encoders to gain the features at different levels from the query image and support the image with its mask. The features are concatenated at a different level and compared to each other. We use Cross Entropy Loss in the model. The experiment shows that our approach has a significant improvement over the baselines.

Image Segmentation
Image segmentation is the process of segmenting digital images into several segments. After semantic segmentation, the original image contains several colors, each of which represents a category.
In the computer vision field, recognizing and locating objects are import goals of many modules, and image segmentation can locate the sets of pixels for different objects. Networks used in image segmentation tasks, like FCN, U-net and Mask R-CNN, have their own advantages in different areas, and we should get the suitable one.

Semantic Segmentation
Recognizing objects is an important goal of many computer vision tasks, so people propose Semantic Segmentation to recognize the specific classes of images. Semantic Segmentation will separate the image into different categories, and objects from the same categories will share the same features. Semantic Segmentation is widely used in various applications, including autonomous driving, image search engines, and human-machine interaction.

One-shot learning
One-shot learning [8,9] aims to learn information about object categories from one or a few training samples. A large data-set is virtually impossible in some conditions, and this is where one-shot learning comes into play. In contrast, many non-parametric models allow novel examples to be rapidly assimilated, whilst not suffering from catastrophic overfitting [10]. Instead of treating the task as a classification problem, one-shot learning turns it into a different evaluation problem. The learning model takes two images and returns a value that shows the similarity between the two images when adjusted for one-shot learning [11]. There are many types of one-shot learning: metric learning methods, metalearning [12,13] method, data-augment method, and semantic-based method. The metric learning method learns the mapping images from the same class closed together and away from the different classes. Meta-learning aims to learn new concepts and skills fast with a few training data. Data-augment will generate more useful samples from the few labeled objects. Semantic-based methods aiming to find the relationship between the objective and object properties are introduced in zero-shot learning.

One-shot Segmentation
One-shot object detection and segmentation aim to develop a model that can localize and segment objects from arbitrary categories when provided with a single visual example from that category. Oneshot segmentation takes sparse labeled images as the training set and segments an image from an unseen class in the training process. Few-shot segmentation avoids over-fitting and time-consuming, so it is applied to solve many computer vision problems. Many useful modules have already been proposed.

Siamese Network
Siamese networks [14] are an effective method for contrast learning that use contrastive loss to embed semantically similar samples closer together and dissimilar images further away [8,15]. Siamese Network works in tandem, taking two images as input and output the difference value. Once the Siamese network is trained, the model will tell them these images are similar enough to be the same category or not if people provide two new images.

Method
In this paper, we build a two-branch network based on U-net [1]. The overview of our model is shown in Figure 1. The query branch will take the RGB image as input. In the support branch, the product of support image and support mask is taken as input. The two branches are built based on Siamese network. They share the same weights and bias. By using Siamese network in the structure, we can reduce the load and time for training. The two branches will encode the features of the support and the query into different levels. Then the features will be compared and decoded for predicting segmentation masks. The predicted mask will be used for calculating Cross Entropy Loss with the true mask.  Figure 2. Each of the convolution layers is followed by a rectified linear unit (ReLU) and batch normalization. This module is used as the core part of the whole network. There are five double convolution modules in the support branch, query branch, and decode branch.  Figure 2. Double convolution module The structure and feature sizes of the query branch and support branch are shown in Figure 3. As we are using Siamese network, the two branches share the same structure. The input of the branch will be encoded into different levels for later comparison and decoding.  Finally, the output layer, which is a convolution layer, will calculate the probability of being the object or the background. Then we use the softmax function of finalizing the class of every pixel to get the predicted mask. The predicted mask will be used for calculating the Cross Entropy Loss for the training process and evaluation.

Data
The data set PASCAL-5i is constructed for few-shot image segmentation tasks, which are proposed in Shaban et al.'s research [6]. The data set is constructed based on PASCAL VOC 2012 and extra masks from SDS [17]. The data is split into four folds for training and evaluation. The classed contained in the folds are shown in Table 1.  [224,224]. There are mainly two classes in the task: the target object and the background-the optimizer used in the experiment in Adam with a 0.003 learning rate. The IoU and the loss of every image in the evaluation part will be recorded. We decided to use Cross Entropy Loss as our loss function. Our network is tested with the four splits of the dataset. The results are evaluated using mean Intersection-over-Union (mean IoU) and shown in Table 2 below. In three folds of the data set, our model can work much better. However, in split 1, the model is worse than the baseline. The reason can be the content in the split. Good predictions are shown below. In some categories, the model has a very good performance. The general shape of the object can be predicted. Only some small error exists in the prediction.

Conclusion
In this paper, we proposed a method of one-shot segmentation, which is based on U-net. The structure of U-Net turned out to be good in the setting of one-shot segmentation. It works better than expected. However, there are still some problems. For example, some of the masks are stripped. The pixels of the overall object are not continuous, but the general shape of the object is correct. The problem can be handled by adding additional refinement modules for the masks. For future work, we are planning to design a suitable refinement module for the network. The refinement module aims to rearrange the pixels in the predicted images for improving the performance.